[SOLVED] Pacman package cache: sharing inside home net

ua4000 · 2026-03-31 12:24:18

I have several arch machines at home, normal desktop machines, no servers, not always online, they might differ by installed packages.
My aim is to save download volume by sharing the recent downloads in /var/cache/pacman/pkg/ between my clients, like a mesh.
Knowing a cache miss will happen now and then, because the machines may not be online always at the same time and differ in packages.
I read https://wiki.archlinux.org/title/Package_proxy_cache in the wiki and tried out some in the last years.

But none of them feels like the correct way for me, my idea:

- no central server
- no machine which is always online
- no extra package
- no extra daemon running, which open access to the machine
- no hole in sshd_config for sftp user without password etc. (idea: CacheServer = sftp://pkg@machine1.local:2222 ... and the other machines)
- no unnecessary transfer of unused packages (which is side effect of "Two-way with rsync or FTP" or Syncthing)
- best effort caching, but no need to reach 100%

New idea:

* sync the database:

# pacman -Sy

* get a list of exact(!) required filenames:

# pacman -Sup --print-format '%f'

* try to fetch missing files from home net machines, rsync over ssh:
(via dedicated ssh user+key)

missing= **list of filenames**
hosts= **list of local machines**
cache_dir="/var/cache/pacman/pkg"
for pkg in $missing; do
  for host in "${hosts[@]}"; do
    rsync -az --timeout=2 "$host:$cache_dir/$pkg" "$cache_dir/" && break
  done
done

* perform the usual update

# pacman -Syu

Is this idea worth a shot ?
Any fatal errors or flawed design ?
Does anybody have this solution maybe already in place ?

Last edited by ua4000 (2026-04-04 10:01:47)

twelveeighty · 2026-03-31 14:58:32

Whenever I see `pacman -Sy` I get nervous, because of the dire warning. Instead, have you considered using `checkupdates -d` against a proxy cache? Or at least use checkupdates instead of pacman -Sy?

seth · 2026-03-31 15:35:47

Afaiu the OP wants to implement https://wiki.archlinux.org/title/Packag … only_cache w/o running a http or ftp server.
The idea would be to atomically call -Sy, populate the cache and then -Syu - this would not result in a partial update or pre-updated database than any failing/aborted -Syu

A flaw (next to your psudocode not working ) would be that the subsequent database update might expose changes meaning your might have transferred stale cache data.
Quick sanity check: your setup does not allow eg.

CacheServer = rsync://basement/var/cache/pacman/pkg/
CacheServer = rsync://attic/var/cache/pacman/pkg/
CacheServer = rsync://garage/var/cache/pacman/pkg/

ua4000 · 2026-03-31 16:47:52

seth wrote:

meaning your might have transferred stale cache data.

I'm aware, that my idea is not 100% and may led now and then to a cache miss, or a larger miss when I pick the wrong minute to update.

seth wrote:

Quick sanity check: your setup does not allow eg.
CacheServer = rsync://basement/var/cache/pacman/pkg/

I did not try, but researched it:
pacman uses in default config libcurl, which is based on curl.
"curl -V " does not offer rsync protocol at the moment.

In addition I don't want to run an UNencrypted rsync dameon.
rsync over ssh looks promising, but where to put the ssh config + key, accessible for the DownloadUser alpm ?
That would be the same problem too, when thinking about CacheServer = sftp://...

Edit: encrypted --> UNencrypted

Last edited by ua4000 (2026-04-01 13:21:05)

ua4000 · 2026-04-01 13:34:23

twelveeighty wrote:

have you considered using `checkupdates -d`

Yes, this was my first idea, but: the output does not contain the full filename, only a part. For the rsync I need the full exact filename.

Also I forget in my solution to fetch the associated *.sig files.
I'm thinking of a plain textfile with the file list, rsync can work with:

--files-from=FILE        read list of source-file names from FILE

I only need to patch this file with the .sig files.

And the 1st

# pacman -Sy

can be ommited, by putting the "y" in the 2nd part:

# pacman -Syup --print-format '%f'

So the update procedure would be

pacman -Syup --print-format '%f' > list
add .sig files to list
rsync ...machine1..x ... --files-from=list
pacman -Syu

And an optimization:
Only machines with an gigabit lan conenction could be rsync sender, wlan clients will become only rsync receiver.

ua4000 · 2026-04-02 19:32:19

During testing, I decided to separate the database sync again: simply to get a clean download list without verbose output from database sync.
1st testing worked for me:

1. rsync sender:
add ssh account with ssh-key, on each machine which is suitable as sender.

2. rsync receiver
root user: add ssh key and .ssh/config for the "rsync sender" account

3. run immediately before the "real" pacman update run:

cache_get() {
	senders="machine1 machine2 machine3"
	f_upd="$(mktemp)"
	d_pkg="/var/cache/pacman/pkg/"

	sudo pacman --sync --refresh --noconfirm --noprogressbar 
	sudo pacman --sync           --noconfirm --noprogressbar --sysupgrade --print --print-format '%f' | awk '{print $0; print $0".sig";}' > "${f_upd}"

	for x in $senders; do
		printf '[%s] \t' "${x}"
		rsync --archive --atimes --inplace --partial --no-compress --no-perms --no-owner --no-group --timeout=2 --ignore-missing-args \
		"${x}:${d_pkg}" "${d_pkg}" --files-from="${f_upd}" --verbose  ## --dry-run
	done
	
	rm "${f_upd}" 
	true
}

Any ideas to make it better ?
e.g. instead root user, let DownloadUser = alpm fetch the files, but alpm has no home dir, where to put the ssh config ?

seth · 2026-04-02 21:05:14

sudo -u alpm -H /path/to/wherever rsync

ua4000 · 2026-04-03 12:06:47

Thanks seth, but this will not work:

$ sudo -u alpm -H /tmp rsync
sudo: /tmp: command not found

Also switching to su (the update script will run as root) won't work:

# su --login alpm --command "df -h"                                                                                       
This account is currently not available.

My idea was also, to re-use alpm user also as rsync sender (allow ssh login access), but with the issues above I made it the other way around:
rsync sender is a new normal user "pkg", same as rsync receiver user, which I use in the master script with su:

su --login pkg --command  'rsync ...

So rsync sender is running as normal user,
and rsync receiver is also a normal user.
After the rsync receiver is fished I pick as root the received files from local /var/cache/pacman/pkg/tmp.O9amw6tn5o ... and move them one level higher ...

seth · 2026-04-03 12:35:55

Ah, shit - -H just sets the target $HOME from /etc/passwd

sudo -u alpm env HOME=/tmp rsync

?

ua4000 · 2026-04-03 13:12:00

Yes, thanks, this would work.

But the user still needs some permanent place for the ssh config, and ssh key, and a shell for the rsync,
seems alpm user is not suited for my idea to use him as sender as well as receiver:

after giving alpm user an public ssh key:

$ rsync alpm:
protocol version mismatch -- is your shell clean?
(see the rsync manpage for an explanation)
rsync error: protocol incompatibility (code 2) at compat.c(622) [Receiver=3.4.1]

$ ssh alpm
This account is currently not available.

So a "normal" additional user, with own home, with default shell, solves both my problems.

bulletmark · 2026-04-03 22:03:45

ua4000 wrote:

Does anybody have this solution maybe already in place ?

Here's my approach: https://github.com/bulletmark/pacpush

ua4000 · 2026-04-04 10:01:18

Thanks for sharing, interesting solution!
For me it is does not feel like a way to go, because of "You need to have root ssh access to the remote machines for pacpush to work."

For my solution I had a 1st test version run, and it worked fine, and saved already ~85% download volume.

While the update script is straight forward, simple and easy to read and maintain, the ssh config is not.
It's the drawback in my solution to configure n times n:

on every local machine
   create ssh user
   authorized_keys configuration
   for every remote machine
      add entry in ssh_config
      add ssh key
      test ssh connection, to get known_hosts populated

Adding a machine, or replacing one will result also in manual work...

Edit: marked as [SOLVED], thanks for all feedback!

Last edited by ua4000 (2026-04-04 10:02:20)

Arch Linux

#1 2026-03-31 12:24:18

[SOLVED] Pacman package cache: sharing inside home net

#2 2026-03-31 14:58:32

Re: [SOLVED] Pacman package cache: sharing inside home net

#3 2026-03-31 15:35:47

Re: [SOLVED] Pacman package cache: sharing inside home net

#4 2026-03-31 16:47:52

Re: [SOLVED] Pacman package cache: sharing inside home net

#5 2026-04-01 13:34:23

Re: [SOLVED] Pacman package cache: sharing inside home net

#6 2026-04-02 19:32:19

Re: [SOLVED] Pacman package cache: sharing inside home net

#7 2026-04-02 21:05:14

Re: [SOLVED] Pacman package cache: sharing inside home net

#8 2026-04-03 12:06:47

Re: [SOLVED] Pacman package cache: sharing inside home net

#9 2026-04-03 12:35:55

Re: [SOLVED] Pacman package cache: sharing inside home net

#10 2026-04-03 13:12:00

Re: [SOLVED] Pacman package cache: sharing inside home net

#11 2026-04-03 22:03:45

Re: [SOLVED] Pacman package cache: sharing inside home net

#12 2026-04-04 10:01:18

Re: [SOLVED] Pacman package cache: sharing inside home net

Board footer