pacserve - easily share Pacman packages between computers

Xyne · 2021-11-12 01:33:18

aureooms wrote:

Hi Xyne,
If I restart pacserve peers can be discovered again. Is there a standard way to
automate recovering from this error?

I've added a handler for that exception so it should just keep retrying at the configured multicast announcement interval without the need to restart.

aureooms · 2021-11-14 11:16:25

Xyne wrote:

I've added a handler for that exception so it should just keep retrying at the configured multicast announcement interval without the need to restart.

Thanks a lot! I suppose this is a recent addition available with version 2021.11-1 of python3-threaded_servers. Your system at https://xyne.archlinux.ca/ looks down for me however so cannot update. But do not worry, I will be patient!

Last edited by aureooms (2021-11-14 11:16:45)

Xyne · 2021-11-14 21:00:12

aureooms wrote:

Thanks a lot! I suppose this is a recent addition available with version 2021.11-1 of python3-threaded_servers. Your system at https://xyne.archlinux.ca/ looks down for me however so cannot update. But do not worry, I will be patient!

I don't host the site myself. It's been graciously hosted by Dusty for over a decade but he hasn't really been involved in the community for a while so the free ride may be over.

linuxninja · 2021-11-15 02:34:35

The secondary mirror for Xyne's repo is still up.
Xyne: If you need a place to move your primary hosting, the offer still stands.
I was unable to email you due to the domain DNS being unresolvable.

Xyne · 2021-11-15 14:46:36

linuxninja wrote:

The secondary mirror for Xyne's repo is still up.
Xyne: If you need a place to move your primary hosting, the offer still stands.
I was unable to email you due to the domain DNS being unresolvable.

I appreciate that, thanks! I'm going to sort out a new email address and I'll get back to you as soon as that's done.

toothandnail · 2021-12-13 13:28:41

This morning I upgraded (using pacserve...), getting a whole lot of python package upgrades. After a restart, pacserve is failing consistently:

[paulm@rigel ~]$ systemctl status pacserve
● pacserve.service - Pacserve
     Loaded: loaded (/usr/lib/systemd/system/pacserve.service; enabled; vendor preset: disabled)
     Active: activating (auto-restart) (Result: exit-code) since Mon 2021-12-13 13:05:44 GMT; 7s ago
    Process: 1835 ExecStart=/usr/bin/pacserve $PACSERVE_ARGS (code=exited, status=1/FAILURE)
   Main PID: 1835 (code=exited, status=1/FAILURE)
        CPU: 35ms

The error doesn't give me much, so I've no idea which upgrade caused the problem. One of my desktops, which hasn't had a restart still has an operational pacserve.

Xyne · 2021-12-13 20:42:32

What's the output of journalctl -xeu pacserve.service? What happens when you run pacserve directly from the command line?

toothandnail · 2021-12-14 07:51:01

Xyne wrote:

What's the output of journalctl -xeu pacserve.service? What happens when you run pacserve directly from the command line?

  journalctl -xeu pacserve.service

Dec 14 07:19:03 rigel systemd[1]: Started Pacserve.
░░ Subject: A start job for unit pacserve.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit pacserve.service has finished successfully.
░░ 
░░ The job identifier is 7077.
Dec 14 07:19:03 rigel pacserve[16324]: /usr/bin/python3: Error while finding module specification for 'ThreadedServers.Pacserve' (ModuleNotFoundError: No module named 'ThreadedServers')
Dec 14 07:19:03 rigel systemd[1]: pacserve.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit pacserve.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Dec 14 07:19:03 rigel systemd[1]: pacserve.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit pacserve.service has entered the 'failed' state with result 'exit-code'.
Dec 14 07:19:18 rigel systemd[1]: pacserve.service: Scheduled restart job, restart counter is at 42.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ Automatic restarting of the unit pacserve.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Dec 14 07:19:18 rigel systemd[1]: Stopped Pacserve.
░░ Subject: A stop job for unit pacserve.service has finished
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A stop job for unit pacserve.service has finished.
░░ 
░░ The job identifier is 7190 and the job result is done.
Dec 14 07:19:18 rigel systemd[1]: Started Pacserve.
░░ Subject: A start job for unit pacserve.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit pacserve.service has finished successfully.
░░ 
░░ The job identifier is 7190.
Dec 14 07:19:18 rigel pacserve[16350]: /usr/bin/python3: Error while finding module specification for 'ThreadedServers.Pacserve' (ModuleNotFoundError: No module named 'ThreadedServers')
Dec 14 07:19:18 rigel systemd[1]: pacserve.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit pacserve.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Dec 14 07:19:18 rigel systemd[1]: pacserve.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit pacserve.service has entered the 'failed' state with result 'exit-code'

[paulm@rigel ~]$ pacserve
/usr/bin/python3: Error while finding module specification for 'ThreadedServers.Pacserve' (ModuleNotFoundError: No module named 'ThreadedServers')

I need to read the journalctl man page - would be nice to be able to strip the colour codes for this sort of thing..... The journalctl entry is only a segment, since the unit attempts multiple restarts.

Just did a pacman update and added the new python3-threaded-servers. Back to normal output from systemctl status pacserve. Thanks for the quick fix

zoqaeski · 2021-12-28 23:52:08

I'm having an issue where my computers don't fetch the latest repository updates, because they always prefer to download the repository itself off the local network. So then pacman tells me that there are no updates and I can't update anything because the repo dbs are out of sync. If one of the computers is offline, the other one tries to get its updates from itself and never finds any updated packages.

The behaviour seemed to have changed in the last month or so, and I didn't notice until I had no new updates in almost two weeks. Telling pacserve to prefer the newest files (with --find-newest) breaks hosting of my local repo that I use for AUR packages.

Xyne · 2021-12-30 11:13:13

I haven't (intentionally) changed anything in the way Pacserve handles requests. It should pass through requests for database files to the first configured external mirror. Did you remove all other mirrors from your pacman configuration?

zoqaeski · 2021-12-31 09:19:13

Xyne wrote:

I haven't (intentionally) changed anything in the way Pacserve handles requests. It should pass through requests for database files to the first configured external mirror. Did you remove all other mirrors from your pacman configuration?

Nope, and I'm using Reflector to periodically update the mirrorlist (I think the systemd timer runs once a fortnight?). The only way I could get them to update was to disable Pacserve, run `pacman -Syy` to force refresh the mirrors followed by a `pacman -Syu` to download the packages to one machine, then reenable Pacserve so the other one could get its updates (repeating the process of disabling, refreshing, and re-enabling Pacserve). I'm holding the updates for a couple of packages due to non-repo dependencies that are incompatible with newer library versions, but that shouldn't be causing issues with the rest. All of my AUR and non-repo packages are built on the more powerful machine and served up from a custom repo via Pacserve (using the optional `<filepath>` argument). Makepkg builds the custom packages to /var/cache/pacman/makepkg/<repo-name>, and Pacserve is supposed to share that directory on the network.

Xyne · 2022-01-01 03:04:59

What's the output of the following command (replace the host and port with the values for the local pacserve server)?

curl -I 'http://localhost:15678/pacman/core/x86_64/core.db'

Are you using the --trust-pacserve-peers option? In that case you should basically have one system configured to act as a local hub by downloading the databases from the mirror directly.

zoqaeski · 2022-01-05 13:19:31

Xyne wrote:

What's the output of the following command (replace the host and port with the values for the local pacserve server)?
curl -I 'http://localhost:15678/pacman/core/x86_64/core.db'

The following command gives:

$ curl -I 'http://localhost:15678/pacman/core/x86_64/core.db'
HTTP/1.1 303 See Other
Server: ThreadedServers.Pacserve/2021.11.20
Date: Wed, 05 Jan 2022 13:12:19 GMT
Content-Type: text/plain; charset=UTF-8
Content-Length: 58
Location: https://mirror.osbeck.com/archlinux/core/os/x86_64/core.db
Connection: close

Are you using the --trust-pacserve-peers option? In that case you should basically have one system configured to act as a local hub by downloading the databases from the mirror directly.

No, I am not using that option on either peer. The pacserve command run by `pacserve.service` is:

pacserve --avahi --multicast /var/cache/pacman/makepkg/

I've just finished moving house and I haven't had the time to upgrade any of my computers for over a week, so I'll try updating on the weekend and see what happens.

Last edited by zoqaeski (2022-01-05 13:20:24)

Xyne · 2022-01-05 23:53:42

The command output shows that pacserve is correctly forwarding the request to the mirror. Try switching mirrors if the problem persists.

sysedwinistrator · 2022-01-25 22:33:27

Hi xyne,
thank you for this useful piece of software. I'm currently deploying it on my cluster of Arch Linux / Arch Linux ARM hosts.
Some of my hosts have multiple IP addresses on a bridge (one has a VRRP IP on its main interface, others have a bridge with dozens of addresses created by a Kubernetes networking plugin).
On those nodes, "pacserve --multicast" fails with "[Errno 98] Address already in use". An strace reveals the cause:

# LANG=C strace -f -e trace=network pacserve --multicast
strace: Process 3465684 attached
[pid 3465684] socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 3
[pid 3465684] setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
[pid 3465684] bind(3, {sa_family=AF_INET, sin_port=htons(15678), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid 3465684] getsockname(3, {sa_family=AF_INET, sin_port=htons(15678), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0
[pid 3465684] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
[pid 3465684] connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 3465684] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
[pid 3465684] connect(4, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 3465684] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
[pid 3465684] connect(4, {sa_family=AF_UNIX, sun_path="/run/systemd/resolve/io.systemd.Resolve"}, 42) = -1 ENOENT (No such file or directory)
[pid 3465684] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 4
[pid 3465684] connect(4, {sa_family=AF_UNIX, sun_path="/run/systemd/resolve/io.systemd.Resolve"}, 42) = -1 ENOENT (No such file or directory)
[pid 3465684] listen(3, 5)              = 0
[pid 3465684] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_IP) = 4
[pid 3465684] bind(4, {sa_family=AF_INET, sin_port=htons(15679), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
[pid 3465684] socket(AF_INET, SOCK_DGRAM|SOCK_CLOEXEC, IPPROTO_IP) = 5
[pid 3465684] setsockopt(4, SOL_IP, IP_ADD_MEMBERSHIP, {imr_multiaddr=inet_addr("224.3.45.67"), imr_interface=inet_addr("127.0.0.1")}, 8) = 0
[pid 3465684] setsockopt(4, SOL_IP, IP_ADD_MEMBERSHIP, {imr_multiaddr=inet_addr("224.3.45.67"), imr_interface=inet_addr("192.168.2.4")}, 8) = 0
[pid 3465684] setsockopt(4, SOL_IP, IP_ADD_MEMBERSHIP, {imr_multiaddr=inet_addr("224.3.45.67"), imr_interface=inet_addr("192.168.2.52")}, 8) = -1 EADDRINUSE (Address already in use)
[Errno 98] Address already in use
[pid 3465684] getsockname(3, {sa_family=AF_INET, sin_port=htons(15678), sin_addr=inet_addr("0.0.0.0")}, [16]) = 0
[pid 3465684] getpeername(3, 0xffffd654f458, [16]) = -1 ENOTCONN (Transport endpoint is not connected)
[pid 3465684] +++ exited with 71 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3465684, si_uid=0, si_status=71, si_utime=48, si_stime=15} ---
+++ exited with 71 +++

By default - when --multicast-server-address is not specified - pacserve tries to add each and every IP address the host has to the multicast group. However, on interfaces that have more than one IP address, it will always fail on the second one - because there already is an listening socket for the same multicast group on the same interface.

I first attempted to work around this by adding --multicast-server-address <interface>, in which case pacserve would only add the single interface's primary IP address to the multicast group and start. But with this configuration the hosts wouldn't see each other.
I think that is an example of what is explained in this stackoverflow answer. In short: Pacserve isn't receiving multicast traffic when specifying --multicast-server-address <name or IP address of some interface> because its socket is only bound to a regular unicast IP address, not to a multicast address. Even though the traffic shows up on the host (confirmed via tcpdump) after the interface in question has been added to the multicast group, the socket still needs to be bound either to the multicast address or to a wildcard address in order to receive it. Due to this the option --multicast-server-address can't perform its function of having the pacserve listening to multicast on one interface: If one specifies an interface name or an IP address of one, pacserve will correctly only add that interface to the multicast group, but bind its socket to the interface's IP address which will never receive multicast. If one specifies an multicast address (the group address), the socket would correctly be bound to an multicast address, but since no interface should have an multicast address directly assigned to it, pacserve will fail to find one to add to the multicast group.
This leads me to the conclusion that the address the multicast listening socket binds to (either wildcard or the multicast group) and the interfaces (or IP addresses thereof) that are added to the multicast group should be two different options, since they can't be the same.
Even in the default setting of --multicast-server-address 0.0.0.0 the socket is bound to the wildcard address, but the wildcard address is considered "unbound" - because adding 0.0.0.0 to a multicast group will result in the interface that has the default route being added to the that group and not in every interface currently available being added - and will not be added to the multicast group. Instead all of the host's IP addresses are being added to the multicast group individually:

ThreadedServers/common.py:
   285	def unbound_address(ip):
   286	    '''Check if the given IP is considered unbound.'''
   287	    return ip is None or str(ip) in ('', '0.0.0.0', '::')

ThreadedServers/Multicast.py, class MulticastServer:
   197	    def server_bind(self):
   198	        # The IP protocol version type.
   199	        ip_type = type(ipaddress.ip_address(self.multicast_group))
   200	
   201	        # Determine which addresses are available.
   202	        if unbound_address(self.server_address[0]):
   203	            addresses = (ip for (_, ip) in get_all_interfaces())
   204	        else:
   205	            addresses = (ipaddress.ip_address(self.server_address[0]),)
   206	
   207	        if self.allow_reuse_address:
   208	            self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
   209	        self.socket.bind((str(self.server_address[0]), self.server_address[1]))
   210	
   211	        for address in addresses:
   212	            # Skip mismatched IP versions.
   213	            if not isinstance(address, ip_type):
   214	                continue
   215	            mreq = ipaddress.ip_address(self.multicast_group).packed + address.packed
   216	            self.socket.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)

And this fails on any interface with more than one IP address. For that the solution would be to only add the primary IP addresses of the host's interfaces to the multicast group. I haven't looked into how to do that yet.

As a fix for --multicast-server-address I created a patch for ThreadedServers that always binds the multicast listening socket to the multicast group address instead of the specified --multicast-server-address. As per the quote in the stackoverflow answer I linked above, binding a multicast listening socket to a wildcard address is not particulary useful because it will receive any UDP traffic to that port and it should therefore be bound only to the multicast group address it will receive traffic from:

From 247646a95b8fec2e07e1a1517e3c5792dfc3cf16 Mon Sep 17 00:00:00 2001
From: Edwin Mackenzie-Owen <edwin.mowen@gmail.com>
Date: Tue, 25 Jan 2022 21:14:36 +0100
Subject: [PATCH] Make multicast listening socket always listen on multicast
 group address

---
 ThreadedServers/Multicast.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ThreadedServers/Multicast.py b/ThreadedServers/Multicast.py
index 305a7e0..134c88f 100644
--- a/ThreadedServers/Multicast.py
+++ b/ThreadedServers/Multicast.py
@@ -206,7 +206,7 @@ class MulticastServer(socketserver.UDPServer):
 
         if self.allow_reuse_address:
             self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-        self.socket.bind((str(self.server_address[0]), self.server_address[1]))
+        self.socket.bind((self.multicast_group, self.server_address[1]))
 
         for address in addresses:
             # Skip mismatched IP versions.
-- 
2.34.1

This suffices for my own usage of pacserve. For that use case of ThreadedServers, an option to specify the multicast listening socket's bind address is superflous: It can only be wildcard or the multicast group address, and I prefer the latter for the aforementioned reason. With my patch, the option --multicast-server-address only affects which interfaces are added to the multicast group.

Last edited by sysedwinistrator (2022-01-25 22:37:33)

Xyne · 2022-01-27 23:20:02

@sysedwinistrator
Before I got to the end of your post, I was worried that I would have to try to wrap my head around multicasting again after not having touched it for a while. The clear explanation of the issue and the patch were very much appreciated. The reasoning looks good to me so I've applied it. I'm pushing the update now.

Thanks!

hexadecagram · 2022-05-16 09:15:47

Coincidentally, I have a setup very similar to sysedwinstrator, with the exception that I am using CARP in place of VRRP and Docker without Kubernetes. But unfortunately even with his patch, I am getting "[Errno 98] Address already in use".

In my case, I have a bridge br0 on top of a bonding interface bond0, and not 1, not 2, but 5 (3 inet + 2 inet6) addresses assigned to br0.

I'm currently working around this by passing one of br0's addresses to --multicast-server-address, which enables pacserve to start. If I do not do this, it will fail to start.

If I understand the issue correctly, it can arise with any interface having more than 1 inet address. In my case, Docker is configured with "ipv6": true in daemon.json, so docker0 has 4 addresses (1 inet and 3 inet6), lo and a couple more more bridges (br-hexnum from libvirtd and docker_gwbridge) that have 2 addresses (1 inet + 1 inet6), and a whole lot of virtual interfaces with just 1 inet6 address. pacserve doesn't trip over any of those, and I surmise that the reason for that is because inet6 is unaffected by any of this. (If it isn't there already, perhaps inet6 support should be added at some point?)

It would seem that an adjustment is needed to accomodate interfaces with greater than 2 inet addresses (i.e., 3 or more).

It's probably also worth pointing out that with virtual IPs a la CARP and VRRP, these extra addresses can come and go as machines transition between online and offline.

Last edited by hexadecagram (2022-05-16 19:39:41)

hexadecagram · 2022-05-16 20:08:45

The plot thickens.

This afternoon, I deployed pacserve to all the Arch nodes in my LAN and I had a second machine require --multicast-server-address. It has a nearly identical network config as the other machine (br0 on bond0) and it's also runing CARP and Docker (but no libvirtd). The difference is, its br0 has 2 inet + 2 inet6 addresses. Another difference is that it does not require iptables -P FORWARD ACCEPT because it is not running libvirtd, in other words, this second machine retains the default DROP policy for the FORWARD chain per Docker defaults. Nevertheless, I have to use --multicast-server-address to get it to start.

So maybe inet6 addresses do have something to do with this after all?

Last edited by hexadecagram (2022-05-16 20:12:12)

Xyne · 2022-05-16 21:35:37

Does it work as expected when you pass -multicast-server-address? Are there multiple instances of pacserve running on the same physical machine?
I'll make a note of this but I probably won't be able to dig into this myself for a while. If you figure out what the issue is before then though, I'll try to implement it.

Condor · 2022-08-27 10:58:44

Hi,

I have an issue with pacserve 2021-4.

Recently, I set up aurutils with a custom local repository for my AUR packages and included this repository in pacman.conf. Since then, pacman -Syu throws this error:

:: Synchronizing package databases...
 core is up to date
 extra                                      1826,7 KiB  1627 KiB/s 00:01 [########################################] 100%
 community                                     6,8 MiB  4,62 MiB/s 00:01 [########################################] 100%
 multilib is up to date
 aur-repo is up to date
error: failed retrieving file 'core.db' from localhost:15678 : Empty reply from server
error: failed retrieving file 'multilib.db' from localhost:15678 : Empty reply from server
error: failed retrieving file 'aur-repo.db' from localhost:15678 : Empty reply from server
warning: too many errors from localhost:15678, skipping for the remainder of this transaction
error: failed retrieving file 'extra.db' from localhost:15678 : Empty reply from server
error: failed retrieving file 'community.db' from localhost:15678 : Empty reply from server

Thereafter, the update continues normally and finishes with success. Installing a single package is also without problems. If the package is not available locally, normal redirection to a mirror occurs.

The custom repository in pacman.conf has been added like this:

[aur-repo]
SigLevel = Optional TrustAll
Include = /etc/pacman.d/pacserve
Server = file:///home/condor/Software/aur-repo

In line with the ArchWiki description on setting up pacman to use pacserve, I have added the pacserve include in this section as well.

When performing pacman -Syu, this error shows in pacserve's log file, among others (see full log):

pacserve[7696]:   File "/usr/lib/python3.10/site-packages/ThreadedServers/Quickserve.py", line 432, in resolve_path
pacserve[7696]:     st = os.stat(localpath)
pacserve[7696]:   File "/usr/lib/python3.10/site-packages/ThreadedServers/HTTPS.py", line 854, in do_GET
pacserve[7696]:     self.do_authenticated_GET()
pacserve[7696]: PermissionError: [Errno 13] Permission denied: '/home/condor/Software/aur-repo/community.db'
pacserve[7696]:   File "/usr/lib/python3.10/site-packages/ThreadedServers/Quickserve.py", line 696, in do_authenticated_GET
pacserve[7696]:     self.do_authenticated_GET_or_HEAD()

Why does pacserve even look for core.db, community.db, extra.db and multilib.db at this location?
And why does a request for aur-repo.db fail? The file is present, although a world-readable symlink pointing to the world-readable aur-repo.tar.gz.

Xyne · 2022-09-11 21:55:44

@Condor
Please post your full pacman.conf file. There is nothing in Pacserve than can co-opt database requests. If pacman is requesting the database from pacserve then you have somehow configured pacman do to so.

The pacserve errors tell you that pacserve does not have permission to access the files under /home/condor/Software/aur-repo/community.db. Pacserve runs with limited priviliges, not as the root user. If you want it to work, you need to give it access to those files.

However, there is absolutely no point in enabling Pacserve for a local repo with a file:// URL. The entire point of pacserve is to download files from the cache of other systems on the local network before sending the request to a server. If the repo is already in a local directory, you gain nothing.

cloverskull · 2022-09-11 22:54:32

Hey friends, question for those of you that use this. I like to build my own tkg kernel but it takes some time and some decent compute cycles on my laptop. What I would love to do is build it in a cloud VM to my specifications and then serve it as a package to my laptop. Is that something I could do using this?

Condor · 2022-09-12 06:17:44

Please find the full pacman.conf here.
Note that there is one change to pacman.conf since my previous post: The line Include = /etc/pacman.d/pacserve has been commented out.

The errors reported by pacman -Syu and in pacserve's log file however remain the same. They only stop when commenting out the pacserve include from all active repositories.
Why would pacserve still try to access and serve /home/condor/Software/aur-repo/community.db?
Further, why should pacserve try to find a community.db (or core.db, multilib.db or extra.db) in the local repo folder at all? They are not there to begin with, there is only a aur-repo.db. Am I missing something here?

Xyne wrote:

However, there is absolutely no point in enabling Pacserve for a local repo with a file:// URL. The entire point of pacserve is to download files from the cache of other systems on the local network before sending the request to a server. If the repo is already in a local directory, you gain nothing.

Several PCs on my network use the same AUR packages. A couple of them take long to compile, which is quickest on my desktop PC. My intention was to share the built packages across my network, but I see that my approach does not work with pacserve this way.

Xyne · 2022-09-12 18:55:54

If it's looking for any database other than aur-repo.db under /home/condor/Software/aur-repo/ then it's a bug. That path is only specified in 2 locations in your configuration file:

CacheDir    = /var/cache/pacman/pkg/ /home/condor/Software/aur-repo/
# ...
[aur-repo]
SigLevel = Optional TrustAll
#Include = /etc/pacman.d/pacserve
Server = file:///home/condor/Software/aur-repo

So either it's erroneously looking in the cache directory for database files or it's somehow confusing servers for different repos. I suspect the former but I have to look into it. I'll post an update once I have an answer and hopefully a fix.
Just to check, have you configured it trust the peers for the database? (--trust-pacserve-peers)

For sharing the built package, you really just need a simple http server to serve the built packages from your desktop pc. Quickserve could do it, but you could probably easily find even simpler solutions.

@cloverskull
This is aimed at sharing packages between computers on the same lan. For the cloud VM I would also recommend running a little http server to serve the built packages directly to pacman.

edit
@Condor
The reason it's looking in the cache directory is because I implemented database retrieval in a very lazy way by just appending the DBPath to the CacheDir array and allowed a passthrough. Given that databases and packages use different suffixes there is no risk of a name collision error and it kept the code simple. The error in this case is that it fails when trying to read the unreadable cache directory before trying to read the database directory.

As refactoring pacserve to use modern third-party libraries is on my todo list I'm going to leave the current lazy implementation as is. I'll add an exception handler for the permission errors for now. I would also recommend not using the option to share databases unless a few MB really make a difference on your connection.

Last edited by Xyne (2022-09-12 19:41:13)

Condor · 2022-09-13 15:20:44

Xyne wrote:

So either it's erroneously looking in the cache directory for database files or it's somehow confusing servers for different repos.

I tried some more changes to the pacman.conf to narrow it down:

Modification of the CacheDir option (going back to the default value):
```
CacheDir    = /var/cache/pacman/pkg/
```
==> No change. Pacserve is still trying to look up {core,community,extra,multilib}.db in /home/condor/Software/aur-repo and erroring out. I guess the cache directory approach is not a likely cause.
Modification of the Include option of the repos, commenting out the pacserve include:
```
[core]
#Include = /etc/pacman.d/pacserve
Include = /etc/pacman.d/mirrorlist
```
==> Change in behavior: The previously displayed error “failed retrieving file 'core.db' from localhost:15678 : Empty reply from server” is gone. For {extra,community,multilib}.db, still displayed. Similar, when removing the pacserve include from the other repos, their errors will stop.

I hope that helps to narrow down the cause.

Xyne wrote:

Just to check, have you configured it trust the peers for the database? (--trust-pacserve-peers)

I don't have that option configured. The only configuration set in /etc/pacserve/pacserve.service.conf is the --multicast option.

Thank you for looking up the possible cause of the error (and pacserve itself!). I will work around the issue until a fix is available.

Arch Linux

#251 2021-11-12 01:33:18

Re: pacserve - easily share Pacman packages between computers

#252 2021-11-14 11:16:25

Re: pacserve - easily share Pacman packages between computers

#253 2021-11-14 21:00:12

Re: pacserve - easily share Pacman packages between computers

#254 2021-11-15 02:34:35

Re: pacserve - easily share Pacman packages between computers

#255 2021-11-15 14:46:36

Re: pacserve - easily share Pacman packages between computers

#256 2021-12-13 13:28:41

Re: pacserve - easily share Pacman packages between computers

#257 2021-12-13 20:42:32

Re: pacserve - easily share Pacman packages between computers

#258 2021-12-14 07:51:01

Re: pacserve - easily share Pacman packages between computers

#259 2021-12-28 23:52:08

Re: pacserve - easily share Pacman packages between computers

#260 2021-12-30 11:13:13

Re: pacserve - easily share Pacman packages between computers

#261 2021-12-31 09:19:13

Re: pacserve - easily share Pacman packages between computers

#262 2022-01-01 03:04:59

Re: pacserve - easily share Pacman packages between computers

#263 2022-01-05 13:19:31

Re: pacserve - easily share Pacman packages between computers

#264 2022-01-05 23:53:42

Re: pacserve - easily share Pacman packages between computers

#265 2022-01-25 22:33:27

Re: pacserve - easily share Pacman packages between computers

#266 2022-01-27 23:20:02

Re: pacserve - easily share Pacman packages between computers

#267 2022-05-16 09:15:47

Re: pacserve - easily share Pacman packages between computers

#268 2022-05-16 20:08:45

Re: pacserve - easily share Pacman packages between computers

#269 2022-05-16 21:35:37

Re: pacserve - easily share Pacman packages between computers

#270 2022-08-27 10:58:44

Re: pacserve - easily share Pacman packages between computers

#271 2022-09-11 21:55:44

Re: pacserve - easily share Pacman packages between computers

#272 2022-09-11 22:54:32

Re: pacserve - easily share Pacman packages between computers

#273 2022-09-12 06:17:44

Re: pacserve - easily share Pacman packages between computers

#274 2022-09-12 18:55:54

Re: pacserve - easily share Pacman packages between computers

#275 2022-09-13 15:20:44

Re: pacserve - easily share Pacman packages between computers

Board footer