pacserve - easily share Pacman packages between computers

Pebby · 2014-09-04 20:50:40

Hi Xyne: First, thank you for this valuable tool. Helps me save a ton of bandwidth on my multi-machine environment. I did want to report a bug (or feature?) when dealing with certain packages that have odd formatting/names that causes pacman to get stuck in an infinite loop of failed queries. For example, dialog-1:1.2_20140219-1 causes the following (infinite) output:

Sep 04 13:46:10 d2sparrow pacserve[387]: [2014-09-04 13:46:10 PDT] INFO: 10.0.0.142 "POST / HTTP/1.1" 200 -
Sep 04 13:46:10 d2sparrow pacserve[387]: [2014-09-04 13:46:10 PDT] INFO: POSTing to http://10.0.0.1:15678/ [type: file check]
Sep 04 13:46:10 d2sparrow pacserve[387]: [2014-09-04 13:46:10 PDT] INFO: 10.0.0.142 redirecting to http://d2sparrow:15678/pkg/?repo=core&arch=x86_64&file=/dialog-1:1.2_20140219-1-x86_64.pkg.tar.xz
Sep 04 13:46:10 d2sparrow pacserve[387]: [2014-09-04 13:46:10 PDT] INFO: 10.0.0.142 tell the Pacman devs to url-decode their redirects
Sep 04 13:46:10 d2sparrow pacserve[387]: [2014-09-04 13:46:10 PDT] INFO: 10.0.0.142 "GET /pkg/?repo=core&arch=x86_64&file=/dialog-1:1.2_20140219-1-x86_64.pkg.tar.xz HTTP/1.1" 303 -
Sep 04 13:46:11 d2sparrow pacserve[387]: [2014-09-04 13:46:11 PDT] INFO: POSTing to http://10.0.0.142:15678/ [type: file check]

It's not a big deal since I can just work around it in this rare case, but let me know if there's anything I can do to assist.

Xyne · 2014-09-04 22:37:11

I am unable to reproduce this. With the local pacserver as my first mirror in the mirrorlist and a second pacserve with the dialog package on another system I am able to retrieve the package with Pacman.

Normally it should prevent infinite loops by tracking previous queries. There may be a bug due to the absence of url-encoding when dealing with Pacman (see the 4th line in your output: there's a bug in Pacman that requires a workaround in Pacserve).

Are d2sparrow and 10.0.0.142 different systems? Does either system have the dialog package? It shouldn't be redirecting after a file check unless the package was found. What is the output on the other system?

schmoken · 2015-03-26 04:25:36

I have multiple machines running pacserve. Is there a way to set a priority as to which machine gets queried first?
I would prefer the machines on the wired ethernet serve the files before those only on wi-fi.

Xyne · 2015-03-26 20:49:22

@schmoken
I have to check to be sure but I don't think there is any way at the moment. I like the concept though and I have an idea of how to implement it (each pacserver could report an arbitrary "preference" value that the querying server could use to order servers that it detects). I need to look at how queries are propagated and give it some more thought.

ploub · 2015-10-06 09:06:17

Hi,
Pacserve stopped working for me on both my desktop and laptop with this error output:

Oct 06 10:32:26 mesh systemd[1]: Started Pacserve.
Oct 06 10:32:26 mesh pacserve[9466]: /usr/sbin/python3: Error while finding spec for 'ThreadedServers.Pacserve' (<class 'ImportError'>: No module named 'ThreadedServers')
Oct 06 10:32:26 mesh systemd[1]: pacserve.service: Main process exited, code=exited, status=1/FAILURE
Oct 06 10:32:26 mesh systemd[1]: pacserve.service: Unit entered failed state.
Oct 06 10:32:26 mesh systemd[1]: pacserve.service: Failed with result 'exit-code'.
Oct 06 10:32:42 mesh systemd[1]: pacserve.service: Service hold-off time over, scheduling restart.
Oct 06 10:32:42 mesh systemd[1]: Stopped Pacserve.

This repeats every 15 seconds. I'm not sure when this started but I think it was OK a month ago. I have -Syu'd and also re-installed python3-threaded_servers with no change.
Help anyone?

olebowle · 2015-10-06 10:34:09

Due to the new python 3.5 release you need to rebuild python3-threaded_servers from the AUR. I flagged the package out-of-date, but it wasn't updated yet. Xyne is probably busy right now.

Edit: Ok, I didn't read your post carefully enough. What is the output of:

$ pacman -Q python
$ pacman -Ql python3-threaded_servers

Do you use a repo provided by Xyne?

Last edited by olebowle (2015-10-06 10:49:34)

ploub · 2015-10-06 11:03:48

olebowle wrote:

Due to the new python 3.5 release you need to rebuild python3-threaded_servers from the AUR. I flagged the package out-of-date, but it wasn't updated yet. Xyne is probably busy right now.
Edit: Ok, I didn't read your post carefully enough. What is the output of:
$ pacman -Q python
$ pacman -Ql python3-threaded_servers
Do you use a repo provided by Xyne?

Yes I use xyne-x86_64. Anyway I rebuilt from the AUR and all seems to be well.

Thanks - no error messages so I think I'm now up and running.

zoqaeski · 2015-10-14 04:29:07

I'm getting a weird response from pacserve. Each time I update, it'll work fine for the first few packages, and then suddenly seemingly stop responding and pacman will spit out error messages for each package along the lines of "error: failed retrieving file 'PACKAGE.pkg.tar.xz' from localhost:15678 : Operation too slow. Less than 1 bytes/sec transferred the last 10 seconds".

Xyne · 2015-10-18 17:34:13

Describe your setup. Are you running Pacserve on the same machine or another machine. What exactly leads to the error. Run Pacserve in a terminal and post the output. You haven't given me any information to help you.

Ian Kelling · 2015-11-28 23:01:39

patch: make idempotent pacman.conf-insert_pacserve

--- /usr/bin/pacman.conf-insert_pacserve	2015-11-28 07:16:56.000000000 -0800
+++ /a/bin/pacman.conf-insert_pacserve	2015-11-28 07:45:44.199596186 -0800
@@ -13,7 +13,9 @@
       if line and line[0] == '[' and line[-1] == ']':
         section = line[1:-1]
       elif section and section != 'options':
-        if line[:8] == 'Include ' or line[:7] == 'Server ':
+        if line == PACSERVER:
+          section = None
+        elif line[:8] == 'Include ' or line[:7] == 'Server ':
           config += PACSERVER + '\n'
           section = None
       config += line + '\n'

Xyne · 2015-11-30 15:48:00

Done. I'll upload shortly.

gunzy83 · 2015-12-06 11:49:57

Latest update from your repo gives the following error:

error: pacserve: signature from "Xyne. (key #3) <xyne@archlinux.ca>" is invalid
:: File /var/cache/pacman/pkg/pacserve-2015.11-1-any.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] n
error: failed to commit transaction (invalid or corrupted package)

I saw that that his same error has been posted before months ago on another version of the package but just thought you would like to know.

Lastly, thank you for the packages (in particular pacserve, reflector and quickserve), they are fantastic.:)

Xyne · 2015-12-06 15:35:51

It should be fixed now. I have added additional checks to my release scripts to try to detect such errors.

gunzy83 · 2015-12-07 12:39:47

Xyne wrote:

It should be fixed now. I have added additional checks to my release scripts to try to detect such errors.

Working again now. Thanks

toothandnail · 2017-10-09 15:14:08

Been a while since the last post in this area....

I've been using pacserver for a while. Very useful since I've got multiple Arch installs. In the last few days, I've started getting this error on every attempt to use it:

error: failed retrieving file 'core.db' from localhost:15678 : Connection timed out after 10972 milliseconds

I'm not having problems accessing localhost on other ports. Pacserver is running on all the Arch machines, so I'm at a bit of a loss to know what is going on. Can anyone point me in the right direction to find what is causing the problem? Its happening on all the Arch installs. Normal repo access is working fine, but I'd like to get back the ability to share downloaded packages between the machines.

Paul.

Xyne · 2017-10-09 17:28:51

Hi Paul,

The database requests should be forwarded to the first mirror in your mirrorlist. Try changing mirrors to see if this is due to a transient server error. You can test the download directly after changing the mirrorlist with http://localhost:15678/pacman/core/x86_64/core.db (replace x86_64 with i686 if you're running i686, and change 15678 if you are using a different port).

toothandnail · 2017-10-10 05:48:59

Xyne wrote:

Hi Paul,
The database requests should be forwarded to the first mirror in your mirrorlist. Try changing mirrors to see if this is due to a transient server error. You can test the download directly after changing the mirrorlist with http://localhost:15678/pacman/core/x86_64/core.db (replace x86_64 with i686 if you're running i686, and change 15678 if you are using a different port).

Thanks Xyne. That did it, once I'd remembered to restart the pacserve process. Seems that one of the German mirrors has gone offline completely, even though its normally faster than the more local UK mirrors. I must have got lax in updating my mirrolist.

Looks as though that mirror may have been having problems for a while - I notice that it is now a lot quicker starting to retrieve packages than it has been for some time.

Paul.

kete · 2017-10-13 13:55:26

I don't understand why I'm getting an "ERROR: POST to http://10.10.1.2:15678/ failed [reason: [Errno 111] Connection refused]". pacserve-ports is active on the 10.10.1.2 computer. (Packages still download from the mirrors.) I used [xyne-x86_64]
Server = http://xyne.archlinux.ca/repos/xyne on the host computer and the AUR package on the client.

PS—It seems to be working the other way. It's much faster, but systemctl status still says it's redirecting to a mirror. There's no error. One computer doesn't have iptables set up yet. Maybe I should manually open this port in iptables.

PPS—I got through by stopping pacserve-ports and adding the rules to the TCP and UDP chains that the Arch wiki demonstrated: so fast, now. (instantaneous)

Last edited by kete (2017-10-14 21:47:30)

Xyne · 2017-10-15 12:43:45

kete wrote:

PPS—I got through by stopping pacserve-ports and adding the rules to the TCP and UDP chains that the Arch wiki demonstrated: so fast, now. (instantaneous)

Can you post the rules that you used? I want to understand why pacserve-ports did not work.

aphirst · 2017-10-15 14:25:32

Hi Xyne. I've been using pacserve for a few months now with no issues, on a handful of machines all running Arch x86_64.

Since pacserve is in the AUR, I decided to try it out on my handful of machines running Archlinux ARM, and things seem to work well between each other.

However, I've just started to notice that sometimes these ALARM machines fail to be able to -Syu, and the reason seems to be that they're seeing -any packages on the vanilla Arch pacserve instances, getting those, and then summarily rejecting them due to checksum differences (since, indeed, Archlinux ARM packages are built entirely independently).

I'm aware that one isn't supposed to bring up Archlinux ARM here, but this thread does seem to be the "official" one for pacserve.

Is there a way to handle this situation, namely the coexistence of pacserve instances running different "flavours" of Arch? All I can think of would be either

for Archlinux ARM to build its -any packages identically to vanilla arch (though since this goes on checksum, and since packages are signed, it'd mean having to literally reuse the compiled package files)
for pacserve to work out a way to distinguish between different Arch sub-distros/flavours
for pacserve to force an upstream retry (or at least a different LAN pacserve client) when getting a checksum failure like this (i.e. indicating this problem)
manually stop pacserve when trying a problematic update

Of course there are no issues when packages differ by -architecture - it's only on this special -any case where things seem to differ, due to the package filename aliasing. Though, I can imagine there being similar problems if one were to use packages from non-official repositories which were "different" to those upstream but used the same name/version/architecture (i.e. if they aliased the official repo packages).

Xyne · 2017-10-15 18:11:58

HI aphirst,

aphirst wrote:

for Archlinux ARM to build its -any packages identically to vanilla arch (though since this goes on checksum, and since packages are signed, it'd mean having to literally reuse the compiled package files)

There should not be any compiled files in an "any" package. The very purpose of the "any" designation is to indicate that the package will run on any architecture precisely because it contains no such files. A common misconception is that a package that can be built for any architecture should use the "any" designation when it should actually explicitly list all of the architectures in the arch array of its PKGBUILD. Is this the case here?

If the error is due to misuse of "any" packages, get the maintainers to correct the PKGBUILDs.
If the error is due to ARM repositories independently building their own "any" packages instead of re-using existing ones (thereby resulting in equivalent packages that differ only due to compression), then either convince the repo maintainer(s) to re-use the official "any" packages, or split your pacserve network in two: one for supported machines that use the official packages, and one for your ARM machines that use separate repositories. There is no point in having a common pacserve network if the "any" packages differ: your ARM machines will never find a valid package on your non-ARM machines and vice versa, but each server will still scan its cache and slow down the retrieval.

Packages do not contain metadata about the repo from which they were downloaded and Pacserve is essentially repo-agnostic. It assumes that the package's filename (<pkgname>-<pkgver>-<pkgrel>-<arch>-<ext>) is unique and that all packages with identical names are interchangeable, with the further assumption that each pacserve server operates within a coherent package ecosystem. While it would be possible to add checksum verifications to pacserve, it would require a lot of overhead and redundancy (e.g. multiple checksumming of the same file, passing checksums between pacserve servers for every file). Checksumming the package is pacman's job.

Until now I have assumed that all machines on the same pacserve network will use compatible repo configurations, but I recognize that it is a valid issue in the case of different repos shadowing each others package names. I will post an update if I find a simple solution.

edit
I realized that there is no way to use the checksum to determine the correct file because pacman does not pass that information when requesting the package. While I could integrate that into the internal pacserve support of powerpill/bauerbill, pacserve has to remain compatible with pacman.

Last edited by Xyne (2017-10-15 18:18:12)

aphirst · 2017-10-15 19:05:55

I think splitting my LAN into two separate networks purely for the sake of pacserve is WAY out of scope for the ~6 machine network I have at home.

If the error is due to ARM repositories independently building their own "any" packages instead of re-using existing ones (thereby resulting in equivalent packages that differ only due to compression)

Would they not also differ due to package signing?
I'll ask the guys at #archlinuxarm if they have any particular reason to not reuse the -any packages from upstream, but one thing which occurs to immediately is that they will often need to be independently repackaged to account for changes in packages on which they depend - and there's no guarantee that the number of pkgrel bumps (or even nature of them!) of an arch-based distro will be synchronised with vanilla arch.

progandy · 2017-10-15 19:45:12

aphirst wrote:

I think splitting my LAN into two separate networks purely for the sake of pacserve is WAY out of scope for the ~6 machine network I have at home.

I think the "pacserve network" can be split by selecting different multicast ports or maybe different multicast groups?

aphirst · 2017-10-15 20:09:09

Hmm, I'm going to have to look into what exactly multicast is and how it works. But, would I be right in assuming that I could set all the ALARM machines' pacserve multicast to, say, 224.3.45.68 (the pacserve default is 224.3.45.67)? Obviously this guess has been made in total ignorance of any actual meaning of the parts of a multicast address, but that's due to the need to read into this.

eschwartz · 2017-10-15 20:23:39

aphirst wrote:

If the error is due to ARM repositories independently building their own "any" packages instead of re-using existing ones (thereby resulting in equivalent packages that differ only due to compression)
Would they not also differ due to package signing?
I'll ask the guys at #archlinuxarm if they have any particular reason to not reuse the -any packages from upstream, but one thing which occurs to immediately is that they will often need to be independently repackaged to account for changes in packages on which they depend - and there's no guarantee that the number of pkgrel bumps (or even nature of them!) of an arch-based distro will be synchronised with vanilla arch.

The signatures are not contained within the package files, the signatures are signatures *of* the package files and cannot be self-referential.

That being said, the PACKAGER and build date metadata will certainly differ, and potentially other parts of the .BUILDINFO in addition to the files themselves having unpredictable mtimes. development versions of pacman can reproducibly build packages though.

The archlinuxarm repositories should consider reusing our "any" packages, but if they really need to change the package just to work on ARM then I suspect they don't qualify as "any" packages.

Arch Linux

#151 2014-09-04 20:50:40

Re: pacserve - easily share Pacman packages between computers

#152 2014-09-04 22:37:11

Re: pacserve - easily share Pacman packages between computers

#153 2015-03-26 04:25:36

Re: pacserve - easily share Pacman packages between computers

#154 2015-03-26 20:49:22

Re: pacserve - easily share Pacman packages between computers

#155 2015-10-06 09:06:17

Re: pacserve - easily share Pacman packages between computers

#156 2015-10-06 10:34:09

Re: pacserve - easily share Pacman packages between computers

#157 2015-10-06 11:03:48

Re: pacserve - easily share Pacman packages between computers

#158 2015-10-14 04:29:07

Re: pacserve - easily share Pacman packages between computers

#159 2015-10-18 17:34:13

Re: pacserve - easily share Pacman packages between computers

#160 2015-11-28 23:01:39

Re: pacserve - easily share Pacman packages between computers

#161 2015-11-30 15:48:00

Re: pacserve - easily share Pacman packages between computers

#162 2015-12-06 11:49:57

Re: pacserve - easily share Pacman packages between computers

#163 2015-12-06 15:35:51

Re: pacserve - easily share Pacman packages between computers

#164 2015-12-07 12:39:47

Re: pacserve - easily share Pacman packages between computers

#165 2017-10-09 15:14:08

Re: pacserve - easily share Pacman packages between computers

#166 2017-10-09 17:28:51

Re: pacserve - easily share Pacman packages between computers

#167 2017-10-10 05:48:59

Re: pacserve - easily share Pacman packages between computers

#168 2017-10-13 13:55:26

Re: pacserve - easily share Pacman packages between computers

#169 2017-10-15 12:43:45

Re: pacserve - easily share Pacman packages between computers

#170 2017-10-15 14:25:32

Re: pacserve - easily share Pacman packages between computers

#171 2017-10-15 18:11:58

Re: pacserve - easily share Pacman packages between computers

#172 2017-10-15 19:05:55

Re: pacserve - easily share Pacman packages between computers

#173 2017-10-15 19:45:12

Re: pacserve - easily share Pacman packages between computers

#174 2017-10-15 20:09:09

Re: pacserve - easily share Pacman packages between computers

#175 2017-10-15 20:23:39

Re: pacserve - easily share Pacman packages between computers

Board footer