pacserve - easily share Pacman packages between computers

eduardo.eae · 2019-02-24 15:42:59

I followed the wiki, opened ports manually (others servers are being added correctly), added the "Include" line to each Repo of my pacman.conf, but pacman seems to enter an ethernal loop. Looking at the logs I get this:

redirecting to http://localhost:15678/pacman/core/x86_64/core.db
tell the Pacman devs to url-decode their redirects
"GET /pacman/core/x86_64/core.db HTTP/1.1" 303 -

Repeated forever

* Fixed. Turns out I had no "127.0.0.1 localhost" in my /etc/hosts file, oops.

Last edited by eduardo.eae (2019-02-27 00:59:34)

Xyne · 2019-02-27 20:41:15

@connection_reset
Thanks for reporting the error and posting patches and links. I've integrated both in today's release of python3-threaded_servers.

I was about to write "welcome to the forum" after seeing your post count, but then I saw the registration date and wondered if I should apologize for breaking your silence with my bug. Reminds me of this scene

@eduardo.eae
Glad you got it working

eduardo.eae · 2019-03-02 15:20:50

Now I have another problem in another network:

WARNING: unrecognized multicast message from 192.168.1.100

I see this message in both PCs in this network.
Maybe my router is doing something funny with multicast?
Multicast was set to disabled in the router. I set it to IGMP v3 and it didn't change anything.

Xyne · 2019-03-03 03:56:40

Did the warnings appear after Wednesday's upgrade? Do the the servers detect each other?
If you provide some details about your network setup, it will help me to try to reproduce the warning and fix it.

eduardo.eae · 2019-03-04 22:49:11

Now it is working as intended, and I have changed nothing

Xyne · 2019-03-06 21:32:41

eduardo.eae wrote:

Now it is working as intended, and I have changed nothing

Strange. If it happens again, post the setup and full console output.

connection_reset · 2019-03-17 10:00:19

eduardo.eae wrote:

WARNING: unrecognized multicast message from 192.168.1.100

This happened to me too just now.
I updated one installation (out of multiple) from python3-threaded_servers version 2019.2.3 (with my patches) to Xyne's latest 2019.2.27 and got that warning and no peers.
The multicast message contains an identifier with the version and is discarded (with that warning) if it doesn't match, so apparently different versions don't work together.

alex.forencich · 2019-06-06 04:49:08

Xyne wrote:

@alex.forencich
The underlying HTTP server is the one provided by the Python standard library and it is limited to listening to either all or a single interface at a time (unless that's changed since I last checked). If you bind it to a non-local interface, update /etc/pacman.d/pacserve, /etc/powerpill/powerpill.json or whatever else uses pacserve to point to the bound interface instead of localhost.
The configuration files are located in /etc/pacserve. I have updated the project page.

How do I do that when the interface address comes from DHCP?

Edit: I also tried playing around with some of the command line parameters. Adding ---multicast-interface eno1 seems to do nothing. Adding --multicast-server-address eno1 causes pacserve to fail with:

pacserve[11855]: Traceback (most recent call last):
pacserve[11855]:   File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
pacserve[11855]:     "__main__", mod_spec)
pacserve[11855]:   File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
pacserve[11855]:     exec(code, run_globals)
pacserve[11855]:   File "/usr/lib/python3.7/site-packages/ThreadedServers/Pacserve.py", line 325, in <module>
pacserve[11855]:     run(main)
pacserve[11855]:   File "/usr/lib/python3.7/site-packages/ThreadedServers/common.py", line 413, in run
pacserve[11855]:     main()
pacserve[11855]:   File "/usr/lib/python3.7/site-packages/ThreadedServers/Pacserve.py", line 314, in main
pacserve[11855]:     page_generators,
pacserve[11855]:   File "/usr/lib/python3.7/site-packages/ThreadedServers/Pacserve.py", line 108, in __init__
pacserve[11855]:     super().__init__(*args, **kwargs)
pacserve[11855]:   File "/usr/lib/python3.7/site-packages/ThreadedServers/PeeredQuickserve.py", line 207, in __init__
pacserve[11855]:     self.start_multicast_threads()
pacserve[11855]:   File "/usr/lib/python3.7/site-packages/ThreadedServers/Multicast.py", line 324, in start_multicast_threads
pacserve[11855]:     self.options.multicast_group
pacserve[11855]:   File "/usr/lib/python3.7/site-packages/ThreadedServers/Multicast.py", line 214, in __init__
pacserve[11855]:     super().__init__(*args, **kwargs)
pacserve[11855]:   File "/usr/lib/python3.7/site-packages/ThreadedServers/Multicast.py", line 176, in __init__
pacserve[11855]:     socketserver.UDPServer.__init__(self, server_address, handler, *args, **kwargs)
pacserve[11855]:   File "/usr/lib/python3.7/socketserver.py", line 452, in __init__
pacserve[11855]:     self.server_bind()
pacserve[11855]:   File "/usr/lib/python3.7/site-packages/ThreadedServers/Multicast.py", line 195, in server_bind
pacserve[11855]:     self.socket.bind(self.server_address)
pacserve[11855]: TypeError: str, bytes or bytearray expected, not IPv4Address

Last edited by alex.forencich (2019-06-06 04:53:19)

Xyne · 2019-06-15 03:06:12

Passing the interface name is the right way to handle dhcp. The bug in --multicast-server-address should be fixed with today's release.

The --multicast-interface selects the interface to which multicast announcements are sent, not received.

toothandnail · 2020-12-02 07:55:18

Just installed 400+ MiB of updates. Including python-3.9.0. I'm now getting pacserve failures:

[paulm@rigel build]$ systemctl status pacserve
● pacserve.service - Pacserve
     Loaded: loaded (/usr/lib/systemd/system/pacserve.service; enabled; vendor preset: disabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2020-12-02 07:46:55 GMT; 4s ago
    Process: 10110 ExecStart=/usr/bin/pacserve $PACSERVE_ARGS (code=exited, status=1/FAILURE)
   Main PID: 10110 (code=exited, status=1/FAILURE)

Rebuilt both pacserve and python3-threaded_servers, still get the same failure.

Xyne · 2020-12-02 23:19:51

The error was due to a built-in library function's renaming. It's been fixed in today's update.

toothandnail · 2020-12-03 08:34:21

Xyne wrote:

The error was due to a built-in library function's renaming. It's been fixed in today's update.

Thanks. Working fine again....

beroal · 2021-04-13 10:27:07

Hi, Xyne. Thank you for your program. Would you like to add to the wiki how to check that `pacserve` sees other computers?

zoqaeski · 2021-06-07 02:57:04

With the most recent update of pacman (6.0.0), I'm getting lots of HTTP 400 errors from pacserve which is causing pacman to fail to update anything unless pacserve is disabled. It usually downloads about three or four packages, and then spits out a bunch of "Failed retrieving file '<PACKAGE>' from localhost:15678 : The requested URL returned error: 400" and gives up.

Xyne · 2021-06-18 22:35:45

@zoqaeski
What errors does pacserve report on the console?

I've just noticed an error in the multicast configuration introduced by a recent update but I haven't had the time to figure it out yet. I don't know if that's related but I'll try to solve that first.

edit: The multicast issue should be fixed now. If the 400 errors persist, please post the console or log output from pacserve so that I can try to debug it.

Last edited by Xyne (2021-06-19 09:28:00)

zoqaeski · 2021-06-28 02:53:29

Xyne wrote:

@zoqaeski
What errors does pacserve report on the console?
I've just noticed an error in the multicast configuration introduced by a recent update but I haven't had the time to figure it out yet. I don't know if that's related but I'll try to solve that first.
edit: The multicast issue should be fixed now. If the 400 errors persist, please post the console or log output from pacserve so that I can try to debug it.

It's really irregular and sporadic, so I had issues and then they went away without Pacman or Pacserve being updated in the meantime. Then I go to install a package today, and it fails with:

❯ sudo pacman -S ocrfeeder
[sudo] password for root:
resolving dependencies...
looking for conflicting packages...

Package (6)                 New Version  Net Change  Download Size

community/goocanvas         3.0.0-1        2.86 MiB       0.28 MiB
extra/python-defusedxml     0.7.1-2        0.13 MiB       0.03 MiB
community/python-odfpy      1.4.1-4        1.33 MiB       0.20 MiB
community/python-reportlab  3.5.67-1       8.59 MiB       2.32 MiB
community/python-sane       2.8.3-6        0.06 MiB       0.02 MiB
community/ocrfeeder         0.8.3-5        2.27 MiB       0.78 MiB

Total Download Size:    3.63 MiB
Total Installed Size:  15.25 MiB

:: Proceed with installation? [Y/n]
:: Retrieving packages...
 python-sane-2.8.3-6-x86_64                                                                          18.2 KiB  79.1 KiB/s 00:00 [##############################################################################] 100%
 python-defusedxml-0.7.1-2-any                                                                       33.5 KiB   144 KiB/s 00:00 [##############################################################################] 100%
 python-odfpy-1.4.1-4-any                                                                           201.6 KiB   738 KiB/s 00:00 [##############################################################################] 100%
 goocanvas-3.0.0-1-x86_64                                                                           284.1 KiB  1026 KiB/s 00:00 [##############################################################################] 100%
 ocrfeeder-0.8.3-5-any                                                                              797.5 KiB  4.40 MiB/s 00:00 [##############################################################################] 100%
 python-reportlab-3.5.67-1-x86_64                                                                     2.3 MiB  5.19 MiB/s 00:00 [##############################################################################] 100%
 Total (6/6)                                                                                          3.6 MiB  7.83 MiB/s 00:00 [##############################################################################] 100%
error: failed retrieving file 'python-reportlab-3.5.67-1-x86_64.pkg.tar.zst.sig' from 192.168.1.197:15678 : The requested URL returned error: 400
warning: failed to retrieve some files
error: failed to commit transaction (failed to retrieve some files)
Errors occurred, no packages were upgraded.

I last updated on the weekend (Saturday?), and I don't think the turnaround for updates on remote mirrors is that quick. Running the same command a second time, and it works without a hitch:

❯ sudo pacman -S ocrfeeder
resolving dependencies...
looking for conflicting packages...

Package (6)                 New Version  Net Change

community/goocanvas         3.0.0-1        2.86 MiB
extra/python-defusedxml     0.7.1-2        0.13 MiB
community/python-odfpy      1.4.1-4        1.33 MiB
community/python-reportlab  3.5.67-1       8.59 MiB
community/python-sane       2.8.3-6        0.06 MiB
community/ocrfeeder         0.8.3-5        2.27 MiB

Total Installed Size:  15.25 MiB

:: Proceed with installation? [Y/n]
:: Retrieving packages...
 python-reportlab-3.5.67-1-x86_64 is up to date
(6/6) checking keys in keyring                                                                                                  [##############################################################################] 100%
(6/6) checking package integrity                                                                                                [##############################################################################] 100%
(6/6) loading package files                                                                                                     [##############################################################################] 100%
(6/6) checking for file conflicts                                                                                               [##############################################################################] 100%
(6/6) checking available disk space                                                                                             [##############################################################################] 100%
:: Running pre-transaction hooks...
(1/1) Performing snapper pre snapshots for the following configurations...
==> root: 617
:: Processing package changes...
(1/6) installing goocanvas                                                                                                      [##############################################################################] 100%
(2/6) installing python-defusedxml                                                                                              [##############################################################################] 100%
(3/6) installing python-odfpy                                                                                                   [##############################################################################] 100%
(4/6) installing python-reportlab                                                                                               [##############################################################################] 100%
(5/6) installing python-sane                                                                                                    [##############################################################################] 100%
(6/6) installing ocrfeeder                                                                                                      [##############################################################################] 100%
Optional dependencies for ocrfeeder
    cuneiform: character recognition
    gocr: character recognition
    ocrad: character recognition
    tesseract: character recognition [installed]
:: Running post-transaction hooks...
(1/4) Arming ConditionNeedsUpdate...
(2/4) Updating icon theme caches...
(3/4) Updating the desktop file MIME type cache...
(4/4) Performing snapper post snapshots for the following configurations...
==> root: 618

~                                                                                                                                                                         took 3s at 2021-06-28 12:53:44
❯

Last edited by zoqaeski (2021-06-28 02:55:23)

Xyne · 2021-07-06 00:51:19

Can you post your mirrorlist and the pacserve URL that you're using? I think the error may be due to a faulty assumptions in pacserve about the mirror's URL format.

zoqaeski · 2021-07-16 07:31:50

Xyne wrote:

Can you post your mirrorlist and the pacserve URL that you're using? I think the error may be due to a faulty assumptions in pacserve about the mirror's URL format.

My mirrorlist is auto-generated using Reflector:

################################################################################
################# Arch Linux mirrorlist generated by Reflector #################
################################################################################

# With:       reflector @/etc/xdg/reflector/reflector.conf
# When:       2021-07-11 17:00:14 UTC
# From:       https://www.archlinux.org/mirrors/status/json/
# Retrieved:  2021-07-11 16:57:55 UTC
# Last Check: 2021-07-11 16:36:16 UTC

Server = https://mirror.osbeck.com/archlinux/$repo/os/$arch
Server = https://mirrors.niyawe.de/archlinux/$repo/os/$arch
Server = https://archlinux.thaller.ws/$repo/os/$arch
Server = https://archlinux.mailtunnel.eu/$repo/os/$arch
Server = https://mirror.f4st.host/archlinux/$repo/os/$arch
Server = https://mirror.cyberbits.eu/archlinux/$repo/os/$arch
Server = https://mirror.chaoticum.net/arch/$repo/os/$arch
Server = https://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch
Server = https://mirror.pseudoform.org/$repo/os/$arch
Server = https://arch.jensgutermuth.de/$repo/os/$arch
Server = https://mirror.pkgbuild.com/$repo/os/$arch
Server = https://phinau.de/arch/$repo/os/$arch
Server = https://arch.mirror.zachlge.org/$repo/os/$arch
Server = https://mirror.bethselamin.de/$repo/os/$arch
Server = https://mirror.mikrogravitation.org/archlinux/$repo/os/$arch
Server = https://mirror.moson.org/arch/$repo/os/$arch
Server = https://mirror.ubrco.de/archlinux/$repo/os/$arch
Server = https://ftp.myrveln.se/pub/linux/archlinux/$repo/os/$arch
Server = https://mirrors.celianvdb.fr/archlinux/$repo/os/$arch
Server = https://packages.oth-regensburg.de/archlinux/$repo/os/$arch

The pacserve URL in my pacman.conf is just the default http://localhost:15678/pacman/$repo/$arch

Xyne · 2021-08-02 19:13:48

@zoqaeski
It doesn't seem to be the error that I suspected. Do you see any errors or warnings in the pacserve output (e.g. via "systemctl status pacserve.service" or "journalctl -feu pacserve.service" if running with systemd or directly in the console if started manually)?

antimokoko · 2021-08-19 06:56:50

Hi, Xyne. Thank you for creating this great software.

I have used it for a long time, but in the weeks it started showing some issues.

When I use pacserv, it downloads the first package from a mirror than throws an error like so:

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 iana-etc-20210728-1-any is up to date
 libxcrypt-4.4.25-1-x86_64                                                                          88.6 KiB  12.4 MiB/s 00:00 [#############################################################################] 100%
 Total ( 2/74)                                                                                     473.7 MiB  49.1 MiB/s 00:10 [#############################################################################] 100%
error: failed retrieving file 'libxcrypt-4.4.25-1-x86_64.pkg.tar.zst.sig' from 192.xx.xx.10:15678 : The requested URL returned error: 400
warning: failed to retrieve some files
error: failed to commit transaction (failed to retrieve some files)
Errors occurred, no packages were upgraded.

I tried journalctl -feu pacserve.service and I get the following logs:

Aug 19 07:27:45 pc pacserve[5814]: [2021-08-19 07:27:45 CET] INFO: 192.xx.xx.20 "GET /pkg/libxcrypt-4.4.25-1-x86_64.pkg.tar.zst HTTP/1.1" 200 -
Aug 19 07:27:45 pc pacserve[5814]: [2021-08-19 07:27:45 CET] INFO: 192.xx.xx.20 88.56 KiB / 88.56 KiB transferred [/var/cache/pacman/pkg/libxcrypt-4.4.25-1-x86_64.pkg.tar.zst]
Aug 19 07:27:45 pc pacserve[5814]: [2021-08-19 07:27:45 CET] INFO: POSTing to http://192.xx.xx.6:15678/ [type: file check]
Aug 19 07:27:48 pc pacserve[5814]: [2021-08-19 07:27:48 CET] ERROR: POST to http://192.xx.xx.6:15678/ failed (error: <urlopen error timed out>)
Aug 19 07:27:48 pc pacserve[5814]: [2021-08-19 07:27:48 CET] INFO: POSTing to http://192.xx.xx.9:15678/ [type: file check]
Aug 19 07:27:51 pc pacserve[5814]: [2021-08-19 07:27:51 CET] ERROR: POST to http://192.xx.xx.9:15678/ failed (error: <urlopen error timed out>)
Aug 19 07:27:51 pc pacserve[5814]: [2021-08-19 07:27:51 CET] INFO: POSTing to http://192.xx.xx.20:15678/ [type: file check]
Aug 19 07:27:51 pc pacserve[5814]: [2021-08-19 07:27:51 CET] ERROR: 192.xx.xx.20 code 400, message Bad Request

I saw at least one person on the repository, reporting the same issue:

C0rn3j commented on 2021-06-08 08:01
Seems that after pacman 6.0 update, things are a bit wonky, I am repeatedly getting one package downloaded on a -Syu, then the next one failing with 400

:: Retrieving packages...
speex-1.2.0-3-x86_64 is up to date
flac-1.3.3-3-x86_64 294.0 KiB 14.4 MiB/s 00:00 [--] 100%
Total ( 2/13) 47.5 MiB 432 MiB/s 00:00 [--] 100%
error: failed retrieving file 'flac-1.3.3-3-x86_64.pkg.tar.zst.sig' from 192.168.1.10:15678 : The requested URL returned error: 400

Any ideas how to resolve this?
Thank you

Last edited by antimokoko (2021-08-19 06:59:01)

ghthor · 2021-08-22 17:22:53

I'm noticed this same error concerning 400s. What I've noticed is that when the machine serving the package file doesn't have the corresponding signature file this is when pacserve returns a 400 to the remote client and then the installation fails out on the client.

Xyne · 2021-09-08 23:06:43

The failed signature file requests helped me to pinpoint and fix the bug. Please upgrade to python3-threaded_servers 2019-9 once it's up and let me know if it works.

The bug was due to pacman's new behavior of requesting package signature files. Previously it fully relied on the PGP signature data in the database files. It seems that when Pacman is redirected to a Pacserve server, it makes a direct request for the signature file by appending ".sig" to the redirection URL, instead of to the initial request URL (which includes the repo name). Pacserve only ever expected 2 types of requests: a Pacman request with the repo info, or a request for an existing file, so it threw a 400 error (bad request) because the client was requesting something unexpected.

Anyway, it will now throw 404 errors in response to requests for missing sigs and anything else that it doesn't have or can't redirect. Signature files are correctly served if it has them.

ghthor · 2021-09-18 16:45:54

I'm NOW getting the 404 error but it's still causing pacman to bail out of a install entirely. I guess at this point it might be out of pacserve's domain to handle this as this is likely the way pacman is handling the situation, though I would have thought it would just try another mirror instead of bailing completely.

error: failed retrieving file 'spirv-tools-2021.1-1-x86_64.pkg.tar.zst.sig' from 192.168.10.180:15678 : The requested URL returned error: 404
warning: failed to retrieve some files
error: failed to commit transaction (failed to retrieve some files)
Errors occurred, no packages were upgraded.

Xyne · 2021-10-04 22:59:29

Do you have other mirrors enabled? Unless pacman's behavior has changed, it should just move on to the next server in the list after a 404 error.

aureooms · 2021-10-09 23:34:44

Hi Xyne,

Thanks for developping pacserve! It allows me to reduce bandwith usage by a lot.

Often I have this error at the service's startup and then peer discovery does
not seem to work at all (I am using the default configuration, no iptables,
just `systemctl enable pacserve`).

Oct 07 12:47:26 pandora systemd[1]: Started Pacserve.
Oct 07 12:47:27 pandora pacserve[384]: [2021-10-07 12:47:27 CEST] INFO: sending multicast message to (224.3.45.67, 15679) via 127.0.0.1
Oct 07 12:47:27 pandora pacserve[384]: Exception in thread Thread-2:
Oct 07 12:47:27 pandora pacserve[384]: Traceback (most recent call last):
Oct 07 12:47:27 pandora pacserve[384]:   File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner
Oct 07 12:47:27 pandora pacserve[384]:     self.run()
Oct 07 12:47:27 pandora pacserve[384]:   File "/usr/lib/python3.9/threading.py", line 910, in run
Oct 07 12:47:27 pandora pacserve[384]:     self._target(*self._args, **self._kwargs)
Oct 07 12:47:27 pandora pacserve[384]:   File "/usr/lib/python3.9/site-packages/ThreadedServers/Multicast.py", line 165, in multicast_announcer
Oct 07 12:47:27 pandora pacserve[384]:     multicast(message_prefix + message, group, ports, bind_address)
Oct 07 12:47:27 pandora pacserve[384]:   File "/usr/lib/python3.9/site-packages/ThreadedServers/Multicast.py", line 94, in multicast
Oct 07 12:47:27 pandora pacserve[384]:     send_to_sock(socket.AF_INET, address)
Oct 07 12:47:27 pandora pacserve[384]:   File "/usr/lib/python3.9/site-packages/ThreadedServers/Multicast.py", line 90, in send_to_sock
Oct 07 12:47:27 pandora pacserve[384]:     sock.sendto(message, (group, port))
Oct 07 12:47:27 pandora pacserve[384]: OSError: [Errno 101] Network is unreachable

If I restart pacserve peers can be discovered again. Is there a standard way to
automate recovering from this error?

Arch Linux

#226 2019-02-24 15:42:59

Re: pacserve - easily share Pacman packages between computers

#227 2019-02-27 20:41:15

Re: pacserve - easily share Pacman packages between computers

#228 2019-03-02 15:20:50

Re: pacserve - easily share Pacman packages between computers

#229 2019-03-03 03:56:40

Re: pacserve - easily share Pacman packages between computers

#230 2019-03-04 22:49:11

Re: pacserve - easily share Pacman packages between computers

#231 2019-03-06 21:32:41

Re: pacserve - easily share Pacman packages between computers

#232 2019-03-17 10:00:19

Re: pacserve - easily share Pacman packages between computers

#233 2019-06-06 04:49:08

Re: pacserve - easily share Pacman packages between computers

#234 2019-06-15 03:06:12

Re: pacserve - easily share Pacman packages between computers

#235 2020-12-02 07:55:18

Re: pacserve - easily share Pacman packages between computers

#236 2020-12-02 23:19:51

Re: pacserve - easily share Pacman packages between computers

#237 2020-12-03 08:34:21

Re: pacserve - easily share Pacman packages between computers

#238 2021-04-13 10:27:07

Re: pacserve - easily share Pacman packages between computers

#239 2021-06-07 02:57:04

Re: pacserve - easily share Pacman packages between computers

#240 2021-06-18 22:35:45

Re: pacserve - easily share Pacman packages between computers

#241 2021-06-28 02:53:29

Re: pacserve - easily share Pacman packages between computers

#242 2021-07-06 00:51:19

Re: pacserve - easily share Pacman packages between computers

#243 2021-07-16 07:31:50

Re: pacserve - easily share Pacman packages between computers

#244 2021-08-02 19:13:48

Re: pacserve - easily share Pacman packages between computers

#245 2021-08-19 06:56:50

Re: pacserve - easily share Pacman packages between computers

#246 2021-08-22 17:22:53

Re: pacserve - easily share Pacman packages between computers

#247 2021-09-08 23:06:43

Re: pacserve - easily share Pacman packages between computers

#248 2021-09-18 16:45:54

Re: pacserve - easily share Pacman packages between computers

#249 2021-10-04 22:59:29

Re: pacserve - easily share Pacman packages between computers

#250 2021-10-09 23:34:44

Re: pacserve - easily share Pacman packages between computers

Board footer