pacserve - easily share Pacman packages between computers

Xyne · 2017-10-15 22:47:06

The pacserve network can indeed be split by using different multicast groups. Yeah, you should be able to just set the ARM group to 224.3.45.68. I haven't looked at multicast configurations for a while, but the pacserve group was chosen as an arbitrary group in "AD-HOC block 2". As long as nothing else on your network is using that group, it should be fine.

The package signature is not part of the package. It only appears in the repository database when the package is added with repo-add.

The ARM packagers have set up a parallel system if they are building their own "any" packages. Given what you have said about them (different dependencies, different pkgrel bumps), you cannot expect them to be compatible. If the "any" packages don't match, and the architecture-specific packages do not overlap, there is no reason for an ARM machine to ever query the pacserver on a non-ARM machine.

kete · 2017-10-17 23:06:43

Xyne wrote:

Can you post the rules that you used? I want to understand why pacserve-ports did not work.

# Generated by iptables-save v1.6.1 on Sat Oct 14 17:00:37 2017
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [48:5139]
:TCP - [0:0]
:UDP - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p ipv6 -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p icmp -m icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -p udp -m conntrack --ctstate NEW -j UDP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j TCP
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A INPUT -j REJECT --reject-with icmp-proto-unreachable
-A TCP -p tcp -m tcp --dport 22 -j ACCEPT
-A TCP -p tcp -m tcp --dport 15678 -m comment --comment pacserve -j ACCEPT
-A UDP -p udp -m udp --dport 5353 -m comment --comment "Avahi daemon" -j ACCEPT
-A UDP -p udp -m pkttype --pkt-type multicast -m udp --dport 15679 -m comment --comment pacserve -j ACCEPT
COMMIT
# Completed on Sat Oct 14 17:00:37 2017

based on https://wiki.archlinux.org/index.php/Si … l_firewall
PS—edited to probably format comments

Last edited by kete (2017-10-23 13:12:31)

Xyne · 2017-10-21 11:41:31

edit: misread the configuration, revised comment

Ok, thanks, that makes sense. The default service adds the rules to the INPUT table. I had actually forgotten that the package provides a configuration file for changing them (/etc/pacserve/pacserve-ports.conf).

Last edited by Xyne (2017-10-21 11:41:59)

zoqaeski · 2017-12-25 23:59:56

As of last night, pacserve is refusing to start under systemd, but starts normally if I run it from the terminal. The following messages are repeated in the journal, ad infinitum:

Dec 26 00:17:29 tencendor systemd[1]: Started Pacserve.
Dec 26 00:17:32 tencendor pacserve[489]: Failed to import the site module
Dec 26 00:17:32 tencendor pacserve[489]: Traceback (most recent call last):
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/site.py", line 544, in <module>
Dec 26 00:17:32 tencendor pacserve[489]:     main()
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/site.py", line 530, in main
Dec 26 00:17:32 tencendor pacserve[489]:     known_paths = addusersitepackages(known_paths)
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/site.py", line 282, in addusersitepackages
Dec 26 00:17:32 tencendor pacserve[489]:     user_site = getusersitepackages()
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/site.py", line 258, in getusersitepackages
Dec 26 00:17:32 tencendor pacserve[489]:     user_base = getuserbase() # this will also set USER_BASE
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/site.py", line 248, in getuserbase
Dec 26 00:17:32 tencendor pacserve[489]:     USER_BASE = get_config_var('userbase')
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/sysconfig.py", line 601, in get_config_var
Dec 26 00:17:32 tencendor pacserve[489]:     return get_config_vars().get(name)
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/sysconfig.py", line 558, in get_config_vars
Dec 26 00:17:32 tencendor pacserve[489]:     _CONFIG_VARS['userbase'] = _getuserbase()
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/sysconfig.py", line 205, in _getuserbase
Dec 26 00:17:32 tencendor pacserve[489]:     return joinuser("~", ".local")
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/sysconfig.py", line 184, in joinuser
Dec 26 00:17:32 tencendor pacserve[489]:     return os.path.expanduser(os.path.join(*args))
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/posixpath.py", line 247, in expanduser
Dec 26 00:17:32 tencendor pacserve[489]:     userhome = pwd.getpwuid(os.getuid()).pw_dir
Dec 26 00:17:32 tencendor pacserve[489]: KeyError: 'getpwuid(): uid not found: 65534'
Dec 26 00:17:32 tencendor systemd[1]: pacserve.service: Main process exited, code=exited, status=1/FAILURE
Dec 26 00:17:32 tencendor systemd[1]: pacserve.service: Failed with result 'exit-code'.
Dec 26 00:17:47 tencendor systemd[1]: pacserve.service: Service hold-off time over, scheduling restart.
Dec 26 00:17:47 tencendor systemd[1]: pacserve.service: Scheduled restart job, restart counter is at 1.
Dec 26 00:17:47 tencendor systemd[1]: Stopped Pacserve.

I did a system update yesterday before this started happening, but the only python packages that were updated were python-setuptools and python-jedi; could either of those have caused issues? and it seems that an upgrade to systemd from 235.38-4 to 236.0-2 broke something.

Solution

The issue appears to be related to running pacserve as nobody; I fixed it by creating a system user pacserve, and running the service as that user by modifying the unit file accordingly.

Last edited by zoqaeski (2017-12-27 05:08:40)

The Loko · 2017-12-31 00:18:10

The nobody uid was changed. Check this bug report

Last edited by The Loko (2017-12-31 00:18:38)

Xyne · 2017-12-31 06:18:14

Thanks for reporting the issue with the traceback and bug report. I went with zoqaeski's approach of creating a pacserve use and group to run the service. The error should be fixed with pacserve-2017-1.

toothandnail · 2018-01-02 14:28:11

Xyne wrote:

Thanks for reporting the issue with the traceback and bug report. I went with zoqaeski's approach of creating a pacserve use and group to run the service. The error should be fixed with pacserve-2017-1.

Currently I have 3 Arch installs on my laptop, plus 3 desktop machines that have Arch installs. Two of the installs (one desktop and one of the laptop installs) failed to start pacserve. In those cases, adding the user/group and editing the systemd unit file has worked fine. The other 4 installs show no sign of the problem - pacserve is starting and running correctly, even though they are still set to use nobody/nobody. All the installs are up to date, so I don't understand why some work and others don/t....

Paul.

zosodk69 · 2018-01-03 15:17:14

On my system the UID for pacserve is 1002. Aren't UIDs 0-999 reserved for system accounts and UIDs 1000-65534 for users?

eschwartz · 2018-01-03 15:30:00

1) Yes, useradd should be used with [-r, --system] when creating a user for a service.

2) useradd should not be used at all, since Arch Linux is trying to make a concerted effort to move to systemd-sysusers: https://www.archlinux.org/todo/switch-t … -sysusers/

3) The AUR package has not been updated, and neither has the PKGBUILD on the homepage at https://xyne.archlinux.ca/projects/pacserve/

In fact, the homepage still lists the 2016 version, which awkwardly enough is no longer available in src/

STEELBAS · 2018-01-10 17:43:58

Hello,

I'm having trouble running the script pacman.conf-insert_pacserve:

$  pacman.conf-insert_pacserve 
Traceback (most recent call last):
  File "/usr/bin/pacman.conf-insert_pacserve", line 29, in <module>
    config = insert_pacserver(fpath)
  File "/usr/bin/pacman.conf-insert_pacserve", line 11, in insert_pacserver
    for line in f:
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3169: ordinal not in range(128)
code 1

I'm sure this is something I'm doing wrong, because the script does work on the other computers in my network, but I can't for the life of me figure out what it is. As a result of the script not working, I get the following when I try to run pacsrv normally:

$  sudo pacsrv -Syu
Traceback (most recent call last):
  File "/usr/bin/pacman.conf-insert_pacserve", line 29, in <module>
    config = insert_pacserver(fpath)
  File "/usr/bin/pacman.conf-insert_pacserve", line 11, in insert_pacserver
    for line in f:
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3169: ordinal not in range(128)
error: no usable package repositories configured.
code 1

The pacserve systemd service seems to be running without problems.

Xyne · 2018-01-15 02:06:04

@zosodk69 & Eschwartz
The package has been updated to use systemd-sysusers. I will upload it shortly and double-check the links.

@STEELBAS
The file is normally UTF-8 but it appears to be falsely detected as ascii on your system. Try opening it in a text editor and checking the encoding. Save it again as UTF-8.

STEELBAS · 2018-01-22 16:47:47

Xyne wrote:

@STEELBAS
The file is normally UTF-8 but it appears to be falsely detected as ascii on your system. Try opening it in a text editor and checking the encoding. Save it again as UTF-8.

Right, thanks. This gave me the pointers I needed - turns out my locale wasn't set right.

aphirst · 2018-01-26 11:17:03

It's been a while, so I figured I should report back.

Regarding the ARM machines on the network, this is now a non-issue again since I decommissioned one, but before doing that I did change the multicast to the suggested value and it did indeed achieve the expected goal of "separating" the pacserve networks.

I've come specifically however to report some odd behaviour, which, whether or not it turns out to be a bug, has given me the idea of a feature request.

I have two (normal) Arch machines which connect to my home LAN via ethernet. If I simply turn them both on, with the pacserve services enabled, pacserve works as expected (by e.g. running -Syu on one machine via SSH, using a tool like bmon on it to view the transmission rate over ethernet, then running -Syu on the local machine). The problems seem to happen once I either sleep/suspend one of the machines, or temporarily disconnect one from the network (usually by undocking from my docking station and re-connecting to another elsewhere in the house; but sometimes by switching over to WLAN).

Important to note is that the docking stations just act as port replicators (so take on the ethernet MAC of the connected device), and that I also have my DHCP/DNS server (pi-hole) configured to assign the same IP/hostname to both the ethernet AND wireless for each device (along with a netctl entry to toggle the wireless on/off via rfkill whenever the ethernet port changes state).

Essentially, sometimes I run -Syu on the two machines in turn, and the 2nd machine doesn't pull from the first, instead connecting to the internet - and, the reason it irritates me, wasting my bandwidth cap. Obviously I'm using pacserve not merely to make updates faster, but to save WAN bandwidth - which happens fast due to it being fibre. If, however, I manually re-run dhcpcd on both machines, pacserve seems to work fine.

Is there a way to view other pacserve clients on the network? If there is, is there any way to (optionally) integrate this into the -S and -Sy(u) process, so that the user can get some advance warning about whether this issue is going to occur, and where the data's coming from?

Of course, I intend to try to pin down my issue in more detail, but in the meantime I was just hoping for some comments and discussion to see whether e.g. something I'm doing is fundamentally misguided.

aphirst · 2018-02-02 22:59:06

I have another comment regarding package deltas. I'm using an arch mirror which uses deltas, but because of the order in which the logic is applied, re-downloading a delta from the internet is prioritised over simply transferring the updated full package across the LAN from another machine which already has it. Is there any way to circumvent this in the pacserve model?

eschwartz · 2018-02-04 01:47:31

I guess you could disable deltas.

pacserve has no way of messaging pacman to prioritize full packages over deltas. pacman's native logic assumes that all mirrors are the same, and that all mirrors have the same files. pacserve is pretty awesome about serving you the same files you would be downloading anyway, from a LAN connection instead of the public internet... but it cannot tell pacman to download different files than it would otherwise have done.

OTOH if pacserve is tracking the delta as well, then it should be able to offer you that. If your other machine is also configured to use deltas, then as long as pacserve knows how to handle deltas this "should" work seamlessly, but since I have never used pacserve, I can only take a wild guess based on the fact that it isn't working for you, that pacserve does not currently handle deltas.

Xyne · 2018-04-26 11:29:03

I've added Avahi support alongside multicast support for peer detection. There are probably still some bugs to work out but it works so far in my initial tests.

Speranskiy · 2018-04-27 03:44:26

Hi Xyne! Thanks for the project!
After the latest update it's started to consume 100% of CPU (this process -> python3 -mThreadedServers.Pacserve --multicast
). Here is the log of pacserve. Package Database Sync hangs. What could be wrong? How could I debug?

Xyne · 2018-04-27 16:08:10

Hi Speranskiy,

Are you using the latest version of python3-threaded_servers?

Speranskiy · 2018-04-28 12:41:16

Nope, it was python3-threaded_servers 2018.4.26-1 I was using. Updated to 2018.4.27-1, the issue is still there on both machines.

Xyne · 2018-04-28 14:20:11

Ok, it should be fixed in python3-threaded_servers-2018.4.28. Thanks for reporting the issue and providing the log.

Speranskiy · 2018-04-28 20:56:51

Yep, fixed! Thanks!

aphirst · 2018-04-29 13:07:41

It'd be a shame if "disable deltas" is the only workaround.

The ideal outcome would be, whichever machine Syu's first gets deltas from upstream and generates the new complete package file itself, and all the subsequent machines ignore the delta and just get the complete package from the first one; with all machines able to serve as the "first" and get the update via delta if needed.

The current situation either has me waste bandwidth for that first complete package download (where a delta would have saved it), OR has me waste CPU cycles with every machine generating it via delta, OR forces me to pre-commit to which machine can obtain via deltas (and which must always be Syu'd first, otherwise you lose one of the two benefits).

It seems odd that, regardless of a machine's delta setting, it can't "see" that the full package is available from some other LAN pacserve instance.

As an aside, a few months ago I used the flags config to hardcode the details for my pacserve-capable devices, since sometimes pacserve would "lose sight of" the other machines whenever the network was dis+reconnected. I'm not sure whether this is more robust with the new avahi changes, but the hardcoding seems to be working for me.

Last edited by aphirst (2018-04-29 13:10:16)

pdc · 2018-07-23 18:07:52

The pacserve package conflicts with pacredir in [community], since both packages include files /usr/bin/pacserve and /usr/lib/systemd/system/pacserve.service

Xyne · 2018-07-30 23:44:06

I'll contact the maintainer/developer to discuss solutions to the naming conflict. "Pacserve" is admittedly not very original, but I've been using it longer

toothandnail · 2018-08-06 17:52:19

I've been using pacserve for quite a while - have 5 Arch installs, so its a big help when it comes to keeping them updated. I've got a couple of problems, one of which I've seen before.

Quite suddenly today, every attempt I make to use 'pacsrv -Syu' results in strings of this error:

error: failed retrieving file 'core.db' from localhost:15678 : Failed to connect to localhost port 15678: Connection refused

Last time it happened, I was told it indicated a problem with one of the mirrors. I've run reflector to generate a new mirrorlist, but I'm still getting the same error every time I try and use pacsrv.

Second problem. I recently got a new (to me anyway) laptop. I've just installed Arch on it, and one of the first things I installed was pacserve, since I should already have most of the packages that I want on other machines in my local network. Before the error above came up, it was running without any apparent problems. However, it did not look as if it was using any of the locally available packages - even on a wifi connection, there is a very visible difference in download speed. When pacserve is working, I normally get around 7.5 mbs transfers on anything in my local network. On raw internet, I normally get around 3.5 mbs. Ever since I installed pacserve on the new machine, I'm getting no better than the normal internet transfer rate. So it looks as though pacserve is not picking anything up from the local network.

Any ideas how to fix these two problems? I'd really like to have pacserve up and running again....

Paul.

Arch Linux

#176 2017-10-15 22:47:06

Re: pacserve - easily share Pacman packages between computers

#177 2017-10-17 23:06:43

Re: pacserve - easily share Pacman packages between computers

#178 2017-10-21 11:41:31

Re: pacserve - easily share Pacman packages between computers

#179 2017-12-25 23:59:56

Re: pacserve - easily share Pacman packages between computers

Solution

#180 2017-12-31 00:18:10

Re: pacserve - easily share Pacman packages between computers

#181 2017-12-31 06:18:14

Re: pacserve - easily share Pacman packages between computers

#182 2018-01-02 14:28:11

Re: pacserve - easily share Pacman packages between computers

#183 2018-01-03 15:17:14

Re: pacserve - easily share Pacman packages between computers

#184 2018-01-03 15:30:00

Re: pacserve - easily share Pacman packages between computers

#185 2018-01-10 17:43:58

Re: pacserve - easily share Pacman packages between computers

#186 2018-01-15 02:06:04

Re: pacserve - easily share Pacman packages between computers

#187 2018-01-22 16:47:47

Re: pacserve - easily share Pacman packages between computers

#188 2018-01-26 11:17:03

Re: pacserve - easily share Pacman packages between computers

#189 2018-02-02 22:59:06

Re: pacserve - easily share Pacman packages between computers

#190 2018-02-04 01:47:31

Re: pacserve - easily share Pacman packages between computers

#191 2018-04-26 11:29:03

Re: pacserve - easily share Pacman packages between computers

#192 2018-04-27 03:44:26

Re: pacserve - easily share Pacman packages between computers

#193 2018-04-27 16:08:10

Re: pacserve - easily share Pacman packages between computers

#194 2018-04-28 12:41:16

Re: pacserve - easily share Pacman packages between computers

#195 2018-04-28 14:20:11

Re: pacserve - easily share Pacman packages between computers

#196 2018-04-28 20:56:51

Re: pacserve - easily share Pacman packages between computers

#197 2018-04-29 13:07:41

Re: pacserve - easily share Pacman packages between computers

#198 2018-07-23 18:07:52

Re: pacserve - easily share Pacman packages between computers

#199 2018-07-30 23:44:06

Re: pacserve - easily share Pacman packages between computers

#200 2018-08-06 17:52:19

Re: pacserve - easily share Pacman packages between computers

Board footer