You are not logged in.

#176 2017-10-15 22:47:06

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,318
Website

Re: pacserve - easily share Pacman packages between computers

The pacserve network can indeed be split by using different multicast groups. Yeah, you should be able to just set the ARM group to 224.3.45.68. I haven't looked at multicast configurations for a while, but the pacserve group was chosen as an arbitrary group in "AD-HOC block 2". As long as nothing else on your network is using that group, it should be fine.

The package signature is not part of the package. It only appears in the repository database when the package is added with repo-add.

The ARM packagers have set up a parallel system if they are building their own "any" packages. Given what you have said about them (different dependencies, different pkgrel bumps), you cannot expect them to be compatible. If the "any" packages don't match, and the architecture-specific packages do not overlap, there is no reason for an ARM machine to ever query the pacserver on a non-ARM machine.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#177 2017-10-17 23:06:43

kete
Member
Registered: 2015-01-01
Posts: 36

Re: pacserve - easily share Pacman packages between computers

Xyne wrote:

Can you post the rules that you used? I want to understand why pacserve-ports did not work.

# Generated by iptables-save v1.6.1 on Sat Oct 14 17:00:37 2017
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [48:5139]
:TCP - [0:0]
:UDP - [0:0]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p ipv6 -j ACCEPT
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p icmp -m icmp --icmp-type 8 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -p udp -m conntrack --ctstate NEW -j UDP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j TCP
-A INPUT -p udp -j REJECT --reject-with icmp-port-unreachable
-A INPUT -p tcp -j REJECT --reject-with tcp-reset
-A INPUT -j REJECT --reject-with icmp-proto-unreachable
-A TCP -p tcp -m tcp --dport 22 -j ACCEPT
-A TCP -p tcp -m tcp --dport 15678 -m comment --comment pacserve -j ACCEPT
-A UDP -p udp -m udp --dport 5353 -m comment --comment "Avahi daemon" -j ACCEPT
-A UDP -p udp -m pkttype --pkt-type multicast -m udp --dport 15679 -m comment --comment pacserve -j ACCEPT
COMMIT
# Completed on Sat Oct 14 17:00:37 2017

based on https://wiki.archlinux.org/index.php/Si … l_firewall
PS—edited to probably format comments

Last edited by kete (2017-10-23 13:12:31)

Offline

#178 2017-10-21 11:41:31

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,318
Website

Re: pacserve - easily share Pacman packages between computers

edit: misread the configuration, revised comment

Ok, thanks, that makes sense. The default service adds the rules to the INPUT table. I had actually forgotten that the package provides a configuration file for changing them (/etc/pacserve/pacserve-ports.conf).

Last edited by Xyne (2017-10-21 11:41:59)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#179 2017-12-25 23:59:56

zoqaeski
Member
From: /earth/australia/.
Registered: 2009-09-30
Posts: 125

Re: pacserve - easily share Pacman packages between computers

As of last night, pacserve is refusing to start under systemd, but starts normally if I run it from the terminal. The following messages are repeated in the journal, ad infinitum:

Dec 26 00:17:29 tencendor systemd[1]: Started Pacserve.
Dec 26 00:17:32 tencendor pacserve[489]: Failed to import the site module
Dec 26 00:17:32 tencendor pacserve[489]: Traceback (most recent call last):
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/site.py", line 544, in <module>
Dec 26 00:17:32 tencendor pacserve[489]:     main()
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/site.py", line 530, in main
Dec 26 00:17:32 tencendor pacserve[489]:     known_paths = addusersitepackages(known_paths)
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/site.py", line 282, in addusersitepackages
Dec 26 00:17:32 tencendor pacserve[489]:     user_site = getusersitepackages()
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/site.py", line 258, in getusersitepackages
Dec 26 00:17:32 tencendor pacserve[489]:     user_base = getuserbase() # this will also set USER_BASE
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/site.py", line 248, in getuserbase
Dec 26 00:17:32 tencendor pacserve[489]:     USER_BASE = get_config_var('userbase')
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/sysconfig.py", line 601, in get_config_var
Dec 26 00:17:32 tencendor pacserve[489]:     return get_config_vars().get(name)
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/sysconfig.py", line 558, in get_config_vars
Dec 26 00:17:32 tencendor pacserve[489]:     _CONFIG_VARS['userbase'] = _getuserbase()
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/sysconfig.py", line 205, in _getuserbase
Dec 26 00:17:32 tencendor pacserve[489]:     return joinuser("~", ".local")
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/sysconfig.py", line 184, in joinuser
Dec 26 00:17:32 tencendor pacserve[489]:     return os.path.expanduser(os.path.join(*args))
Dec 26 00:17:32 tencendor pacserve[489]:   File "/usr/lib/python3.6/posixpath.py", line 247, in expanduser
Dec 26 00:17:32 tencendor pacserve[489]:     userhome = pwd.getpwuid(os.getuid()).pw_dir
Dec 26 00:17:32 tencendor pacserve[489]: KeyError: 'getpwuid(): uid not found: 65534'
Dec 26 00:17:32 tencendor systemd[1]: pacserve.service: Main process exited, code=exited, status=1/FAILURE
Dec 26 00:17:32 tencendor systemd[1]: pacserve.service: Failed with result 'exit-code'.
Dec 26 00:17:47 tencendor systemd[1]: pacserve.service: Service hold-off time over, scheduling restart.
Dec 26 00:17:47 tencendor systemd[1]: pacserve.service: Scheduled restart job, restart counter is at 1.
Dec 26 00:17:47 tencendor systemd[1]: Stopped Pacserve.

I did a system update yesterday before this started happening, but the only python packages that were updated were python-setuptools and python-jedi; could either of those have caused issues? and it seems that an upgrade to systemd from 235.38-4 to 236.0-2 broke something.

Solution

The issue appears to be related to running pacserve as nobody; I fixed it by creating a system user pacserve, and running the service as that user by modifying the unit file accordingly.

Last edited by zoqaeski (2017-12-27 05:08:40)

Offline

#180 2017-12-31 00:18:10

The Loko
Member
From: Spain
Registered: 2014-07-23
Posts: 80

Re: pacserve - easily share Pacman packages between computers

The nobody uid was changed. Check this bug report

Last edited by The Loko (2017-12-31 00:18:38)

Offline

#181 2017-12-31 06:18:14

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,318
Website

Re: pacserve - easily share Pacman packages between computers

Thanks for reporting the issue with the traceback and bug report. I went with zoqaeski's approach of creating a pacserve use and group to run the service. The error should be fixed with pacserve-2017-1.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#182 2018-01-02 14:28:11

toothandnail
Member
From: Oxfordshire, UK
Registered: 2012-05-05
Posts: 64

Re: pacserve - easily share Pacman packages between computers

Xyne wrote:

Thanks for reporting the issue with the traceback and bug report. I went with zoqaeski's approach of creating a pacserve use and group to run the service. The error should be fixed with pacserve-2017-1.

Currently I have 3 Arch installs on my laptop, plus 3 desktop machines that have Arch installs. Two of the installs (one desktop and one of the laptop installs) failed to start pacserve. In those cases, adding the user/group and editing the systemd unit file has worked fine. The other 4 installs show no sign of the problem - pacserve is starting and running correctly, even though they are still set to use nobody/nobody. All the installs are up to date, so I don't understand why some work and others don/t....

Paul.

Offline

#183 2018-01-03 15:17:14

zosodk69
Member
Registered: 2005-01-30
Posts: 28

Re: pacserve - easily share Pacman packages between computers

On my system the UID for pacserve is 1002.  Aren't UIDs 0-999 reserved for system accounts and UIDs 1000-65534 for users?


-=[dave]=-

Offline

#184 2018-01-03 15:30:00

Eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 1,846

Re: pacserve - easily share Pacman packages between computers

1) Yes, useradd should be used with [-r, --system] when creating a user for a service.

2) useradd should not be used at all, since Arch Linux is trying to make a concerted effort to move to systemd-sysusers: https://www.archlinux.org/todo/switch-t … -sysusers/

3) The AUR package has not been updated, and neither has the PKGBUILD on the homepage at https://xyne.archlinux.ca/projects/pacserve/

In fact, the homepage still lists the 2016 version, which awkwardly enough is no longer available in src/

Offline

#185 2018-01-10 17:43:58

STEELBAS
Member
From: The Netherlands
Registered: 2010-01-18
Posts: 73

Re: pacserve - easily share Pacman packages between computers

Hello,

I'm having trouble running the script pacman.conf-insert_pacserve:

$  pacman.conf-insert_pacserve 
Traceback (most recent call last):
  File "/usr/bin/pacman.conf-insert_pacserve", line 29, in <module>
    config = insert_pacserver(fpath)
  File "/usr/bin/pacman.conf-insert_pacserve", line 11, in insert_pacserver
    for line in f:
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3169: ordinal not in range(128)
code 1

I'm sure this is something I'm doing wrong, because the script does work on the other computers in my network, but I can't for the life of me figure out what it is. As a result of the script not working, I get the following when I try to run pacsrv normally:

$  sudo pacsrv -Syu
Traceback (most recent call last):
  File "/usr/bin/pacman.conf-insert_pacserve", line 29, in <module>
    config = insert_pacserver(fpath)
  File "/usr/bin/pacman.conf-insert_pacserve", line 11, in insert_pacserver
    for line in f:
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3169: ordinal not in range(128)
error: no usable package repositories configured.
code 1

The pacserve systemd service seems to be running without problems.

Offline

#186 2018-01-15 02:06:04

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,318
Website

Re: pacserve - easily share Pacman packages between computers

@zosodk69 & Eschwartz
The package has been updated to use systemd-sysusers. I will upload it shortly and double-check the links.

@STEELBAS
The file is normally UTF-8 but it appears to be falsely detected as ascii on your system. Try opening it in a text editor and checking the encoding. Save it again as UTF-8.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#187 2018-01-22 16:47:47

STEELBAS
Member
From: The Netherlands
Registered: 2010-01-18
Posts: 73

Re: pacserve - easily share Pacman packages between computers

Xyne wrote:

@STEELBAS
The file is normally UTF-8 but it appears to be falsely detected as ascii on your system. Try opening it in a text editor and checking the encoding. Save it again as UTF-8.

Right, thanks. This gave me the pointers I needed - turns out my locale wasn't set right.

Offline

#188 2018-01-26 11:17:03

aphirst
Member
From: Hull, England
Registered: 2008-06-30
Posts: 96
Website

Re: pacserve - easily share Pacman packages between computers

It's been a while, so I figured I should report back.

Regarding the ARM machines on the network, this is now a non-issue again since I decommissioned one, but before doing that I did change the multicast to the suggested value and it did indeed achieve the expected goal of "separating" the pacserve networks.

I've come specifically however to report some odd behaviour, which, whether or not it turns out to be a bug, has given me the idea of a feature request.

I have two (normal) Arch machines which connect to my home LAN via ethernet. If I simply turn them both on, with the pacserve services enabled, pacserve works as expected (by e.g. running -Syu on one machine via SSH, using a tool like bmon on it to view the transmission rate over ethernet, then running -Syu on the local machine). The problems seem to happen once I either sleep/suspend one of the machines, or temporarily disconnect one from the network (usually by undocking from my docking station and re-connecting to another elsewhere in the house; but sometimes by switching over to WLAN).

Important to note is that the docking stations just act as port replicators (so take on the ethernet MAC of the connected device), and that I also have my DHCP/DNS server (pi-hole) configured to assign the same IP/hostname to both the ethernet AND wireless for each device (along with a netctl entry to toggle the wireless on/off via rfkill whenever the ethernet port changes state).

Essentially, sometimes I run -Syu on the two machines in turn, and the 2nd machine doesn't pull from the first, instead connecting to the internet - and, the reason it irritates me, wasting my bandwidth cap. Obviously I'm using pacserve not merely to make updates faster, but to save WAN bandwidth - which happens fast due to it being fibre. If, however, I manually re-run dhcpcd on both machines, pacserve seems to work fine.

Is there a way to view other pacserve clients on the network? If there is, is there any way to (optionally) integrate this into the -S and -Sy(u) process, so that the user can get some advance warning about whether this issue is going to occur, and where the data's coming from?

Of course, I intend to try to pin down my issue in more detail, but in the meantime I was just hoping for some comments and discussion to see whether e.g. something I'm doing is fundamentally misguided.


ArchLinux | x86_64 | linux-ck-ivybridge
OpenBox3 | tint2 | sakura | PCManFM | Chromium | Thunderbird | Code::Blocks | GCC
Lenovo Thinkpad X230 | 12.5" | i5-3320M (2.5GHz) | HD 4000 | 16GB (1066MHz) | 256GB mSATA SSD | 2TB HDD

Offline

#189 2018-02-02 22:59:06

aphirst
Member
From: Hull, England
Registered: 2008-06-30
Posts: 96
Website

Re: pacserve - easily share Pacman packages between computers

I have another comment regarding package deltas. I'm using an arch mirror which uses deltas, but because of the order in which the logic is applied, re-downloading a delta from the internet is prioritised over simply transferring the updated full package across the LAN from another machine which already has it. Is there any way to circumvent this in the pacserve model?


ArchLinux | x86_64 | linux-ck-ivybridge
OpenBox3 | tint2 | sakura | PCManFM | Chromium | Thunderbird | Code::Blocks | GCC
Lenovo Thinkpad X230 | 12.5" | i5-3320M (2.5GHz) | HD 4000 | 16GB (1066MHz) | 256GB mSATA SSD | 2TB HDD

Offline

#190 2018-02-04 01:47:31

Eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 1,846

Re: pacserve - easily share Pacman packages between computers

I guess you could disable deltas. smile

pacserve has no way of messaging pacman to prioritize full packages over deltas. pacman's native logic assumes that all mirrors are the same, and that all mirrors have the same files. pacserve is pretty awesome about serving you the same files you would be downloading anyway, from a LAN connection instead of the public internet... but it cannot tell pacman to download different files than it would otherwise have done.

OTOH if pacserve is tracking the delta as well, then it should be able to offer you that. If your other machine is also configured to use deltas, then as long as pacserve knows how to handle deltas this "should" work seamlessly, but since I have never used pacserve, I can only take a wild guess based on the fact that it isn't working for you, that pacserve does not currently handle deltas.

Offline

#191 2018-04-26 11:29:03

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,318
Website

Re: pacserve - easily share Pacman packages between computers

I've added Avahi support alongside multicast support for peer detection. There are probably still some bugs to work out but it works so far in my initial tests.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#192 2018-04-27 03:44:26

Speranskiy
Member
From: Moscow
Registered: 2013-11-10
Posts: 50

Re: pacserve - easily share Pacman packages between computers

Hi Xyne! Thanks for the project!
After the latest update it's started to consume 100% of CPU (this process -> python3 -mThreadedServers.Pacserve --multicast
). Here is the log of pacserve. Package Database Sync hangs. What could be wrong? How could I debug?

Offline

#193 2018-04-27 16:08:10

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,318
Website

Re: pacserve - easily share Pacman packages between computers

Hi Speranskiy,

Are you using the latest version of python3-threaded_servers?


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#194 2018-04-28 12:41:16

Speranskiy
Member
From: Moscow
Registered: 2013-11-10
Posts: 50

Re: pacserve - easily share Pacman packages between computers

Nope, it was python3-threaded_servers 2018.4.26-1 I was using. Updated to 2018.4.27-1, the issue is still there on both machines.

Offline

#195 2018-04-28 14:20:11

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,318
Website

Re: pacserve - easily share Pacman packages between computers

Ok, it should be fixed in python3-threaded_servers-2018.4.28. Thanks for reporting the issue and providing the log.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#196 2018-04-28 20:56:51

Speranskiy
Member
From: Moscow
Registered: 2013-11-10
Posts: 50

Re: pacserve - easily share Pacman packages between computers

Yep, fixed! Thanks!

Offline

#197 2018-04-29 13:07:41

aphirst
Member
From: Hull, England
Registered: 2008-06-30
Posts: 96
Website

Re: pacserve - easily share Pacman packages between computers

It'd be a shame if "disable deltas" is the only workaround.

The ideal outcome would be, whichever machine Syu's first gets deltas from upstream and generates the new complete package file itself, and all the subsequent machines ignore the delta and just get the complete package from the first one; with all machines able to serve as the "first" and get the update via delta if needed.

The current situation either has me waste bandwidth for that first complete package download (where a delta would have saved it), OR has me waste CPU cycles with every machine generating it via delta, OR forces me to pre-commit to which machine can obtain via deltas (and which must always be Syu'd first, otherwise you lose one of the two benefits).

It seems odd that, regardless of a machine's delta setting, it can't "see" that the full package is available from some other LAN pacserve instance.

As an aside, a few months ago I used the flags config to hardcode the details for my pacserve-capable devices, since sometimes pacserve would "lose sight of" the other machines whenever the network was dis+reconnected. I'm not sure whether this is more robust with the new avahi changes, but the hardcoding seems to be working for me.

Last edited by aphirst (2018-04-29 13:10:16)


ArchLinux | x86_64 | linux-ck-ivybridge
OpenBox3 | tint2 | sakura | PCManFM | Chromium | Thunderbird | Code::Blocks | GCC
Lenovo Thinkpad X230 | 12.5" | i5-3320M (2.5GHz) | HD 4000 | 16GB (1066MHz) | 256GB mSATA SSD | 2TB HDD

Offline

Board footer

Powered by FluxBB