You are not logged in.

#201 2018-08-06 19:17:23

morarch
Member
Registered: 2015-10-06
Posts: 4

Re: pacserve - easily share Pacman packages between computers

Python aur packages need to be rebuilt since Arch Linux updated Python to version 3.7.
In case of pacserve run:

aurman -S python3-threaded_servers pacserve

to rebuild and reinstall the necessary python packages.

Then restart the pacserve server:

sudo systemctl start pacserve.service

Last edited by morarch (2018-08-06 19:18:29)

Offline

#202 2018-08-06 19:58:23

toothandnail
Member
From: Oxfordshire, UK
Registered: 2012-05-05
Posts: 74

Re: pacserve - easily share Pacman packages between computers

morarch wrote:

Python aur packages need to be rebuilt since Arch Linux updated Python to version 3.7.
In case of pacserve run:

aurman -S python3-threaded_servers pacserve

to rebuild and reinstall the necessary python packages.

Then restart the pacserve server:

sudo systemctl start pacserve.service

Thanks. Should have thought of that - I saw quite a number of python updates come through earlier and didn't twig. I'll have to wait and see if that solves the problem of the new install not picking up on locally available files.

Paul.

Offline

#203 2018-08-07 00:27:31

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,412
Website

Re: pacserve - easily share Pacman packages between computers

All Python packages in my repo have been rebuilt with Python 3.7. AUR users while have to update manually as suggested above.

toothandnail wrote:

Second problem. I recently got a new (to me anyway) laptop. I've just installed Arch on it, and one of the first things I installed was pacserve, since I should already have most of the packages that I want on other machines in my local network. Before the error above came up, it was running without any apparent problems. However, it did not look as if it was using any of the locally available packages - even on a wifi connection, there is a very visible difference in download speed. When pacserve is working, I normally get around 7.5 mbs transfers on anything in my local network. On raw internet, I normally get around 3.5 mbs. Ever since I installed pacserve on the new machine, I'm getting no better than the normal internet transfer rate. So it looks as though pacserve is not picking anything up from the local network.

If the second problem is unrelated to the first, check systemctl status pacserve.service (or the console output if you are running it manually) on different systems to see which Pacserve instances are detected. Check your firewall settings too to make sure that you are allowing everything through (the Pacserve HTTP(S) port, and the multicast ports if you use that). If you use Avahi, make sure that it's up and running too.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#204 2018-10-26 12:55:50

julesm
Member
Registered: 2014-07-29
Posts: 70

Re: pacserve - easily share Pacman packages between computers

Pacserve has stopped working for me since updating from version 2018-9 to version 2018-11. My system is fully up to date - including python3-threaded_servers. What happens is that the pacserve service fails but pacserve-ports starts successfully. I don't see any errors in the journal. Here's some data. Not sure where to look next. Pointers/ideas/suggestions would be most appreciated.

xyne-x86_64/pacserve 2018-11 [installed]
    Easily share Pacman packages between computers. A replacement for PkgD.
xyne-x86_64/python3-threaded_servers 2018.6-3 [installed]
    Threaded server modules (ThreadedHTTPSServer, ThreadedMulticastServer, Quickserve, Pacserve).
systemctl status pacserve.service
● pacserve.service - Pacserve
   Loaded: loaded (/usr/lib/systemd/system/pacserve.service; enabled; vendor preset: disabled)
   Active: activating (auto-restart) (Result: exit-code) since Fri 2018-10-26 14:40:54 CEST; 2s ago
  Process: 4229 ExecStart=/usr/bin/pacserve $PACSERVE_ARGS (code=exited, status=1/FAILURE)
 Main PID: 4229 (code=exited, status=1/FAILURE)

systemctl status pacserve-ports.service
● pacserve-ports.service - Open main and multicast ports for Pacserve
   Loaded: loaded (/usr/lib/systemd/system/pacserve-ports.service; enabled; vendor preset: disabled)
   Active: active (exited) since Fri 2018-10-26 14:29:58 CEST; 11min ago
 Main PID: 2490 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/pacserve-ports.service

This is the contents of my pacserve.service.conf file:

PACSERVE_ARGS="--multicast"

Cheers,
Jules

Offline

#205 2018-10-31 20:54:20

romax
Member
Registered: 2018-10-31
Posts: 1

Re: pacserve - easily share Pacman packages between computers

I have found an issue with pacserv which always gets me as it's happening once in a while and then I forget about it.

The issue is: *-any.pkg.tar.xz packages are signed by different build keys on different architectures.

In my home network, there are multiple instances of x64, 32bit and ARM Arch Linuxes, all set up to use pacserv. Usually, they get along pretty well. However, if a architecture-independent package is updated and cached by an pacserve instance on a different architecture, one gets a signature error, as the used key is not stored in the keyring on the local, to be updated machine. That may be obvious, and a design choice of Arch, but to me, that thwarts the idea of an universal, architecture-independent *-any.pkg.tar.xz idea.

Example of today:

error: perl-error: signature from "Felix Yan <felixonmars@archlinux.org>" is invalid
:: File /var/cache/pacman/pkg/perl-error-0.17027-1-any.pkg.tar.xz is corrupted (invalid or corrupted package (PGP signature)).
Do you want to delete it? [Y/n] Y

The only way of updating then is deleting the package and using pacman instead of pacserve.

It is totally possible, in fact quite probably, a misunderstanding on my side. But still, I don't remember seeing a note on this somewhere that you should not mix architectures.

Last edited by romax (2018-10-31 21:53:43)

Offline

#206 2018-10-31 22:06:38

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 2,853

Re: pacserve - easily share Pacman packages between computers

When Arch still supported i686, we shared -any packages across both.

The problem is that archlinuxarm and archlinux32 are different Linux distributions, and they rebuild their entire repository from scratch in order to be self-sufficient. The packages can share architectures, but rebuilding the same identical, -any package will still result in different checksums due to packaging metadata like file timestamps and the PACKAGER information, if nothing else.

One fix would be for those architectures to directly import our package pool when possible -- but that means less flexibility in rebuilding packages on their schedule.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#207 2018-11-02 08:30:31

julesm
Member
Registered: 2014-07-29
Posts: 70

Re: pacserve - easily share Pacman packages between computers

My issue (above) was resolved when I changed the following kernel parameters:

from

ipv6.disable=1

to

ipv6.disable_ipv6=1

It seems that the pacserve service will fail if the whole IPv6 stack is disabled.

Offline

#208 2018-11-04 03:01:05

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,412
Website

Re: pacserve - easily share Pacman packages between computers

@romax
As eschwartz has pointed out, you're mixing different package ecosystems. If the 32-bit and ARM "any" packages use the same PKGBUILDs as the official repos, then identical package versions should be interchangeable despite having been built by different packagers. If you can guarantee that, or are willing to risk breakage due to unforeseen incompatibility, then you should be able to pool packager keys from each distribution on each system to validate the "any" packages. The extra hassle and (minor) risk is probably not worth it though.

@julesm
Thanks for the follow-up and workaround. The IPv6 support in the pacserve backend is still wonky and untested. I'll make a note to try to fix it the next time I sit down with the code.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#209 2018-11-19 02:34:22

alex.forencich
Member
Registered: 2011-05-29
Posts: 91

Re: pacserve - easily share Pacman packages between computers

I have a question about how to restrict network interface use with pacserve. 

I have a peer-to-peer VPN that connects most of my machines (zerotier) and I want to restrict pacserve to operate over the local network only and not over the VPN, so I need to force pacserve to only listen on specific interfaces.  However, it seems that pacserve has to listen on both the localhost interface as well as the external interface, while the command line only allows one interface to be selected.  Is there another way of configuring what interfaces pacserve uses, or this this a bug in pacserve?

Also, the documentation refers to /etc/conf.d/pacserve , but this file does not appear to exist.  Was this deprecated?

Last edited by alex.forencich (2018-11-19 02:43:37)

Offline

#210 2018-11-24 21:22:15

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,412
Website

Re: pacserve - easily share Pacman packages between computers

@alex.forencich
The underlying HTTP server is the one provided by the Python standard library and it is limited to listening to either all or a single interface at a time (unless that's changed since I last checked). If you bind it to a non-local interface, update /etc/pacman.d/pacserve, /etc/powerpill/powerpill.json or whatever else uses pacserve to point to the bound interface instead of localhost.

The configuration files are located in /etc/pacserve. I have updated the project page.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#211 2018-12-11 09:24:13

C0rn3j
Member
Registered: 2016-02-14
Posts: 10

Re: pacserve - easily share Pacman packages between computers

I've had this issue with pacserve presenting itself on multiple systems - it seems that the mirrorlist will get downloaded from one of the not-uptodate pacserve hosts and then pacman will think there's no updates. If I disable pacserve, new mirrorlists and packages are downloaded. (reboot also seems to work)

https://haste.rys.pw/raw/doxeguguri

Offline

#212 2018-12-17 22:03:28

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,412
Website

Re: pacserve - easily share Pacman packages between computers

There seems to be some confusion of terminology here. Pacserve doesn't handle mirrorlists. Neither does Pacman. The mirrorlist is the file in /etc/pacman.d that you configure manually to specify your servers. There is a mirrorlist in the mirrorlist package, but when the package is installed, the ".pacserve" extension is added to the new mirrorlist and it is installed next to the current mirrorlist. During normal operation with pacserve and pacman, nothing changes your current mirrorlist.

The output that you posted indicates a problem with pacserve's configuration as pacman is unable to connect to it. How did you configure and start it? Are there any error messages in the output of systemctl or journalctl? If you provide information about your setup and what you are trying to achieve, it will make it easier to help you.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#213 2018-12-21 01:06:59

C0rn3j
Member
Registered: 2016-02-14
Posts: 10

Re: pacserve - easily share Pacman packages between computers

>The output that you posted indicates a problem with pacserve's configuration as pacman is unable to connect to it.

You can see that is AFTER I disabled the service. Only then I downloaded current repo data.

And yeah, I am confusing mirrorlists with repo updates in the post above.

With pacserve enabled the repo updates to repo data from what I assume is an outdated pacserve host on the network.

Offline

#214 2018-12-24 05:33:32

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,412
Website

Re: pacserve - easily share Pacman packages between computers

C0rn3j wrote:

>The output that you posted indicates a problem with pacserve's configuration as pacman is unable to connect to it.

You can see that is AFTER I disabled the service. Only then I downloaded current repo data.

Sorry, you're right. I was in a hurry and rushed through the post.

C0rn3j wrote:

With pacserve enabled the repo updates to repo data from what I assume is an outdated pacserve host on the network.

Pacserve serves the first file that it finds. The database files are not versioned, so requesting e.g. core.db from the local pacserver will return the one in /var/lib/pacman/sync. Pacman will detect that the modification time of the file offered by the server is the same as the local database and skip the download.

At least that's how it was supposed to work. Looking over the code, I noticed that it never actually serves the sync databases. I must have removed that functionality the last time that I refactored the code. I've updated the code (in the backend package python3-threaded_servers) to serve database files when the --trust-pacserve-peers option is given, and I have added a new option --find-newest to query all peers for the most recent version of each requested file. Add those two options to /etc/pacserve/pacserve.service.conf to share databases between pacservers if that's what you want to do.

Note that you will still need to refresh the databases directly from a mirror on at least one system. If you use the pacsrv wrapper then you can sync with the mirror using "pacman -Syu". If you have configured pacman to use pacserve directly then you can create a script or alias to run "pacman -Syu" with a different configuration file on one system.

I may add some logic later to compare the newest database file on the network against the modification time reported by the mirror but I am leaving it as-is for now. Handling that properly would require a little refactoring and extra code to handle the multiple protocols used by different mirrors.


Given that the existing code didn't handle databases at all, I don't actually understand how it could affect your database refresh. Maybe trizen/pacman/something else failed to handle the redirect. If it was a general issue then I would expect to have had more reports.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#215 2018-12-24 12:48:21

C0rn3j
Member
Registered: 2016-02-14
Posts: 10

Re: pacserve - easily share Pacman packages between computers

I don't alias pacserve, I add the redirects to the pacman.conf - https://gitlab.com/C0rn3j/arch/blob/mas … h#L229-234 (CTRL+F for xyne if the lines are not aligned anymore due to me updating the script)

I've had this happen on two different networks with different hosts (with the same setup), but it seems to happen after a few days or at least some time, not immediately.

> I've updated the code (in the backend package python3-threaded_servers) to serve database files when the --trust-pacserve-peers option is given, and I have added a new option --find-newest to query all peers for the most recent version of each requested file. Add those two options to /etc/pacserve/pacserve.service.conf to share databases between pacservers if that's what you want to do.

I actually do not want to share the database, just the packages. From what I understand you couldn't figure out why this happened so it may happen again on my setup?

If(When) it happens, how do I troubleshoot it to get you needed info to fix it? Check mod times in /var/lib/pacman/sync and try to turn off pacserve on other hosts and see if it works after that perhaps?

Offline

#216 2018-12-24 17:10:57

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,412
Website

Re: pacserve - easily share Pacman packages between computers

Tests that you can try:

  • local downloads: run pacserve, open http://localhost:15678/pkg/ in your browser and download a package by clicking on its link.

  • peer detection: navigate to http://localhost:15678/?page=peers and check that the other pacserve servers have been detected.

  • peer accessibility: follow the peer link and try downloading packages directly from them.

  • pacman downloads: try downloading a package with your browser, wget, curl or whatever with the URL format "http://localhost:15678/pacman/<repo>/<arch>/<pkgname>", e.g. "http://localhost:15678/pacman/core/x86_64/pacman-5.1.1-1-x86_64.pkg.tar.xz" (using a package that is available locally or on one of the peers).

  • database redirects: using the same method as the previous test, try to download a database with e.g. "http://localhost:15678/pacman/core/x86_64/core.db" and check that it redirects to a mirror.

If any of those don't work, run pacserve in a terminal and post the full output. If peers don't show up or refuse connections, check your settings (multicast or avahi? firewall settings?).

Also check that pacserve is using the same mirrorlist as pacman or whatever wrapper you are using. If not, it may be that pacserve redirects to a stale mirror with old databases that pacman correctly ignores, but when running without pacserve disabled it checks a different mirror first with fresher database files and thus downloads them. With the default settings, pacserve redirects all non-package requests to the first mirror so I don't know what else it could be right now. Does trizen use a separate configuration file?


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#217 2018-12-26 23:11:07

C0rn3j
Member
Registered: 2016-02-14
Posts: 10

Re: pacserve - easily share Pacman packages between computers

>Does trizen use a separate configuration file?

Sorry for not being clear about that, pacman has identical wrong behavior as trizen, it's not a trizen issue.

I'll check out what you posted when I run into this again.

There should be no firewall except for NAT.

Offline

#218 2019-01-30 05:33:57

Jarhmander
Member
Registered: 2016-08-10
Posts: 7

Re: pacserve - easily share Pacman packages between computers

OK, I have issues myself with pacserve lately, like, in the last month probably?

But, first things first! Thank you Xyne for that utility, it bothered me to have my laptop and my server run Arch Linux and yet download the same packages on both when upgrading. I searched a bit (really, not much!), found pacserve, and it worked flawlessly from the beginning (since 2018-03-10 according to /var/log/pacman.log) without effort until, well I don't know precisely, but it's quite recent, like maybe a month ago. Pardon my lack of rigor on keeping track of time.

The issue is quite simple actually: I can upgrade packages, but somehow there is no sharing in my network. In my case it's quite noticeable: I have a 10Mbit internet connection, so sharing files locally has a ×10 speedup (yeah, only 100Mbit).

Here's what I think matters in the testing: lately I uninstalled and reinstalled pacserve and its dependencies (python3-threaded_servers and even pyalpm!) and I verified that the newly installed pacserve is running on both my server and my laptop. I double checked on both that /etc/pacman.conf is correct, in fact it didn't change since I last touched it. I normally use pacaur (yeah, I know, it's deprecated!) but for troubleshooting here, I use plain vanilla pacman, just to be sure.

Your last post contains a bullet list of things to check. Well, I can tell the first three bullets work: the service is indeed up, the http server is running and I can browse my packages as well as the other peer's, and download whatever file I see. So, I think that completely rules out firewall issues and the like.

Now the interesting bits: testing from the laptop, using at the time of writing the package pacman-5.1.2-2-x86_64.pkg.tar.xz as an example; if I have it locally, trying to download http://localhost:15678/pacman/core/pacman-5.1.2-2-x86_64.pkg.tar.xz works, and is instant because, well, I already have it. Then, if I delete it from /var/cache/pacman/pkg, I get a weird redirect that is broken, http://archlinux.mirror.rafal.ca/core/o … pkg.tar.xz; archlinux.mirror.rafal.ca is my first mirror on the laptop... Seems like it used $repo/os/$pkg/$pkg instead of $repos/os/$arch/$pkg, but that's not exactly the problem at hand I think.

Ok, upon trying right now, it seems I can't see peers on the peers page if I run pacserve directly on each computer, so I fall back to start the service and look at the output of journalctl --follow.

So, if I run "wget http://localhost:15678/pacman/core/pacman-5.1.2-2-x86_64.pkg.tar.xz" on my laptop, I see this on the laptop:

Jan 30 00:13:41 jarhmander-tabletpc pacserve[612]: [2019-01-30 00:13:41 EST] INFO: POSTing to http://XXXXX:15678/ [type: file check]
Jan 30 00:13:41 jarhmander-tabletpc pacserve[612]: [2019-01-30 00:13:41 EST] INFO: 127.0.0.1 redirecting to http://archlinux.mirror.rafal.ca/core/os/pacman-5.1.2-2-x86_64.pkg.tar.xz/pacman-5.1.2-2-x86_64.pkg.tar.xz
Jan 30 00:13:41 jarhmander-tabletpc pacserve[612]: [2019-01-30 00:13:41 EST] INFO: 127.0.0.1 "GET /pacman/core/pacman-5.1.2-2-x86_64.pkg.tar.xz HTTP/1.1" 303 -

At the same time, this happens on the server (obvious when looking at the timestamp):

Jan 30 00:13:41 compaq-trash2 pacserve[19074]: [2019-01-30 00:13:41 EST] INFO: YYYYY "POST / HTTP/1.1" 200 -

Where XXXXX is the server's IP address (compaq-trash2) and YYYYY is the laptop's (jarhmander-tabletpc). The faulty redirect is seen here, after POSTing to the server.

While we can see that somehow the communication works, something is going on with the POST and the package is not sent afterward. Obviously, there's not enough info above to diagnose what's went wrong; I need guidance to further debug this.

Offline

#219 2019-01-31 22:08:38

jdn06
Member
Registered: 2010-02-01
Posts: 62

Re: pacserve - easily share Pacman packages between computers

For me downgrading python3-threaded_servers to 2018.12.24.1 solved this issue.

Last edited by jdn06 (2019-01-31 22:09:09)

Offline

#220 2019-01-31 23:56:44

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,412
Website

Re: pacserve - easily share Pacman packages between computers

I found a bug in the changes made when --find-newest was introduced, but that raised an unhandled Python value errors. After fixing that, redirects to other pacservers and to mirrors work for me. Here's the server output for a wget request for a package that exists and one that doesn't.


> wget http://localhost:12345/pacman/core/x86_64/pacman-5.1.2-2-x86_64.pkg.tar.xz
# ... retrieves package from peered server

> wget http://localhost:12345/pacman/core/x86_64/pacman-5.1.2-3-x86_64.pkg.tar.xz
# ... get redirected to mirror and 404's
> pacserve --port 12345 --peer http://127.0.0.1:15678 --pacman-conf ./pacman.conf
PacserveServer
  PID              *****
  Addresses
    lo:   127.0.0.1
    lo:   ::1
    wlo1: 192.168.0.11
  Port             12345
  Filelist         None
  Filterlist       None
  MOTD             None
  Upload directory None
  Tar              Disabled
  Paths            None
  Multicast        False
  Avahi            False
  Static Peers     http://127.0.0.1:15678

Press ctrl+C to exit.
[2019-02-01 00:34:43 CET] INFO: announcing presence via POST to http://127.0.0.1:15678
[2019-02-01 00:34:43 CET] INFO: POSTing to http://127.0.0.1:15678/ [type: nudge]
[2019-02-01 00:34:43 CET] INFO: http://127.0.0.1:15678 acknowledged POST
[2019-02-01 00:34:50 CET] INFO: POSTing to http://127.0.0.1:15678/ [type: file check]
[2019-02-01 00:34:50 CET] INFO: 127.0.0.1 redirecting to http://127.0.0.1:15678/pkg/pacman-5.1.2-2-x86_64.pkg.tar.xz
[2019-02-01 00:34:50 CET] INFO: 127.0.0.1 "GET /pacman/core/x86_64/pacman-5.1.2-2-x86_64.pkg.tar.xz HTTP/1.1" 303 -
[2019-02-01 00:34:55 CET] INFO: POSTing to http://127.0.0.1:15678/ [type: file check]
[2019-02-01 00:34:55 CET] INFO: 127.0.0.1 redirecting to https://mirrors.kernel.org/archlinux/core/os/x86_64/pacman-5.1.2-3-x86_64.pkg.tar.xz
[2019-02-01 00:34:55 CET] INFO: 127.0.0.1 "GET /pacman/core/x86_64/pacman-5.1.2-3-x86_64.pkg.tar.xz HTTP/1.1" 303 -

I have no idea right now why it's repeating the package name in the redirect on your system. I'll try a few more things here but I'm scratching my head at the moment. I'll post an update if I find anything.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#221 2019-02-01 04:29:15

Jarhmander
Member
Registered: 2016-08-10
Posts: 7

Re: pacserve - easily share Pacman packages between computers

The redirect is fixed!

Jan 31 23:24:23 jarhmander-tabletpc pacserve[28321]: [2019-01-31 23:24:23 EST] INFO: POSTing to http://XXXXX:15678/ [type: file check]
Jan 31 23:24:23 jarhmander-tabletpc pacserve[28321]: [2019-01-31 23:24:23 EST] INFO: 127.0.0.1 redirecting to http://archlinux.mirror.rafal.ca/core/os/x86_64/pacman-5.1.2-2-x86_64.pkg.tar.xz                  
Jan 31 23:24:23 jarhmander-tabletpc pacserve[28321]: [2019-01-31 23:24:23 EST] INFO: 127.0.0.1 "GET /pacman/core/x86_64/pacman-5.1.2-2-x86_64.pkg.tar.xz HTTP/1.1" 303 -

...but it's still redirecting even though the server has the file; if I replace "localhost" with the server's IP address in the wget command I can download that file from the server. My intuition tells me that something goes wrong with the POST; what can I do to diagnose that transaction?

EDIT: on the server, I stopped pacserve, ran "netcat -l -p 15678" and receive this after I try the "wget" command:

POST / HTTP/1.1
Accept-Encoding: identity
Content-Type: application/x-www-form-urlencoded
Content-Length: 129
Host: XXXXX:15678
User-Agent: Python-urllib/3.7
Connection: close

json=%7B%22type%22%3A+%22file+check%22%2C+%22paths%22%3A+%5B%22pkg%2Fpacman-5.1.2-2-x86_64.pkg.tar.xz%22%5D%2C+%22hops%22%3A+0%7D

... which is once url-decoded this:

json={"type":+"file+check",+"paths":+["pkg/pacman-5.1.2-2-x86_64.pkg.tar.xz"],+"hops":+0}

Are the plus signs normal in here?

Regardless, afterwards I re-started pacserve on the server, and on the laptop I sent that POST to the server thru netcat, and got this answer:

HTTP/1.1 200 OK
Server: ThreadedServers.Pacserve/2013.05.12
Date: Fri, 01 Feb 2019 04:44:17 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 96
Connection: close

{"pkg/pacman-5.1.2-2-x86_64.pkg.tar.xz": ["pkg/pacman-5.1.2-2-x86_64.pkg.tar.xz", 1546707906.0]}

... which looks OK. The "Server:" line is a bit perplexing though. Maybe not important, but what I did to send that POST is to "netcat XXXXX 15678", paste the entire POST contents and press enter.

Last edited by Jarhmander (2019-02-01 04:58:44)

Offline

#222 2019-02-02 18:40:05

Jarhmander
Member
Registered: 2016-08-10
Posts: 7

Re: pacserve - easily share Pacman packages between computers

The freshly updated version of python3-threaded_servers (2019.2-1) fixed the sharing problem, thank you Xyne! 100Mb/s download between my machines again!

Out of curiosity, what changed in this version? I can't see on your changelog, and can't seem to find your code hosted on a public SCM. I diff'd your two last releases, and I see more documentation added, minor refactorings, the returned dict is a bit different in `search_pkgs` and there's an early exit in `remote_file_check`, but you can understand I have no clue what all those changes mean.

Offline

#223 2019-02-03 01:38:38

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,412
Website

Re: pacserve - easily share Pacman packages between computers

Adding the --find-newest option required the file queries to pass around modification times. I rushed the changes and ended up passing along tuples to some places that expected a string and vice versa. Yesterday's update fixes those inconsistencies.

I agree that the server line is strange. I'll fix that in the next release. As for the plus signs, they are treated as spaces in URL query components and are automatically added by the url encoding functions.

Thanks for reporting the error and confirming the fix.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#224 2019-02-03 15:26:59

Jarhmander
Member
Registered: 2016-08-10
Posts: 7

Re: pacserve - easily share Pacman packages between computers

Thanks for the info. So in the reworking, internally the data types wouldn't match at places where the queries are handled and it would act like there's an error in the communication.

Shit happens; that's cool. If I wasn't expecting something to break occasionally —because of progression— I wouldn't use Arch in the first place. Thanks again for the support!

Offline

#225 2019-02-22 18:56:52

connection_reset
Member
Registered: 2017-04-25
Posts: 2

Re: pacserve - easily share Pacman packages between computers

Hello,

I'm running pacserve for a few containers with multiple interfaces and had issues with multicast.
Pacserve should only work over the host-only network (eth0 in the containers).
I run pacserve with `pacserve --multicast` on the host and in the containers.
Inside the containers I keep pacserve off the other interfaces with iptables `iptables -A OUTPUT -p udp --dport 15679 -o eth1 -j REJECT`.

The multicast_announcer would then crash due to an uncaught exception `PermissionError: [Errno 1] Operation not permitted` due to the iptables rule.
I just catch them all now.

--- a/ThreadedServers/Multicast.py
+++ b/ThreadedServers/Multicast.py
@@ -98,7 +98,7 @@ def multicast(message, group, ports, bind_address=None):
       else:
         logging.warning('multicast function only accepts addresses that are instances of ipaddress.IPv4Address or ipaddress.IPV6Address: ignoring address of type {}'.format(type(address)))
         continue
-    except socket.gaierror as e:
+    except Exception as e:
 #       if e.errno == socket.EAI_NONAME:
 #         continue
 #       else:

Pacserve would not join the multicast group on the right interface and would not find peers.
I copied the address enumeration from the `multicast` function and join every (multicast_group, address) combination (see https://www.tldp.org/HOWTO/Multicast-HOWTO-2.html#ss2.4).
I also explicitly set the interface to send multicast messages out of but I don't know if that is necessary since that socket is already bound to an address.

--- a/ThreadedServers/Multicast.py
+++ b/ThreadedServers/Multicast.py
@@ -74,6 +74,8 @@ def multicast(message, group, ports, bind_address=None):
 
     def send_to_sock(sock, address, *extra_bind_args):
       sock.bind((address.exploded, 0, *extra_bind_args))
+      in_addr = address.packed
+      sock.setsockopt(socket.SOL_IP, socket.IP_MULTICAST_IF, in_addr)
       for port in ports:
         logging.info('sending multicast message to ({}, {:d}) via {}'.format(
           group, port, address
@@ -171,15 +173,25 @@ class MulticastServer(BaseServer, socketserver.UDPServer, object):
 
 
   def server_bind(self):
+    # This is used to get the IP type of the group (IPv4 or IPv6).
+    group_ip_type = type(ipaddress.ip_address(self.multicast_group))
+
+    if unbound_address(self.server_address[0]):
+      addresses = (ip for (_, ip) in get_all_interfaces())
+    else:
+      addresses = (ipaddress.ip_address(sef.server_address[0]),)
+
     if self.allow_reuse_address:
       self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
     self.socket.bind(self.server_address)
-    mreq = struct.pack(
-      "4sl",
-      ipaddress.ip_address(self.multicast_group).packed,
-      socket.INADDR_ANY
-    )
-    self.socket.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
+
+    for address in addresses:
+      # Skip mismatched IP versions (e.g. IPv4 multicast groups on IPv6 networks).
+      if type(address) != group_ip_type:
+        continue
+
+      mreq = ipaddress.ip_address(self.multicast_group).packed + address.packed
+      self.socket.setsockopt(socket.SOL_IP, socket.IP_ADD_MEMBERSHIP, mreq)
 #     self.server_address = self.get_address_and_port()
 
 

Offline

Board footer

Powered by FluxBB