You are not logged in.

#1 2011-04-17 00:49:13

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

pacserve - easily share Pacman packages between computers

Project Info

http://xyne.archlinux.ca/projects/pacserve

Pacserve is a local HTTP server that can communicate with other Pacserve servers on your LAN (or beyond). It will both save bandwidth and speed up package downloads by transferring packages between local computers so you don't need to download them from a remote server.

Peers can be detected automatically using UDP multicast so setting up a local Pacserve network is as simple as enabling a couple of systemd services and running a command.

Powerpill natively supports Pacserve.

News

Pacserve has been completely rewritten in Python 3 and now inherits many features from Quickserve.


Questions, comments and other feedback are welcome as always. Enjoy!

Last edited by Xyne (2013-05-10 22:51:34)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#2 2011-04-17 02:32:02

ataraxia
Member
From: Pittsburgh
Registered: 2007-05-06
Posts: 1,553

Re: pacserve - easily share Pacman packages between computers

If two machines have each other's pacserves in their pacman mirrorlists, or list each other as peer pacserve mirrors directly, requests for files will bounce around forever as each pacserve forwards them back where they came from.

Also, pacman complains about the 307 response codes, rather than just following them quietly.

In any case, don't hurry on my account, as I'll probably not actually use this in practice, though it's cool.

Offline

#3 2011-04-17 03:05:51

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: pacserve - easily share Pacman packages between computers

It should prevent infinite recursion by tracking servers. There might be an exception if you have two mirrors set up on localhost (due to some special handling), but if you set up two servers on the same machine and have them check each other then you don't deserve those CPU cycles to begin with. tongue

I'll look into it though to see if there is a way to improve it.



As for the pacman errors, that's out of my hands. I've already filed a bug report: https://bugs.archlinux.org/task/23800
You can use the XferCommand to avoid those errors as both curl and wget handle them correctly. Curl may need the "-L" option, although it seemed to work without it when I tried.



edit: fixed the recursion

Last edited by Xyne (2011-04-17 03:30:34)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#4 2011-04-17 20:40:38

whitethorn
Member
Registered: 2010-05-02
Posts: 153

Re: pacserve - easily share Pacman packages between computers

Hi, yay this is exactly what I was looking for.  When starting pacserve on my laptop everything works fine.  I can't get it to start on my PC though.  Here's the output I get.

# pacserve -p 5000
Traceback (most recent call last):
  File "/usr/bin/pacserve", line 470, in <module>
    main()
  File "/usr/bin/pacserve", line 428, in main
    server.localhost = socket.gethostbyname( server.server_name )
socket.gaierror: [Errno -2] Name or service not known

Figured it out, looks like I didn't have my /etc/hosts properly setup.  Adding the same hostname as in /etc/rc.conf to localhost line fixed the error. This is wicked.  Thx.

Last edited by whitethorn (2011-04-17 20:41:42)

Offline

#5 2011-04-18 18:53:48

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: pacserve - easily share Pacman packages between computers

I've updated pacserve to catch that error and print out a more informative error message. Thanks for posting the solution.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#6 2011-04-18 21:41:45

whitethorn
Member
Registered: 2010-05-02
Posts: 153

Re: pacserve - easily share Pacman packages between computers

I'm not quite sure but after updating pacserve I can't get it to start.  As far as I can tell my /etc/hosts is correct. Here's the output from pacserve and /etc/hosts.

~ $ pacserve
error: host lookup for "localhost.localdomain" failed: [Errno -2] Name or service not known

This can sometimes be fixed by adding the host to your hosts file (/etc/hosts).
$ cat /etc/hosts
#
# /etc/hosts: static lookup table for host names
#

#<ip-address>    <hostname.domain.org>    <hostname>
127.0.0.1    localhost.localdomain localhost wt-archmini 
::1        localhost.localdomain    localhost
192.168.0.2 salem
# End of file

Offline

#7 2011-04-18 22:13:18

toofishes
Developer
From: Chicago, IL
Registered: 2006-06-06
Posts: 602
Website

Re: pacserve - easily share Pacman packages between computers

Not to hijack, but I thought I would finally push my script somewhere that I created oh so long ago:
https://github.com/toofishes/multipkg

The basic idea of it was to be zero configuration- if you just start the script on your various machines, they find each other and off you go with your shared package cache.

Xyne: regarding redirects, if you just use 301/302 instead, I know for a fact pacman will follow those correctly, since that is exactly what I do in my script. The situation isn't totally dire.

Offline

#8 2011-04-19 06:19:22

ploub
Member
Registered: 2007-05-16
Posts: 133

Re: pacserve - easily share Pacman packages between computers

@ Xyne: thanks for reviving pkgd, I've missed it wink
But I now have the same error as whitethorn. My /etc/hosts is correct AFAICT, and is in line with HOSTNAME in /etc/rc.conf.
Help?

Offline

#9 2011-04-19 09:03:40

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: pacserve - easily share Pacman packages between computers

@whitethorn & ploub
Sorry about that. When I was debugging it, I set the host to something like "dklfjskfjsdf" just to see if the error was caught, and I forgot to remove it when I updated the package. It should be fixed now.


@toofishes
Using 301/302 is against the HTTP standard. The redirects will most often be HEAD requests and thus a 307 is required to indicate that the redirect is temporary and that the request should be resent using the same method. I won't use a kludge that will break standard compliance if the only thing that fails to follow the standard is Pacman's internal downloader. Both curl and wget work as expected via XferCommand, and apparently Pacman should too:  https://bugs.archlinux.org/task/23800


As for multipkg, I like your zero-configuration approach (I think you contacted me about that at some point). I may implement similar detection for pacserve but in that case I will add filter options. For now, I haven't even re-implemented PkgDs master notification protocol. *edit* I am considering re-implementing that as it makes sense to support a central "go to" server that can track other dynamic servers (e.g. laptops that aren't always connected). Multicast may be the best way to do that, but I have never done anything with it before so I would need to do some research.

Last edited by Xyne (2011-04-19 11:46:15)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#10 2011-04-19 10:47:38

ploub
Member
Registered: 2007-05-16
Posts: 133

Re: pacserve - easily share Pacman packages between computers

working now, thanks

Offline

#11 2011-04-19 13:21:06

toofishes
Developer
From: Chicago, IL
Registered: 2006-06-06
Posts: 602
Website

Re: pacserve - easily share Pacman packages between computers

Xyne wrote:

@toofishes
Using 301/302 is against the HTTP standard. The redirects will most often be HEAD requests and thus a 307 is required to indicate that the redirect is temporary and that the request should be resent using the same method. I won't use a kludge that will break standard compliance if the only thing that fails to follow the standard is Pacman's internal downloader. Both curl and wget work as expected via XferCommand, and apparently Pacman should too:  https://bugs.archlinux.org/task/23800

I didn't realize you were doing HEAD redirects, so excuse me there- I haven't dealt with those and redirects at the same time for the most part. I also disagree that it could be construed against standards, but many browsers implemented it oddly from the beginning so further codes were introduced to resolve the ambiguity. Isn't it a bit silly to throw 90% of your users to the wind because you're unwilling to do a simple workaround for a bug in software two layers below yours?

And it is libfetch that does things wrong here. I found the bug this morning looking closer at it- it doesn't handle the code at all in another switch statement. I'll put details in said bug report of ours.

Offline

#12 2011-04-19 13:38:36

toofishes
Developer
From: Chicago, IL
Registered: 2006-06-06
Posts: 602
Website

Re: pacserve - easily share Pacman packages between computers

@Xyne: can you give this a try and see if it resolves things? A patched libfetch with the bug report patch I sent upstream. http://dev.archlinux.org/~dan/libfetch- … src.tar.gz

Offline

#13 2011-04-19 17:05:35

Ravenman
Member
Registered: 2009-07-03
Posts: 236

Re: pacserve - easily share Pacman packages between computers

I can't update from another machine in the LAN:

Server side messages:

[root@testserver ~]# pacserve
Starting Pacserve...
  address: all interfaces
  port: 8080
  pid: 2114
Press <Ctrl-C> to exit.

[2011-04-19 11:42:48] 192.168.2.3:50671 "HEAD /request/core/i686/core.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:42:48] 192.168.2.3:50672 "HEAD /request/extra/i686/extra.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:42:48] 192.168.2.3:50673 "HEAD /request/community/i686/community.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:44:23] 192.168.2.3:50684 "HEAD /request/core/i686/core.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:44:23] 192.168.2.3:50685 "HEAD /request/extra/i686/extra.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:44:23] 192.168.2.3:50686 "HEAD /request/community/i686/community.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:44:40] 192.168.2.3:50689 "HEAD /request/core/i686/core.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:44:40] 192.168.2.3:50690 "HEAD /request/extra/i686/extra.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:44:40] 192.168.2.3:50691 "HEAD /request/community/i686/community.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:47:39] 192.168.2.3:45652 "GET /request/core/i686/core.db.tar.gz HTTP/1.0" 404 -
[2011-04-19 11:47:41] 192.168.2.3:45653 "GET /request/extra/i686/extra.db.tar.gz HTTP/1.0" 404 -
[2011-04-19 11:47:43] 192.168.2.3:45654 "GET /request/community/i686/community.db.tar.gz HTTP/1.0" 404 -
[2011-04-19 11:47:58] 192.168.2.3:45655 "GET /request/core/i686/core.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:47:58] 192.168.2.3:45656 "GET /request/extra/i686/extra.db.tar.gz HTTP/1.1" 404 -
[2011-04-19 11:48:02] 192.168.2.3:45657 "GET /request/community/i686/community.db.tar.gz HTTP/1.1" 404 -

Client side messages:

[root@testworkstation ~]# pacman -Sy
:: Synchronizing package databases...
error: failed retrieving file 'core.db.tar.gz' from 192.168.2.2 : Not Found
error: failed to update core (Not Found)
error: failed retrieving file 'extra.db.tar.gz' from 192.168.2.2 : Not Found
error: failed to update extra (Not Found)
error: failed retrieving file 'community.db.tar.gz' from 192.168.2.2 : Not Found
error: failed to update community (Not Found)
error: failed to synchronize any databases
[root@testworkstation ~]# nano /etc/pacman.conf 
[root@testworkstation ~]# pacman -Sy
:: Synchronizing package databases...
--2011-04-19 11:46:39--  http://192.168.2.2:8080/request/core/i686/core.db.tar.gz
Connecting to 192.168.2.2:8080... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-04-19 11:46:39 ERROR 404: Not Found.

error: failed to update core (error invoking external downloader)
--2011-04-19 11:46:41--  http://192.168.2.2:8080/request/extra/i686/extra.db.tar.gz
Connecting to 192.168.2.2:8080... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-04-19 11:46:41 ERROR 404: Not Found.

error: failed to update extra (error invoking external downloader)
--2011-04-19 11:46:43--  http://192.168.2.2:8080/request/community/i686/community.db.tar.gz
Connecting to 192.168.2.2:8080... connected.
HTTP request sent, awaiting response... 404 Not Found
2011-04-19 11:46:43 ERROR 404: Not Found.

error: failed to update community (error invoking external downloader)
error: failed to synchronize any databases
[root@testworkstation ~]# nano /etc/pacman.conf 
[root@testworkstation ~]# pacman -Sy
:: Synchronizing package databases...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     9  100     9    0     0   4515      0 --:--:-- --:--:-- --:--:--  9000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     9  100     9    0     0   4502      0 --:--:-- --:--:-- --:--:--  9000
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     9  100     9    0     0   4249      0 --:--:-- --:--:-- --:--:--  9000
[root@testworkstation ~]# 

Offline

#14 2011-04-19 18:24:40

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: pacserve - easily share Pacman packages between computers

toofishes wrote:

Isn't it a bit silly to throw 90% of your users to the wind because you're unwilling to do a simple workaround for a bug in software two layers below yours?

I was assuming that the error would be trivial to fix and would be included in the next release of Pacman. I thought that everyone could just ignore the errors or use the XferCommand until then. After all, PkgD just returned 404 and people didn't mind that.

toofishes wrote:

@Xyne: can you give this a try and see if it resolves things? A patched libfetch with the bug report patch I sent upstream. http://dev.archlinux.org/~dan/libfetch- … src.tar.gz

That works. Thanks for the quick fix.

Btw, I never realized that you were Dan hmm




@Ravenman
Database requests should redirect you to external mirrors, but it needs to read them from pacman.conf. Either you don't have any mirrors on the system running the server, or it isn't finding your pacman.conf file. Check that it's located at /etc/pacman.conf and that you have uncommented mirrors.

*edit*
Update to the latest version first before you try again.




@everyone else
I've added multicast support with filtering options, and changed the default port. Let me know how it works. There's a simple setup example on the project page now.

Last edited by Xyne (2011-04-19 18:25:21)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#15 2011-04-20 14:59:09

n17ikh
Member
Registered: 2008-09-15
Posts: 11

Re: pacserve - easily share Pacman packages between computers

I'm trying out pacserve and I'm using the example multicast configuration - two systems running "pacserve --multicast" and "http://localhost:15678/request/$repo/i686" as the first mirror for every repo, replacing the line I had there for pkgd. It worked for a few packages but then started throwing this error on the machine with the packages:

----------------------------------------
Exception happened during processing of request from ('192.168.5.10', 55148)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/XyneHTTPServer.py", line 70, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 639, in __init__
    self.handle()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 337, in handle
    self.handle_one_request()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 325, in handle_one_request
    method()
  File "/usr/lib/python2.7/site-packages/XyneHTTPServer.py", line 406, in do_GET
    self.do_authenticated_GET()
  File "/usr/bin/pacserve", line 259, in do_authenticated_GET
    self.do_authenticated_GET_or_HEAD(True)
  File "/usr/bin/pacserve", line 348, in do_authenticated_GET_or_HEAD
    if e.strerror[0] == errno.ECONNREFUSED and server in pc.multicast_servers:
TypeError: 'NoneType' object is not subscriptable
---------------------------------------

And this error on the machine running pacman -Syu:

----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 57112)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/XyneHTTPServer.py", line 70, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 639, in __init__
    self.handle()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 337, in handle
    self.handle_one_request()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 325, in handle_one_request
    method()
  File "/usr/lib/python2.7/site-packages/XyneHTTPServer.py", line 406, in do_GET
    self.do_authenticated_GET()
  File "/usr/bin/pacserve", line 259, in do_authenticated_GET
    self.do_authenticated_GET_or_HEAD(True)
  File "/usr/bin/pacserve", line 348, in do_authenticated_GET_or_HEAD
    if e.strerror[0] == errno.ECONNREFUSED and server in pc.multicast_servers:
TypeError: 'NoneType' object is not subscriptable
----------------------------------------

In pacman, I see this:

--2011-04-20 10:38:40--  (try: 2)  http://localhost:15678/request/extra/i686/akonadi-1.5.2-1-i686.pkg.tar.xz
Connecting to localhost|127.0.0.1|:15678... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

--2011-04-20 10:38:43--  (try: 3)  http://localhost:15678/request/extra/i686/akonadi-1.5.2-1-i686.pkg.tar.xz
Connecting to localhost|127.0.0.1|:15678... connected.
HTTP request sent, awaiting response... No data received.
Retrying.

If I stop pacserve on the system with the packages, pacserve on the system running pacman redirects to the mirrors in the mirrorlist as it should.
It seems to work just fine with the traditional "--pacserve" directives. Any suggestions?

Last edited by n17ikh (2011-04-20 15:00:15)

Offline

#16 2011-04-20 18:43:16

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: pacserve - easily share Pacman packages between computers

That was a side-effect of a workaround for an upstream bug. I've pushed another update. Try that and let me know if it works.



As a note to anyone looking at the output code wondering wtf I'm using "e.strerror[0] == errno.ECONNREFUSED", take a look at that section of code and the comments around it. The "connection refused" error caught in that block is the expected exception or tuple and e.errno is just the string "socket error". After trying several things to identify and catch the error reliably I finally found that e.strerror contains the expected tuple, at which point I uttered several words that would break forum etiquette and then kludged it.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#17 2011-04-20 18:56:55

Ravenman
Member
Registered: 2009-07-03
Posts: 236

Re: pacserve - easily share Pacman packages between computers

Xyne wrote:

@Ravenman
Database requests should redirect you to external mirrors, but it needs to read them from pacman.conf. Either you don't have any mirrors on the system running the server, or it isn't finding your pacman.conf file. Check that it's located at /etc/pacman.conf and that you have uncommented mirrors.

*edit*
Update to the latest version first before you try again.

@everyone else
I've added multicast support with filtering options, and changed the default port. Let me know how it works. There's a simple setup example on the project page now.

In the server, I have only one uncommented mirror:

[root@testserver ~]# cat /etc/pacman.d/mirrorlist | grep "ftp.archlinux"
#Server = ftp://ftp.hosteurope.de/mirror/ftp.archlinux.org/$repo/os/$arch
#Server = http://ftp.hosteurope.de/mirror/ftp.archlinux.org/$repo/os/$arch
#Server = ftp://ftp.mfa.kfki.hu/pub/mirrors/ftp.archlinux.org/$repo/os/$arch
#Server = ftp://ftp.heanet.ie/mirrors/ftp.archlinux.org/$repo/os/$arch
#Server = http://ftp.heanet.ie/mirrors/ftp.archlinux.org/$repo/os/$arch
#Server = ftp://ftp.piotrkosoft.net/pub/mirrors/ftp.archlinux.org/$repo/os/$arch
#Server = http://piotrkosoft.net/pub/mirrors/ftp.archlinux.org/$repo/os/$arch
Server = ftp://ftp.archlinux.org/$repo/os/$arch
[root@testserver ~]# 

I have the latest version installed:

[root@testserver ~]# pacman -Q python-xynehttpserver pacserve
python-xynehttpserver 2011.04.19.1-1
pacserve 2011.04.19.2-1

But the failure still continues sad

Client side messages:

[root@testworkstation ~]# pacman -Sy
:: Synchronizing package databases...
error: failed retrieving file 'core.db.tar.gz' from 192.168.2.2 : Not Found
error: failed to update core (Not Found)
error: failed retrieving file 'extra.db.tar.gz' from 192.168.2.2 : Not Found
error: failed to update extra (Not Found)
error: failed retrieving file 'community.db.tar.gz' from 192.168.2.2 : Not Found
error: failed to update community (Not Found)
error: failed to synchronize any databases

Server side messages:

[root@testserver ~]# pacserve --multicast
Starting Pacserve...
  address: all interfaces
  port: 15678
  pid: 5945
Press <Ctrl-C> to exit.

[2011-04-20 11:23:09] multicast group and port: (224.3.45.67, 15679)
[2011-04-20 11:23:09] announcing presence via multicast
[2011-04-20 11:23:10] listening for multicasts on port 15679, all interfaces
[2011-04-20 11:23:20] 192.168.2.3:45807 "HEAD /request/core/i686/core.db.tar.gz HTTP/1.1" 404 -
[2011-04-20 11:23:20] 192.168.2.3:45808 "HEAD /request/extra/i686/extra.db.tar.gz HTTP/1.1" 404 -
[2011-04-20 11:23:20] 192.168.2.3:45809 "HEAD /request/community/i686/community.db.tar.gz HTTP/1.1" 404 -

Offline

#18 2011-04-20 19:17:01

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: pacserve - easily share Pacman packages between computers

@Ravenman

edit
It should be fixed now. I had assumed that all database files were named "*.db".

Incidentally, what version of pacman are you using?

Last edited by Xyne (2011-04-20 19:32:39)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#19 2011-04-21 20:55:06

moose jaw
Member
From: Milwaukee
Registered: 2007-08-20
Posts: 104

Re: pacserve - easily share Pacman packages between computers

Just a quick note to say THANKS! Pacserve is great. With two Arch boxes in the house, this is something I've wished I had (and wished I had the programming chops to write myself) for a long time. Much like quickserve, it really scratches an itch.

Offline

#20 2011-04-22 15:17:40

moose jaw
Member
From: Milwaukee
Registered: 2007-08-20
Posts: 104

Re: pacserve - easily share Pacman packages between computers

Okay, now one question: I've set up both boxes according to the instructions on your site and run 'pacserve --multicast' on both. But it seems that the server pool only contains the second box. That is:
1. Run 'pacserve --multicast' on Box 1
2. Run 'pacserve --multicast' on Box 2
3. Box 1 shows output: "adding: [Box 2 ip]:15678 to server pool"

But there is no comparable output on Box 2. And while Box 1 can now download packages from Box 2, Box 2 doesn't seem able to download packages from Box 1. Is this a general issue? Will a host running pacserve only be able to "see"/download packages from other hosts where pacserve has been started later than it? Forgive me if this is some basic issue in multicasting that I simply don't understand, or if I'm not following the directions right. As I said above, this is a great tool!

Offline

#21 2011-04-24 01:34:31

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: pacserve - easily share Pacman packages between computers

Ok, after multiple facepalms and "wtf?" moments, I've figured out the problem and a solution.

To build the initial network, Pacserve was sending out a multicast announcement and then checking replies to build a list of existing servers. The problem was that that new outbound sockets use random ports and these were presumably blocked by the firewall, so replies were being dropped. Multicast is UDP and UDP is stateless, so the normal "RELATED,ESTABLISHED" rules do not apply to the replies. I missed this in testing because I just enabled all incoming UDP traffic.

I've reconfigured the multicast subroutine to avoid direct replies and use multicasting instead, so it should only need the multicast port to be open for detection to work (edit: obviously the HTTP port needs to be open for it to serve packages).

If anyone needs a starting point for configuring iptables, I've added an example to the project page: http://xyne.archlinux.ca/projects/pacse … es-example

Last edited by Xyne (2011-04-24 01:37:41)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#22 2011-04-24 03:58:11

moose jaw
Member
From: Milwaukee
Registered: 2007-08-20
Posts: 104

Re: pacserve - easily share Pacman packages between computers

Ooh, nice! Just updated and now both boxes are "adding: [other box]:15678 to server pool". Will test out actual serving of packages, etc, in a day or two after there have been some updates of various packages to test it with. But so far so good!

Update, a day later: It works! Box 2 can download packages from Box 1 even when its instance of pacserve is started later than Box 1's.

Last edited by moose jaw (2011-04-25 00:39:26)

Offline

#23 2011-04-24 04:04:18

ataraxia
Member
From: Pittsburgh
Registered: 2007-05-06
Posts: 1,553

Re: pacserve - easily share Pacman packages between computers

I'm still seeing the infinite recursion problem. The easiest way to reproduce it to to follow the "Simple Setup Example" on your pacserve page, then request from this pacserve any package it doesn't happen to already have in the local cache. As it sees itself as the first pacman mirror, it will query itself, and off it goes until it runs the machine out of source ports.

Offline

#24 2011-04-24 05:11:58

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,965
Website

Re: pacserve - easily share Pacman packages between computers

ataraxia wrote:

I'm still seeing the infinite recursion problem. The easiest way to reproduce it to to follow the "Simple Setup Example" on your pacserve page, then request from this pacserve any package it doesn't happen to already have in the local cache. As it sees itself as the first pacman mirror, it will query itself, and off it goes until it runs the machine out of source ports.

Check that you're using the latest versions of both pacserve and python-xynehttpserver. I noticed one recursion scenario as I was reworking multicasting but I think I fixed that.

It would also help if you were specific. Post the output so I can see exactly where it's recursing. Give me some more details about how you've set it up too, as you may be doing something differently and not mentioning it because you don't think it should matter (e.g. running two servers on the same machine to test them, using something other than "localhost" in the "Server" line). Also check that you have "localhost" or whatever you're using in your hosts file.

You could also try using 127.0.0.1 instead of "localhost".

It should skip over itself when querying the mirrors by looking up the target host and comparing it against itself.

The only way that I can reproduce this is by not using something that resolves to 127.0.0.1 in the Server line.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#25 2011-04-24 17:14:34

ataraxia
Member
From: Pittsburgh
Registered: 2007-05-06
Posts: 1,553

Re: pacserve - easily share Pacman packages between computers

Ok, here's exactly what I'm doing:

- installed pacserve 2011.04.24.2-1 and python-xynehttpserver 2011.04.19.1-1 from your xyne-any repo.
- added "Server = http://localhost:15678/request/$repo/$arch" as the first line in /etc/pacman.d/mirrorlist. (Using "localhost" vs "127.0.0.1" doesn't make any difference, FWIW.)
- ran "pacserve"
- in another window, ran "curl http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz". (This is a package I don't have in my cache.)
- waited a while with no response in the curl window. Looked at the pacserve window and saw lots of output, like this:

Starting Pacserve...
  address: all interfaces
  port: 15678
  pid: 9318
Press <Ctrl-C> to exit.

[2011-04-24 12:52:59] 127.0.0.1:45333 querying http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz
[2011-04-24 12:52:59] 127.0.0.1:45334 querying http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz
[2011-04-24 12:52:59] 127.0.0.1:45335 querying http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz
[2011-04-24 12:52:59] 127.0.0.1:45336 querying http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz
[2011-04-24 12:52:59] 127.0.0.1:45337 querying http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz
... (lots more of this, one for every port in sequence between 45337 above and 45838 below)
[2011-04-24 12:53:00] 127.0.0.1:45838 querying http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz
[2011-04-24 12:53:00] 127.0.0.1:45839 querying http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz
[2011-04-24 12:53:00] 127.0.0.1:45840 querying http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz
[2011-04-24 12:53:00] 127.0.0.1:45841 querying http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz
[2011-04-24 12:53:00] 127.0.0.1:45842 querying http://localhost:15678/request/core/x86_64/rpcbind-0.2.0-3-x86_64.pkg.tar.xz

No other output was shown.

So, I don't think I'm doing anything odd at all. No multiple instances on one machine, no weird or overly clever pacman Server entries.

/etc/hosts looks reasonable to me also:

127.0.0.1    localhost.localdomain    localhost
128.2.126.84    akherou.cc.cmu.edu    akherou
67.186.14.223    amaranth

Anything further I can provide?

Offline

Board footer

Powered by FluxBB