PkgD has been deprecated along with perl-xyne-arch. Use Pacserve is a replacement for PkgD.
Info page: http://xyne.archlinux.ca/info/pkgd
I saw awesome_welles' post and ended up thinking "that should be possible". I ended up coming up with pkgd (and finally got around to writing my first official daemon ).
The info page explains it. Post questions, comments, feedback, odes, etc here.
Btw, this is fully functional for the network topology listed on the info page, but I intend to try to scale and see what I end up with. One possibility that's floating around in my head is a decentralized p2p repository that should facilitate finding old pkgs. Not sure how feasible that is though once security comes into it.
Btw, pkgd is quite straight-forward. I have some idiot-checks in there, but don't try creating a bunch of cyclical master/slave relationships or mix pkg architectures.
Last edited by Xyne (2011-04-28 14:46:35)
I like it. It saves me having to sync my cache to a custom repo and do a repo_add. Seems to work fine but two points:
sudo pacman -Sy claws-mail :: Synchronising package databases... custom is up to date error: failed retrieving file 'core.db.tar.gz' from 192.168.123.114 : Success
and so on. I guess the error message is irrelevant as it seems to download fine from the server.
Shaman doesn't like the additional lines in /etc/pacman.conf
peter ~ $ shaman Translations are enabled. Loading translations from "/usr/share/shaman/translations/" Parsing "custom" Parsing "core" Segmentation fault
Otherwise a roaring success I would say. Thanks!
I think the error messages feel a bit unclean too. I've left it that way initially because I was more interested in getting the basic LAN topology worked out. I'll look into the possibility of having pkgd retrieve packages from the mirrors itself or at least forward pacman to the next mirror in the list (the latter shouldn't be too hard).
I don't know what shaman is complaining about. You could try running the master pkgd in a console with "pkgd" instead of "/etc/rc.d/pkgdd start" and see what request it's getting from shaman. Post it here and I'll see if I can figure it out.
For a while I was sharing my pacman cache via NFS. I guess this would work the same?
If the package isnt found on the server but is found on a client can you get the server to download the file? To speed the process if its needed again
For now, the best thing is to have a pkgd running on each server that's using the master pkgd. That way, any pkg that's downloaded will be available to all the others.
I had originally thought of making pkgd capable of downloading pkgs and creating a cache (including database files), but there are some complexities:
a) the main pkgd then ends up storing copies of all pkgs requested by other hosts, which leads to some unnecessary redundancy
As long as it's optional though, that's not really a problem
b) the package has to be downloaded, saved and forwarded at the same time
I need to learn a bit more http protocol to do this (e.g. how to tell pacman to wait with keep-alive responses while pkgd starts downloading the pkg), plus there's the issue of configuring external downloaders. That's not really difficult, but it will take some time and testing. There's a good chance that I'll do this though, because I would want to use metalink downloads like powerpill does.
c) the main mirror would need to have a mirrorlist for all repos in order to know where to get the packages
What I really need to do is work out better pkgd-pkgd communication methods so that mirrorlists etc can be sent with requests (e.g. check if you have pkgX and download it here if you don't). I also want to add checks for master/slave cycles, enable optional network cross-overs, slave-sharing, etc.
Excellent! Seems to be working perfectly.
Can you divert the master log output from vc/1? Pkgdd outputs the request lines to vc/1, when client makes a query.
btw, this software rocks
I've added log support now. Set a log file in /etc/pkgd.conf. I've also fixed the "client xxx.xxx.xxx.xxx has requested y.pkg.tar.gz" output (I had missed a print statement).
Last edited by Xyne (2009-02-03 13:01:18)
Nice and simple, I needed something like this. I was thinking of implementing something similar with shared cache and python, but your solution is much better. Too bad that I don't know Perl to contribute and customize it to my needs.
Do you have any suggestion for the folowing configuration. I have WiFi network at home, two of my laptops have Arch64 installed and they are not alway turned on. I would want to always check when downloading packages if the package is already donwloaded on the other laptop if it is connected. How would I configure pkgd? Will it work if I run pkgd on both laptops and configure on each laptop that the other laptop is the master.
For laptop on 192.168.0.2 the master is 192.168.0.3
For laptop on 192.168.0.3 the master is 192.168.0.2
Topology (arrows point to the direction of downloading):
laptop1 <------> laptop2 ^ ^ \ / \ / \ / arch mirror
If I manage to configure this two laptops, it would be easy to configure my other two machines running Arch i686 in the same way.
Last edited by SnapShot (2009-02-05 05:36:24)
I think it would work to set up a reciprocal master-slave relationship. As it is right now, the pkgd first checks if it has the pkg itself, then it sends a simple request to each slave to ask if the slave has it. The slaves don't yet query their own slaves, so there shouldn't be an infinite query loop. I will eventually add that as a feature, but I'll also add checks to prevent such loops.
Just try it in non-daemon mode ("pkgd" from a terminal) first and see what happens, then report back to let me know how it goes.
I found pkgd to use 100% CPU and reach speeds of 50-100KiB/s when serving files from Pentium III 650MHz and Atom N270 1.6GHz systems. Couldn't get to profile pkgd, maybe because of the use of threads.
Ah, also, pkgd dies with SIGPIPE when you Ctrl-C pacman while downloading.
Last edited by CptYossarian (2009-02-05 12:12:02)
Nice concept man! To make it really distributed/no-spof you could get rid of the master-slave concept and use something like avahi
< Daenyth> and he works prolifically
4 8 15 16 23 42
Can you give me some more information about your setup and what you're doing when it uses 100% CPU? (network topology, etc)
Nvm, I just noticed the relatively high cpu usage here too. I'll try to fix that.
I don't really know anything about avahi but I can look into it.
Last edited by Xyne (2009-02-05 14:22:33)
Now it seems to work just fine.
Thanks Xyne, it works with recirpocal master-slave configuration.
I have just tought something, it seems like it is not necessary to configure master on each of the laptop. If I just add the other laptop as a Server in pacman.conf it should always query if the package is available on the other laptop. I have not tested it yet because the master-slave configuration worked perfectly. Do you think it's fine for simple 2 PCs configuration (1<----->1) to just run pkgd on each of them without any master-slave configuration?
Last edited by SnapShot (2009-02-05 17:52:51)
That's great. It will become very useful when package signature will be implemented in Pacman, because without it you have to trust all the other people using it on your LAN... But that's a step in the good direction, I love the idea of sharing resources like that.
That would work too. The only difference that I can think of right now is that you'll get up to 2 error messages if the package can't be found, but that's only an aesthetic problem.
Yeah, it's still very simplistic at the moment. I added the password option in order to provide some very basic security/filtering, but I'll need to do a bit more to make it robust. If someone did manage to insert a poisoned pkg though, they would still need to duplicate the md5sum to get it past pacman. Hopefully nobody has anyone that determined to harm their system on their LAN.
In the future I intend to add more integrity checks and other features to enable safe(r?) scaling.
Last edited by Xyne (2009-02-06 11:56:43)
Thanks very much for this useful contribution, Xyne! This is exactly what I've been looking for!
what goes up must come down
Xyne, do you have any plans to make pkgd work with powerpill in the near future, as mentioned on your website?
what goes up must come down