You are not logged in.

#1 2009-02-01 13:49:02

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

pkgd: a daemon for sharing pkgs over your LAN

*edit*
PkgD has been deprecated along with perl-xyne-arch. Use Pacserve is a replacement for PkgD.




Info page: http://xyne.archlinux.ca/info/pkgd

I saw awesome_welles' post and ended up thinking "that should be possible". I ended up coming up with pkgd (and finally got around to writing my first official daemon big_smile).

The info page explains it. Post questions, comments, feedback, odes, etc here.

Btw, this is fully functional for the network topology listed on the info page, but I intend to try to scale and see what I end up with. One possibility that's floating around in my head is a decentralized p2p repository that should facilitate finding old pkgs. Not sure how feasible that is though once security comes into it.


Btw, pkgd is quite straight-forward. I have some idiot-checks in there, but don't try creating a bunch of cyclical master/slave relationships or mix pkg architectures.

Enjoy.

/Xyne

Last edited by Xyne (2011-04-28 14:46:35)

Offline

#2 2009-02-01 15:45:44

Shaika-Dzari
Member
From: Québec, Canada
Registered: 2006-04-14
Posts: 436
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

It's look pretty cool.
I will try and give some feedback Xyne. smile

Offline

#3 2009-02-01 16:24:41

ploub
Member
Registered: 2007-05-16
Posts: 128

Re: pkgd: a daemon for sharing pkgs over your LAN

I like it. It saves me having to sync my cache to a custom repo and do a repo_add. Seems to work fine but two points:

 sudo pacman -Sy claws-mail
:: Synchronising package databases...
 custom is up to date
error: failed retrieving file 'core.db.tar.gz' from 192.168.123.114 : Success

and so on. I guess the error message is irrelevant as it seems to download fine from the server.
Shaman doesn't like the additional lines in /etc/pacman.conf

peter ~  $  shaman
Translations are enabled. 
Loading translations from "/usr/share/shaman/translations/" 
Parsing  "custom" 
Parsing  "core" 
Segmentation fault

Otherwise a roaring success I would say. Thanks!

Offline

#4 2009-02-01 23:00:48

fukawi2
Forum Moderator
From: .vic.au
Registered: 2007-09-28
Posts: 5,283
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Looks good. Nice work Xyne smile

Offline

#5 2009-02-02 01:25:03

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Thanks all. smile

@ploub
I think the error messages feel a bit unclean too. I've left it that way initially because I was more interested in getting the basic LAN topology worked out. I'll look into the possibility of having pkgd retrieve packages from the mirrors itself or at least forward pacman to the next mirror in the list (the latter shouldn't be too hard).

I don't know what shaman is complaining about. You could try running the master pkgd in a console with "pkgd" instead of "/etc/rc.d/pkgdd start" and see what request it's getting from shaman. Post it here and I'll see if I can figure it out.

Offline

#6 2009-02-02 02:37:03

sand_man
Member
From: Australia
Registered: 2008-06-10
Posts: 2,164

Re: pkgd: a daemon for sharing pkgs over your LAN

For a while I was sharing my pacman cache via NFS. I guess this would work the same?


neutral

Offline

#7 2009-02-02 03:36:37

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I think so.

The main difference that I can think is that each host running a pkgd can clear its own cache without affecting the cache of others.

Offline

#8 2009-02-02 03:42:12

Zariel
Member
Registered: 2008-10-07
Posts: 446

Re: pkgd: a daemon for sharing pkgs over your LAN

If the package isnt found on the server but is found on a client can you get the server to download the file? To speed the process if its needed again

Offline

#9 2009-02-02 04:53:50

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

For now, the best thing is to have a pkgd running on each server that's using the master pkgd. That way, any pkg that's downloaded will be available to all the others.

I had originally thought of making pkgd capable of downloading pkgs and creating a cache (including database files), but there are some complexities:

a) the main pkgd then ends up storing copies of all pkgs requested by other hosts, which leads to some unnecessary redundancy
As long as it's optional though, that's not really a problem

b) the package has to be downloaded, saved and forwarded at the same time
I need to learn a bit more http protocol to do this (e.g. how to tell pacman to wait with keep-alive responses while pkgd starts downloading the pkg), plus there's the issue of configuring external downloaders. That's not really difficult, but it will take some time and testing. There's a good chance that I'll do this though, because I would want to use metalink downloads like powerpill does.

c) the main mirror would need to have a mirrorlist for all repos in order to know where to get the packages
What I really need to do is work out better pkgd-pkgd communication methods so that mirrorlists etc can be sent with requests (e.g. check if you have pkgX and download it here if you don't). I also want to add checks for master/slave cycles, enable optional network cross-overs, slave-sharing, etc.

Offline

#10 2009-02-02 23:54:38

awesome_welles
Member
Registered: 2008-12-12
Posts: 25

Re: pkgd: a daemon for sharing pkgs over your LAN

Excellent! Seems to be working perfectly.

Offline

#11 2009-02-03 07:28:06

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

big_smile

empty.png

Last edited by Xyne (2009-02-03 07:31:53)

Offline

#12 2009-02-03 11:10:31

Purch
Member
From: Finland
Registered: 2006-02-23
Posts: 229

Re: pkgd: a daemon for sharing pkgs over your LAN

Can you divert the master log output from vc/1? Pkgdd outputs the request lines to vc/1, when client makes a query.

btw, this software rocks big_smile

Offline

#13 2009-02-03 12:44:03

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I've added log support now. Set a log file in /etc/pkgd.conf. I've also fixed the "client xxx.xxx.xxx.xxx has requested y.pkg.tar.gz" output (I had missed a print statement).

Last edited by Xyne (2009-02-03 13:01:18)

Offline

#14 2009-02-05 05:34:23

SnapShot
Member
Registered: 2008-08-26
Posts: 43

Re: pkgd: a daemon for sharing pkgs over your LAN

Nice and simple, I needed something like this. I was thinking of implementing something similar with shared cache and python, but your solution is much better. Too bad that I don't know Perl to contribute and customize it to my needs.

Do you have any suggestion for the folowing configuration. I have WiFi network at home, two of my laptops have Arch64 installed and they are not alway turned on. I would want to always check when downloading packages if the package is already donwloaded on the other laptop if it is connected. How would I configure pkgd? Will it work if I run pkgd on both laptops and configure on each laptop that the other laptop is the master.

For example:
For laptop on 192.168.0.2 the master is 192.168.0.3
For laptop on 192.168.0.3 the master is 192.168.0.2

Topology (arrows point to the direction of downloading):

laptop1 <------> laptop2
     ^            ^
      \          /
       \        /
        \      /
      arch mirror

If I manage to configure this two laptops, it would be easy to configure my other two machines running Arch i686 in the same way.

Last edited by SnapShot (2009-02-05 05:36:24)

Offline

#15 2009-02-05 10:15:10

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I think it would work to set up a reciprocal master-slave relationship. As it is right now, the pkgd first checks if it has the pkg itself, then it sends a simple request to each slave to ask if the slave has it. The slaves don't yet query their own slaves, so there shouldn't be an infinite query loop. I will eventually add that as a feature, but I'll also add checks to prevent such loops.

Just try it in non-daemon mode ("pkgd" from a terminal) first and see what happens, then report back to let me know how it goes.

Offline

#16 2009-02-05 12:09:16

CptYossarian
Member
Registered: 2009-02-05
Posts: 5

Re: pkgd: a daemon for sharing pkgs over your LAN

I found pkgd to use 100% CPU and reach speeds of 50-100KiB/s when serving files from Pentium III 650MHz and Atom N270 1.6GHz systems. Couldn't get to profile pkgd, maybe because of the use of threads.

Ah, also, pkgd dies with SIGPIPE when you Ctrl-C pacman while downloading.

Last edited by CptYossarian (2009-02-05 12:12:02)

Offline

#17 2009-02-05 12:26:19

Dieter@be
Forum Fellow
From: Belgium
Registered: 2006-11-05
Posts: 1,959
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Nice concept man!  To make it really distributed/no-spof you could get rid of the master-slave concept and use something like avahi


< Daenyth> and he works prolifically
4 8 15 16 23 42

Offline

#18 2009-02-05 14:15:56

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

@CptYossarian
Can you give me some more information about your setup and what you're doing when it uses 100% CPU? (network topology, etc)
*edit*
Nvm, I just noticed the relatively high cpu usage here too. I'll try to fix that.

@Dieter@be
Thanks.
I don't really know anything about avahi but I can look into it.

Last edited by Xyne (2009-02-05 14:22:33)

Offline

#19 2009-02-05 14:57:13

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

@CptYossarian
I think I've found and fixed both problems. Can you test the latest version and let me know if either persist?

Offline

#20 2009-02-05 15:54:46

CptYossarian
Member
Registered: 2009-02-05
Posts: 5

Re: pkgd: a daemon for sharing pkgs over your LAN

Now it seems to work just fine.

Thanks!

Offline

#21 2009-02-05 17:51:22

SnapShot
Member
Registered: 2008-08-26
Posts: 43

Re: pkgd: a daemon for sharing pkgs over your LAN

Thanks Xyne, it works with recirpocal master-slave configuration.

I have just tought something, it seems like it is not necessary to configure master on each of the laptop. If I just add the other laptop as a Server in pacman.conf it should always query if the package is available on the other laptop. I have not tested it yet because the master-slave configuration worked perfectly. Do you think it's fine for simple 2 PCs configuration (1<----->1) to just run pkgd on each of them without any master-slave configuration?

Last edited by SnapShot (2009-02-05 17:52:51)

Offline

#22 2009-02-05 18:04:23

catwell
Member
From: Bretagne, France
Registered: 2008-02-20
Posts: 207
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

That's great. It will become very useful when package signature will be implemented in Pacman, because without it you have to trust all the other people using it on your LAN... But that's a step in the good direction, I love the idea of sharing resources like that.

Offline

#23 2009-02-06 11:56:29

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,688
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

@SnapShot
That would work too. The only difference that I can think of right now is that you'll get up to 2 error messages if the package can't be found, but that's only an aesthetic problem.

@catwell
Yeah, it's still very simplistic at the moment. I added the password option in order to provide some very basic security/filtering, but I'll need to do a bit more to make it robust. If someone did manage to insert a poisoned pkg though, they would still need to duplicate the md5sum to get it past pacman. Hopefully nobody has anyone that determined to harm their system on their LAN. wink

In the future I intend to add more integrity checks and other features to enable safe(r?) scaling.

Last edited by Xyne (2009-02-06 11:56:43)

Offline

#24 2009-03-09 21:52:48

brazzmonkey
Member
From: between keyboard and chair
Registered: 2006-03-16
Posts: 818
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Thanks very much for this useful contribution, Xyne! This is exactly what I've been looking for!


what goes up must come down

Offline

#25 2009-03-12 11:00:42

brazzmonkey
Member
From: between keyboard and chair
Registered: 2006-03-16
Posts: 818
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Xyne, do you have any plans to make pkgd work with powerpill in the near future, as mentioned on your website?


what goes up must come down

Offline

Board footer

Powered by FluxBB