You are not logged in.

#26 2009-03-12 16:23:45

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,534
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I still haven't thought of a completely clean way to do it and the best way I've thought of so far will require restructuring the powerpill code which I haven't had the time to do. From the looks of it, I don't expect to get this done in the near future, sorry. It's on my mind again though, so we'll see.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#27 2009-03-13 11:01:58

brazzmonkey
Member
From: between keyboard and chair
Registered: 2006-03-16
Posts: 818
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Xyne wrote:

I don't expect to get this done in the near future, sorry.

Hey, no need to apologize! Your contributions in their current state are already very much appreciated!


what goes up must come down

Offline

#28 2009-04-11 13:11:08

Zariel
Member
Registered: 2008-10-07
Posts: 446

Re: pkgd: a daemon for sharing pkgs over your LAN

How will this work in this situation:

My main server is i686 and my workstation and a laptop are x86_64, will it still work as intended? (Sorry if this is obvious lol)

Offline

#29 2009-04-11 16:43:30

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,534
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Zariel wrote:

How will this work in this situation:

My main server is i686 and my workstation and a laptop are x86_64, will it still work as intended? (Sorry if this is obvious lol)

My first reaction was "just don't mix and match" but I realized that you probably want to run the master pkgd on the main server and have the workstation and laptop connect to that. You can either try removing "/var/cache/pacman/pkg" from the %CACHE% list in the config file or try replacing it with a path to an empty directory or /dev/null. That should cause it to only check with the slave pkgds running on the other server.

You could also just bypass the main server and directly connect to the pkgd on the laptop from the workstation and vice versa.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#30 2009-04-11 20:24:41

toofishes
Developer
From: Chicago, IL
Registered: 2006-06-06
Posts: 602
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Have you thought about using multicast to make this decentralized?

For your pacman.conf entry, you would always add http://localhost:pkdport/$repo as the entry. When your local pkgd instance got a request, it would send out the query on the multicast address and listen for responses, waiting a very short period of time. If it got any ACK responses, it would pick one and send the redirect back to pacman. If there were all NACK responses or no responses at all, then it would send the 404. Finally, you would want the code in the script to subscribe/listen to that multicast address for requests and send responses as necessary.

Offline

#31 2009-04-11 23:43:39

Zariel
Member
Registered: 2008-10-07
Posts: 446

Re: pkgd: a daemon for sharing pkgs over your LAN

Well it worked fine, the laptop downloaded from the workstation just fine.

Offline

#32 2009-04-12 01:29:31

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,534
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

toofishes wrote:

Have you thought about using multicast to make this decentralized?

For your pacman.conf entry, you would always add http://localhost:pkdport/$repo as the entry. When your local pkgd instance got a request, it would send out the query on the multicast address and listen for responses, waiting a very short period of time. If it got any ACK responses, it would pick one and send the redirect back to pacman. If there were all NACK responses or no responses at all, then it would send the 404. Finally, you would want the code in the script to subscribe/listen to that multicast address for requests and send responses as necessary.

I hadn't considered that mostly because I've never done any real network programming. I'll look into it but my first thought is that it may be problematic when dealing with i686 and x86_64 networks on the same LAN.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#33 2009-04-12 07:38:48

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,534
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

The new pkgd is now in xyne-any. It's not compatible with the older versions but it now works with powerpill big_smile


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#34 2009-04-12 07:40:50

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: pkgd: a daemon for sharing pkgs over your LAN

Xyne wrote:

I hadn't considered that mostly because I've never done any real network programming. I'll look into it but my first thought is that it may be problematic when dealing with i686 and x86_64 networks on the same LAN.

I suppose each pc on the network either has i686 packages or x86_64 packages, so they can't share together anyway, so you could just use two different ports. So this is not a problem at all, is it?

If you suppose some pc can have both and you want to share everything, then it might be problematic, but does this case exist?

Anyway, I guess the problem is this stupid community repo which is still dropping the arch from all package filenames?
Otherwise you would not have to consider the arch, only the filenames.

What you could do instead is adding the arch in the query. Then each pc checks if it has a package. If yes, it checks if that package has the correct arch : bsdtar xOf foo.pkg.tar.gz .PKGINFO | grep -q $arch


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#35 2009-04-12 07:55:02

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,534
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I hadn't considered that broadcasts would be port-specific when I replied. Designating different default ports for each architecture might be a simple solution.

I just had to deal with the community upload stripping the architecture out of the package name... I don't see the point of that.

Adding a query introduces a whole new level of complexity. On top of that it could create too much overhead if a server has to unpack and check lots of (large) packages.


Anyway, I'll give this some though but I doubt that I"ll change it in the near future. I've spent way too much time refactoring powerpill/reflector/pkgd, adding man pages to everything and changing my upload scripts/layout.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#36 2009-04-12 08:37:13

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: pkgd: a daemon for sharing pkgs over your LAN

Xyne wrote:

I hadn't considered that broadcasts would be port-specific when I replied. Designating different default ports for each architecture might be a simple solution.

Adding a query introduces a whole new level of complexity. On top of that it could create too much overhead if a server has to unpack and check lots of (large) packages.

You only have to check one package, the one in the query.
And you don't have to unpack it fully, you only unpack a very small file from it, the PKGINFO one.
So I doubt that is too much overhead.

Anyway, I was just saying I thought it was possible and giving some tips to achieve it.
I think its useless so I dont support that solution, I prefer the port specific one which seems simpler and easier.


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#37 2009-04-12 08:59:48

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,534
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

shining wrote:

Anyway, I was just saying I thought it was possible and giving some tips to achieve it.
I think its useless so I dont support that solution, I prefer the port specific one which seems simpler and easier.

It's interesting to explore the idea anyway, so thanks for the suggestions. I like the idea of designated default ports too and I will (eventually) look into that and multicasting. wink


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#38 2009-04-12 11:24:35

Zariel
Member
Registered: 2008-10-07
Posts: 446

Re: pkgd: a daemon for sharing pkgs over your LAN

Are the depends set correctly? I got lots of errors trying to run the deamon, and the deamon wont start or stop on any of my servers (but pkgd works)

/e using `pidof pkgd` works /e well no it doesnt but it starts it atlest, using pkgd --get-pid it gives, Unable to connect error

pkgd --get-pid really should just spit out nothing then for the deamon to work?

/e change line 163 to this -> || return "";

Or something better but this fixes the problem

/e (Working feverishly here)

It doesnt work with powerpill, it says exception URI=http://192.168.0.5:14687/Pkgd/$pkg not found, but im pretty sure it is there. The line in powerpill conf

[Pkgd]
Server = http://192.168.0.5:14687/$repo

And the server isnt getting any requests, doesnt even have the Client connection log entry

Last edited by Zariel (2009-04-12 11:47:26)

Offline

#39 2009-04-12 20:51:02

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,534
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Thanks for the feedback, Zariel.

I think I've fixed it. I wasn't thinking about the script in /etc/rc.d when I added the pid error message (removing it was the best solution, as you suggested). I've just tested with powerpill and it seems to work. Let me know if you still have issues.

I've updated the man page and conf file too (somehow forgot to do that).


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#40 2009-04-13 10:58:28

Purch
Member
From: Finland
Registered: 2006-02-23
Posts: 229

Re: pkgd: a daemon for sharing pkgs over your LAN

some problems here with the latest 1.6-1 (perl 5.10.0-4)

$ sudo /etc/rc.d/pkgdd restart
Can't locate Xyne/Arch/PkgD.pm in @INC (@INC contains: /usr/lib/perl5/site_perl/5.10.0 /usr/share/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl /usr/lib/perl5/current/i686-linux-thread-multi /usr/lib/perl5/current /usr/lib/perl5/site_perl/current/i686-linux-thread-multi /usr/lib/perl5/site_perl/current .) at /usr/bin/pkgd line 22.
BEGIN failed--compilation aborted at /usr/bin/pkgd line 22.
Can't locate Xyne/Arch/PkgD.pm in @INC (@INC contains: /usr/lib/perl5/site_perl/5.10.0 /usr/share/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl /usr/lib/perl5/current/i686-linux-thread-multi /usr/lib/perl5/current /usr/lib/perl5/site_perl/current/i686-linux-thread-multi /usr/lib/perl5/site_perl/current .) at /usr/bin/pkgd line 22.
BEGIN failed--compilation aborted at /usr/bin/pkgd line 22.
:: Stopping pkgdd                                                                                                             [FAIL]
Can't locate Xyne/Arch/PkgD.pm in @INC (@INC contains: /usr/lib/perl5/site_perl/5.10.0 /usr/share/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl /usr/lib/perl5/current/i686-linux-thread-multi /usr/lib/perl5/current /usr/lib/perl5/site_perl/current/i686-linux-thread-multi /usr/lib/perl5/site_perl/current .) at /usr/bin/pkgd line 22.
BEGIN failed--compilation aborted at /usr/bin/pkgd line 22.
:: Starting pkgdd                                                                                                             [BUSY] Can't locate Xyne/Arch/PkgD.pm in @INC (@INC contains: /usr/lib/perl5/site_perl/5.10.0 /usr/share/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl /usr/lib/perl5/current/i686-linux-thread-multi /usr/lib/perl5/current /usr/lib/perl5/site_perl/current/i686-linux-thread-multi /usr/lib/perl5/site_perl/current .) at /usr/bin/pkgd line 22.
BEGIN failed--compilation aborted at /usr/bin/pkgd line 22.
                                                                                                                              [DONE]

Offline

#41 2009-04-13 13:05:19

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,534
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Um, silly question, but do you actually have the dependencies installed (perl-xyne-common, perl-xyne-arch)?
Check with "pacman -Qs perl-xyne".


Did you build from the AUR or download from the xyne-any repo on my site?


Purch wrote:
Can't locate Xyne/Arch/PkgD.pm in @INC (@INC contains: ... /usr/share/perl5/vendor_perl ...
$pacman -Ql perl-xyne-arch
...
perl-xyne-arch /usr/share/perl5/vendor_perl/Xyne/Arch/PkgD.pm
...

Last edited by Xyne (2009-04-13 13:25:50)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#42 2009-04-13 15:46:43

Purch
Member
From: Finland
Registered: 2006-02-23
Posts: 229

Re: pkgd: a daemon for sharing pkgs over your LAN

Shouldn't those be dependencies? I installed it from AUR. Only 'depends' is perl.

Last edited by Purch (2009-04-13 15:54:26)

Offline

#43 2009-04-13 16:07:34

Purch
Member
From: Finland
Registered: 2006-02-23
Posts: 229

Re: pkgd: a daemon for sharing pkgs over your LAN

PID stuff does not work in the daemon script, because It is never zero.

$ /usr/bin/pkgd --get-pid
detected Quiet

$ /usr/bin/pkgd --get-pid
detected Quiet
32755

Commenting Quiet from pkgd.conf helps.

Last edited by Purch (2009-04-13 16:14:19)

Offline

#44 2009-04-13 18:59:43

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,534
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Sorry about that... I had left a debugging line in and had forgotten to updatie the dependencies array. The AUR is now synced with the repos on my site. Please test the latest version and let me know if it works as expected.

*should probably get some sleep soon*

Last edited by Xyne (2009-04-13 19:00:40)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#45 2009-09-02 09:42:10

brazzmonkey
Member
From: between keyboard and chair
Registered: 2006-03-16
Posts: 818
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I've got several computers at home (32 and 64-bit) and my internet bandwidth will be limited in the near future (I'll move ton countryside).

I noticed some packages are now marked as "any" architecture. If I understand things right, this means the same package could be used for both x86_64 and i686.
Would there be a way to share these "any" packages between both architectures with pkgd?

Thx


what goes up must come down

Offline

#46 2009-09-02 10:37:29

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 6,534
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

I think so but not yet. I don't remember exactly how I've implemented it but pkgd should only care about the package name. Previously packages in [community] had the architecture removed from their names (e.g. foo-1.4-5-i686.pkg.tar.gz -> foo-1.4-5.pkg.tar.gz) which would create problems for mixed architectures due to name overlapping.

Once all of the community packages have been upgraded they should all have non-overlapping names so it should be ok to mix architectures with pkgd. The problem is that it will take a while to phase out the old naming scheme because I don't think the packages are being actively upgraded just to change the package name.

A workaround in the meantime would be to create an "any" cache on each computer outside of the standard cache directory (you would need to specify the additional cache in pacman.conf so pacman can find those packages). You could automatically move all "*-any.pkg.*" to the "any" cache and then run a second instance of pkgd on each machine on a different port. You would then have 3 pkgd networks, one for each architecture and one for "any". You would then add the "x86_64" and "any" master pkgds to pacman.conf on your x86_64 machines and "i686" and "any" master pkgds to pacman.conf on your i686 machines.

The downside to the approach is that it probably feels a bit complicated and that you would often get timeout errors when searching for "any" packages as pacman checks the non-"any" network. You might be able to get around that though by using powerpill's built-in pkgd support. I actually haven't used it myself (other than in limited testing) but I think it would transparently handle both networks. I'd be interested to hear of the results if you try it.

You should also consider setting up a regular repo for all of your "any" packages. You would need to copy all of your "any" packages to that, run "repo-add" and probably prune old packages. If you decide to go that route then you might find "makerepo" useful. It depends on what you consider the simplest solution.


Btw, to move the packages from the standard cache to an "any" cache, you could use something like this in a cronjob:

find /var/cache/pacman/pkg -name "*-any.pkg.*" -exec mv -t /path/to/any/cache {} \;

That's untested but it should work.

Last edited by Xyne (2009-09-02 10:39:58)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#47 2009-09-04 07:15:06

brazzmonkey
Member
From: between keyboard and chair
Registered: 2006-03-16
Posts: 818
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Thanks for your interesting answer, Xyne.
Actually I hoped for something like "just do this and that and you're done", or "no, that's not possible". because I don't have much time available to explore your suggestion right now. Hopefully in the near future...

Thanks again.


what goes up must come down

Offline

#48 2009-09-06 13:26:15

wantilles
Member
From: Athens - Greece
Registered: 2007-03-29
Posts: 327

Re: pkgd: a daemon for sharing pkgs over your LAN

Why do you need to make a whole daemon, when you can accomplish this over a LAN, for as many computers as you wish with:

Server (exported via nfs or samba -> ie /etc/exports or /etc/samba/smb.conf)
Clients (imported in fstab)
- 1 nfs (or samba) share (per architecture)

- 3 symlinks (per architecture)

/etc/pacman.d
/var/cache/pacman/pkg
/var/lib/pacman/sync

- 2 more optional symlinks (per architecture)

/etc/pacman.conf
/etc/makepkg.conf

And two more for abs:
- 2 symlinks (per architecture)

/var/abs
/etc/abs.conf

You are making a lot of fuss about nothing.

Where is the KISS philosophy?

If you share all the above, and you put in your common (per architecture) pacman.conf file:

[options]
CleanMethod = KeepCurrent

You will never have to download even twice, a package that has been downloaded once, for any machine on the LAN (this goes for each architecture).

Pacman databases are always updated on the server (pacman -Sy - this goes only for its own architecture), and clients will only have to update themselves (pacman -Su).

Offline

#49 2009-09-06 14:01:49

shining
Pacman Developer
Registered: 2006-05-10
Posts: 2,043

Re: pkgd: a daemon for sharing pkgs over your LAN

wantilles wrote:

Why do you need to make a whole daemon, when you can accomplish this over a LAN, for as many computers as you wish with:

Server (exported via nfs or samba -> ie /etc/exports or /etc/samba/smb.conf)
Clients (imported in fstab)
- 1 nfs (or samba) share (per architecture)

If you had bothered reading the first few comments, you would have noticed someone already asked this and Xyne already answered. smile

Xyne wrote:

The main difference that I can think is that each host running a pkgd can clear its own cache without affecting the cache of others.

so the main advantage is that this solution is more decentralized.

Then I don't know if all pkgd users are aware of this.


pacman roulette : pacman -S $(pacman -Slq | LANG=C sort -R | head -n $((RANDOM % 10)))

Offline

#50 2009-09-06 14:05:08

Allan
Member
From: Brisbane, AU
Registered: 2007-06-09
Posts: 10,946
Website

Re: pkgd: a daemon for sharing pkgs over your LAN

Also, try doing a "pacman -Syu" on multiple systems with that setup.   Two pacman instances trying to download the same package to the cache at the same time really makes things interesting. I ran into that while using multiple chroots with the same pacman cache.

Although, I don't know if Xyne's setup is more robust to that...

Offline

Board footer

Powered by FluxBB