You are not logged in.
line 9 from /etc/rc.d/pkgdd should look like this
su -p pkgd -c '/usr/bin/pkgd --quiet &'
as suggested.
after this, you can have the shell for the pkgd user point to /sbin/nologin and it works as expected..
Offline
updated
It should update the account automatically. Let me know if there are any problems.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
updated
It should update the account automatically. Let me know if there are any problems.
there is a small issue, when started during boot, it fails... my guess is because there is no inherited environment during boot.
tbh, i think this is a widespread issue in arch. net-auto fails the same way in my computer. and so does mythbackend. the daemons run correctly when started manually.
Offline
Would removing the "-p" option solve the issue or prevent the daemon from "logging in"?
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Would removing the "-p" option solve the issue or prevent the daemon from "logging in"?
if -p is removed, su fails to start a shell (now it does not have one) and thus it fails..
im not sure if it only needs an environment...maybe with export $HOME=/root/ would help... but im just doing the same thing as suggested for the mythbackend rc script.
EDIT
i just tried it and it didnt work. im not sure what piece of the environment is needed. if i find it, i'll report back
EDIT2
the beggining of the rc.d script should look like this:
#!/bin/bash
. /etc/rc.conf
. /etc/rc.d/functions
### HACK
export SHELL="/bin/bash"
PID=`/usr/bin/pkgd --get-pid`
...
this works for me
Last edited by eldragon (2009-10-16 19:01:41)
Offline
Thanks, eldragon.
I've included that "export" line in the latest version.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Hey xyne,
thanks for providing a fine package like pkgd
Unfortunatly, I can't get it working. I plan to have my desktop computer as a master pkgd server and my laptop as a slave.
Both boxes use yaourt -> powerpill -> pacman-color as their packaging toolchain.
On my desktop PC I installed pkgd and did not specify any master server and so on, whereas on my laptop I put the following
line into my /etc/powerpill.conf (with 192.168.6.5 being my desktop pc):
[PkgD]
Server = http://192.168.6.5:14687/$repo
Now to the problem that arises: As soon as my laptop starts downloading packages, pkgdd on my desktop instantly exits
without further notice. Neither do I see any logfiles in /var/log/pkgd/ nor is anything printed when run from commandline.
I know this bugreport is pretty fuzzy, but I don't know how to start debugging without any error messages
Thanks in advance,
Vrob
Offline
@Vrob
Try running pkgd directly in a terminal on your desktop (i.e. just run "pkgd", not the daemon script) to get some output. You can also try setting a log path to generate a log file with "--log /path/to/file" on the command line or by uncommenting and editing the relevant line in the configuration file. Normally the output in the terminal will be the same as the log output plus any Perl errors.
Post that and I'll try to debug it.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Ok, I did what you suggested and using the --log param I got some logs but still no output on the terminal (I had already tried starting pkgd directly from the commandline but I did not use the --log parameter so then I got basically nothing... ).
But I cannot derive any interesting information from logs as I did multiple test runs and each run (with wiping the slaves cache inbetween) , the logs were different. There are no error messages, or other messages than "client requested package XYZ" for that matter.
Out of 6 tries pkgd stopped four times with "xulrunner-1.9.1.3-2-x86_64.pkg.tar.gz" and two times with "libpciaccess-0.10.9-1-x86_64.pkg.tar.gz" being the last request.
Greetings,
Vrob
Last edited by Vrob (2009-10-22 17:35:54)
Offline
So sometimes the download operation succeeds and sometimes it doesn't?
Try "pacman -Sp <some package>" to print out a url and then try downloading that url with wget to see if it gives any information. You can also try using the "--debug" flag with pacman to see if it gives any information. This shouldn't be a client-side issue if pkgd is exiting but I really have no idea what could cause this if there are no errors at all.
Without further information I can't help you. The only thing that I can do is add a debug mode to pkgd but I really don't have the time to do that right now. Sorry.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
I also have the same problem. For me it serves three packages and then quits silently. I'd love to see this fixed, as it makes it pretty unusable.
[git] | [AURpkgs] | [arch-games]
Offline
@Daenyth
See my previous post. Run it in a console with full output, get the URLs from pacman and then try getting them with curl or wget to see what errors it generates.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Do you mean to run powerpill -Sp? I only have PkgD defined in powerpill.conf as the documentation advises.
[root@Muspelheimr pkgd]# tail pkgd.log
[2009-11-23 22:13:00] client 10.0.1.7 requested libxaw-1.0.7-1-i686.pkg.tar.gz
[2009-11-23 22:13:00] client 10.0.1.7 requested pygtk-2.16.0-2-i686.pkg.tar.gz
[2009-11-23 22:13:00] client 10.0.1.7 requested jre-6u17-1-i686.pkg.tar.gz
[2009-11-23 22:17:59] listening on port 14687
[2009-11-23 22:18:07] client 10.0.1.7 requested libxaw-1.0.7-1-i686.pkg.tar.gz
[2009-11-23 22:18:07] client 10.0.1.7 requested pygtk-2.16.0-2-i686.pkg.tar.gz
[2009-11-23 22:18:07] client 10.0.1.7 requested jre-6u17-1-i686.pkg.tar.gz
[2009-11-23 22:18:14] listening on port 14687
[2009-11-23 22:18:28] client 10.0.1.7 requested jre-6u17-1-i686.pkg.tar.gz
[2009-11-23 22:18:28] client 10.0.1.7 requested jre-6u17-1-i686.pkg.tar.gz
Last edited by Daenyth (2009-11-24 13:58:13)
[git] | [AURpkgs] | [arch-games]
Offline
Ok, I did some testing and thus I think I could drill down the problem to pkgdds server being imcompatible with aria2c.
To get the download URLs I added the "Server = <Master>" Lines to my /etc/pacman.conf as described in the manpages and started the pkgdds on each computer.
Using pacman -Sup I got the following list of download-URLs:
http://192.168.6.5:14687/extra/cmake-2. … pkg.tar.gz
http://192.168.6.5:14687/extra/dirmngr- … pkg.tar.gz
http://192.168.6.5:14687/extra/gnupg2-2 … pkg.tar.gz
http://192.168.6.5:14687/extra/imagemag … pkg.tar.gz
http://192.168.6.5:14687/extra/kdelibs- … pkg.tar.gz
http://192.168.6.5:14687/extra/rrdtool- … pkg.tar.gz
http://192.168.6.5:14687/extra/lm_senso … pkg.tar.gz
http://192.168.6.5:14687/extra/php-5.3. … pkg.tar.gz
http://192.168.6.5:14687/extra/tdb-3.4. … pkg.tar.gz
http://192.168.6.5:14687/extra/smbclien … pkg.tar.gz
http://192.168.6.5:14687/extra/samba-3. … pkg.tar.gz
http://192.168.6.5:14687/core/tzdata-20 … pkg.tar.gz
http://192.168.6.5:14687/archlinuxfr/vi … kg.tar.bz2
http://192.168.6.5:14687/extra/xmlrpc-c … pkg.tar.gz
I could then use both pacman and wget (using wget -i) to download the packages successfully. Still, using yaourt -> powerpill -> pacman-color -> pacman did not work, so I tried downloading the packages manually with aria2c as it is done by powerpill which immediately crashed the master-pkgdd without further notice and resulted in the following error messages.
~/foo $ aria2c -i ../dl.txt -d .
2009-11-24 15:00:52.155998 ERROR - CUID#7 - Download aborted. URI=http://192.168.6.5:14687/extra/dirmngr-1.0.2-2-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:195] URI=http://192.168.6.5:14687/extra/dirmngr-1.0.2-2-x86_64.pkg.tar.gz
-> [HttpSkipResponseCommand.cc:172] Resource not found
2009-11-24 15:00:52.297734 ERROR - CUID#13 - Download aborted. URI=http://192.168.6.5:14687/extra/rrdtool-1.4.1-1-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:195] URI=http://192.168.6.5:14687/extra/rrdtool-1.4.1-1-x86_64.pkg.tar.gz
-> [HttpResponse.cc:93] Invalid range header. Request: 1,048,576-0/1,073,096, Response: 0-1,073,095/1,073,096
2009-11-24 15:00:52.543183 ERROR - CUID#8 - Download aborted. URI=http://192.168.6.5:14687/extra/gnupg2-2.0.13-3-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:52.543386 ERROR - CUID#9 - Download aborted. URI=http://192.168.6.5:14687/extra/imagemagick-6.5.7.10-1-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:52.611742 ERROR - CUID#12 - Download aborted. URI=http://192.168.6.5:14687/extra/rrdtool-1.4.1-1-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:52.800105 ERROR - CUID#16 - Download aborted. URI=http://192.168.6.5:14687/extra/cmake-2.8.0-1-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:52.800201 ERROR - CUID#17 - Download aborted. URI=http://192.168.6.5:14687/extra/cmake-2.8.0-1-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:52.800287 ERROR - CUID#18 - Download aborted. URI=http://192.168.6.5:14687/extra/cmake-2.8.0-1-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:52.800368 ERROR - CUID#19 - Download aborted. URI=http://192.168.6.5:14687/extra/cmake-2.8.0-1-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:52.800451 ERROR - CUID#20 - Download aborted. URI=http://192.168.6.5:14687/extra/lm_sensors-3.1.1-4-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:52.800530 ERROR - CUID#21 - Download aborted. URI=http://192.168.6.5:14687/extra/php-5.3.1-3-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:52.872663 ERROR - CUID#6 - Download aborted. URI=http://192.168.6.5:14687/extra/cmake-2.8.0-1-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:52.944851 ERROR - CUID#25 - Download aborted. URI=http://192.168.6.5:14687/extra/tdb-3.4.3-3-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:53.041139 ERROR - CUID#30 - Download aborted. URI=http://192.168.6.5:14687/extra/smbclient-3.4.3-3-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:53.041234 ERROR - CUID#31 - Download aborted. URI=http://192.168.6.5:14687/extra/samba-3.4.3-3-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
[#5 SIZE:0B/18.0MiB(0%) CN:1 SPD:0Bs](3more...) [TOTAL SPD:0Bs] [FileAlloc:#5 9.5MiB/18.0MiB(52%)]
2009-11-24 15:00:53.141400 ERROR - CUID#35 - Download aborted. URI=http://192.168.6.5:14687/core/tzdata-2009s-1-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:53.196645 ERROR - CUID#36 - Download aborted. URI=http://192.168.6.5:14687/archlinuxfr/virtualbox_bin-3.0.12-1.pkg.tar.bz2
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:53.302572 ERROR - CUID#37 - Download aborted. URI=http://192.168.6.5:14687/extra/xmlrpc-c-1.06.38-1-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:53.594247 ERROR - CUID#39 - Download aborted. URI=http://192.168.6.5:14687/extra/kdelibs-4.3.3-2-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:53.594443 ERROR - CUID#10 - Download aborted. URI=http://192.168.6.5:14687/extra/kdelibs-4.3.3-2-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:53.595641 ERROR - CUID#38 - Download aborted. URI=http://192.168.6.5:14687/extra/kdelibs-4.3.3-2-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:53.595729 ERROR - CUID#42 - Download aborted. URI=http://192.168.6.5:14687/extra/kdelibs-4.3.3-2-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
2009-11-24 15:00:53.595817 ERROR - CUID#40 - Download aborted. URI=http://192.168.6.5:14687/extra/kdelibs-4.3.3-2-x86_64.pkg.tar.gz
Exception: [AbstractCommand.cc:568] Failed to establish connection, cause: Connection refused
Download Results:
gid|stat|avg speed |path/URI
===+====+===========+===========================================================
2| ERR| n/a|http://192.168.6.5:14687/extra/dirmngr-1.0.2-2-x86_64.pkg.tar.gz
3| ERR| n/a|http://192.168.6.5:14687/extra/gnupg2-2.0.13-3-x86_64.pkg.tar.gz
4| ERR| n/a|http://192.168.6.5:14687/extra/imagemagick-6.5.7.10-1-x86_64.pkg.tar.gz
6| ERR| n/a|./rrdtool-1.4.1-1-x86_64.pkg.tar.gz
7| ERR| n/a|http://192.168.6.5:14687/extra/lm_sensors-3.1.1-4-x86_64.pkg.tar.gz
8| ERR| n/a|http://192.168.6.5:14687/extra/php-5.3.1-3-x86_64.pkg.tar.gz
1| ERR| n/a|./cmake-2.8.0-1-x86_64.pkg.tar.gz
9| ERR| n/a|http://192.168.6.5:14687/extra/tdb-3.4.3-3-x86_64.pkg.tar.gz
10| ERR| n/a|http://192.168.6.5:14687/extra/smbclient-3.4.3-3-x86_64.pkg.tar.gz
11| ERR| n/a|http://192.168.6.5:14687/extra/samba-3.4.3-3-x86_64.pkg.tar.gz
12| ERR| n/a|http://192.168.6.5:14687/core/tzdata-2009s-1-x86_64.pkg.tar.gz
13| ERR| n/a|http://192.168.6.5:14687/archlinuxfr/virtualbox_bin-3.0.12-1.pkg.tar.bz2
14| ERR| n/a|http://192.168.6.5:14687/extra/xmlrpc-c-1.06.38-1-x86_64.pkg.tar.gz
5| ERR| n/a|./kdelibs-4.3.3-2-x86_64.pkg.tar.gz
Status Legend:
(ERR):error occurred.
aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
(dirmngr is in fact not in the master-pkgdds cache so that doesn't really count )
Greets,
Vrob
Last edited by Vrob (2009-11-24 14:04:54)
Offline
Thanks Vrob, that will give me a good starting point. I'll look into this now.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
This was more difficult to fix than anyone will probably ever appreciate.
Basically aria2c no longer plays nicely with PkgD, or rather with Perl's sockets. It was exiting on larger files because instead of aria2c using a single connection and accepting all the data, it was just exiting and reconnecting. Perl, instead of giving some error, just died each time.
After playing around with it for a while, I got to a point where sometimes it would work, and sometimes it wouldn't. There must be some really complicated interplay going on under the hood but whatever it is, it's buggy as fsck in this case.
So I decided to work around it and go with threads to pre-fetch pkgd packages. That didn't work either because LWP wouldn't play nicely with threads and I kept getting nasty blocking situations no matter how I tried. My favorite was this:
$thr->is_joinable()
Returns true if the thread has finished running, is not detached and has not yet been joined. In other words, the thread is ready to be joined, and a call to $thr->join() will not block.
if ($thr->is_joinable())
{
print "about to join thread $thr\n"
$thr->join() ;
print "done\n"
}
Yep, it blocked.
perldoc.perl.org lies
I tried so many different things (detaching threads and using queues to determine when they've finished, etc) but nothing worked consistently.
In the end I went back to basics and extended my custom protocol for PkgD, then used threads and a queue to handle the prefetch. It seems to work and it's actually more efficient than the previous code, but it took way more time and patience than I had expected.
tl;dr
Grab perl-xyne-arch from my site and tell me if it works. I'll move the latest version into [community] after a little more testing.
Last edited by Xyne (2009-11-25 01:19:01)
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Hey Xyne,
thanks for working on this. I got good and bad news and I'm gonna start with the good ones:
The server doesn't crash anymore and is recognized by powerpill correctly.
And now for the bad ones:
When using powerpill I can't see any aria2c output it simply hangs after:
:: Proceed with download [Y/n] Y
I have turned of the Aria2Silent Option in /etc/powerpill.conf, but that doesn't change anything, I still get no output.
So I tried the pacman way again (adding the server line to /etc/pacman.conf, printing URLs, manual download with aria2c, see above), which resulted in the
same header related errors I posted before, but without the server crashes.
Another strange thing I noticed is, that pacman doesn't print protocol information for pkgd served packages. The "http://"-part is missing in the URLs, so I had
to add them manually.
Both, client and server use your new perl-xyne-arch package.
Greets,
Vrob
Offline
Ok, I think I've solved both issues but I'm not sure.
aria2c is very aggressive and rude by default. It sends 5 requests per file but instead of specifying the end ranges for each segment it requests everything from the beginning of the range until the end of the file. The real problem is that once it gets the segment that it actually wants, it closes the socket. Perl then crashes when trying to write to the empty socket. That's probably a bug in Perl and the first time I've truly wished for a real "try, catch" construct (for Perl programmers reading this: I tried using eval...even that crashed it).
I worked around that by forking the sub that returns the file and it seems to work without any catches.
As for the blocking after "proceed with download [Y/n]", I'm not sure it's actually blocking. I think it might be transferring the files. I tested it by running pkgd and downloading "base". It took a little while to copy all of the files but then powerpill continued as usual. I'll try to add a progress indicator to notify the user while the files are transferring.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Wow, thanks for the hard work Xyne! I love this tool
[git] | [AURpkgs] | [arch-games]
Offline
I've added some feedback during the PkgD retrieval stage. I did some quick testing and it all seems to work so I'm heading out now. I'll push the update into [community] this weekend after I get some feedback.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Hey Xyne,
thanks for working so hard and constantly on this, I really appreciate that
Unfortunately the latest version (Edit: I refer to 0.90 here) of perl-xyne-arch fixed neither the header- nor the blocking issue
I still see messages like these in my master servers logs which crash the clients aria2c:
[2009-11-27 19:39:42] client 192.168.6.10 requested ode-0.11.1-1-x86_64.pkg.tar.gz
[2009-11-27 19:39:42] client 192.168.6.10 requested sdl_net-1.2.7-3-x86_64.pkg.tar.gz
[2009-11-27 19:39:42] client 192.168.6.10 requested xmoto-0.5.2-2-x86_64.pkg.tar.gz
[2009-11-27 19:39:44] client 192.168.6.10 requested xmoto-0.5.2-2-x86_64.pkg.tar.gz
[2009-11-27 19:39:44] client 192.168.6.10 requested xmoto-0.5.2-2-x86_64.pkg.tar.gz
[2009-11-27 19:39:44] client 192.168.6.10 requested xmoto-0.5.2-2-x86_64.pkg.tar.gz
[2009-11-27 19:39:44] client 192.168.6.10 requested xmoto-0.5.2-2-x86_64.pkg.tar.gz
(I use xmoto and it's dependencies as a testsuite since the whole package is like 37 MB, so even over ethernet it's not an instant transfer)
Also this blocking issue seems to connected to this somehow. If I comment out all the pkgd-relevant lines in my clients /etc/powerpill.conf I get an aria2c progress info and everything works fine (besides the packages not being shared in the LAN ), whereas it blocks when pkgd support is turned on. But from the flickering of my switches lights I suppose that the transfer actually takes place but the output is somehow not displayed.
Greets,
Vrob
Last edited by Vrob (2009-11-27 18:57:03)
Offline
Just tested the latest version (you pushed 0.91 while I was writing the above post ).
I still see some showstoppers:
First, the blocking issue is not yet resolved, retrieval works (as it did before, at least if you want to interpret the rapid flickering of my switches activity LEDs excatly after I press return to start download that way...) but it simply hangs after retrieval stage. I have absolutely no clue what is causing this.
Second, inbetween the PkgD status messages I see to perl-related errors:
...
Use of uninitialized value $address in concatenation (.) or string at /usr/share/perl5/vendor_perl/Xyne/Arch/PkgD.pm line 805, <STDIN> line 1.
Use of uninitialized value $port in concatenation (.) or string at /usr/share/perl5/vendor_perl/Xyne/Arch/PkgD.pm line 805, <STDIN> line 1.
Vrob
Last edited by Vrob (2009-11-27 18:56:33)
Offline
Post your configs (pkgd, powerpill, pacman). There seems to be something wrong with your setup as those variables should always be initialized in the loop that they're in. I'll update the code to handle uninitalized values there but that should never happen.
You mentioned before that the "http://" protocol was missing from some output... are you sure that you've included that when specifying your PkgD servers?
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Ok, here are the config files.
Master-server
pacman.conf: http://pastebin.com/f28618a85
powerpill.conf: http://pastebin.com/f740e648f
pkgd.conf: http://pastebin.com/f136cb141
Client
pacman.conf: http://pastebin.com/f35fd6550
powerpill.conf: http://pastebin.com/f565ad3a2
pkgd.conf: http://pastebin.com/f30470e51
I have to apologize, you were right, that I simply forgot to add the "http://" in my pacman.conf. *shame*
A couple of minutes ago I checked your makerepo script and notices that it also blocks without any statement or obvious cause.
Greets,
Vrob
Offline
So does it work now?
Does your master-slave setup work too? I was surprised to see that you'd omitted the port when specifying the master.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline