You are not logged in.
Well, not really
Here is my output from powerpill on the client:
$ yaourt -S xmoto
:: The following packages will be downloaded:
[extra]
ode sdl_net xmoto
:: Number of Packages: 3
:: Total Download Size: 36.98 MiB
:: Total Installed Size: 58.33 MiB
:: Proceed with download? [Y/n]
PkgD: requesting sdl_net-1.2.7-3-x86_64.pkg.tar.gz from 192.168.6.5:14687
PkgD: requesting xmoto-0.5.2-2-x86_64.pkg.tar.gz from 192.168.6.5:14687
PkgD: retrieved sdl_net-1.2.7-3-x86_64.pkg.tar.gz
PkgD: retrieved xmoto-0.5.2-2-x86_64.pkg.tar.gz
After that, it simply blocks. I also don't see ode getting requested although I'm sure that it is in my masters cache and my that my clients cache is empty.
The same for makerepo, I just type in:
$ makerepo /var/abs/local/REPOBUILD
And it blocks after pressing return with no output at all.
Greets,
Vrob
Last edited by Vrob (2009-11-28 10:34:45)
Offline
Why would makerepo be related to this? Maybe there's something wrong with your general setup as I really don't have any idea at this point what it would be.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
The description for your perl-xyne-* packages says that they contain common modules and functions, used across your tools (at least that is what I understood - correct me if I'm wrong ).
So I guessed that there might be a function that is used by makerepo and powerpill that causes the blocking. That would make it pretty easy to trace that certain function.
No offense, but I don't see what kind of "general setup error" would cause two specific programs to simply block, as I don't have such problems with other programs. Also, if you take into consideration that this whole blocking issue arose after you changed the backend to be thread based, I'd rather go for a thread-related bug in the master servers part which causes the client to block. On the other hand, I don't see how this makerepo blocking could fit into that picture...
Greets,
Vrob
Last edited by Vrob (2009-11-29 10:27:58)
Offline
If you can make this so it doesn't show the errors in pacman, then I'll immediately start using it full time. I know it's minor, but it's ugly and bugs me I'd use powerpill, but it doesn't offer any speed improvement on my network with the mirror I use, and I like pacman's output better. But, this is still a great program, nice job I just hate downloading the same package multiple times, sometimes I use like 3 GB of bandwidth in one day just downloading updates to KDE or something on different computers. One problem though is I'd still have i686 packages and x86_64 packages to update, but nothing I can do about that
Last edited by doorknob60 (2009-11-29 10:40:02)
Offline
@doorknob60
I don't think that this is possible, cause the error you get is produced by pacman and there out of the scope of what xynes programs can influence. I'd guess that this would need a special
pacman patch. You could also use powerpill even if it offers you no speed improvement as long as you don't dislike it's output more than the pacman errors
@blocking issue
I solved the blocking thing at least in makerepo. It was more or less my fault as I created a subfolder in %LOCAL_PKGS% for each package I want to be built which confused makerepo
When I put everything in %LOCAL_PKGS% without subfolders, it doesn't block and everything is fine
Edit: Wait, that "solution" for makerepo doesn't really make sense, because I could only store one PKGBUILD without using subfolders which I suppose isn't really the purpose of makerepo. But perhaps that's another story and we should - at least in this thread - focus on pkgd
Bye,
Vrob
Last edited by Vrob (2009-11-29 10:53:51)
Offline
The only point of overlap between makerepo and pkgd would be when downloading binary packages using powerpill for inclusion in the makerepo repo. It seems that you've already discovered that the issues are unrelated though.
I'll mention it here instead of jumping over to the makerepo thread: I use makerepo for the xyne-any repo. In my "projects" directory, I have a "pkgbuilds" directory which contains subdirectories for each project, e.g. "powerpill", "makerepo", etc. Each of these contain the respective project's PKGBUILD and local source files. Each line in the %LOCAL_PKGS% section of the REPOBUILD points to one of these subdirectories, e.g. ".../projects/pkgbuilds/powerpill", ".../projects/pkgbuilds/makerepo", etc. I hope that this clarifies the intended setup.
Back to pkgd, after looking at your configs again I noticed that your powerpill.conf on the client points to the pkgd on the client instead of the master pkgd. If you want to use such a setup, you will need to make the "master" pkgd a slave of the client as well to ensure that you can get all packages from both systems. That shouldn't affect the blocking but it might explain why "ode" was not requested in the output of your previous post. Alternatively, simply point the client's powerpill.conf at the master pkgd instead.
As I can't reproduce the blocking myself it's very difficult to debug. I don't have the time right now to include extensive debugging messages so here's a patch that you can apply to try to isolate the problem:
debug.patch
--- /usr/share/perl5/vendor_perl/Xyne/Arch/Powerpill.pm 2009-11-25 02:36:28.000000000 +0100
+++ Powerpill.pm 2009-11-29 16:20:34.393261516 +0100
@@ -510,7 +510,9 @@
if (@pkgd_urls)
{
@pkgs = map {$self->{Pacman}->get_sync_db_filename($_)} @pkgs;
+ print "debugging: about to request packages from pkgd servers\n";
&Xyne::Arch::PkgD::retrieve_and_save_pkg_from_pkgd(\@pkgs, \@pkgd_urls, $cachedir);
+ print "debugging: pkgd pkg retrieval complete\n";
}
}
@@ -532,7 +534,9 @@
$self->switch_to_pacman() if not @pkgs;
exit if not $self->confirm_download(@pkgs);
+ print "debugging: about to check for pkgd servers\n";
$self->maybe_get_pkgd_pkgs(@pkgs) if not $self->is_action('--get-metalink');
+ print "debugging: moving on to the regular download\n";
$self->set_download(@pkgs);
$self->reflect if $self->{PowerpillConf}->get_option('Reflect');
@@ -542,7 +546,9 @@
print $metalink;
exit;
}
+ print "debugging: about to run aria2c\n";
$self->run_aria2($metalink);
+ print "debugging: done running aria2c\n";
my $verified = $self->verify_md5sums(@pkgs);
if (not $verified)
Save it as "debug.patch" and then apply it with:
patch /usr/share/perl5/vendor_perl/Xyne/Arch/Powerpill.pm <debug.patch
Run powerpill again and post the output. Re-install perl-xyne-arch to reset the changes.
@doorknob60
As Vrob wrote, that's just not possible. As far as pacman is concerned, PkgD is just an HTTP server which may or may not have the necessary packages. I may eventually rewrite it so that each system runs it's own PkgD and broadcasts to find others. In that case, the PkgD on the client's system could redirect to the first mirror. That would require a major rewrite though and I just don't have the time for that right now.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Thanks for the clarification on the makerepo topic.
Doing it the way you described, everything is fine
I don't have the time to apply the patch right now, but I'll definitely do it later and post the results.
Vrob
Offline
I don't have the time to apply the patch right now, but I'll definitely do it later and post the results.
Ok, just bump the thread when you do so that I see it.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Ok, I applied the patch on my client and here is the output:
$ yaourt -S xmoto
:: The following packages will be downloaded:
[extra]
ode sdl_net xmoto
:: Number of Packages: 3
:: Total Download Size: 36.98 MiB
:: Total Installed Size: 58.33 MiB
:: Proceed with download? [Y/n]
debugging: about to check for pkgd servers
debugging: about to request packages from pkgd servers
PkgD: requesting sdl_net-1.2.7-3-x86_64.pkg.tar.gz from 192.168.6.5:14687
PkgD: requesting xmoto-0.5.2-2-x86_64.pkg.tar.gz from 192.168.6.5:14687
PkgD: retrieved sdl_net-1.2.7-3-x86_64.pkg.tar.gz
PkgD: retrieved xmoto-0.5.2-2-x86_64.pkg.tar.gz
I don't see any network, HDD or CPU activity after that.
Additionally, I'm not sure if I understood you right, concerning my configs. I just double-checked them and I don't think, that the client has itself in pkgd.conf. My Network config is:
Desktop/Master: 192.168.6.5
Laptop/Client: 192.168.6.10
Greets,
Vrob
Offline
Sorry, I thought that you had the client pointed at the pkgd on the client. My brain registered 2 different IPs without realizing that one was localhost and thus concluded that they were two different systems.
After seeing the output that you posted, I was able to localize the problem to a threading issue which shouldn't arise. After adding code to catch it, I decided to set up a couple of PkgDs to make sure that they were all playing nicely together. In doing so, I discovered the real cause for the previous issue as well as another dormant issue which limited the effective depth of more complicated topologies.
In summary, this was a bitch to debug but it should work now.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Great work Xyne
Sounds as if you really had a hard time debugging this error, but I'm glad, if it's gone now.
Speaking of it, when will you push the fixed packages onto your site/repository?
Greets,
Vrob
Offline
It's already in [xyne-any]. I'm waiting for feedback before I push it to [community].
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Hi, pkgd doesn't work here, i fired up both pkgd client and master and when i start both i can see my slave being accepted by the master, the only problem is that when pacman tries to retrieve the core.db.tar.gz
error: failed retrieving file 'core.db.tar.gz' from 192.168.0.101 : Not Found
My pacman.conf on the slave http://dpaste.com/139407/
It really looks like a great tool.
Offline
Hi, pkgd doesn't work here, i fired up both pkgd client and master and when i start both i can see my slave being accepted by the master, the only problem is that when pacman tries to retrieve the core.db.tar.gz
error: failed retrieving file 'core.db.tar.gz' from 192.168.0.101 : Not Found
My pacman.conf on the slave http://dpaste.com/139407/
It really looks like a great tool.
thats the expected behavior with pacman. your PKGD server doesnt have the package lists... it should fail, and try with the available mirror.
when upgrading / installing, pkgd will pick the local mirror's package copy if there is one.
Offline
you're right it's working, it's working fast thanks Xyne
Offline
Xyne, i have been using the program since the last post here on my server, it works perfect except that the daemon is a bit of a memory leak. It spawns 100+ instances of itself and it ended using 300Mb of RAM in total.
Offline
Wow, great tool. Definitely something i will install.
Offline
@jelly
I've rewritten most of the backend for perl-xyne-arch>=1.0. I don't think any of the changes that I've made will have affected memory usage, but once I move that package into [community], I'll look into this and see what I can do.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
@jelly
I've rewritten most of the backend for perl-xyne-arch>=1.0. I don't think any of the changes that I've made will have affected memory usage, but once I move that package into [community], I'll look into this and see what I can do.
since update to 2.0, pkgd complains with
read() on unopened filehandle at /usr/share/perl5/vendor_perl/Xyne/Arch/PkgD.pm line 656.
Offline
read() on unopened filehandle at /usr/share/perl5/vendor_perl/Xyne/Arch/PkgD.pm line 656.
Can you post some context? Run pkgd in a terminal and paste the output here, otherwise that is going to be difficult to track down.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
eldragon wrote:read() on unopened filehandle at /usr/share/perl5/vendor_perl/Xyne/Arch/PkgD.pm line 656.
Can you post some context? Run pkgd in a terminal and paste the output here, otherwise that is going to be difficult to track down.
sorry for the lack of info..
that happens when starting /etc/rc.d/pkgdd
some test runs:
$ pkgd
^C
running it from my user doesnt complain.
$ sudo /etc/rc.d/pkgdd start
read() on unopened filehandle at /usr/share/perl5/vendor_perl/Xyne/Arch/PkgD.pm line 656.
:: Starting pkgdd [DONE]
enabling logging (uncommenting) provides the following
$ sudo touch /var/log/pkgd.log
$ sudo chown pkgd:users /var/log/pkgd.log
$ sudo chmod 644 /var/log/pkgd.log
$ sudo /etc/rc.d/pkgdd start
read() on unopened filehandle at /usr/share/perl5/vendor_perl/Xyne/Arch/PkgD.pm line 656.
:: Starting pkgdd [DONE]
mind the fiddling that had to be done to enable logging... maybe that file should be created by the install script
the log turned out empty.
starting with
# su -p pkgd -c '/usr/bin/pkgd'
doesnt complain either.
Last edited by eldragon (2010-01-12 17:42:57)
Offline
read() on unopened filehandle at /usr/share/perl5/vendor_perl/Xyne/Arch/PkgD.pm line 656.
I, too, have run into this (on all three of my machines that run pkgd, no less.)
Also, perhaps unrelated, I noticed while upgrading pkgd recently, that I couldn't stop and restart the service (it gave a fail on stopping and hung on starting the new one.)
Some poking around revealed the following:
peter@Yggdrasil ~ % ps aux | grep pkgd
pkgd 2856 0.0 0.1 7368 2948 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3169 0.0 0.1 7368 2416 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3172 0.0 0.1 7368 2416 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3175 0.0 0.1 7368 2400 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3178 0.0 0.1 7368 2476 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3181 0.0 0.1 7368 2416 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3184 0.0 0.1 7368 2400 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3187 0.0 0.1 7368 2416 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3190 0.0 0.1 7368 2460 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3193 0.0 0.1 7368 2456 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3196 0.0 0.1 7368 2400 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3199 0.0 0.1 7368 2416 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3202 0.0 0.1 7368 2416 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3207 0.0 0.1 7368 2400 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3210 0.0 0.1 7368 2416 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3213 0.0 0.1 7368 2416 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 3216 0.0 0.1 7368 2416 ? S Jan02 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 8524 0.0 0.1 7368 2456 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 8525 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 8526 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 8527 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 8586 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 8589 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 8592 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 8597 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9397 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9400 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9403 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9406 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9409 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9412 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9415 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9418 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9421 0.0 0.1 7368 2624 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9423 0.0 0.1 7368 2544 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9424 0.0 0.1 7368 2540 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9425 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9426 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9427 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9428 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9429 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9430 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9431 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9432 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9445 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9448 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9451 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 9454 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10911 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10912 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10913 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10914 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10915 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10916 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10917 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10918 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10919 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10920 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10921 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10922 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10923 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10924 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10925 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10926 0.0 0.1 7368 2544 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10927 0.0 0.1 7368 2544 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10928 0.0 0.1 7368 2504 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10929 0.0 0.1 7368 2464 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10930 0.0 0.1 7368 2456 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10931 0.0 0.1 7368 2456 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10932 0.0 0.1 7368 2460 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10933 0.0 0.1 7368 2456 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10934 0.0 0.1 7368 2404 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10935 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10936 0.0 0.1 7368 2428 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10937 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10938 0.0 0.1 7368 2404 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10939 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10940 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10941 0.0 0.1 7368 2564 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10942 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10943 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10944 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10945 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10946 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10947 0.0 0.1 7368 2672 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10950 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10951 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10952 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10953 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10954 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10955 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10956 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10957 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10958 0.0 0.1 7368 2404 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10959 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10960 0.0 0.1 7368 2404 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10961 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10962 0.0 0.1 7368 2440 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10963 0.0 0.1 7368 2484 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10964 0.0 0.1 7368 2428 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10965 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10968 0.0 0.1 7368 2404 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10969 0.0 0.1 7368 2400 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10970 0.0 0.1 7368 2404 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10971 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10972 0.0 0.1 7368 2440 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10973 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10974 0.0 0.1 7368 2440 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10975 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10976 0.0 0.1 7368 2440 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10977 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10978 0.0 0.1 7368 2440 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10979 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10980 0.0 0.1 7368 2440 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10981 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10982 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10983 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10984 0.0 0.1 7368 2440 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10985 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10986 0.0 0.1 7368 2440 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10987 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10988 0.0 0.1 7368 2496 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10989 0.0 0.1 7368 2544 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10990 0.0 0.1 7368 2488 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10991 0.0 0.1 7368 2544 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10992 0.0 0.1 7368 2488 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10993 0.0 0.1 7368 2536 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10994 0.0 0.1 7368 2536 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10995 0.0 0.1 7368 2488 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10996 0.0 0.1 7368 2488 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10997 0.0 0.1 7368 2472 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10998 0.0 0.1 7368 2472 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 10999 0.0 0.1 7368 2472 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11000 0.0 0.1 7368 2472 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11001 0.0 0.1 7368 2392 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11002 0.0 0.1 7368 2416 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11003 0.0 0.1 7368 2388 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11004 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11005 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11006 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11007 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11008 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11009 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11010 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11011 0.0 0.1 7368 2360 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11012 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11013 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11014 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11015 0.0 0.1 7368 2520 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11016 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11017 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11018 0.0 0.1 7368 2632 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11019 0.0 0.1 7368 2608 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11020 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11023 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11024 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11025 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11026 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11027 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11028 0.0 0.1 7368 2520 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11029 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11030 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11031 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11032 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11033 0.0 0.1 7368 2632 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11034 0.0 0.1 7368 2448 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11035 0.0 0.1 7368 2360 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11036 0.0 0.1 7368 2360 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11037 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11038 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11039 0.0 0.1 7368 2360 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11040 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11041 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11042 0.0 0.1 7368 2360 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11043 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11044 0.0 0.1 7368 2360 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11045 0.0 0.1 7368 2360 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11046 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11047 0.0 0.1 7368 2360 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11048 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11051 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11052 0.0 0.1 7368 2520 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11053 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11054 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11055 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11056 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11057 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11058 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11059 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11060 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11061 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11062 0.0 0.1 7368 2360 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11063 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11064 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11065 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11066 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11067 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11068 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11069 0.0 0.1 7368 2304 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11070 0.0 0.1 7368 2244 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 11071 0.0 0.1 7368 2248 ? S Jan07 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
root 17534 0.1 0.0 4984 1436 pts/2 S+ 22:51 0:00 /bin/bash /etc/rc.d/pkgdd restart
root 17551 0.0 0.0 4988 1456 pts/2 S+ 22:51 0:00 /bin/bash /etc/rc.d/pkgdd start
pkgd 19934 0.0 0.1 7368 2300 ? S 22:36 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 19937 0.0 0.1 7368 2304 ? S 22:36 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 19940 0.0 0.1 7368 2304 ? S 22:36 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 19943 0.0 0.1 7368 2304 ? S 22:36 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20451 0.0 0.1 7368 2820 ? S 22:39 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20454 0.0 0.1 7368 2824 ? S 22:39 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20457 0.0 0.1 7368 2824 ? S 22:39 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20460 0.0 0.1 7368 2824 ? S 22:39 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20467 0.0 0.1 7368 2784 ? S 22:41 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20470 0.0 0.1 7368 2784 ? S 22:41 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20473 0.0 0.1 7368 2784 ? S 22:41 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20476 0.0 0.1 7368 2784 ? S 22:42 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20479 0.0 0.1 7368 2304 ? S 22:42 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20482 0.0 0.1 7368 2304 ? S 22:42 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20485 0.0 0.1 7368 2304 ? S 22:42 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
pkgd 20488 0.0 0.1 7368 2304 ? S 22:42 0:00 /usr/bin/perl /usr/bin/pkgd --quiet
Or, 217 running instances of pkgd. My guess? One for every package downloaded or requested by another computer. This situation was on the master server.
This same situation does not exist on the non-master computers running pkgd - the only configuration difference I noticed is that I have a master server designated in the clients, and not in the master's configuration file:
peter@Yggdrasil ~ % cat /etc/pkgd.conf
# Comments begin with "#"
# The default cache is /var/cache/pacman/pkg
# If a cache is specified here on on the command line,
# it overrides the default cache. Include the default
# cache if you wish to use multiple caches.
#Cache = /var/cache/pacman/pkg/
#Cache = /some/other/cache
# The interval is the time that pkgd sleeps between
# notifications to the master pkgds.
#Interval = 300
# The path to the log file. The log will only
# be written if a path is specified.
Log = /var/log/pkgd/pkgd.log
# The master pkgd servers, listed on separate lines.
#Master = 192.168.0.1:14687
#Master = 192.168.0.2:14687
# The port for this pkgd to use.
# The default is 14687.
#Port = 14687
# Enabling quiet mode will suppress output.
# This is mostly for running pkgd as a daemon.
#Quiet
After updating pkgd, killing all these instances of pkgd, and starting the daemon running again, and doing an update on my other computers which involves asking the master for a whole bunch of packages, I get the following:
peter@Yggdrasil ~ % ps aux | grep pkgd | grep -v grep | wc -l
60
Yep, 60 new instances of zombie pkgd. Any insight on what causes this? Should I designate a master server as itself in its own conf file?
(also, pkgd appears to no longer be writing the log file after the update. I have no idea why.)
Offline
Ok, I think I've fixed both issues.
Sorry about the flood of pkgd processes. As I don't use pkgd myself, I was completely unaware of the issue. To fix this, I've added a thread which will reap child processes to prevent blocking while waiting for the child process to exit. Because the reaper thread operates on a queue, some processes may linger as defunct while earlier processes are still running, e.g. if a large file is requested before a small file, the latter's process will complete before the former's, but it won't be reaped until the former's has been reaped.
I've tested this by launching 766 backgrounded file transfers (every package in my cache, about 1.8 GB) and monitoring the process list. A few hundred defunct pkgd processes showed up but all were reaped once the transfer was complete.
I'll try to fix logging (including file creation) as soon as I have time. I don't think I actually reimplemented it when I rewrote the module.
Please confirm that the latest version of perl-xyne-arch (>=1.0.35) resolves both issues (it will remind me about the logging too).
Last edited by Xyne (2010-01-14 06:01:42)
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
restarted pkgd and the issue is gone. thanks.
Offline
Ok, I think I've fixed both issues.
Sorry about the flood of pkgd processes. As I don't use pkgd myself, I was completely unaware of the issue. To fix this, I've added a thread which will reap child processes to prevent blocking while waiting for the child process to exit. Because the reaper thread operates on a queue, some processes may linger as defunct while earlier processes are still running, e.g. if a large file is requested before a small file, the latter's process will complete before the former's, but it won't be reaped until the former's has been reaped.
I've tested this by launching 766 backgrounded file transfers (every package in my cache, about 1.8 GB) and monitoring the process list. A few hundred defunct pkgd processes showed up but all were reaped once the transfer was complete.
I'll try to fix logging (including file creation) as soon as I have time. I don't think I actually reimplemented it when I rewrote the module.
Please confirm that the latest version of perl-xyne-arch (>=1.0.35) resolves both issues (it will remind me about the logging too).
Indeed, it looks like the defunct process issue and the warning it throws on startup are both fixed. Thanks for writing a patch so quickly!
Logging is still busted, and I have no clue why. Is it a permissions problem? The logfile belongs to user pkgd, group pkgd, both of which exist on my system, and the log has permissions of 777, which might be my fault if that's why it's broken (I chmodded it that way so I could troubleshoot more easily).
Offline