pkgd: a daemon for sharing pkgs over your LAN

Xyne · 2011-01-28 02:05:11

Ravenman wrote:

Workstation:

[root@workstation ~]# pacman -Si firefox
error: package 'firefox' was not found
[root@workstation ~]# pacman -Si thunderbird
error: package 'thunderbird' was not found
[root@workstation ~]# pacman -Si dhcpcd
error: package 'dhcpcd' was not found
[root@workstation ~]#

[root@workstation ~]# powerpill -Syu
:: Synchronizing package databases...
--2011-01-27 16:28:03--  http://mirrors.kernel.org/archlinux/core/os/i686/core.db.tar.gz
Resolving mirrors.kernel.org... 199.6.1.174, 204.152.191.39, 130.239.17.6, ...
Connecting to mirrors.kernel.org|199.6.1.174|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `/var/lib/pacman/core.db.tar.gz.part'

    [ <=>                                                 ] 3,435       --.-K/s   in 0s      

2011-01-27 16:28:03 (118 MB/s) - `/var/lib/pacman/core.db.tar.gz.part' saved [3435]

--2011-01-27 16:28:03--  http://mirrors.kernel.org/archlinux/extra/os/i686/extra.db.tar.gz
Resolving mirrors.kernel.org... 204.152.191.39, 130.239.17.6, 149.20.20.135, ...
Connecting to mirrors.kernel.org|204.152.191.39|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `/var/lib/pacman/extra.db.tar.gz.part'

    [ <=>                                                 ] 3,437       --.-K/s   in 0s      

2011-01-27 16:28:03 (180 MB/s) - `/var/lib/pacman/extra.db.tar.gz.part' saved [3437]

--2011-01-27 16:28:03--  http://mirrors.kernel.org/archlinux/community/os/i686/community.db.tar.gz
Resolving mirrors.kernel.org... 130.239.17.6, 149.20.20.135, 199.6.1.174, ...
Connecting to mirrors.kernel.org|130.239.17.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `/var/lib/pacman/community.db.tar.gz.part'

    [ <=>                                                 ] 3,445       --.-K/s   in 0s      

2011-01-27 16:28:03 (141 MB/s) - `/var/lib/pacman/community.db.tar.gz.part' saved [3445]

:: Starting full system upgrade...
 there is nothing to do
[root@workstation ~]#

This is definitely not a PkgD issue. There's something wrong with your workstation pacman configuration. It seems that it's saving the databases with a ".part" extension (check /var/lib/pacman to be sure, they should be saved as *.db.tar.gz, not *.db.tar.gz.part). I suspect that you have ".part" in your XferCommand.

Ravenman · 2011-01-28 16:34:48

Xyne wrote:

This is definitely not a PkgD issue. There's something wrong with your workstation pacman configuration. It seems that it's saving the databases with a ".part" extension (check /var/lib/pacman to be sure, they should be saved as *.db.tar.gz, not *.db.tar.gz.part). I suspect that you have ".part" in your XferCommand.

I have pacman 3.4.3-1 in both systems and I copied the /etc/pacman.conf file from server to workstation but nothing changes.

This is the /etc/pacman.conf file:

[root@workstation ~]# cat /etc/pacman.conf 
#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives

#
# GENERAL OPTIONS
#
[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir     = /
#DBPath      = /var/lib/pacman/
#CacheDir    = /var/cache/pacman/pkg/
#LogFile     = /var/log/pacman.log
HoldPkg     = pacman glibc
# If upgrades are available for these packages they will be asked for first
SyncFirst   = pacman
XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#XferCommand = /usr/bin/curl -C - %u > %o
#CleanMethod = KeepInstalled
Architecture = auto

# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
IgnorePkg   = intel-dri xf86-video-intel libgl
#IgnoreGroup =

#NoUpgrade   =
#NoExtract   =

# Misc options (all disabled by default)
#UseSyslog
#ShowSize
#UseDelta
#TotalDownload

#
# REPOSITORIES
#   - can be defined here or included from another file
#   - pacman will search repositories in the order defined here
#   - local/custom mirrors can be added here or in separate files
#   - repositories listed first will take precedence when packages
#     have identical names, regardless of version number
#   - URLs will have $repo replaced by the name of the current repo
#   - URLs will have $arch replaced by the name of the architecture
#
# Repository entries are of the format:
#       [repo-name]
#       Server = ServerName
#       Include = IncludePath
#
# The header [repo-name] is crucial - it must be present and
# uncommented to enable the repo.
#

# The testing repositories are disabled by default. To enable, uncomment the
# repo name header and Include lines. You can add preferred servers immediately
# after the header, and they will be used before the default mirrors.

#[testing]
#Include = /etc/pacman.d/mirrorlist

[core]
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /etc/pacman.d/mirrorlist

#[community-testing]
#Include = /etc/pacman.d/mirrorlist

[community]
Include = /etc/pacman.d/mirrorlist

# An example of a custom package repository.  See the pacman manpage for
# tips on creating your own repositories.
#[custom]
#Server = file:///home/custompkgs

[root@workstation ~]#

Last edited by Ravenman (2011-01-28 16:35:17)

Xyne · 2011-01-28 16:50:02

Try commenting out the XferCommand.

What is the output of

ls -lA /var/lib/pacman

on the workstation?

Ravenman · 2011-01-28 17:23:52

Xyne wrote:

Try commenting out the XferCommand.
What is the output of
ls -lA /var/lib/pacman
on the workstation?

With the XferCommand line commented out:

[root@workstation ~]# powerpill -Syu
:: Synchronizing package databases...
downloading core.db.tar.gz...
downloading extra.db.tar.gz...
downloading community.db.tar.gz...
:: Starting full system upgrade...
 there is nothing to do
[root@workstation ~]# ls -lA /var/lib/pacman/
total 80
-rw-r--r--   1 root root  3423 ene 28 11:36 archlinuxfr.db.tar.gz
-rw-r--r--   1 root root  3445 ene 28 12:25 community.db.tar.gz
-rw-r--r--   1 root root  3435 ene 28 12:25 core.db.tar.gz
-rw-r--r--   1 root root  3437 ene 28 12:25 extra.db.tar.gz
drwxr-xr-x 939 root root 57344 ene 28 11:22 local
drwxr-xr-x   6 root root  4096 ene 28 11:36 sync
[root@workstation ~]#

Xyne · 2011-01-28 17:56:16

dwidmann wrote:

I've just recently stumbled onto pkgd and it seems nice Xyne. It took me a few minutes though after initially setting it up that it doesn't like it when I use pacman.conf's XferCommand option (in particular I was using (XferCommand = /usr/bin/curl -C - %u > %o). I don't know if it's mentioned somewhere within this monster thread. Unsetting that helped. I'm not sure if that's a broad issue or a specific issue. I just thought I'd chime in because it doesn't seem to be mentioned in the manpage.

You need to add "-f" to the curl option, otherwise it does not return an error on 404.
The pacman devs have been informed of this.

@Ravenman
And does "pacman -Si firefox" work now?
If not, what is the output of "pacman -Ss"?
What about "ls -R /var/lib/pacman/sync"?

Consider starting a new thread as this really doesn't belong in the PkgD thread. As I understand the problem, your workstation can't find packages and reports that there is nothing to do, so it is clearly a pacman issue. It seems to be downloading databases though, so maybe it's failing to extract them. Otherwise, try a different mirror.

benjamin9999 · 2011-03-09 03:04:55

broken?

pkgd used to work, but it seems broken.

in (self) sub query_file, when a match is found, the originating entry in {query} is removed.
around line 615...

      delete($obj->{'query'}->{$filename});

but in file_query_pkgds, the loop over returned filenames will " skip anything which was not originally queried "
around line 840 or so...

      # skip anything which was not originally queried
      next if not exists($obj->{'query'}->{$filename});

so registered remote pkgd's return found results for queries, but they are ignored.

commenting the delete at line 614 fixes the problem.

Xyne · 2011-03-09 16:05:01

The query entry is only deleted if it's found locally, so it shouldn't be passed to any remote pkgds. Note that any local file is restored to the query object just after the query is send to the remote pkgds (line 625). I haven't looked at the code in a while so I may be missing something, but if this were a pkgd error then I would be surprised that no one else has noticed it.

Can you give me an example of what it is doing vs what you expect it to do?

You can use "print &str($obj,1)" to pretty-print the contents of the $obj at different points in the query_file function (and after) to track changes.

benjamin9999 · 2011-03-09 17:02:09

my reading of update_file_query_obj at 625 is that it just does some manipulations on {found} and nothing to {query}.

my test case, and fwiw i have mixed $arch's but it appears that pkgd doesn't care about that, and it was a simple test since i know that the master (x86_64) contains none of the files on a secondary pkgd node (i686).

testcase goes like this:

pkgd A is secondary, DOES contain core package.xz, and is configured to use pkgd B as master.
pkgd B is master, contains no packages. (well, in reality it contains x86_64 packages)

client X uses WGET to test with pkgd B.
wget http://pkgdB:14687/core/package.xz

B sends query to A.
A receives query, replies with json that includes 'found' {'package.xz':self etc} but the 'query' object is -empty-.
B loops over the received JSON, but skips the entry because of line 835 to "skip anything which was not originally queried"
B returns 404 to client.

...

no one may notice it because the nature of pkgd is that it "fails" silently, and due to the volatile nature of the arch repository, there's always the chance that "I guess it needed to fetch from the source." i never noticed it until i was setting up a new arch root from scratch after just having updated another, knowing that most of the necessary packages would be available on my pkgd network.

thanks

benjamin9999 · 2011-03-09 17:14:07

more on update_file_query_obj...

it's not clear what you're trying to do here (but i'm not really a perl guy, it's not your fault it's unreadable! hah)

in update_file_query_obj loop @ 804

$obj->{'found'}->{$filename} = $found->{$filename} if not defined($obj->{'found'}->{$filename});

perhaps was intended to be

$obj->{'query'}->{$filename} = $found->{$filename} if not defined($obj->{'query'}->{$filename});

without testing it, i assume this also fixes the problem.

even if so, i'm not sure i understand the point of removing the query entries and restoring them, i guess it has to do with what the results look like after passing from master->secondary->secondary etc ?

Xyne · 2011-03-10 05:39:44

Ok, when I replied earlier, I had just come home after a long day with only 2 hours of sleep the night before, so I only gave the code a cursory glance.

I took another look at it now and set up some pkgds servers for testing. It was indeed a code bug. I had forgotten that I had changed the JSON parser recently and I suspect that I may have introduced the bug when propagating the changes (by blindly pasting "($obj, $failed) = ...", thus overwriting the original $obj), but it may have been there before, as the data in $obj should be updated with results from the returned object (mostly "checked").

Regardless, it should be fixed now, along with an unrelated bug in the JSON parser.

I've pushed the updates to my repo. Please test the latest version of perl-xyne-arch and perl-xyne-common and let me know if they work as expected. I'll push them to [community] in a few days if they do.

Thanks for tracking down the bug.

benjamin9999 · 2011-03-10 06:41:57

looks like it works fine. i see pkgd requests that are fulfilled by a secondary. (again, easy to test, just update a i686 box with a x86_64 master)

thanks.

also, these....

after powerpill pkgd download....

Use of uninitialized value $w in scalar chomp        at /usr/share/perl5/vendor_perl/Xyne/Common/Functions.pm line 473.
Use of uninitialized value $w in pattern match (m//) at /usr/share/perl5/vendor_perl/Xyne/Common/Functions.pm line 474.

and

:: Proceed with download? [Y/n]
open2: fork failed: Cannot allocate memory at /usr/share/perl5/vendor_perl/Xyne/Arch/Powerpill.pm line 840

the former not any showstopper apparently, and i already gather that powerpill likes memory. i noticed when trying to use it on a low ram machine (256MB or something) and no swap that it would cause OOM kill and i'd return to the (arch) livecd login prompt. (to be fair, the livecd has the flaw that all packages are downloaded to the ram/tmp/unionfs)

also, the pkgd debug info is dumped to console but you probably know that.

also, and it's not easy to articulate why, but it may be nice for new users if powerpill.conf's PkgD description, and pkgd.conf's Master were both clear about the fact that the default port# will be used if not specified. neither say this, and i thought for a long time that i had to add it everywhere myself, which meant having to go lookup the port # because i don't do it often enough to remember what it is.

i worked on it because it's worth it - the tool is simple and solves a problem without being over complicated. other tools like pbget are also very useful for an arch admin - thanks.

Xyne · 2011-03-10 18:08:36

* suppressed uninitializled value warnings in Functions.pm (although I don't understand why the value was uninitialized instead of an empty string)
* removed leftover debug info (mea culpa, was coding in a hurry )
* added default port info to PkgD and Powerpill (and thus Bauerbill) -> updated man pages and default configuration files

Threads in Perl are unfortunately heavyweight. You can configure/disable them using the --threads/Threads option.

erick.red · 2011-04-28 14:05:19

Hey:

I tried today to installed and it asking for this perl-xyne-arch dependency and I can't found that in the AUR, and besides this link is broken, or not broken but the package isn't found.

eldragon · 2011-04-28 14:13:32

erick.red wrote:

Hey:
I tried today to installed and it asking for this perl-xyne-arch dependency and I can't found that in the AUR, and besides this link is broken, or not broken but the package isn't found.

this package is deprecated.

you should use pacserve instead. i guess a mod should close this thread

Xyne · 2011-04-28 14:44:29

Yep. See https://bbs.archlinux.org/viewtopic.php?id=115660

Use Pacserve as a replacement for PkgD.

Inxsible · 2011-04-28 14:52:31

Closing on request of OP.

Arch Linux

#151 2011-01-28 02:05:11

Re: pkgd: a daemon for sharing pkgs over your LAN

#152 2011-01-28 16:34:48

Re: pkgd: a daemon for sharing pkgs over your LAN

#153 2011-01-28 16:50:02

Re: pkgd: a daemon for sharing pkgs over your LAN

#154 2011-01-28 17:23:52

Re: pkgd: a daemon for sharing pkgs over your LAN

#155 2011-01-28 17:56:16

Re: pkgd: a daemon for sharing pkgs over your LAN

#156 2011-03-09 03:04:55

Re: pkgd: a daemon for sharing pkgs over your LAN

#157 2011-03-09 16:05:01

Re: pkgd: a daemon for sharing pkgs over your LAN

#158 2011-03-09 17:02:09

Re: pkgd: a daemon for sharing pkgs over your LAN

#159 2011-03-09 17:14:07

Re: pkgd: a daemon for sharing pkgs over your LAN

#160 2011-03-10 05:39:44

Re: pkgd: a daemon for sharing pkgs over your LAN

#161 2011-03-10 06:41:57

Re: pkgd: a daemon for sharing pkgs over your LAN

#162 2011-03-10 18:08:36

Re: pkgd: a daemon for sharing pkgs over your LAN

#163 2011-04-28 14:05:19

Re: pkgd: a daemon for sharing pkgs over your LAN

#164 2011-04-28 14:13:32

Re: pkgd: a daemon for sharing pkgs over your LAN

#165 2011-04-28 14:44:29

Re: pkgd: a daemon for sharing pkgs over your LAN

#166 2011-04-28 14:52:31

Re: pkgd: a daemon for sharing pkgs over your LAN

Board footer