powerpill reborn: pacman wrapper for faster downloads

Jristz · 2012-12-07 19:05:30

Xyne wrote:

@jristz
What repo is kde-full in? I do not have access to it on my system.

Not is a repo
is the meta-metapackage for kde
and using it in pacman prompt for select the other metapackages (kde-artwork-meta bla bla)

by default this prompt for select any (or all) the metapackages and then for the phonon (not prompt too)

Xyne · 2012-12-07 19:16:17

Jristz wrote:

Not is a repo
is the meta-metapackage for kde
and using it in pacman prompt for select the other metapackages (kde-artwork-meta bla bla)

$ pacman -S kde-full
error: target not found: kde-full

Regardless of whether it is a package or a group, it exists in some repo. I am asking which one, because it is not in any repo that I have enabled (core, extra, community, multilib, xyne-*).

Anyway, update to the new version of powerpill and try again. I suspect that you have not enabled the output as recommended, but this is now enabled by default in the new package. Of course, if you have edited /etc/powerpill/powerpill.json then you will need to merge changes from the .pacnew file.

Jristz · 2012-12-07 19:22:20

Ok now I see that is kde-meta not kde-full

I test now and a 'powerpill -S kde-meta' work as spected now
and the kde-full problem was solved too

matersci · 2012-12-07 19:35:20

Any ideas why I can't sync my local repo?

powerpill -Syu
2012-12-07 21:19:51.152769 NOTICE - Download GID#2 not complete: /var/lib/pacman/sync/custom.db
2012-12-07 21:19:51.356468 NOTICE - GID#3 - Download has already completed: /var/lib/pacman/sync/extra.db
2012-12-07 21:19:51.356573 NOTICE - Download complete: /var/lib/pacman/sync/extra.db
2012-12-07 21:19:51.376909 NOTICE - GID#4 - Download has already completed: /var/lib/pacman/sync/community.db
2012-12-07 21:19:51.377082 NOTICE - Download complete: /var/lib/pacman/sync/community.db
2012-12-07 21:19:51.686427 NOTICE - GID#1 - Download has already completed: /var/lib/pacman/sync/core.db
2012-12-07 21:19:51.686507 NOTICE - Download complete: /var/lib/pacman/sync/core.db
[#5 SIZE:0B/0B CN:1 SPD:0Bs]
2012-12-07 21:19:56.493472 NOTICE - GID#5 - Download has already completed: /var/lib/pacman/sync/multilib.db
2012-12-07 21:19:56.493556 NOTICE - Download complete: /var/lib/pacman/sync/multilib.db
Download Results:
gid|stat|avg speed |path/URI
===+====+===========+===========================================================
2| ERR| n/a|/var/lib/pacman/sync/custom.db
3| OK| 0B/s|/var/lib/pacman/sync/extra.db
4| OK| 0B/s|/var/lib/pacman/sync/community.db
1| OK| 0B/s|/var/lib/pacman/sync/core.db
5| OK| 0B/s|/var/lib/pacman/sync/multilib.db
Status Legend:
(OK):download completed.(ERR):error occurred.
aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
error: aria2c exited with 1

pacman syncs without errors.

pacman -Syy
:: Synchronizing package databases...
core 106.2 KiB 767K/s 00:00 [####################################] 100%
custom 10.4 KiB 142M/s 00:00 [####################################] 100%
extra 1431.2 KiB 1098K/s 00:01 [####################################] 100%
community 1808.9 KiB 1066K/s 00:02 [####################################] 100%
multilib 93.9 KiB 764K/s 00:00 [####################################] 100%

Last edited by matersci (2012-12-07 19:37:54)

Xyne · 2012-12-07 21:23:30

@matersci
Aria2 does not appear to support "file://" schemes in meta links. I have updated powerpill to copy such files directly. Try the latest version and let me know if it works as expected.

matersci · 2012-12-07 21:44:11

Xyne wrote:

@matersci
Aria2 does not appear to support "file://" schemes in meta links. I have updated powerpill to copy such files directly. Try the latest version and let me know if it works as expected.

nope...

Now it ignores "custom"

powerpill -Syy

2012-12-07 23:40:28.240194 NOTICE - GID#1 - Download has already completed: /var/lib/pacman/sync/core.db

2012-12-07 23:40:28.240309 NOTICE - Download complete: /var/lib/pacman/sync/core.db

2012-12-07 23:40:28.648242 NOTICE - GID#4 - Download has already completed: /var/lib/pacman/sync/multilib.db

2012-12-07 23:40:28.648347 NOTICE - Download complete: /var/lib/pacman/sync/multilib.db
[#2 SIZE:1.2MiB/1.3MiB(86%) CN:1 SPD:455.1KiBs](1more...) [TOTAL SPD:624.6KiBs]                              
2012-12-07 23:40:31.165480 NOTICE - Download complete: /var/lib/pacman/sync/extra.db
[#3 SIZE:1.5MiB/1.7MiB(87%) CN:1 SPD:210.4KiBs ETA:1s]                                                       
2012-12-07 23:40:36.657213 NOTICE - Download complete: /var/lib/pacman/sync/community.db

Download Results:
gid|stat|avg speed  |path/URI
===+====+===========+===========================================================
  1|  OK|       0B/s|/var/lib/pacman/sync/core.db
  4|  OK|       0B/s|/var/lib/pacman/sync/multilib.db
  2|  OK| 456.7KiB/s|/var/lib/pacman/sync/extra.db
  3|  OK| 214.6KiB/s|/var/lib/pacman/sync/community.db

Xyne · 2012-12-07 22:39:30

The internal copy doesn't output any information because it's not a download. Check the file itself.

I can add messages if necessary, but that will either require an additional command-line option or result in hard-coded output.

matersci · 2012-12-07 23:19:04

You are right, everything works now

powerpill -Syu

2012-12-08 01:16:47.345758 NOTICE - GID#1 - Download has already completed: /var/lib/pacman/sync/core.db

2012-12-08 01:16:47.345886 NOTICE - Download complete: /var/lib/pacman/sync/core.db

2012-12-08 01:16:47.698203 NOTICE - Download complete: /var/lib/pacman/sync/multilib.db
[#2 SIZE:304.0KiB/1.3MiB(21%) CN:1 SPD:124.6KiBs ETA:9s](1more...) [TOTAL SPD:634.3KiBs]                     
2012-12-08 01:16:50.650281 NOTICE - Download complete: /var/lib/pacman/sync/community.db
[#2 SIZE:1.2MiB/1.3MiB(91%) CN:1 SPD:152.6KiBs]                                                              
2012-12-08 01:16:56.730427 NOTICE - Download complete: /var/lib/pacman/sync/extra.db

Download Results:
gid|stat|avg speed  |path/URI
===+====+===========+===========================================================
  1|  OK|       0B/s|/var/lib/pacman/sync/core.db
  4|  OK| 268.3KiB/s|/var/lib/pacman/sync/multilib.db
  3|  OK| 562.2KiB/s|/var/lib/pacman/sync/community.db
  2|  OK| 156.5KiB/s|/var/lib/pacman/sync/extra.db

Status Legend:
(OK):download completed.
:: Starting full system upgrade...
resolving dependencies...
looking for inter-conflicts...

Targets (1): amarok-git-20121208-1

Total Installed Size:   33.12 MiB
Net Upgrade Size:       0.00 MiB

Proceed with installation? [Y/n]

Thanks for the quick fix!

b00rt00s · 2012-12-09 18:25:35

Great news. Now I've got some crazy construction on my system. I generally use yaourt, so in /etc/yaourtrc I've put:

PACMAN="powerpill"

In /etc/powerpill/powerpill.json I've put:

"pacman": {
    "config": "/etc/pacman.conf",
    "path": "/usr/bin/pacmatic"
  },

As a result I use 3 pacman wrappers at once. And It rally works great.

Xyne · 2012-12-09 21:03:31

b00rt00s wrote:

As a result I use 3 pacman wrappers at once. And It rally works great.

The golden age of the Pacman onion is back! Behold the Pactryoshka!

edit
Mostly unrelated, but now I strangely want this.

Last edited by Xyne (2012-12-09 21:05:45)

matersci · 2012-12-11 20:51:50

Xyne,
I noticed that if I choose to reinstall a package (that already exists in cache) powerpill downloads it again.

Why is that?

Xyne · 2012-12-11 22:47:09

matersci wrote:

Xyne,
I noticed that if I choose to reinstall a package (that already exists in cache) powerpill downloads it again.
Why is that?

I haven't noticed that and I can't seem to reproduce it with either aria2 or rsync. How do you know that it's downloading the package again? Do you see output or are you just monitoring network traffic?

Rsync probably checks if there is a newer version of the package on the server. Aria2 should just check the local checksums included in the internal metalink and conclude that the file is valid.

Once I know more I may add an internal check to skip valid packages, but at the moment the download decision is handled entirely by Aria2 and Rsync.

matersci · 2012-12-11 23:09:57

I see the output. Look

In cache folder:

# ls linux*
linux-3.6.9-1-x86_64.pkg.tar.xz              linux-firmware-20121118-1-any.pkg.tar.xz
linux-api-headers-3.6.3-1-x86_64.pkg.tar.xz

but

powerpill -S linux
[#1 SIZE:2.7MiB/41.9MiB(6%) CN:8 SPD:989.2KiBs ETA:40s]

on the other hand, pacman simple installs the file

pacman -S linux
warning: linux-3.6.9-1 is up to date -- reinstalling
resolving dependencies...
looking for inter-conflicts...

Targets (1): linux-3.6.9-1

Total Installed Size:   59.70 MiB
Net Upgrade Size:       0.00 MiB

Proceed with installation? [Y/n] 
(1/1) checking package integrity                                  [####################################] 100%
(/1) loading package files                                       [------------------------------------]   0%

Maybe the "problem" is that I am not using the default cache folder (/var/cache/pacman/pkg/) and aria2 doesn't use the path in pacman.conf

Xyne · 2012-12-12 01:56:35

Ok, I was able to reproduce this with the "linux" package. After some initial confusion I concluded that it was due to a newer modification time on the server, but I am not entirely sure. If that were the case I would expect rsync to detect it and not transfer the entire file.

I actually implemented internal sha256 checksum comparisons for cached files but realized that it should be unnecessary in all cases except one (local Pacserve files) so I commented out the code.

I can try to debug this further with the following information:
1) What is the sha256 checksum of the "linux" package currently in the powerpill cache? (sha256sum /path/to/package)
2) What happens if you add "--checksum" to the rsync args in powerpill.json and try again? (If the server complains about the option, remove it and jump to 4 below. Do not change servers yet as a different server may not behave the same.)

3) For 2, if it tries to download the package again with the "--checksum" option, let the download complete and get the new sha256 checksum. Post both checksums and let me know if it still tries to redownload the package.
4) f it does not download the package with the "--checksum" option, remove it and let it redownload the package. Verify that the checksum is that same as above and then run the command one more time to see if it still tries to redownload the file.

I wasn't able to check this here because I aborted a download and truncated the old file.

Vrantheo · 2012-12-12 03:47:15

A few issues here:

1. rsync is very handy for updating the repo databases but there's seldom point in using it to download the packages as well (since they're new), so during -Syu I'd like only aria2c to handle that part without rsync taking over. How do I enforce it?

2. Is it already possible to provide several rsync servers in powerpill.json (rather than just one) so it can fallback in case the first one fails? If not yet, then it's a feature request.

Last edited by Vrantheo (2012-12-12 04:40:24)

matersci · 2012-12-12 12:46:27

Xyne wrote:

Ok, I was able to reproduce this with the "linux" package. After some initial confusion I concluded that it was due to a newer modification time on the server, but I am not entirely sure. If that were the case I would expect rsync to detect it and not transfer the entire file.
I actually implemented internal sha256 checksum comparisons for cached files but realized that it should be unnecessary in all cases except one (local Pacserve files) so I commented out the code.
I can try to debug this further with the following information:
1) What is the sha256 checksum of the "linux" package currently in the powerpill cache? (sha256sum /path/to/package)
2) What happens if you add "--checksum" to the rsync args in powerpill.json and try again? (If the server complains about the option, remove it and jump to 4 below. Do not change servers yet as a different server may not behave the same.)
3) For 2, if it tries to download the package again with the "--checksum" option, let the download complete and get the new sha256 checksum. Post both checksums and let me know if it still tries to redownload the package.
4) f it does not download the package with the "--checksum" option, remove it and let it redownload the package. Verify that the checksum is that same as above and then run the command one more time to see if it still tries to redownload the file.
I wasn't able to check this here because I aborted a download and truncated the old file.

1)

sha256sum /media/Tmp/cache/linux-3.6.9-1-x86_64.pkg.tar.xz 
c6b585e52a1b2a2847dbdece3f191603f9418093fd2208186446c5aee38fedf9  /media/Tmp/cache/linux-3.6.9-1-x86_64.pkg.tar.xz

2) added this in powerpill.json

"rsync": {
"args": [
"--no-motd",
"--progress",
"--checksum",
"verbose"
],

but same result.

3) sha256sum is the same

sha256sum /media/Tmp/cache/linux-3.6.9-1-x86_64.pkg.tar.xz 
c6b585e52a1b2a2847dbdece3f191603f9418093fd2208186446c5aee38fedf9  /media/Tmp/cache/linux-3.6.9-1-x86_64.pkg.tar.xz

Last edited by matersci (2012-12-12 12:47:42)

Xyne · 2012-12-12 14:37:40

Vrantheo wrote:

1. rsync is very handy for updating the repo databases but there's seldom point in using it to download the packages as well (since they're new), so during -Syu I'd like only aria2c to handle that part without rsync taking over. How do I enforce it?

In my experience that is not true. I have noticed that several mirrors appear to throttle HTTP transfers but not Rsync transfers, so even for new files the transfer is much faster. As mentioned elsewhere, it also avoids separate connections per file.

Nevertheless, I have added a new option: rsync/db only
If true (the current default), Rsync will only be used for database downloads.

Vrantheo wrote:

2. Is it already possible to provide several rsync servers in powerpill.json (rather than just one) so it can fallback in case the first one fails? If not yet, then it's a feature request.

Done.
rsync/server is now a list named rsync/servers in powerpill.json. Servers will be tried in order and Powerpill will fall back on Aria2 if they all fail due to transfer issues.

Let me know if it works as expected.

@matersci
I consider this an rsync/server error so there isn't much that I can do. You can try one or more of the following:
* use a different rsync server
* check the rsync man page for options that might affect the download decision
* disable rsync for package downloads with the new "rsync/db" only option (I recognize that this is not a solution)

If you tell me which rsync mirror you're using, I will test it here to see if I find a solution.

Vrantheo · 2012-12-12 15:42:48

Xyne wrote:

In my experience that is not true. I have noticed that several mirrors appear to throttle HTTP transfers but not Rsync transfers, so even for new files the transfer is much faster. As mentioned elsewhere, it also avoids separate connections per file.
Nevertheless, I have added a new option: rsync/db only
If true (the current default), Rsync will only be used for database downloads.

I'm inside a campus network myself so I can take full advantage when using sources from educational networks too, so multi-threaded downloads via aria2 is fastest for me. (our school admin limits public web bandwidth quite a bit)

Xyne wrote:

Done.
rsync/server is now a list named rsync/servers in powerpill.json. Servers will be tried in order and Powerpill will fall back on Aria2 if they all fail due to transfer issues.
Let me know if it works as expected.

FYI I've been using the yaourt | powerpill | pacman-color | pacget wrapping chain.

The default powerpill.json you provided in the package didn't reflect the "server -> servers" change. You might want to fix that little typo.

All else is so far so good. Kudos and keep up the good work.

Last edited by Vrantheo (2012-12-12 15:58:05)

Klink-a-dink-dink · 2012-12-12 16:27:01

I'm getting an error when running powerpill. It says there's a syntax problem with my config file, but I haven't changed it at all.

error: failed to load /etc/powerpill/powerpill.json [Expecting property name enclosed in double quotes: line 44 column 3 (char 860)]
Check the file for syntax errors.%

Xyne · 2012-12-12 16:59:11

The errors in the default powerpill.json file should be fixed now.

There's also a new option: powerpill/ask
It can be used to enable/disable package selection dialogues for groups. When I have more time I may try to improve the dialogue (readline or curses support).

zoqaeski · 2012-12-13 11:36:33

Nifty. Certainly speeds up the downloads, aria's parallel downloading feature is great.

Though powerpill seems to ignore the IgnorePkg and IgnoreGroup options in /etc/pacman.conf . It goes ahead and downloads everything pacman says is out-of-date, even if I'm choosing to hold updates for some reason (such as the catalyst driver for a misbehaving graphics card).

Xyne · 2012-12-13 12:24:24

@zoqaeski
It should now respect IgnorePkg and IgnoreGroup settings. Please try the latest version.

Bonus: ignored packages are now initially deselected in the group dialogue.

dapolinario · 2012-12-14 01:55:30

Where's the powerpill 2012.12.13-1 of xyne-any repository? Has in AUR but not in your repository.

Xyne · 2012-12-15 23:49:33

@dapolinario
I uploaded it yesterday after I saw your post. I didn't have time to reply.

oboedad55 · 2012-12-16 18:41:30

@xyne: I'm using heftig's repo for Firefox Aurora but powerpill doesn't like it. How can I resolve this error?

 2012-12-16 13:35:06.843242 ERROR - CUID#10 - Download aborted. URI=http://pkgbuild.com/~heftig/repo/x86_64/heftig.db.sig
Exception: [AbstractCommand.cc:303] errorCode=3 URI=http://pkgbuild.com/~heftig/repo/x86_64/heftig.db.sig
  -> [HttpSkipResponseCommand.cc:211] errorCode=3 Resource not found

Arch Linux

#26 2012-12-07 19:05:30

Re: powerpill reborn: pacman wrapper for faster downloads

#27 2012-12-07 19:16:17

Re: powerpill reborn: pacman wrapper for faster downloads

#28 2012-12-07 19:22:20

Re: powerpill reborn: pacman wrapper for faster downloads

#29 2012-12-07 19:35:20

Re: powerpill reborn: pacman wrapper for faster downloads

#30 2012-12-07 21:23:30

Re: powerpill reborn: pacman wrapper for faster downloads

#31 2012-12-07 21:44:11

Re: powerpill reborn: pacman wrapper for faster downloads

#32 2012-12-07 22:39:30

Re: powerpill reborn: pacman wrapper for faster downloads

#33 2012-12-07 23:19:04

Re: powerpill reborn: pacman wrapper for faster downloads

#34 2012-12-09 18:25:35

Re: powerpill reborn: pacman wrapper for faster downloads

#35 2012-12-09 21:03:31

Re: powerpill reborn: pacman wrapper for faster downloads

#36 2012-12-11 20:51:50

Re: powerpill reborn: pacman wrapper for faster downloads

#37 2012-12-11 22:47:09

Re: powerpill reborn: pacman wrapper for faster downloads

#38 2012-12-11 23:09:57

Re: powerpill reborn: pacman wrapper for faster downloads

#39 2012-12-12 01:56:35

Re: powerpill reborn: pacman wrapper for faster downloads

#40 2012-12-12 03:47:15

Re: powerpill reborn: pacman wrapper for faster downloads

#41 2012-12-12 12:46:27

Re: powerpill reborn: pacman wrapper for faster downloads

#42 2012-12-12 14:37:40

Re: powerpill reborn: pacman wrapper for faster downloads

#43 2012-12-12 15:42:48

Re: powerpill reborn: pacman wrapper for faster downloads

#44 2012-12-12 16:27:01

Re: powerpill reborn: pacman wrapper for faster downloads

#45 2012-12-12 16:59:11

Re: powerpill reborn: pacman wrapper for faster downloads

#46 2012-12-13 11:36:33

Re: powerpill reborn: pacman wrapper for faster downloads

#47 2012-12-13 12:24:24

Re: powerpill reborn: pacman wrapper for faster downloads

#48 2012-12-14 01:55:30

Re: powerpill reborn: pacman wrapper for faster downloads

#49 2012-12-15 23:49:33

Re: powerpill reborn: pacman wrapper for faster downloads

#50 2012-12-16 18:41:30

Re: powerpill reborn: pacman wrapper for faster downloads

Board footer