powerpill reborn: pacman wrapper for faster downloads

Gustorn · 2015-10-19 01:39:16

After the latest upgrade all powerpill functionality is seemingly gone for me. It has the exact same output as pacman, and gives no sign of trying to download things in parallel. I'm not sure what information to provide: there's no error message, nothing in the journal, etc.

EDIT: Just in case I screwed something up I reinstalled powerpill and all of its dependencies, but it didn't change anything.

Last edited by Gustorn (2015-10-19 01:45:07)

Ropid · 2015-10-19 02:16:58

Same thing happens here for me, same output as pacman while there's no error messages and nothing to see in the journal.

Gustorn · 2015-10-20 02:38:22

The problem seems to be in Powerpill.py, on line 161 (which leads to line 133). The try...except block silences an exception, where the first short argument fails to parse, because it still includes the leading "-". It can be easily verified by doing:

def expand_recognized_pacman_short_options(args):
  for a in args:
    yield RECOGNIZED_PACMAN_SHORT_OPTIONS[a]

And calling powerpill with only short options. A makeshift workaround actually fixes the problem:

def expand_recognized_pacman_short_options(args):
  for a in args:
    workaround = a.replace("-", "")
    yield RECOGNIZED_PACMAN_SHORT_OPTIONS[workaround]

This, of course, breaks the long version of the command line arguments, but confirms the problem (with this workaround powerpill works as expected, as long as only short pacman options are used).

Last edited by Gustorn (2015-10-20 02:40:01)

Xyne · 2015-10-21 23:18:48

Sorry for the letting through such a huge bug and the delay in fixing it. Please try the latest versions of powerpill and python3-xcpf and let me know if I missed anything else.

Gustorn · 2015-10-21 23:27:36

It works fine now, thanks for the fix!

Xyne · 2015-10-21 23:50:18

Thanks for confirming. I'll go sit in the corner of shame now.

*edit* Oh, hi Allan.

Last edited by Xyne (2015-10-21 23:51:18)

Ropid · 2015-11-04 17:30:14

I noticed strange behavior here. When I do "powerpill -S linux-headers", powerpill downloads two packages "linux-ck-atom-4.1.12-1-x86_64.pkg.tar.xz" and "linux-ck-atom-headers-4.1.12-1-x86_64.pkg.tar.xz", then afterwards correctly proposes to install "linux-headers" like I intended.

$ sudo powerpill -S linux-headers

[...]

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
a22b26|OK  |    83KiB/s|/var/cache/pacman/pkg/linux-ck-atom-headers-4.1.12-1-x86_64.pkg.tar.xz
ad775b|OK  |   573KiB/s|/var/cache/pacman/pkg/linux-ck-atom-4.1.12-1-x86_64.pkg.tar.xz

Status Legend:
(OK):download completed.

[...]

warning: linux-headers-4.3-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Package (1)            Old Version  New Version  Net Change

testing/linux-headers  4.3-1        4.3-1          0.00 MiB

Total Installed Size:  33.29 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n]

In "/etc/pacman.conf", I have "repo-ck" as the first repository, followed by "testing" and "core". Those two "linux-ck-atom*" packages that were downloaded are out of that repo-ck repository.

Xyne · 2015-11-09 00:25:00

What's the output of

pacman -Ss linux-headers
pacman -Si linux-headers linux-ck-atom{,-headers}

I think I know what I need to change. I'll post a follow-up when I figure it out.

Ropid · 2015-11-09 02:17:12

$ pacman -Ss linux-headers

http://pastie.org/private/dxf9xwlgjsnqodcp8leesq

$ pacman -Si linux-headers linux-ck-atom{,-headers}

http://pastie.org/private/ouguguaotbm1aqqgifinw

Before I ran those two commands, I made sure that the bug I mentioned is still there and that powerpill tries to download those two linux-ck-atom* packages I mentioned.

Another thing that might perhaps be involved is that my pacman.conf looks like this:

[repo-ck]
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch
Server = http://repo-ck.com/$arch

[testing]
Include = /etc/pacman.d/mirrorlist

[core]
#Include = /etc/pacman.d/pacserve
Include = /etc/pacman.d/mirrorlist

There's all those repeated lines in the [repo-ck] section. Those are there because that web-server regularly has issues and closes connections. With those repeated lines, pacman tries to reconnect a bunch of times when that happens instead of quitting with an error. EDIT: I just tried removing all those lines and powerpill still downloads those weird packages so I guess it's not important.

Last edited by Ropid (2015-11-09 02:21:06)

Xyne · 2015-11-09 04:00:54

Thanks for posting the requested information. I have updated pm2ml and python3-xcpf to implement the expected behavior.

The repetition of the server lines should have no effect on powerpill. The repeated URLs should appear in the internal metalink but after that it's up to aria2c to determine how to handle transient errors. You can test it with

pm2ml linux-ck-atom > metalink
aria2c -M metalink

Ropid · 2015-11-09 06:52:26

"powerpill -S linux-headers" now works like expected after updating those two packages.

Spyhawk · 2015-11-09 15:50:01

This is a long shot, but would it be possible to implement support for the undocumented "--ask bitmask" option of pacman? User case can be found in this post. To be honest, this is quite a specific user case for one specific script, and I'm not sure if implementing it is worth it - your call.

Xyne · 2015-11-12 00:00:19

@Spyhawk

Done

I'm tempted to dig through the source code to see what it does but that will have to wait until another day.

Spyhawk · 2015-11-12 09:29:48

Xyne> Thank you! You can find some pointers on that parameter here.

Xyne · 2015-11-14 02:28:22

Thanks!

txtsd · 2015-11-15 15:51:20

@Xyne
I'm having a problem with yaourt using pacman instead of powerpill to download packages even when its PACMAN variable is set to powerpill. I thought it was a yaourt problem so I created an issue on yaourt's github, but apparently it's a powerpill issue. Could you take a look please?
https://github.com/archlinuxfr/yaourt/i … -156814915

Xyne · 2015-11-21 07:04:31

Please update powerpill, pm2ml and python3-xcpf and let me know if it works as expected.

I'm almost surprised that this hasn't come up in the last 3 years. The previous incarnation of Powerpill handled repo prefixes. I'm embarrassed that I completely forgot about them.

3 years...
over 7 on the forum...
f#$@^(& time, where does it go?

txtsd · 2015-11-21 14:22:35

Xyne wrote:

Please update powerpill, pm2ml and python3-xcpf and let me know if it works as expected.
I'm almost surprised that this hasn't come up in the last 3 years. The previous incarnation of Powerpill handled repo prefixes. I'm embarrassed that I completely forgot about them.
3 years...
over 7 on the forum...
f#$@^(& time, where does it go?

Just programmer things ( ͡° ͜ʖ ͡°)

It works now! Thanks!

txtsd · 2015-11-23 05:28:49

@Xyne

Ok so installing and searching works, but updating doesn't.

EDIT:

┌[txtsd@dungeon-of-data]─[~]                                                                              [15-11-25 16:33:24]
└─▶ powerpill -Syu                                                                                                            
XcpfError: failed to create database lock file (/var/lib/pacman/db.lck) [[Errno 13] Permission denied: '/var/lib/pacman/db.lck'] [[Errno 13] Permission denied: '/var/lib/pacman/db.lck']

Last edited by txtsd (2015-11-25 11:03:54)

txtsd · 2015-11-25 11:04:24

@Xyne
I'm an idiot. I edited the last post with powerpill's output.

txtsd · 2015-11-30 07:31:24

Powerpill sometimes fails because of invalid signatures from unofficial repos.
But running pacman doesn't throw the same error, and powerpill works normally after that.

┌[txtsd@dungeon-of-data]─[~/Downloads]                                                                    [15-11-30 11:22:08]
└─▶ powerpill -Syu                                                                                                                 

11/30 12:52:47 [NOTICE] GID#fc59aa5210b04390 - Download has already completed: /var/lib/pacman/sync/infinality-bundle-multilib.db

11/30 12:52:47 [NOTICE] Download complete: /var/lib/pacman/sync/infinality-bundle-multilib.db

11/30 12:52:47 [NOTICE] GID#e97a4fd2426a96f1 - Download has already completed: /var/lib/pacman/sync/multilib.db

11/30 12:52:47 [NOTICE] Download complete: /var/lib/pacman/sync/multilib.db

11/30 12:52:47 [NOTICE] GID#cc4deb7a5cafc388 - Download has already completed: /var/lib/pacman/sync/infinality-bundle.db

11/30 12:52:47 [NOTICE] Download complete: /var/lib/pacman/sync/infinality-bundle.db

11/30 12:52:47 [NOTICE] GID#b6f6aa17959d23ce - Download has already completed: /var/lib/pacman/sync/extra.db

11/30 12:52:47 [NOTICE] Download complete: /var/lib/pacman/sync/extra.db

11/30 12:52:48 [NOTICE] GID#df041ce55c66b2bc - Download has already completed: /var/lib/pacman/sync/repo-ck.db

11/30 12:52:48 [NOTICE] Download complete: /var/lib/pacman/sync/repo-ck.db
[DL:0B][#d18f36 0B/0B][#04b40a 0B/0B]                                                                                        
11/30 12:52:48 [NOTICE] GID#d18f368c6005e2d2 - Download has already completed: /var/lib/pacman/sync/community.db

11/30 12:52:48 [NOTICE] Download complete: /var/lib/pacman/sync/community.db

11/30 12:52:48 [NOTICE] GID#04b40a5a637c71e3 - Download has already completed: /var/lib/pacman/sync/core.db

11/30 12:52:48 [NOTICE] Download complete: /var/lib/pacman/sync/core.db

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
fc59aa|OK  |       0B/s|/var/lib/pacman/sync/infinality-bundle-multilib.db
e97a4f|OK  |       0B/s|/var/lib/pacman/sync/multilib.db
cc4deb|OK  |       0B/s|/var/lib/pacman/sync/infinality-bundle.db
b6f6aa|OK  |       0B/s|/var/lib/pacman/sync/extra.db
df041c|OK  |       0B/s|/var/lib/pacman/sync/repo-ck.db
d18f36|OK  |       0B/s|/var/lib/pacman/sync/community.db
04b40a|OK  |       0B/s|/var/lib/pacman/sync/core.db

Status Legend:
(OK):download completed.

11/30 12:53:02 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/freetype2-infinality-ultimate-2.6.2-3-x86_64.pkg.tar.xz

11/30 12:53:02 [NOTICE] Download complete: /var/cache/pacman/pkg/freetype2-infinality-ultimate-2.6.2-3-x86_64.pkg.tar.xz

11/30 12:53:03 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/gimp-2.8.16-1-x86_64.pkg.tar.xz

11/30 12:53:03 [NOTICE] Download complete: /var/cache/pacman/pkg/gimp-2.8.16-1-x86_64.pkg.tar.xz

11/30 12:53:03 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/lib32-freetype2-infinality-ultimate-2.6.2-2-x86_64.pkg.tar.xz

11/30 12:53:03 [NOTICE] Download complete: /var/cache/pacman/pkg/lib32-freetype2-infinality-ultimate-2.6.2-2-x86_64.pkg.tar.xz

11/30 12:53:03 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/mono-4.2.1.102-1-x86_64.pkg.tar.xz

11/30 12:53:03 [NOTICE] Download complete: /var/cache/pacman/pkg/mono-4.2.1.102-1-x86_64.pkg.tar.xz

11/30 12:53:03 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/openal-1.17.0-2-x86_64.pkg.tar.xz

11/30 12:53:03 [NOTICE] Download complete: /var/cache/pacman/pkg/openal-1.17.0-2-x86_64.pkg.tar.xz

11/30 12:53:03 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/python2-urwid-1.3.1-2-x86_64.pkg.tar.xz

11/30 12:53:03 [NOTICE] Download complete: /var/cache/pacman/pkg/python2-urwid-1.3.1-2-x86_64.pkg.tar.xz
[DL:0B][#75f183 36MiB/36MiB(100%)][#f641c1 125KiB/125KiB(100%)] [Checksum:#75f183 11MiB/36MiB(32%)](+1)                      
11/30 12:53:04 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/wine-1.8rc2-1-x86_64.pkg.tar.xz

11/30 12:53:04 [NOTICE] Download complete: /var/cache/pacman/pkg/wine-1.8rc2-1-x86_64.pkg.tar.xz

11/30 12:53:04 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/winetricks-20151116-1-any.pkg.tar.xz

11/30 12:53:04 [NOTICE] Download complete: /var/cache/pacman/pkg/winetricks-20151116-1-any.pkg.tar.xz

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
2a26c5|OK  |       0B/s|/var/cache/pacman/pkg/freetype2-infinality-ultimate-2.6.2-3-x86_64.pkg.tar.xz
abd88d|OK  |       0B/s|/var/cache/pacman/pkg/gimp-2.8.16-1-x86_64.pkg.tar.xz
8cc136|OK  |       0B/s|/var/cache/pacman/pkg/lib32-freetype2-infinality-ultimate-2.6.2-2-x86_64.pkg.tar.xz
516147|OK  |       0B/s|/var/cache/pacman/pkg/mono-4.2.1.102-1-x86_64.pkg.tar.xz
55b7af|OK  |       0B/s|/var/cache/pacman/pkg/openal-1.17.0-2-x86_64.pkg.tar.xz
602506|OK  |       0B/s|/var/cache/pacman/pkg/python2-urwid-1.3.1-2-x86_64.pkg.tar.xz
75f183|OK  |       0B/s|/var/cache/pacman/pkg/wine-1.8rc2-1-x86_64.pkg.tar.xz
f641c1|OK  |       0B/s|/var/cache/pacman/pkg/winetricks-20151116-1-any.pkg.tar.xz

Status Legend:
(OK):download completed.
error: infinality-bundle: signature from "bohoomil (dev key) <bohoomil@zoho.com>" is invalid
error: infinality-bundle-multilib: signature from "bohoomil (dev key) <bohoomil@zoho.com>" is invalid
error: database 'infinality-bundle' is not valid (invalid or corrupted database (PGP signature))
error: database 'infinality-bundle-multilib' is not valid (invalid or corrupted database (PGP signature))

┌[txtsd@dungeon-of-data]─[~/Downloads] [1]                                                                [15-11-30 12:17:10]
└─▶ pacman -Syyu                                                                                                              
[sudo] password for txtsd: 
error: infinality-bundle: signature from "bohoomil (dev key) <bohoomil@zoho.com>" is invalid
error: infinality-bundle-multilib: signature from "bohoomil (dev key) <bohoomil@zoho.com>" is invalid
:: Synchronizing package databases...
 core                                              122.1 KiB  88.3K/s 00:01 [###########################################] 100%
 extra                                            1807.0 KiB  83.7K/s 00:22 [###########################################] 100%
 community                                           3.1 MiB   118K/s 00:27 [###########################################] 100%
 multilib                                          144.1 KiB   182K/s 00:01 [###########################################] 100%
 infinality-bundle                                   9.2 KiB  8.98M/s 00:00 [###########################################] 100%
 infinality-bundle.sig                             287.0   B  0.00B/s 00:00 [###########################################] 100%
 infinality-bundle-multilib                          2.6 KiB  0.00B/s 00:00 [###########################################] 100%
 infinality-bundle-multilib.sig                    287.0   B  0.00B/s 00:00 [###########################################] 100%
 repo-ck                                            50.9 KiB   139K/s 00:00 [###########################################] 100%
:: Starting full system upgrade...
warning: bdf-unifont: local (8.0.01-1) is newer than extra (7.0.06-2)
warning: libtorrent-rasterbar: local (1:1.0.7-1) is newer than extra (1:1.0.6-2)
warning: miredo: local (1.3.0-1) is newer than community (1.2.6-3)
resolving dependencies...
looking for conflicting packages...

Package (8)                                                     Old Version  New Version  Net Change

infinality-bundle/freetype2-infinality-ultimate                 2.6.2-1      2.6.2-3        0.00 MiB
extra/gimp                                                      2.8.14-4     2.8.16-1       1.47 MiB
infinality-bundle-multilib/lib32-freetype2-infinality-ultimate  2.6.2-1      2.6.2-2        0.00 MiB
extra/mono                                                      4.0.4.1-1    4.2.1.102-1   12.54 MiB
extra/openal                                                    1.17.0-1     1.17.0-2       0.00 MiB
extra/python2-urwid                                             1.3.0-3      1.3.1-2       -0.02 MiB
multilib/wine                                                   1.8rc1-1     1.8rc2-1      -0.12 MiB
community/winetricks                                            20150826-1   20151116-1     0.01 MiB

Total Installed Size:  537.11 MiB
Net Upgrade Size:       13.87 MiB

:: Proceed with installation? [Y/n]

Xyne · 2015-11-30 15:50:27

That's just a consequence of how the downloads are handled. Just add the following to each official repo section:

SigLevel = PackageRequired

You could probably just dump that directly in the mirrorlist given that it's included in each section.

txtsd · 2015-11-30 16:09:58

Xyne wrote:

That's just a consequence of how the downloads are handled. Just add the following to each official repo section:
SigLevel = PackageRequired
You could probably just dump that directly in the mirrorlist given that it's included in each section.

I have that on all my repos.

Xyne · 2015-11-30 16:17:47

Sorry, I read through your posts too quickly and missed the "unofficial" keyword. Maybe those unofficial repos are on servers that fail to correctly report the last modification time of the remote file and thus powerpill (via aria2c) fails to detect the update. Does "pacman -Syu" (without the additional "-y") work?

If yes, then try playing around with the aria2 options in the powerpill configuration file.

If no, then just use "powerpill -Syyu". It should work.

txtsd · 2015-11-30 16:23:36

Xyne wrote:

Sorry, I read through your posts too quickly and missed the "unofficial" keyword. Maybe those unofficial repos are on servers that fail to correctly report the last modification time of the remote file and thus powerpill (via aria2c) fails to detect the update. Does "pacman -Syu" (without the additional "-y") work?
If yes, then try playing around with the aria2 options in the powerpill configuration file.
If no, then just use "powerpill -Syyu". It should work.

I put the extra y in on instinct. I'll try it without it the next time this happens.

Arch Linux

#176 2015-10-19 01:39:16

Re: powerpill reborn: pacman wrapper for faster downloads

#177 2015-10-19 02:16:58

Re: powerpill reborn: pacman wrapper for faster downloads

#178 2015-10-20 02:38:22

Re: powerpill reborn: pacman wrapper for faster downloads

#179 2015-10-21 23:18:48

Re: powerpill reborn: pacman wrapper for faster downloads

#180 2015-10-21 23:27:36

Re: powerpill reborn: pacman wrapper for faster downloads

#181 2015-10-21 23:50:18

Re: powerpill reborn: pacman wrapper for faster downloads

#182 2015-11-04 17:30:14

Re: powerpill reborn: pacman wrapper for faster downloads

#183 2015-11-09 00:25:00

Re: powerpill reborn: pacman wrapper for faster downloads

#184 2015-11-09 02:17:12

Re: powerpill reborn: pacman wrapper for faster downloads

#185 2015-11-09 04:00:54

Re: powerpill reborn: pacman wrapper for faster downloads

#186 2015-11-09 06:52:26

Re: powerpill reborn: pacman wrapper for faster downloads

#187 2015-11-09 15:50:01

Re: powerpill reborn: pacman wrapper for faster downloads

#188 2015-11-12 00:00:19

Re: powerpill reborn: pacman wrapper for faster downloads

#189 2015-11-12 09:29:48

Re: powerpill reborn: pacman wrapper for faster downloads

#190 2015-11-14 02:28:22

Re: powerpill reborn: pacman wrapper for faster downloads

#191 2015-11-15 15:51:20

Re: powerpill reborn: pacman wrapper for faster downloads

#192 2015-11-21 07:04:31

Re: powerpill reborn: pacman wrapper for faster downloads

#193 2015-11-21 14:22:35

Re: powerpill reborn: pacman wrapper for faster downloads

#194 2015-11-23 05:28:49

Re: powerpill reborn: pacman wrapper for faster downloads

#195 2015-11-25 11:04:24

Re: powerpill reborn: pacman wrapper for faster downloads

#196 2015-11-30 07:31:24

Re: powerpill reborn: pacman wrapper for faster downloads

#197 2015-11-30 15:50:27

Re: powerpill reborn: pacman wrapper for faster downloads

#198 2015-11-30 16:09:58

Re: powerpill reborn: pacman wrapper for faster downloads

#199 2015-11-30 16:17:47

Re: powerpill reborn: pacman wrapper for faster downloads

#200 2015-11-30 16:23:36

Re: powerpill reborn: pacman wrapper for faster downloads

Board footer