powerpill reborn: pacman wrapper for faster downloads

Xyne · 2012-11-29 04:38:21

For those of you that remember, powerpill is back.

Powerpill speeds up package and database downloads in several ways:
* parallel and segmented downloads with aria2c
* support for rsync downloads from official repos
* support for pacserve to save bandwidth

It's based on the Parisync script that I have been using for a while without issue. Wrapping that up and adding option handling has probably introduced a few bugs though.

For now the aria2c and rsync output is suppressed during downloads because most people think it is too spammy. Feel free to suggest ways to deal with that for reporting progress. If you want to play around with the code, just search for "stdout=DEVNULL" and remove it from the Popen calls.

I'll likely tweak option support as the current ones were written without too much thought.

Feedback is appreciated.

Last edited by Xyne (2012-12-07 21:29:00)

ngoonee · 2012-11-29 07:11:29

Sorry for the 'ask-a-quick-question-without-testing', but does this bring back downloading multiple packages at the same time (which we lost with the last pacman update/powerpill hibernation)?

Jristz · 2012-11-29 07:18:18

Powerpill revive

Yes powerpill was the first think that I install from the AUR in my first time using Arch
I love it (Powerpill multiple download at same time)

Thanks for the package

PD: wrong md5 and sha256 here

Last edited by Jristz (2012-11-29 07:40:23)

skanky · 2012-11-29 10:15:39

Excellent, used to use this a lot. Will give the new version a try.

I think aria2 needs to added as a dependency? I didn't have it installed and it gave me a missing file error (the aria2 executable) when I tried to run it. Once installed it ran fine - though I was up to date so not much of a test.

EDIT2 - forget it, worked it out.

Last edited by skanky (2012-11-29 17:33:43)

pogeymanz · 2012-11-29 19:50:49

I was just thinking about this the other day! I'm glad it's back.

...But now I have to be that guy. Any hope of bauerbill's revival in the foreseeable future? Nothing ever came close. I've been using yaourt since bauerbill died. >:(

oboedad55 · 2012-11-29 20:54:02

@Xyne: I'm getting this error, after which things work fine;
warning: failed to query Pacserve: [Errno 111] Connection refused

Xyne · 2012-11-29 22:19:04

ngoonee wrote:

Sorry for the 'ask-a-quick-question-without-testing', but does this bring back downloading multiple packages at the same time (which we lost with the last pacman update/powerpill hibernation)?

Simple answer: of course, it wouldn't be powerpill if it didn't download packages simultaneously

Not so simple answer: it depends on your configuration. If you only use official repos and enable rsync then it will not use parallel connections to download the packages. The Rsync algorithm only transfers a single file at a time. It's still faster than the default Pacman downloader (in my experience) because it doesn't need to renegotiate a connection for each package and it can quickly check for file changes (e.g. pacman -Sy is very fast). Of course, if you don't add an rsync server to the configuration file then it will use aria2c for everything, which will download all packages in parallel (up to the limit set in the options).

Pacserve connections are also single-package, sequential downloads, but they are over the LAN so they should be blazing fast. After getting all the files that it can from Pacserve it will then use multiple connections to retrieve packages from mirrors.

@Jristz
There's a new package with correct checksums.

@skanky
Thanks, the package now depends on aria2

@oboedad55
The pacserve server is no longer enabled by default, so the error should not appear now.

@pogeymanz
I'd like to create a replacement and I have a very clear idea of how to do it. It's actually part of something even bigger that I would like to do. The problem is that I barely have time to write little things like powerpill. Bigger projects are completely out of my reach right now and will likely remain that way for the foreseeable future.

@everyone
Just a heads up: I refactored the code and added much more extensive option parsing* to make it behave nicely as a Pacman wrapper. Lots of changes = good chance of (new) bugs.

The powerpill.conf file has also been replaced with a standard json file because it provides much more configurability (e.g. passing arguments to reflector, aria2c, rsync) without forcing me to re-invent the wheel and write a complicated parser from scratch. I also like json syntax and use it internally for a lot of things on the site. If you really don't like json then sorry, but hey, it could have been worse... I could have used xml

Check the man page (man powerpill.json) for information about the available options. You can use it to set up rsync, pacserve, reflector, aria2, etc. You can also enable output from aria2 and/or rsync so you see something happening while waiting for the download to complete.

Last edited by Xyne (2012-11-29 22:21:04)

Yannick_LM · 2012-11-29 23:07:47

\o/

I thought this day will never come. It takes great courage to rewrite a piece of software in an other language, kudos !

ngoonee · 2012-11-30 00:21:59

Feature request:- xml configuration =p

oboedad55 · 2012-11-30 00:23:13

Thanks Xyne, everything works great now. Thanks for resurrecting this!

skanky · 2012-11-30 09:36:36

Excellent, again.

One minor powerpill.json man page issue:

The rsync options section says:

args The list of arguments to pass to Reflector.

Should that be "to pass to rsync"?

Nazfellun · 2012-11-30 13:13:08

Xyne wrote:

@pogeymanz
I'd like to create a replacement and I have a very clear idea of how to do it. It's actually part of something even bigger that I would like to do. The problem is that I barely have time to write little things like powerpill. Bigger projects are completely out of my reach right now and will likely remain that way for the foreseeable future.

Any chance you might elaborate on your design? Even if you don't have the time to implement it, having a decent high-level design already planned out may inspire someone else .

Xyne · 2012-11-30 18:31:48

@skanky
fixed, thanks

@Nazfellun
I may eventually publish some roadmaps I have once I accept that I will never be able to do it myself.

Jristz · 2012-11-30 19:09:27

Feature request:
A verbose way to know if my database, packages or anythink IS downloading or not
Actually you have a sinple blank space with no message

Xyne · 2012-11-30 20:59:50

Jristz wrote:

Feature request:
A verbose way to know if my database, packages or anythink IS downloading or not
Actually you have a sinple blank space with no message

powerpill.json man page wrote:

Download Progress
By default no output will be displayed during downloads while waiting for Aria2 and Rsync to complete. To enable Aria2 output, remove the --quiet option from the Aria2 arguments list. To enable output from Rsync, add e.g. --progress and/or --verbose to the Rsync arguments list.

Nazfellun · 2012-11-30 22:36:09

Humm, works awesomely on most of my boxes, but on one I am getting the following on simply trying to install a package with it:

Traceback (most recent call last):
  File "/usr/bin/powerpill", line 566, in <module>
    run_main()
  File "/usr/bin/powerpill", line 559, in run_main
    main(args)
  File "/usr/bin/powerpill", line 546, in main
    powerpill.download(pm2ml_args)
  File "/usr/bin/powerpill", line 413, in download
    conf=self.pacman_conf
TypeError: build_download_queue() got an unexpected keyword argument 'conf'

Xyne · 2012-12-01 23:52:08

@Nazfellun
Powerpill expects the latest version of pm2ml. You seem to have an older version on that box. I've added a versioned dependency to powerpill to prevent that error in the future.

dapolinario · 2012-12-02 02:22:25

@Xyne
You could add bash-completion to complete the package names.

Xyne · 2012-12-02 03:08:48

dapolinario wrote:

@Xyne
You could add bash-completion to complete the package names.

done

It doesn't support powerpill's own options becauseI really have no idea how to write a custom bash-completion file. I'll fix that when I have time. If someone wants to submit a better bash-completion file or point me to a quick and clear tutorial, please do.

Tarqi · 2012-12-02 20:04:22

Xyne wrote:

If someone wants to submit a better bash-completion file or point me to a quick and clear tutorial, please do.

Check out this article, especially Part 2. It's surprisingly quite easy, once understood.

matersci · 2012-12-05 14:34:55

Is it possible to use the first mirrors of pacman/d/mirrorlist instead of using reflector?

Xyne · 2012-12-05 20:32:58

matersci wrote:

Is it possible to use the first mirrors of pacman/d/mirrorlist instead of using reflector?

It is now if you remove the reflector arguments from /etc/powerpill.json, but why would you want to do that?

matersci · 2012-12-05 20:43:03

To use only the mirrors I have tested (speed, sync time etc).

Jristz · 2012-12-07 18:05:12

I test this

sudo powerpill -Sw kde-full $(pacman -Slq core) $(pacman -Slq multilib)

and powerpill freeze with no output about the kde-full

not sure if bug or I

Xyne · 2012-12-07 18:56:45

@jristz
What repo is kde-full in? I do not have access to it on my system.

You will not see any output during the download unless you enable aria2 and rsync output as described above. Have you done that?

@everyone
I am going to enable output by default to avoid further confusion.

Arch Linux

#1 2012-11-29 04:38:21

powerpill reborn: pacman wrapper for faster downloads

#2 2012-11-29 07:11:29

Re: powerpill reborn: pacman wrapper for faster downloads

#3 2012-11-29 07:18:18

Re: powerpill reborn: pacman wrapper for faster downloads

#4 2012-11-29 10:15:39

Re: powerpill reborn: pacman wrapper for faster downloads

#5 2012-11-29 19:50:49

Re: powerpill reborn: pacman wrapper for faster downloads

#6 2012-11-29 20:54:02

Re: powerpill reborn: pacman wrapper for faster downloads

#7 2012-11-29 22:19:04

Re: powerpill reborn: pacman wrapper for faster downloads

#8 2012-11-29 23:07:47

Re: powerpill reborn: pacman wrapper for faster downloads

#9 2012-11-30 00:21:59

Re: powerpill reborn: pacman wrapper for faster downloads

#10 2012-11-30 00:23:13

Re: powerpill reborn: pacman wrapper for faster downloads

#11 2012-11-30 09:36:36

Re: powerpill reborn: pacman wrapper for faster downloads

#12 2012-11-30 13:13:08

Re: powerpill reborn: pacman wrapper for faster downloads

#13 2012-11-30 18:31:48

Re: powerpill reborn: pacman wrapper for faster downloads

#14 2012-11-30 19:09:27

Re: powerpill reborn: pacman wrapper for faster downloads

#15 2012-11-30 20:59:50

Re: powerpill reborn: pacman wrapper for faster downloads

#16 2012-11-30 22:36:09

Re: powerpill reborn: pacman wrapper for faster downloads

#17 2012-12-01 23:52:08

Re: powerpill reborn: pacman wrapper for faster downloads

#18 2012-12-02 02:22:25

Re: powerpill reborn: pacman wrapper for faster downloads

#19 2012-12-02 03:08:48

Re: powerpill reborn: pacman wrapper for faster downloads

#20 2012-12-02 20:04:22

Re: powerpill reborn: pacman wrapper for faster downloads

#21 2012-12-05 14:34:55

Re: powerpill reborn: pacman wrapper for faster downloads

#22 2012-12-05 20:32:58

Re: powerpill reborn: pacman wrapper for faster downloads

#23 2012-12-05 20:43:03

Re: powerpill reborn: pacman wrapper for faster downloads

#24 2012-12-07 18:05:12

Re: powerpill reborn: pacman wrapper for faster downloads

#25 2012-12-07 18:56:45

Re: powerpill reborn: pacman wrapper for faster downloads

Board footer