You are not logged in.

#1 2012-11-29 04:38:21

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,724
Website

powerpill reborn: pacman wrapper for faster downloads

For those of you that remember, powerpill is back. smile

Powerpill speeds up package and database downloads in several ways:
* parallel and segmented downloads with aria2c
* support for rsync downloads from official repos
* support for pacserve to save bandwidth


It's based on the Parisync script that I have been using for a while without issue. Wrapping that up and adding option handling has probably introduced a few bugs though.

For now the aria2c and rsync output is suppressed during downloads because most people think it is too spammy. Feel free to suggest ways to deal with that for reporting progress. If you want to play around with the code, just search for "stdout=DEVNULL" and remove it from the Popen calls.

I'll likely tweak option support as the current ones were written without too much thought.

Feedback is appreciated.

Last edited by Xyne (2012-12-07 21:29:00)

Offline

#2 2012-11-29 07:11:29

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 6,854

Re: powerpill reborn: pacman wrapper for faster downloads

Sorry for the 'ask-a-quick-question-without-testing', but does this bring back downloading multiple packages at the same time (which we lost with the last pacman update/powerpill hibernation)?


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#3 2012-11-29 07:18:18

Jristz
Member
From: America/Santiago
Registered: 2011-06-11
Posts: 924

Re: powerpill reborn: pacman wrapper for faster downloads

Powerpill revive

Yes powerpill was the first think that I install from the AUR in my first time using Arch
I love it (Powerpill multiple download at same time)

Thanks for the package

PD: wrong md5 and sha256 here

Last edited by Jristz (2012-11-29 07:40:23)


Well, I suppose that this is somekind of signature, no?

Offline

#4 2012-11-29 10:15:39

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,839

Re: powerpill reborn: pacman wrapper for faster downloads

Excellent, used to use this a lot. Will give the new version a try.

I think aria2 needs to added as a dependency? I didn't have it installed and it gave me a missing file error (the aria2 executable) when I tried to run it. Once installed it ran fine - though I was up to date so not much of a test.


EDIT2 - forget it, worked it out.

Last edited by skanky (2012-11-29 17:33:43)


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

#5 2012-11-29 19:50:49

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: powerpill reborn: pacman wrapper for faster downloads

I was just thinking about this the other day! I'm glad it's back.

...But now I have to be that guy. Any hope of bauerbill's revival in the foreseeable future? Nothing ever came close. I've been using yaourt since bauerbill died. >:(

Offline

#6 2012-11-29 20:54:02

oboedad55
Member
From: Baku
Registered: 2011-04-05
Posts: 389

Re: powerpill reborn: pacman wrapper for faster downloads

@Xyne: I'm getting this error, after which things work fine;
warning: failed to query Pacserve: [Errno 111] Connection refused


Registered Linux user #436067

Offline

#7 2012-11-29 22:19:04

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,724
Website

Re: powerpill reborn: pacman wrapper for faster downloads

ngoonee wrote:

Sorry for the 'ask-a-quick-question-without-testing', but does this bring back downloading multiple packages at the same time (which we lost with the last pacman update/powerpill hibernation)?

Simple answer: of course, it wouldn't be powerpill if it didn't download packages simultaneously


Not so simple answer: it depends on your configuration. If you only use official repos and enable rsync then it will not use parallel connections to download the packages. The Rsync algorithm only transfers a single file at a time. It's still faster than the default Pacman downloader (in my experience) because it doesn't need to renegotiate a connection for each package and it can quickly check for file changes (e.g. pacman -Sy is very fast). Of course, if you don't add an rsync server to the configuration file then it will use aria2c for everything, which will download all packages in parallel (up to the limit set in the options).

Pacserve connections are also single-package, sequential downloads, but they are over the LAN so they should be blazing fast. After getting all the files that it can from Pacserve it will then use multiple connections to retrieve packages from mirrors.



@Jristz
There's a new package with correct checksums.


@skanky
Thanks, the package now depends on aria2


@oboedad55
The pacserve server is no longer enabled by default, so the error should not appear now.


@pogeymanz
I'd like to create a replacement and I have a very clear idea of how to do it. It's actually part of something even bigger that I would like to do. The problem is that I barely have time to write little things like powerpill. Bigger projects are completely out of my reach right now and will likely remain that way for the foreseeable future. hmm


@everyone
Just a heads up: I refactored the code and added much more extensive option parsing*  to make it behave nicely as a Pacman wrapper. Lots of changes = good chance of (new) bugs.

The powerpill.conf file has also been replaced with a standard json file because it provides much more configurability (e.g. passing arguments to reflector, aria2c, rsync) without forcing me to re-invent the wheel and write a complicated parser from scratch. I also like json syntax and use it internally for a lot of things on the site. If you really don't like json then sorry, but hey, it could have been worse... I could have used xml tongue

Check the man page (man powerpill.json) for information about the available options. You can use it to set up rsync, pacserve, reflector, aria2, etc. You can also enable output from aria2 and/or rsync so you see something happening while waiting for the download to complete.

Last edited by Xyne (2012-11-29 22:21:04)

Offline

#8 2012-11-29 23:07:47

Yannick_LM
Member
Registered: 2008-12-22
Posts: 142

Re: powerpill reborn: pacman wrapper for faster downloads

\o/

I thought this day will never come. It takes great courage to rewrite a piece of software in an other language, kudos !

Offline

#9 2012-11-30 00:21:59

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 6,854

Re: powerpill reborn: pacman wrapper for faster downloads

Feature request:- xml configuration =p


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#10 2012-11-30 00:23:13

oboedad55
Member
From: Baku
Registered: 2011-04-05
Posts: 389

Re: powerpill reborn: pacman wrapper for faster downloads

Thanks Xyne, everything works great now. Thanks for resurrecting this!


Registered Linux user #436067

Offline

#11 2012-11-30 09:36:36

skanky
Member
From: WAIS
Registered: 2009-10-23
Posts: 1,839

Re: powerpill reborn: pacman wrapper for faster downloads

Excellent, again.

One minor powerpill.json man page issue:

The rsync options section says:

args   The list of arguments to pass to Reflector.

Should that be "to pass to rsync"?


"...one cannot be angry when one looks at a penguin."  - John Ruskin
"Life in general is a bit shit, and so too is the internet. And that's all there is." - scepticisle

Offline

#12 2012-11-30 13:13:08

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: powerpill reborn: pacman wrapper for faster downloads

Xyne wrote:

@pogeymanz
I'd like to create a replacement and I have a very clear idea of how to do it. It's actually part of something even bigger that I would like to do. The problem is that I barely have time to write little things like powerpill. Bigger projects are completely out of my reach right now and will likely remain that way for the foreseeable future. hmm

Any chance you might elaborate on your design? Even if you don't have the time to implement it, having a decent high-level design already planned out may inspire someone else smile.

Offline

#13 2012-11-30 18:31:48

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,724
Website

Re: powerpill reborn: pacman wrapper for faster downloads

@skanky
fixed, thanks

@Nazfellun
I may eventually publish some roadmaps I have once I accept that I will never be able to do it myself.

Offline

#14 2012-11-30 19:09:27

Jristz
Member
From: America/Santiago
Registered: 2011-06-11
Posts: 924

Re: powerpill reborn: pacman wrapper for faster downloads

Feature request:
A verbose way to know if my database, packages or anythink IS downloading or not
Actually you have a sinple blank space with no message


Well, I suppose that this is somekind of signature, no?

Offline

#15 2012-11-30 20:59:50

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,724
Website

Re: powerpill reborn: pacman wrapper for faster downloads

Jristz wrote:

Feature request:
A verbose way to know if my database, packages or anythink IS downloading or not
Actually you have a sinple blank space with no message

powerpill.json man page wrote:

Download Progress

By default no output will be displayed during downloads while waiting for Aria2 and Rsync to complete. To enable Aria2 output, remove the --quiet option from the Aria2 arguments list. To enable output from Rsync, add e.g. --progress and/or --verbose to the Rsync arguments list.

Offline

#16 2012-11-30 22:36:09

Nazfellun
Member
Registered: 2012-10-22
Posts: 19

Re: powerpill reborn: pacman wrapper for faster downloads

Humm, works awesomely on most of my boxes, but on one I am getting the following on simply trying to install a package with it:

Traceback (most recent call last):
  File "/usr/bin/powerpill", line 566, in <module>
    run_main()
  File "/usr/bin/powerpill", line 559, in run_main
    main(args)
  File "/usr/bin/powerpill", line 546, in main
    powerpill.download(pm2ml_args)
  File "/usr/bin/powerpill", line 413, in download
    conf=self.pacman_conf
TypeError: build_download_queue() got an unexpected keyword argument 'conf'

Offline

#17 2012-12-01 23:52:08

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,724
Website

Re: powerpill reborn: pacman wrapper for faster downloads

@Nazfellun
Powerpill expects the latest version of pm2ml. You seem to have an older version on that box. I've added a versioned dependency to powerpill to prevent that error in the future.

Offline

#18 2012-12-02 02:22:25

dapolinario
Member
From: Brasil
Registered: 2010-03-18
Posts: 48

Re: powerpill reborn: pacman wrapper for faster downloads

@Xyne
You could add bash-completion to complete the package names.


Arch Linux: The power in your hands!

Offline

#19 2012-12-02 03:08:48

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,724
Website

Re: powerpill reborn: pacman wrapper for faster downloads

dapolinario wrote:

@Xyne
You could add bash-completion to complete the package names.

done

It doesn't support powerpill's own options becauseI really have no idea how to write a custom bash-completion file. I'll fix that when I have time. If someone wants to submit a better bash-completion file or point me to a quick and clear tutorial, please do.

Offline

#20 2012-12-02 20:04:22

Tarqi
Member
From: Ixtlan
Registered: 2012-11-27
Posts: 106
Website

Re: powerpill reborn: pacman wrapper for faster downloads

Xyne wrote:

If someone wants to submit a better bash-completion file or point me to a quick and clear tutorial, please do.

Check out this article, especially Part 2. It's surprisingly quite easy, once understood.


Knowing others is wisdom, knowing yourself is enlightenment. ~Lao Tse

Offline

#21 2012-12-05 14:34:55

matersci
Member
From: Hellas
Registered: 2006-01-19
Posts: 159

Re: powerpill reborn: pacman wrapper for faster downloads

Is it possible to use the first mirrors of pacman/d/mirrorlist instead of using reflector?

Offline

#22 2012-12-05 20:32:58

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,724
Website

Re: powerpill reborn: pacman wrapper for faster downloads

matersci wrote:

Is it possible to use the first mirrors of pacman/d/mirrorlist instead of using reflector?

It is now if you remove the reflector arguments from /etc/powerpill.json, but why would you want to do that?

Offline

#23 2012-12-05 20:43:03

matersci
Member
From: Hellas
Registered: 2006-01-19
Posts: 159

Re: powerpill reborn: pacman wrapper for faster downloads

To use only the mirrors I have tested (speed, sync time etc).

Offline

#24 2012-12-07 18:05:12

Jristz
Member
From: America/Santiago
Registered: 2011-06-11
Posts: 924

Re: powerpill reborn: pacman wrapper for faster downloads

I test this

sudo powerpill -Sw kde-full $(pacman -Slq core) $(pacman -Slq multilib)

and powerpill freeze with no output about the kde-full

not sure if bug or I


Well, I suppose that this is somekind of signature, no?

Offline

#25 2012-12-07 18:56:45

Xyne
Moderator/TU
Registered: 2008-08-03
Posts: 5,724
Website

Re: powerpill reborn: pacman wrapper for faster downloads

@jristz
What repo is kde-full in? I do not have access to it on my system.

You will not see any output during the download unless you enable aria2 and rsync output as described above. Have you done that?


@everyone
I am going to enable output by default to avoid further confusion.

Offline

Board footer

Powered by FluxBB