Powerpill: an improved download accelerator for Pacman.

Square · 2008-07-30 05:37:53

I'm pretty sure it's the Windows boxes on my network reserving bandwidth that's the problem. That, or my router causing problems. I plugged directly into my modem instead of my router and came out with 1400KiB/s using powerpill.

Looks like I have to go back over and remember how to stop that again.

Last edited by Square (2008-07-30 05:39:10)

ProtoXyne · 2008-07-30 05:47:42

Square wrote:

I'm pretty sure it's the Windows boxes on my network reserving bandwidth that's the problem. That, or my router causing problems. I plugged directly into my modem instead of my router and came out with 1400KiB/s using powerpill.

Square wrote:

Looks like I have to go back over and remember how to stop that again.

Look at the bright side, you discovered that your bandwidth was throttled and you know where to look to fix it. Plus getting 1400KiB/s on a connection rated at less than half that isn't too bad either... and now we know where the mysterious 900 KiB/s came from.

Last edited by ProtoXyne (2008-07-30 07:27:04)

faelar · 2008-07-30 10:10:48

screenshot
I'll do a speedtest, but I'm on my eeepc, WPA wireless

Update :

Last edited by faelar (2008-07-30 10:17:18)

ProtoXyne · 2008-07-31 06:14:54

Nice, thanks faelar.

I should probably add some screenshots in the OP, thanks for the idea.

MetalheadGautham · 2008-08-02 01:39:12

w00t! this is just what I was looking for!

1. So let me see.... I need to install aria2c, and then copy this script to my bin folder, and then start using it right ?

2. Are there any commands for pacman which don't work if I use powerpill instead ?

3. Is it safe to use this script in combination with Reflector script with mirrorrank ?

4. do I need to use -Syy and -Syu for this too to keep stuff in sync ?

5. Are all the mirrors always synchronised properly ? I don't want to end up with parts of an older version combined with parts of a newer one.

6. Is it possible to use this same script to download from ONE mirror only, and do aggressive multi-threaded downloading, the way a standard download accelerator like firefox's DownThemAll does ?

7. I know this is offtopic, but do you know of a simple no nonsense CLI download manager, which has the features and robustness of wget and curl, but supports splitting of the file into parts by specifying something like -t 5 for 5 paralell threads for the same download ?

rooloo · 2008-08-02 03:02:18

ARIA2:   1|  OK|/var/cache/pacman/pkg/libwpd-0.8.14-1-x86_64.pkg.tar.gz
ARIA2:   2|  OK|/var/cache/pacman/pkg/openoffice-base-2.4.1-2-x86_64.pkg.tar.gz
ARIA2: Status Legend:
ARIA2:  (OK):download completed.(ERR):error occurred.(INPR):download in-progress.
==> SUCCESS: Checksum verified: libwpd-0.8.14-1-x86_64.pkg.tar.gz
==> SUCCESS: Checksum verified: openoffice-base-2.4.1-2-x86_64.pkg.tar.gz
==> 107.29 MiB downloaded and verified in 200 s.
==> Average rate: 549.31 kiB/s
==> Maximum rate: 613.26 kiB/s

I was using strictly the rit server before. I get 20 - 40 kib's more using this script. I have never got anymore then ~600 kib's ever when downloading, whether that be from one server or multiple servers.

not sure I see the benefit. but I will give it some time to see if I see a benefit. I think this would be helpful only when you need to download multiple && huge files. As I understand it this will download files concurrently, unlike pacman which is succesively? Yes/no?

I have to start the script like

./powerpill.php -S

some special reason why I need add the .php part?

ProtoXyne · 2008-08-02 12:33:09

MetalheadGautham wrote:

w00t! this is just what I was looking for!
1. So let me see.... I need to install aria2c, and then copy this script to my bin folder, and then start using it right ?
2. Are there any commands for pacman which don't work if I use powerpill instead ?
3. Is it safe to use this script in combination with Reflector script with mirrorrank ?
4. do I need to use -Syy and -Syu for this too to keep stuff in sync ?
5. Are all the mirrors always synchronised properly ? I don't want to end up with parts of an older version combined with parts of a newer one.
6. Is it possible to use this same script to download from ONE mirror only, and do aggressive multi-threaded downloading, the way a standard download accelerator like firefox's DownThemAll does ?
7. I know this is offtopic, but do you know of a simple no nonsense CLI download manager, which has the features and robustness of wget and curl, but supports splitting of the file into parts by specifying something like -t 5 for 5 paralell threads for the same download ?

1. Yes and no. You need to install aria2, but you don't need to put powerpill in your bin folder. If you put it in ~ for example, you could just run it directly with a ./powerpill command. Putting it in bin means that you can always run it by entering "powerpill" anywhere, but it's up to you.

2. I think it should be fine with any commands. Anything that doesn't involve a download just skips the powerpill part and passes the arguments to pacman. If you run into a problem, let me know.

3. It should be (that's what I do anyway). One important note though... keep your trusted mirror at the top so that pacman always gets the database from that one.

4. Yeah, because this doesn't directly do anything to your installation or databases. Powerpill just downloads the packages that pacman tells it to when you run it. You still have to run -Syy and -Syu to keep your system up to date. Doing it through powerpill will make the downloads faster though (powerpill -Syu).

5. The mirrors aren't always synced properly, but that doesn't matter as long as the mirror you normally use is at the top. That's the mirror that pacman will use for the databases and powerpill will check the downloaded packages against the database checksums, so even if a few of the mirrors in your list are out of sync, all that will happen is that their packages will be discarded and it will get them from another server that is in sync. Reflector tries to make sure that your mirrors are in sync by only taking the ones that are up-to-date.

6. Yeah, but please don't. That just puts unnecessary load on the server and that's now what this is meant for. Just use all your mirrors... you'll get better overall speed and there's no risk that you'll get mixed versions.

7. Yeah... aria2

rooloo wrote:

ARIA2:   1|  OK|/var/cache/pacman/pkg/libwpd-0.8.14-1-x86_64.pkg.tar.gz
ARIA2:   2|  OK|/var/cache/pacman/pkg/openoffice-base-2.4.1-2-x86_64.pkg.tar.gz
ARIA2: Status Legend:
ARIA2:  (OK):download completed.(ERR):error occurred.(INPR):download in-progress.
==> SUCCESS: Checksum verified: libwpd-0.8.14-1-x86_64.pkg.tar.gz
==> SUCCESS: Checksum verified: openoffice-base-2.4.1-2-x86_64.pkg.tar.gz
==> 107.29 MiB downloaded and verified in 200 s.
==> Average rate: 549.31 kiB/s
==> Maximum rate: 613.26 kiB/s
I was using strictly the rit server before. I get 20 - 40 kib's more using this script. I have never got anymore then ~600 kib's ever when downloading, whether that be from one server or multiple servers.
not sure I see the benefit. but I will give it some time to see if I see a benefit. I think this would be helpful only when you need to download multiple && huge files. As I understand it this will download files concurrently, unlike pacman which is succesively? Yes/no?
I have to start the script like
./powerpill.php -S
some special reason why I need add the .php part?

If you're only using one server, you won't see much of a benefit. The script is written to take advantage of multiple servers (also, check your bandwidth using an online speedtest to make sure that you actually can get more speed). See my reply above for more info about using more mirrors and using reflector.

You'll notice the greatest different when downloading many and/or big files, but unlike the other download accelerators for pacman, it won't slow down single small downloads. Yes, it downloads files concurrently, and downloads larger files in segments, which is why you get a better result from using multiple servers.

You didn't save the script correctly. Just change the name to "powerpill" (mv powerpill.php powerpill).

faelar · 2008-08-02 21:15:25

Another screenshot
This time it's my desktop over ethernet.
The speedtest is almost the same :

Maybe I should try to order servers in the list, but I think it performs well enought already.
Great job !

MetalheadGautham · 2008-08-03 04:09:25

K. My speed jumped from 15kbps to 50kbps. But I am on a 2mbps line.
I am supposed to be downloading usually at around 240kbps.
Just some time back, CentOS 5.2 DVD from internode.net was downloading on DownThemAll at 222kbps.

ProtoXyne · 2008-08-03 08:17:58

faelar wrote:

Another screenshot
This time it's my desktop over ethernet.
The speedtest is almost the same :
http://www.speedtest.net/result/304107461.png
Maybe I should try to order servers in the list, but I think it performs well enought already.
Great job !

From the output it looks like you're using an older version of the script. There were a few mistakes that have been corrected since then which should affect the speed (plus now it will also report the maximum speed attained).

MetalheadGautham wrote:

K. My speed jumped from 15kbps to 50kbps. But I am on a 2mbps line.
I am supposed to be downloading usually at around 240kbps.
Just some time back, CentOS 5.2 DVD from internode.net was downloading on DownThemAll at 222kbps.

First, check your bandwidth at http://www.speedtest.net to make sure that it is what you think it is, then make sure that you have at least 10 mirrors uncommented in your mirror list. Also, make sure that you're using the latest version of powerpill.

MetalheadGautham · 2008-08-03 10:42:11

ProtoXyne wrote:

MetalheadGautham wrote:
K. My speed jumped from 15kbps to 50kbps. But I am on a 2mbps line.
I am supposed to be downloading usually at around 240kbps.
Just some time back, CentOS 5.2 DVD from internode.net was downloading on DownThemAll at 222kbps.
First, check your bandwidth at http://www.speedtest.net to make sure that it is what you think it is, then make sure that you have at least 10 mirrors uncommented in your mirror list. Also, make sure that you're using the latest version of powerpill.

I was downloading CentOS moments ago at 222kbps as reported by DTA. I mean kilo BYTES per second, not kilo BITS.

I used mirrorlist generated by ./reflector --rank

ProtoXyne · 2008-08-03 10:51:46

MetalheadGautham wrote:

I was downloading CentOS moments ago at 222kbps as reported by DTA. I mean kilo BYTES per second, not kilo BITS.
I used mirrorlist generated by ./reflector --rank

What are you downloading with powerpill? Just as a test, try this:

mkdir /tmp/cache
powerpill -Sw --cachedir /tmp/cache/ base

If that doesn't approach your maximum download speed, there's something wrong with your settings.

Maybe you have an old powerpill.conf file. Regenerate it if that's the case (powerpill --gen-conf) and let it overwrite the old one.

Army · 2008-08-03 11:48:04

Don't you want to provide a package in AUR?

ProtoXyne · 2008-08-03 12:07:38

Army wrote:

Don't you want to provide a package in AUR?

I'm new to both Linux and Arch, give me some time. I'll get to learning about how to make packages soon enough.

Daenyth · 2008-08-03 13:49:01

There are prototype PKGBUILDS in /usr/share/pacman to use as a base. The wiki has many articles on how to make a package also.

rooloo · 2008-08-03 14:36:52

If you're only using one server, you won't see much of a benefit. The script is written to take advantage of multiple servers (also, check your bandwidth using an online speedtest to make sure that you actually can get more speed). See my reply above for more info about using more mirrors and using reflector.

I am using multiple mirrors, I was just saying before powerpill, I was using only one mirror and I was getting same bandwidth.

I will try with reflector, idk what that is. but I will try.

Xeno, nice to see you on arch.

One thing though, is there a way to change the default behavior of powerpill. More or less right now it ask you twice if you would like to d/l the packages.

==> The following packages were specified:
==> ghostscript-8.63-2
==> intltool-0.40.3-1
==> lighttpd-1.4.19-4
==> pm-utils-1.1.2.4-1
==> rarian-0.8.0-2
==> speex-1.2rc1-1
==> ttf-dejavu-2.26-1
==> xkeyboard-config-1.3-1
==> bzip2-1.0.5-3
==> fakeroot-1.9.5-1
==> glib2-2.16.5-1
==> man-pages-3.05-1
==> sqlite3-3.6.0-1
Would you like to download all of them? (y/n) y
==> Loading your pacman settings.
==> Checking permissions for given cache directories.
==> Found and verified: /var/cache/pacman/pkg/ghostscript-8.63-2-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/intltool-0.40.3-1-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/lighttpd-1.4.19-4-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/pm-utils-1.1.2.4-1-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/rarian-0.8.0-2-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/speex-1.2rc1-1-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/ttf-dejavu-2.26-1-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/xkeyboard-config-1.3-1-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/bzip2-1.0.5-3-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/fakeroot-1.9.5-1-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/glib2-2.16.5-1-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/man-pages-3.05-1-x86_64.pkg.tar.gz
==> Found and verified: /var/cache/pacman/pkg/sqlite3-3.6.0-1-x86_64.pkg.tar.gz

==> No files to download. Will invoke pacman and exit.

the second part would ask to download again, if I had not already had them d/l'd at the time. This to me is an annoyance. Why not set default behavior to always update every package and use a switch to change that default behavior? Does that makes sense?

Last edited by rooloo (2008-08-03 14:57:56)

ProtoXyne · 2008-08-03 15:14:55

Daenyth wrote:

There are prototype PKGBUILDS in /usr/share/pacman to use as a base. The wiki has many articles on how to make a package also.

I know. I don't think it will be difficult, but it will take a little time, so I'll get to it when I can sit down and read through it.

rooloo wrote:

I am using multiple mirrors, I was just saying before powerpill, I was using only one mirror and I was getting same bandwidth.
I will try with reflector, idk what that is. but I will try.

Powerpill will always be at least as fast as pacman for single files and will always be faster for multiple files and/or large files. Try tweaking the settings if you really don't see an improvement, but I can't imagine that you wouldn't see an improvement unless your bandwidth is already maxed out..

rooloo wrote:

I have never got anymore then ~600 kib's ever when downloading

According to your speed test, your limit is 575.75 kiB/s. If you already get that when downloading normally, nothing can make it go faster (except upgrading your connection )

rooloo wrote:

Xeno, nice to see you on arch.

Thanks

rooloo wrote:

One thing though, is there a way to change the default behavior of powerpill. More or less right now it ask you twice if you would like to d/l the packages.
the second part would ask to download again, if I had not already had them d/l'd at the time. This to me is an annoyance. Why not set default behavior to always update every package and use a switch to change that default behavior? Does that makes sense?

Yeah, it makes sense. The way powerpill works is that it calls pacman with the passed options and gets the URIs from pacman, parses them, downloads them, then calls pacman again for installation. I could skip the part that asks if you'd like to download the packages, but I think more people would prefer to see how much they're about to download before they do (and it's pacman's default behavior). The second part is pacman asking you, so I can't change that. It's a consequence of how the script works... the download isn't integrated into pacman, it runs on its own and then passes control back to pacman. Pacman doesn't know how the packages got there, so he just sees that there's nothing to download and asks if you'd like to continue with the installation.

I could add a "quiet" option so that it powerpill doesn't bother to ask, but then you don't get to check things before they're downloaded.

Last edited by ProtoXyne (2008-08-03 15:16:59)

rooloo · 2008-08-03 15:25:18

I decided to try a PKGBUILD for this powerpill. I knew it would be easy and I needed the practice. Now all it needs is a post_install script to tell everyone to uncomment atleast 'x' mirrors in pacman.d/mirrorlist

# Contributor: Your Name <youremail@domain.com>
pkgname=powerpill
pkgver=6.0
pkgrel=1
pkgdesc="Powerpill is a wrapper for pacman that utilizes aria2 to max download speeds on packages"
arch=('i686' 'x86_64')
url=""
license=('GPL')
depends=('aria2')
provides=('powerpill')
source=(http://xeno.planetnexuiz.com/xeno/$pkgname.php)
md5sums=('97738aa508b3915b5f158cef22715ab6')

build() {
  cd "$srcdir/"
  mv powerpill.php powerpill
  
  mkdir -p $startdir/pkg/usr/bin
  install -m655 powerpill $startdir/pkg/usr/bin/${pkgname}
}

# vim:set ts=2 sw=2 et:

If u don't wanna submit it to AUR, I will go ahead and do it, but I will leave that up to you. It's your package after all.

Last edited by rooloo (2008-08-03 15:31:53)

ProtoXyne · 2008-08-03 15:49:31

I do want to submit it to AUR and I will. I'll try to get to it later today if I can.

Also, the file should never be named "powerpill.php". I'm not sure how that happened.

rooloo · 2008-08-03 16:33:33

ProtoXyne wrote:

Also, the file should never be named "powerpill.php". I'm not sure how that happened.

Here's the download link: http://xeno.planetnexuiz.com/xeno/powerpill.php

you change the link and then you can remove the mv line in the PKGBUILD.

Xyne · 2008-08-04 11:45:33

I've submitted the package to AUR: http://aur.archlinux.org/packages.php?ID=18882

(changed my name too)

faelar · 2008-08-06 12:11:14

Oh ! Sorry I'm not using the latest version, but I found a bug when trying to update today :

...
==> Added to download queue: sqlite3-3.6.0-1 (567.47 kiB)
Can't use an undefined value as an ARRAY reference at /bin/powerpill line 865.

Is there a changelog somewhere ? I don't want to report already corrected problem
Two packages which were available for update were from a custom repos.

Another improvement request, when you do 'powerpill -Ss' you have twice the output for the package list, it's a bit annoying.

Now I'm going to remove the old script, I'll add my vote to AUR

Xyne · 2008-08-06 17:31:51

There was a changelog in the original post of this thread but I thought it might be a bit trivial so I removed it.

I just updated the AUR package again with a change to avoid the double invocation when doing a search (-s/--search)... missed that one somehow when I added the other catches.

I'm working on the custom repo error right now... just figured out how to fix it but I'm on my way out so I'll have to do it later.

ProtoXyne · 2008-08-06 22:10:58

Sorry for the double-post (I really wish I could just change the name on this account ).

faelar, the error with the custom repos should be fixed in the latest version.

Poisonblack · 2008-08-07 15:44:07

When I use powerpill to do full system upgrade, I get following message:

Can't use an undefined value as an ARRAY reference at ./powerpill line 865.

Powerpill works fine for all other jobs and really does speed up downloads while at it.!

Arch Linux

#26 2008-07-30 05:37:53

Re: Powerpill: an improved download accelerator for Pacman.

#27 2008-07-30 05:47:42

Re: Powerpill: an improved download accelerator for Pacman.

#28 2008-07-30 10:10:48

Re: Powerpill: an improved download accelerator for Pacman.

#29 2008-07-31 06:14:54

Re: Powerpill: an improved download accelerator for Pacman.

#30 2008-08-02 01:39:12

Re: Powerpill: an improved download accelerator for Pacman.

#31 2008-08-02 03:02:18

Re: Powerpill: an improved download accelerator for Pacman.

#32 2008-08-02 12:33:09

Re: Powerpill: an improved download accelerator for Pacman.

#33 2008-08-02 21:15:25

Re: Powerpill: an improved download accelerator for Pacman.

#34 2008-08-03 04:09:25

Re: Powerpill: an improved download accelerator for Pacman.

#35 2008-08-03 08:17:58

Re: Powerpill: an improved download accelerator for Pacman.

#36 2008-08-03 10:42:11

Re: Powerpill: an improved download accelerator for Pacman.

#37 2008-08-03 10:51:46

Re: Powerpill: an improved download accelerator for Pacman.

#38 2008-08-03 11:48:04

Re: Powerpill: an improved download accelerator for Pacman.

#39 2008-08-03 12:07:38

Re: Powerpill: an improved download accelerator for Pacman.

#40 2008-08-03 13:49:01

Re: Powerpill: an improved download accelerator for Pacman.

#41 2008-08-03 14:36:52

Re: Powerpill: an improved download accelerator for Pacman.

#42 2008-08-03 15:14:55

Re: Powerpill: an improved download accelerator for Pacman.

#43 2008-08-03 15:25:18

Re: Powerpill: an improved download accelerator for Pacman.

#44 2008-08-03 15:49:31

Re: Powerpill: an improved download accelerator for Pacman.

#45 2008-08-03 16:33:33

Re: Powerpill: an improved download accelerator for Pacman.

#46 2008-08-04 11:45:33

Re: Powerpill: an improved download accelerator for Pacman.

#47 2008-08-06 12:11:14

Re: Powerpill: an improved download accelerator for Pacman.

#48 2008-08-06 17:31:51

Re: Powerpill: an improved download accelerator for Pacman.

#49 2008-08-06 22:10:58

Re: Powerpill: an improved download accelerator for Pacman.

#50 2008-08-07 15:44:07

Re: Powerpill: an improved download accelerator for Pacman.

Board footer