Powerpill: an improved download accelerator for Pacman.

ProtoXyne · 2008-07-23 16:45:50

I've move the official powerpill thread to http://bbs.archlinux.org/viewtopic.php?pid=431164 to avoid having to log into this account every time I update it.

Mods, please lock this thread.

Last edited by ProtoXyne (2008-10-08 21:24:34)

Daenyth · 2008-07-23 16:48:51

You get my support for using such an awesome name.

dolby · 2008-07-23 20:23:37

Hah! I was about to say the exact same thing.

ProtoXyne · 2008-07-27 06:34:53

Has anyone tried it yet?

If anyone has any questions about any part of the script, just ask and I'll try to explain what it's doing and why. I'm guessing that most people are (rightfully) wary of running a script like this from an unknown source. If anyone has actually tried it, please post a comment (e.g. "fast as f***, didn't bork my system").

I may eventually try to make a package for it too, mostly just to learn learn how.

Last edited by ProtoXyne (2008-07-27 08:33:51)

faelar · 2008-07-27 22:51:42

Installed, I believe it should work, but there is currently no update available for my system.
Everything is well explained.
I just think that having two files, one for the conf, and another for the program should be better, especially if you wish to write a PKGBUILD.
Give you more feedback later

ProtoXyne · 2008-07-28 00:00:05

faelar wrote:

I just think that having two files, one for the conf, and another for the program should be better, especially if you wish to write a PKGBUILD.
Give you more feedback later

Thanks.

I've found the pacman.conf man page and am in the process of rewriting powerpill to parse pacman.conf for settings, paths and servers, including descending into Includes (with a check for cyclic recursion death). Moving the remaining configs out of the script would be no problem (plaintext, xml, <insert other request here>). If it ends up as a package, I agree that it would make much more sense to separate them.

And now back to dealing with the possibilities of multiple CacheDir specifications...

Daenyth · 2008-07-28 04:22:55

Generally I prefer a simple "setting=value" config file. Simpler to parse, and if you keep it that simple you can just source it as a shell script.

ProtoXyne · 2008-07-28 05:15:34

Daenyth wrote:

Generally I prefer a simple "setting=value" config file. Simpler to parse, and if you keep it that simple you can just source it as a shell script.

Hehe, saw this right after I implemented the config that way.

dr.cranium · 2008-07-28 06:12:45

Whats the difference between this and Pacget?

ProtoXyne · 2008-07-28 21:21:19

dr.cranium wrote:

Whats the difference between this and Pacget?

Two main things.

Firstly, pacget uses segmented downloading for every file. If you read through the pacget thread, you'll notice that several people have mentioned that pacget actually slows down pacman when syncing small files. Splitting files takes extra cycles and the gain from downloading small files in segments is negligible. The other aria2 implementation mentioned on the first page of the pacget thread addresses that issue by setting a size threshold for splitting files (5 MB) to avoid it. Powerpill does this as well, using a user-defined minimum size to avoid splitting small files. This way, powerpill can only ever speed up the download. *

Secondly, pacget downloads files sequentially because it's called directly by pacman with each URI. That means that when pacman needs to sync 10 files, it calls pacget and tells it to get the first file... then it waits for pacget to finish and then it calls it again to get the second file... waits again, and so on. That means that if you have lots of (small, but even large) files, it's going to take longer than it needs to. Powerpill gets a list of all the packages at once so that it can create a single metalink file and pipe that to aria2 (again, no writing to disk, no clutter like the other metalink script, which also does it sequentially for each file like pacget, with the same slowdown). Once aria2 gets the complete list, it can work on everything at once (you configure the global connections, etc). This way it can simultaneously download several smalll files. So if you have files a,b,c,d,e,f,g, etc, instead of doing a->b->c->d->etc, it does (a-j) at once. As each connection is freed, it's immediately reused to get the next file so until the download ends, it's always using the user-defined max global connections.

The combined effect of the two is that at a given point in the download, you could be downloading one large file from 5 servers and another 5 small files from 5 different servers, all simultaneously. Once one of the small files finishes off, that connection is immediately reused. Once the big file finishes, those 5 connections will immediately be used to download other files.

If you want a good example, try 'pacman -Sw base' (w so it doesn't get installed), then 'powerpill -Sw base' (make sure you remove the files from your cache both times before you download so that they all get downloaded, for a fair comparison). You should see a clear difference (unless you're on a modem, which would be maxed even by pacman on its own).

Other minor points:
You don't have to change anything. For pacget, you need to add the XferCommand line to pacman.conf and add pacget to your path. Powerpill can be run from anywhere and if you want to use vanilla pacman, you just run it directly. You can add powerpill to your path if you want, of course.

Powerpill splits the load across several servers (never a sync problem because all the files are verified according to the database checksums, and powerpill lets pacman handle database downloads itself).

Powerpill is only called once, and any system calls that it makes are optimized so there are no unnecessary calls (not noticeable on modern systems, but still, it saves cycles and is more elegant). Pacget gets run once for every package, which means it reads the mirrorlist and does all the sed and awk calls over and over again.

As the first 2 comments show, it's got a cool name.

* Before I wrote powerpill, I rewrote pacget in perl and added the threshold option to avoid splitting small files. I can post it if you want (it's a direct drop-in replacement for the pacget bash script).

Last edited by ProtoXyne (2008-07-29 04:17:56)

Square · 2008-07-28 21:36:46

I like it, but there is one suggestion I would make. Instead of having to run the script as root, perhaps you could make it call for a root password when needed? (Such as yaourt does)

Also, it probably could do with being a little more verbose. Doing your example of downloading the base it doesn't say what the whole content is before asking you if you'd like to install the whole content.

Last edited by Square (2008-07-28 21:45:27)

ProtoXyne · 2008-07-29 03:26:16

Square wrote:

I like it, but there is one suggestion I would make. Instead of having to run the script as root, perhaps you could make it call for a root password when needed? (Such as yaourt does)
Also, it probably could do with being a little more verbose. Doing your example of downloading the base it doesn't say what the whole content is before asking you if you'd like to install the whole content.

Done and done

dr.cranium · 2008-07-29 03:45:12

Whoa. Thanks for the fantastic reply. Thats very interesting. I'm usually at school(massive connection) so any way to speed up downloads is great. I used pacget for a while, then switched to just aria. But I'll give powerpill a try. Thanks a lot again for the really good explanation.

Square · 2008-07-29 03:59:56

That was fast. Keep it up!

EDIT: Found another little nuisance. When the system is up to date, and powerpill is used with the Syu command, instead of saying the local database is up to date I got

==> MESSAGE: No files to download. Invoking pacman...
PACMAN: Invoked
error: you cannot perform this operation unless you are root.

This doesn't display when powerpill is run with sudo, but instead the output it normal.
Heh.

Last edited by Square (2008-07-29 04:10:27)

ProtoXyne · 2008-07-29 04:27:09

Fixed.

I forgot about the "no files to download" check.

Btw, what about the speeeeeeeeeeed? *waits eagerly*

Last edited by ProtoXyne (2008-07-29 04:37:30)

Square · 2008-07-29 23:43:38

I haven't really noticed any speed improvement when using powerpill.
I'd like to get a huge boost, but it just isn't so.

ProtoXyne · 2008-07-30 00:37:37

Square wrote:

I haven't really noticed any speed improvement when using powerpill.
I'd like to get a huge boost, but it just isn't so.

oO

What's your bandwidth? What settings are you using and have you tried tweaking them? How many mirrors do you have in your mirror list? Maybe you've only been updating small packages. I'm sorry, but there's just no way that this wouldn't significantly boost your download speed.

I'd really like to know more about your settings and what you're actually doing with powerpill. A "I don't notice a difference" comment like that will discourage people even though I know for a fact that this works as described.

Square · 2008-07-30 02:22:22

I think what I'm seeing may just be the servers being downloading from. My bandwidth is nowhere near being maxed out when using powerpill. I suppose I will have to switch my mirrors around or something to try and get better results. Even so, I'm not seeing the max connections being met.

I've been using open-office base to test the download by the way, since it pulls in libwpd which is over 100 MiB. My download speeds have been around 900KiB/s at high rates, and the most I'm seeing with powerpill is around 236KiB/s.

Also, I should include that I'm using the default powerpill settings.

Last edited by Square (2008-07-30 02:29:32)

ProtoXyne · 2008-07-30 03:50:01

If there's a large single package, try increasing the following:
CONNECTIONS_PER_FILE = x
SERVERS = x
TOTAL_CONNECTIONS = (x or more)

and make sure you have at least x mirrors uncommented in your mirror file. I just tried powerpill -Sw openoffice-base and it doesn't drop below 1000KiB/s for me.

Square · 2008-07-30 04:07:12

I tried increasing those variables earlier with no effect. I have all available mirrors uncommented. Still the same.

ProtoXyne · 2008-07-30 04:26:52

How man connections does aria2c show while it's downloading? (CN:?) I'm honestly surprised that you're not getting a real boost. Have you tried any online speed tests to check your connection?

I'm going to change a bit of code so that it can increase the connections per file depending on the size (e.g. 5 connections for a 10 MB file, 15 for a 50MB) until it reaches the number of servers.

Square · 2008-07-30 04:34:06

It looks like aria2c is getting 4 connections.
I commented out a few servers to get some different server's being used. No luck.
As for my internet speed, here's a test with other things running in the background consuming some bandwidth.

Last edited by Square (2008-07-30 04:38:24)

ProtoXyne · 2008-07-30 05:01:31

Square wrote:

...and the most I'm seeing with powerpill is around 236KiB/s.

Looks like powerpill IS maxing your connection. You reported 236KiB/s (kibibytes), speedtest.net shows kb/s (kilobits or more likely kibibits, but bits anyway)
236 kibibytes = 1888 kibibits = 1933.312 kilobits, which is about what the speedtest shows your max rate to be.

Seems like something else is throttling your bandwidth, but powerpill is grabbing all that's available when you run it.

Last edited by ProtoXyne (2008-07-31 04:26:36)

Square · 2008-07-30 05:20:28

Yup, you're right. I'm not sure what's wrong with me today, I'm not catching all the little details.
Keep up the good work, though. And... looks like I've got another problem to sort out now.
Seems my internet connection is rated for 640 KiB/s, and that's no good if I'm only getting 236 Not sure where I remember 900 from... perhaps I just flipped my 6.

Last edited by Square (2008-07-30 05:23:37)

ProtoXyne · 2008-07-30 05:26:41

Square wrote:

Yup, you're right. I'm not sure what's wrong with me today, I'm not catching all the little details.
Keep up the good work, though. And... looks like I've got another problem to sort out now.

Thanks. I'm just really relieved because I was sitting here thinking "there's no f#$*king way it's that slow".

Good luck sorting out the bandwidth problem. Post back once you've tested an unthrottled powerpill download.

Square wrote:

Edit:
Seems my internet connection is rated for 640 KiB/s, and that's no good if I'm only getting 236 sad Not sure where I remember 900 from... perhaps I just flipped my 6.

Just a few ideas:
Are you on an encrypted wifi connection maybe? That would slow it down. Try testing your bandwidth using an ethernet connection in that case.

If you share your connection, check if anyone else is down-/uploading (including someone hacking your wifi, just in case )

Do you use proxies?

Last edited by ProtoXyne (2008-07-30 05:34:45)

Arch Linux

#1 2008-07-23 16:45:50

Powerpill: an improved download accelerator for Pacman.

#2 2008-07-23 16:48:51

Re: Powerpill: an improved download accelerator for Pacman.

#3 2008-07-23 20:23:37

Re: Powerpill: an improved download accelerator for Pacman.

#4 2008-07-27 06:34:53

Re: Powerpill: an improved download accelerator for Pacman.

#5 2008-07-27 22:51:42

Re: Powerpill: an improved download accelerator for Pacman.

#6 2008-07-28 00:00:05

Re: Powerpill: an improved download accelerator for Pacman.

#7 2008-07-28 04:22:55

Re: Powerpill: an improved download accelerator for Pacman.

#8 2008-07-28 05:15:34

Re: Powerpill: an improved download accelerator for Pacman.

#9 2008-07-28 06:12:45

Re: Powerpill: an improved download accelerator for Pacman.

#10 2008-07-28 21:21:19

Re: Powerpill: an improved download accelerator for Pacman.

#11 2008-07-28 21:36:46

Re: Powerpill: an improved download accelerator for Pacman.

#12 2008-07-29 03:26:16

Re: Powerpill: an improved download accelerator for Pacman.

#13 2008-07-29 03:45:12

Re: Powerpill: an improved download accelerator for Pacman.

#14 2008-07-29 03:59:56

Re: Powerpill: an improved download accelerator for Pacman.

#15 2008-07-29 04:27:09

Re: Powerpill: an improved download accelerator for Pacman.

#16 2008-07-29 23:43:38

Re: Powerpill: an improved download accelerator for Pacman.

#17 2008-07-30 00:37:37

Re: Powerpill: an improved download accelerator for Pacman.

#18 2008-07-30 02:22:22

Re: Powerpill: an improved download accelerator for Pacman.

#19 2008-07-30 03:50:01

Re: Powerpill: an improved download accelerator for Pacman.

#20 2008-07-30 04:07:12

Re: Powerpill: an improved download accelerator for Pacman.

#21 2008-07-30 04:26:52

Re: Powerpill: an improved download accelerator for Pacman.

#22 2008-07-30 04:34:06

Re: Powerpill: an improved download accelerator for Pacman.

#23 2008-07-30 05:01:31

Re: Powerpill: an improved download accelerator for Pacman.

#24 2008-07-30 05:20:28

Re: Powerpill: an improved download accelerator for Pacman.

#25 2008-07-30 05:26:41

Re: Powerpill: an improved download accelerator for Pacman.

Board footer