Powerpill - A wrapper and download accelerator for Pacman

Sharpeee · 2008-10-20 02:43:04

Okay, nice!

I'm completely new to Archlinux and am really impressed by the speed, and this little program certainly helps on that impression!

Thanks for an awesome program!

Xyne · 2008-10-20 02:46:35

I'm glad it's appreciated.

Sharpeee · 2008-10-22 16:04:11

I have a suggestion to an possible improvement.

The files get downloaded in a totally random order, and when you download a lot of files this is sometimes not the best solution I think.
All the way through the list the download is very fast as the connection max always is maxed out, but often when you, or at least me, get to the bottom of the list, the connections starts to drop because there is not many files left. Then if the remaining files is small, below the min size to split the files, only one connection is given per file. If this connection happens to be slow, you have to wait a relative long time for the last couple of files.

If instead the list would be sorted by size, so the largest files comes last, you can, if the largest files is larger than min size to split, keep the number of connections up longer and therefore keep the speed going longer.

I do not know if it is possible to implement this feature, but it would be nice if you could

Regards,
Carsten

Xyne · 2008-10-22 17:49:17

That's a good idea but don't know if it's possible either. I just took a look though the metalink specification and found nothing that would enable a preferential download order. I also created a metalink to test if the order of <file> tags is respected by default but it isn't (it appears to be random).

If you notice that a download of a file slows to a crawl, you can always stop and restart the download. For single files, the mirror used is random and it often happens that some mirrors just have very low bandwidth available ("rankmirrors" or "reflector" with the "--rank" option might be able to help). There's also an option in the powerpill configuration file to specify a minimum download speed.

I agree though that grabbing all of the small files first would probably speed it up even more.

Sharpeee · 2008-10-23 14:11:13

Would you be able to put all the packages in an array and then sort the array by package size?

Xyne · 2008-10-23 17:27:11

Sharpeee wrote:

Would you be able to put all the packages in an array and then sort the array by package size?

Yeah, that's not a problem. The version bump to 11.x was due to a restructuring of the way powerpill stores pkg data internally. It now uses a dictionary object that keeps track of sizes, urls, etc (I had used an array and separate hashes before), so that's easy to sort. I tested the download order of files specified in a metalink because I would have arranged them by increasing size.*

*Actually, it would probably be better to arrange the small files (ones which aren't split) in descending order, followed by the large files, also in descending order. That way if any of the small files are left as a trailing connection, it will be one of the smallest ones and take less time. The same should apply to the segments of the larger files.

fflarex · 2008-10-23 20:29:07

Regarding the --get-metalink option, is the metalink generated by aria2c, or did you program that part of powerpill yourself? I only ask because aria2c's option names are sometimes ambiguous, and several times now I've discovered a new feature I never thought it had just because the option was named something weird (like seeding a previously downloaded torrent).

Great program, by the way. When I first tested it out I could not believe how fast it downloaded all of kde and the texlive packages (which is quite a lot of data).

Xyne · 2008-10-23 21:03:15

fflarex wrote:

Regarding the --get-metalink option, is the metalink generated by aria2c, or did you program that part of powerpill yourself? I only ask because aria2c's option names are sometimes ambiguous, and several times now I've discovered a new feature I never thought it had just because the option was named something weird (like seeding a previously downloaded torrent).

The metalink is generated by powerpill and follows the official documentation afaik (that's what I used as reference anyway).

fflarex wrote:

Great program, by the way. When I first tested it out I could not believe how fast it downloaded all of kde and the texlive packages (which is quite a lot of data).

Last edited by Xyne (2008-10-23 21:03:57)

MetalheadGautham · 2008-10-26 02:51:26

Hey there, I would like to file a few bug reports and feature requests for powerpill.

1. sudo rm /var/lib/pacman/db.lck

I have to do this manually whenever I kill pacman forcefully. Can you try and make this doable from within powerpill ?

2. sudo powerpill -Sw

When the "w" tag is used with "S", powerpill calls pacman a second time as usual after download is finished. This can be avoided, right ?

3. alternate mirrorlist

Is it possible to have an alternate mirrorlist for powerpill to use, which I can continue updating with reflector ? And can reflector code be intergrated into powerpill ? This can keep the default mirrorlist, which contains ALL the available mirrors in regional order untouched.

Xyne · 2008-10-26 12:51:55

MetalheadGautham wrote:

1. sudo rm /var/lib/pacman/db.lck
I have to do this manually whenever I kill pacman forcefully. Can you try and make this doable from within powerpill ?

I could, but I won't. Powerpill never touches the database. It only looks at it to get some information and leaves updating it to pacman. The lock file is there to prevent corruption of the database and afaik the file is only left behind if you kill pacman during a database sync operation. In order to remove it, powerpill would need to check for other instances of pacman running, remove the lock file and rerun pacman to sync the database. I don't want to try to cover all possible scenarios that would leave a lock file behind and I just don't think powerpill has any business doing anything that could possibly help database corruption.

If you really want it though, I could write a script that removes the lockfile if there are no other instances of pacman running, reruns pacman to sync/fix the database, then invokes powerpill. Really though, I'd be more concerned about why I have to kill pacman often enough during a database sync for it to be an issue.

MetalheadGautham wrote:

2. sudo powerpill -Sw
When the "w" tag is used with "S", powerpill calls pacman a second time as usual after download is finished. This can be avoided, right ?

That was fixed a while ago. Update to the latest version.

MetalheadGautham wrote:

3. alternate mirrorlist
Is it possible to have an alternate mirrorlist for powerpill to use, which I can continue updating with reflector ? And can reflector code be intergrated into powerpill ? This can keep the default mirrorlist, which contains ALL the available mirrors in regional order untouched.

That's already possible by using the following pacman argument:

       --config <file>
           Specify an alternate configuration file.

You could copy your pacman.conf file, change the mirror list that it points to, then pass that as the argument to powerpill. Because only the official repos use the same mirrors, it's difficult to just pass powerpill an extra mirrorlist as it would need to know which repos it belongs to.

I don't see the issue with the default mirrorlist. Reflector doesn't overwrite it and the name of a mirrorlist doesn't matter, only that it's specified with the correct name in pacman.conf.

Thinking about it more though, I see a few situations in which it would be good to have a list of additional mirrors. Some mirrors might not be trusted enough to update the database so you wouldn't want to include them in your default pacman.conf. Powerpill can use them though because it verifies packages against the database (which should be from a trusted mirror), so it doesn't matter if it gets a bad file from some other mirror (it will just discard it).

I'll see if I can create an option that can take a file as an argument. The file could use the same syntax as the pacman.conf file (listing mirrors in the file itself or pointing to lists additional mirrors for each repo). It could then include lists used by reflector too. E.g.

[core]
Include = /path/to/reflector_list
[extra]
Include = /path/to/reflector_list
[community]
Include = /path/to/reflector_list

I'll post here once I"ve implemented this.

Once I've implemented the above, I'll consider adding reflector's functionality to powerpill, but I'll probably talk to Gerbra about it first (whose site is used to check the mirror status). I'm not really sure that I like the idea though, but I'll keep it in mind.

MetalheadGautham · 2008-10-27 08:14:36

Hmmm.... How about specifying path to alternate pacman.conf ?
Then you need to implement less stuff all over again.
powerpill.conf can have an arguement saying [location of pacman.conf], where default value can be /etc/pacman.conf and it can be altered.

And about the rm /var/lib/pacman/db.lck, you can atleast try stating that this file exists and this means either pacman is already running or that it was not closed the right way, and ask for an arguement y or n, where y removes the file (with a warning "do this only if you are sure that pacman is not already running") and n exits powerpill. Better yet, you may even add an arguement c, check, to check if pacman or powerpill is already running another instance and THEN again ask the old question, y or n.

Xyne · 2008-10-28 10:32:03

MetalheadGautham wrote:

Hmmm.... How about specifying path to alternate pacman.conf ?

That's what "--config <file>" does.

MetalheadGautham wrote:

powerpill.conf can have an arguement saying [location of pacman.conf], where default value can be /etc/pacman.conf and it can be altered.

It already does. If you're still using an old config file, generate a new one:

powerpill --gen-conf

I still intend to implement a config file for additional mirrors because I believe that will be generally useful, I just haven't had the time yet.

MetalheadGautham wrote:

And about the rm /var/lib/pacman/db.lck, you can atleast try stating that this file exists and this means either pacman is already running or that it was not closed the right way, and ask for an arguement y or n, where y removes the file (with a warning "do this only if you are sure that pacman is not already running") and n exits powerpill. Better yet, you may even add an arguement c, check, to check if pacman or powerpill is already running another instance and THEN again ask the old question, y or n.

I don't think that powerpill should do this. Powerpill is a wrapper around pacman and all database maintenance is entirely left up to pacman. If pacman leaves a lockfile, pacman should deal with removing it. I think you should submit a feature request to the pacman devs if you want to have this implemented.

Last edited by Xyne (2008-10-28 10:33:15)

MetalheadGautham · 2008-10-30 08:59:45

Xyne wrote:

I don't think that powerpill should do this. Powerpill is a wrapper around pacman and all database maintenance is entirely left up to pacman. If pacman leaves a lockfile, pacman should deal with removing it. I think you should submit a feature request to the pacman devs if you want to have this implemented.

Well, alright then.

And about the mirrors part, I really think KDEmod needs a redesigning of its mirrors. The problem is that first you have directories for the repo, and then each repo has a x86_64 and i686 subdirectory. So I need to manually type out the mirrors for each of the three repositories into pacman.conf. I can't do a simple include "filename" in each of the lines as in the standard arch repos, where the architecture directories are placed at a higher level than the various repo directories.

Xyne · 2008-10-30 20:10:58

MetalheadGautham wrote:

And about the mirrors part, I really think KDEmod needs a redesigning of its mirrors. The problem is that first you have directories for the repo, and then each repo has a x86_64 and i686 subdirectory. So I need to manually type out the mirrors for each of the three repositories into pacman.conf. I can't do a simple include "filename" in each of the lines as in the standard arch repos, where the architecture directories are placed at a higher level than the various repo directories.

It seems that all they would need to do is add "kdemod-" to the repo name in the uri to enable a single external mirrorlist to be used. Maybe they would consider doing that if you presented the argument.

I've added the option to specify extra mirrors now btw, and I've uploaded reflector to the AUR and added an option to it to grab the x fastest mirrors too. Check the example in the OP to see how to use them.

Xiong Chiamiov · 2008-11-04 00:50:21

I was thinking today, while updating, that it would be cool if I could pass a flag to powerpill (or specify an option in the configuration file) to hide a lot of the aria messages. I especially get a lot of errors from mirrors that aren't synced yet, and it fills up my screen quite a bit. Even without those, there are long lists of files to be downloaded, files being added to the download queue, files completed, files downloaded?, and files checksummed. After the initial pacman check of which files I want to update, I don't really want to see anything else besides the progress bar until it passes the packages to pacman to install.

Xyne · 2008-11-04 01:49:43

Xiong Chiamiov wrote:

I was thinking today, while updating, that it would be cool if I could pass a flag to powerpill (or specify an option in the configuration file) to hide a lot of the aria messages.

I've added a command line option ("--silent-aria") for now. I might add a config file option later.

faelar · 2008-11-06 00:46:46

You can call --gen-conf as user, it will add the config files in your home directory.
But only root is allowed to sync, update etc...
Having files in the user's folder is a bit pointless since most of the options are download related.
I don't now what you should do, maybe restrict --gen-conf option to root, or move files somewhere else... ?

Xyne · 2008-11-06 09:33:12

faelar wrote:

You can call --gen-conf as user, it will add the config files in your home directory.
But only root is allowed to sync, update etc...
Having files in the user's folder is a bit pointless since most of the options are download related.
I don't now what you should do, maybe restrict --gen-conf option to root, or move files somewhere else... ?

[xyne@~]$ sudo powerpill -Syu
==> /home/xyne/.config/powerpill/powerpill.conf loaded

It correctly loads the user's config files when using sudo. If you're switching to root with "su", it will generate a config file in root's home directory and use that when run as root.

Have I misunderstood something?

faelar · 2008-11-06 12:17:31

Yes, I don't use sudo, so my own directory change from /home/faelar to /root when I su. I never think about sudo, you can forget about the "pointless" thing, but it stills anoying for the non-sudo user to handle the configuration. Maybe /etc should be a better option for the config ?

Powerpill in itself works well, thank you again for all what you do

Xyne · 2008-11-06 18:25:56

faelar wrote:

Yes, I don't use sudo, so my own directory change from /home/faelar to /root when I su. I never think about sudo, you can forget about the "pointless" thing, but it stills anoying for the non-sudo user to handle the configuration. Maybe /etc should be a better option for the config ?

In your case, putting the config in /etc/ would require you to use root to edit it. If you're going to use root anyway, why not just create it in /root/ and edit it there?

I suppose that I could make it check for /etc/powerpill/ and load that if it's present. I'd like to know if anyone else thinks this would be useful first though.

faelar wrote:

Powerpill in itself works well, thank you again for all what you do

Thanks

Last edited by Xyne (2008-11-06 18:49:55)

fflarex · 2008-11-06 21:24:17

What about something like a --config option to override the default? Anybody who's annoyed by the config location can just set an alias with this option and be happy.

Xyne · 2008-11-06 21:46:07

*nods*

That sounds like a good idea, plus it should be quick and easy to implement. I'm still considering adding the configs to /etc/powerpill too.

kaola_linux · 2008-12-05 02:09:57

Just want to say "thank you" for developing such cool app!!!!

It really speeds up my downloads!!!

Is the default config of powerpill optimized already? I mean, are there still some tweaks?

Loved your work xyne
More power to POWERPILL and I hope this app will soon be included on the [community] repos...:D

Xyne · 2008-12-05 03:04:57

Thanks kaola_linux

I haven't tweaked the default settings much at all, so play around with them to see if you can improve performance (mainly TOTAL_CONNECTIONS and CONNECTIONS_PER_FILE). The overall speed will also depend on the number of mirrors available and the speed that you can get using each one.

MindTooth · 2008-12-05 08:27:23

Your site is down, again

If you show me how to mirror, I would gladly do so. Got 100GB bw.

Birger

Arch Linux

#26 2008-10-20 02:43:04

Re: Powerpill - A wrapper and download accelerator for Pacman

#27 2008-10-20 02:46:35

Re: Powerpill - A wrapper and download accelerator for Pacman

#28 2008-10-22 16:04:11

Re: Powerpill - A wrapper and download accelerator for Pacman

#29 2008-10-22 17:49:17

Re: Powerpill - A wrapper and download accelerator for Pacman

#30 2008-10-23 14:11:13

Re: Powerpill - A wrapper and download accelerator for Pacman

#31 2008-10-23 17:27:11

Re: Powerpill - A wrapper and download accelerator for Pacman

#32 2008-10-23 20:29:07

Re: Powerpill - A wrapper and download accelerator for Pacman

#33 2008-10-23 21:03:15

Re: Powerpill - A wrapper and download accelerator for Pacman

#34 2008-10-26 02:51:26

Re: Powerpill - A wrapper and download accelerator for Pacman

#35 2008-10-26 12:51:55

Re: Powerpill - A wrapper and download accelerator for Pacman

#36 2008-10-27 08:14:36

Re: Powerpill - A wrapper and download accelerator for Pacman

#37 2008-10-28 10:32:03

Re: Powerpill - A wrapper and download accelerator for Pacman

#38 2008-10-30 08:59:45

Re: Powerpill - A wrapper and download accelerator for Pacman

#39 2008-10-30 20:10:58

Re: Powerpill - A wrapper and download accelerator for Pacman

#40 2008-11-04 00:50:21

Re: Powerpill - A wrapper and download accelerator for Pacman

#41 2008-11-04 01:49:43

Re: Powerpill - A wrapper and download accelerator for Pacman

#42 2008-11-06 00:46:46

Re: Powerpill - A wrapper and download accelerator for Pacman

#43 2008-11-06 09:33:12

Re: Powerpill - A wrapper and download accelerator for Pacman

#44 2008-11-06 12:17:31

Re: Powerpill - A wrapper and download accelerator for Pacman

#45 2008-11-06 18:25:56

Re: Powerpill - A wrapper and download accelerator for Pacman

#46 2008-11-06 21:24:17

Re: Powerpill - A wrapper and download accelerator for Pacman

#47 2008-11-06 21:46:07

Re: Powerpill - A wrapper and download accelerator for Pacman

#48 2008-12-05 02:09:57

Re: Powerpill - A wrapper and download accelerator for Pacman

#49 2008-12-05 03:04:57

Re: Powerpill - A wrapper and download accelerator for Pacman

#50 2008-12-05 08:27:23

Re: Powerpill - A wrapper and download accelerator for Pacman

Board footer