Powerpill - A wrapper and download accelerator for Pacman

MindTooth · 2009-03-06 23:27:37

Adding --nomessage is now disable the whole output of i.e. yaourt. Maybe it broke with the latest update of yaourt. If I remove --nomessage, I can see the text outputed.

Birger

Xyne · 2009-03-07 01:04:35

Thanks MindTooth.

I've changed the recommendation for "--nomessage" usage on the info page to a mention that it might help with some wrappers but that the user should try it without that option first. I've also changed some text in the help message and in other places (and discovered a bug in makerepo while I was at it ).

MindTooth · 2009-03-07 15:07:40

So, I can't use nomessage with yaourt anymore? Not a big deal though.

Birger

Ed1t: Your welcome

Last edited by MindTooth (2009-03-07 15:07:54)

Xyne · 2009-03-08 02:13:21

I've just tried using "--nomessages" when wrapping powerpill with yaourt and it works as expected here (powerpill output is suppressed, yaourt output is shown). How have you configured yaourt and powerpill? On my system, I've set PacmanBin in /etc/yaourtrc to /usr/bin/powerpill (well, actually it points to the latest working copy, but it's effectively the same) and to test "--nomessages", I've added "ADDITIONAL_POWERPILL_ARGUMENTS = '--nomessages'" to ~/.config/powerpill/powerpill.conf

If "--nomessages" really doesn't work for you, check if "--noconfirm" and "--silent-aria" work. You could even try to regenerate the powerpill configuration file to see if that changes anything (it should transfer your old settings to the new file, but back up the old one if you're afraid of losing anything in it).

MindTooth · 2009-03-08 12:12:29

Ed1t: I seems that I was to quick on that conclusion. Still doesn't work. Will just wait to see how this evolves.

Birger

Last edited by MindTooth (2009-03-08 13:52:34)

skottish · 2009-03-08 14:43:28

Stickied by me.

MindTooth · 2009-03-26 17:59:31

I have now installed Powerpill on several new computers, and I still experience problems. Seems that it does not play well with Yaourt.
Though I have grown fond of the Powerpill output.

Just a quick action, when adding --silent-aria is added. Do I still need to press "y"?

Birger

Xiong Chiamiov · 2009-03-26 18:17:40

MindTooth wrote:

I have now installed Powerpill on several new computers, and I still experience problems. Seems that it does not play well with Yaourt.
Though I have grown fond of the Powerpill output.
Just a quick action, when adding --silent-aria is added. Do I still need to press "y"?
Birger

If you mean to choose which files to download, yes, at least on the version of powerpill I have (haven't updated AUR in a few weeks).

smartboyathome · 2009-03-26 19:46:01

MindTooth wrote:

I have now installed Powerpill on several new computers, and I still experience problems. Seems that it does not play well with Yaourt.
Though I have grown fond of the Powerpill output.
Just a quick action, when adding --silent-aria is added. Do I still need to press "y"?
Birger

I haven't had any issues with yaourt and powerpill. It just takes a few seconds to start up after it is called.

EDIT: Actually, it doesn't get used when installing dependancies for a PKGBUILD. I just forgot about this.

Last edited by smartboyathome (2009-03-26 19:46:45)

Xyne · 2009-03-27 09:33:56

I've just finished completely refactoring powerpill so I won't be fixing any minor issues with the current version. I need to do some testing (I might creating a testing repo, but I'll post a thread about it if I do) and add some other features (pkgd support, maybe built-in reflector support and some other ideas that I have).

Some of the dialogue handling has also been improved so hopefully that will make most people happy.

I'll update it as soon as I feel it's ready.

MoonSwan · 2009-03-27 10:17:57

I'll happily be a tester for you if I ever get back to irc.

onguarde · 2009-03-29 16:01:42

Just wanna clarify how powerpill picks its mirrors when it is used in conjunction with yaourt and pacman-color.

Is it true that it picks its mirrors from pacman.conf first and after which, also in additional_mirrors.conf. Both of which optionally links to external files through their "Include=" parameter.

This means that I also have to change the default "Include=" parameter in pacman.conf to point to my reflector_mirrorlist else it will default to the unsorted one right?

Also, just curious as to why the config files are not centrally stored in /etc/ as in pacman. Keep needing to "synchronize" the root's .config files with the users'.

Thanks for such a great tool!

Last edited by onguarde (2009-03-29 16:04:01)

Xyne · 2009-03-29 18:23:38

onguarde wrote:

Is it true that it picks its mirrors from pacman.conf first and after which, also in additional_mirrors.conf. Both of which optionally links to external files through their "Include=" parameter.

Powerpill parses the pacman configuration file and creates internal lists of servers for each repo from the "Server =" lines. It also follows "Include =" lines to external files, so it always knows about all mirrors available to pacman. The additional_mirrors.conf specifies mirrors that will be appended to those lists for the download, following the same syntax as pacman.conf. The order itself doesn't matter because the metalink protocol randomly picks servers from the list.*

onguarde wrote:

This means that I also have to change the default "Include=" parameter in pacman.conf to point to my reflector_mirrorlist else it will default to the unsorted one right?

Nope. Just for clarification, the reason that powerpill enables the user to specify additional mirrors is to keep those mirrors separate from pacman. Powerpill doesn't deal with database updates so the idea is to have a small set of relatively known/trusted mirrors in available to pacman which it can use for database updates. Powerpill can use more unknown mirrors (they should all be generally trusted and synced though, but still) because it verifies the integrity of downloaded files against the databases md5sums. This way you can use all of the mirrors retrieved by reflector without worrying about what it might do to your database or if everything is in sync, for example.

onguarde wrote:

Also, just curious as to why the config files are not centrally stored in /etc/ as in pacman. Keep needing to "synchronize" the root's .config files with the users'.

The short answer: the original powerpill script was hacked together just after I switched to Linux and Arch so I made some noobish decisions
As mentioned, I've just finished completely refactoring powerpill. One of the changes is the inclusion of a central config file in /etc
I should have it up soon (for testing at least). I just need to add a few more things.

onguarde wrote:

Thanks for such a great tool!

*It's possible to specify preferences in the metalink url tags and I might include such a feature later, but the risk is that people will abuse it and hammer certain servers. It really shouldn't matter which servers are included in the list as long as they come from the official Arch mirrorlist.

onguarde · 2009-03-30 09:06:39

Thx for the quick reply and excellent clarificaiton!

The order itself doesn't matter because the metalink protocol randomly picks servers from the list.

I think the problem I face at times with such "randomness" in mirror picking is that, at times, a particularly slow mirror is picked. And even though all the other packages are done downloading, this last teeny weeny package is still crawling at dismal speed. Rarely do I see aria2 splitting that last package, possibly because there are so many connections to begin with.

I guess one way outta this is to remove all but a few fast mirrors supposing there's no limit on number of connections per mirror. Unfortunately, the no. of mirrors is proportional to the the "maxing" potential. Another way would be to prioritize the mirrors higher up in the list instead of random picking. I guess this would only be natural since reflector sorts mirrors in descending order of speed. Otherwise, I don't really see a point in this sorting.

I'm thiking maybe the bigger sized packages can use the faster mirrors and the slower ones can settle on the slower mirrors. Once the fast mirrors are free, some of the smaller packages can take its place(resume connection - change mirror).

Also, I'm not sure how the rankmirrors script really work(I presume mostly by access time since it times the download of a 50k file) but the sorting does not really take into account actual stable speeds. Some mirrors seems to be more throttled than others.

P.s I think it'll be useful if the documentation mentions a bit of this. I used to think that since powerpill is called during the download process instead of pacman, so only the mirrors from powerpill's side is used. Apart from the first few listed in pacman.conf for database sync perhaps.

Last edited by onguarde (2009-03-30 09:07:41)

Xiong Chiamiov · 2009-03-30 17:25:55

That's why we have reflector. I have an alias that polls the mirrors and gets the fastest 10 that have synced in the last 12 hours, and puts those into my additional_mirrors.conf. It only takes a few minutes, and running it every now and again gives you generally fast mirrors. If you can't figure out Xyne's documentation, I can give you what I use when I get home.

One of these days, I'll make some sort of database of times and availabilities, so I can get the ones that are most commonly "good", but that'll take a bit more effort than it's worth.

Xyne · 2009-03-31 12:15:25

The powerpill.conf file allows you to set the minimal download speed per server. I recommend using that to avoid those crawling mirrors (10K is normally enough for me). As Xiong Chiamiov mentioned, it's also possible to use reflector with rankmirrors to pull the fastest x servers from the latest mirrorlist which should give you a decent set from which to download.

Anyway, in other news, I just need to sort out my packaging scripts and then I should be able to put the new version up for testing.

onguarde · 2009-04-08 06:03:55

Ah, the minimum speed limit thing seems to be working. I see a lot of slow ones getting rejected.

Regarding the discussion on installing packages as soon as they're downloaded a while back,

Xyne wrote:

Actually, it would probably be possible to install packages through a thread with pacman and wait for the thread to exit before launching another one to avoid simultaneous database access conflicts, but that made me realize the greatest limitation to doing this: the packages are not downloaded in any particular order in the metalink, so some dependencies may be the last packages to be downloaded, which nullifies this approach.

I think a way around this is to let pacman resolve the dependencies first and force install everything downloaded sequentially i.e start force installing separately, each time the package download is done.

Last edited by onguarde (2009-04-08 06:07:53)

Xiong Chiamiov · 2009-04-08 18:21:55

onguarde wrote:

Xyne wrote:
Actually, it would probably be possible to install packages through a thread with pacman and wait for the thread to exit before launching another one to avoid simultaneous database access conflicts, but that made me realize the greatest limitation to doing this: the packages are not downloaded in any particular order in the metalink, so some dependencies may be the last packages to be downloaded, which nullifies this approach.
I think a way around this is to let pacman resolve the dependencies first and force install everything downloaded sequentially i.e start force installing separately, each time the package download is done.

Do you really want to force-install everything, though? That's kinda like catching all exceptions, rather than just the one you want; you'll be overriding errors caused by other things than the dependencies missing.

I suppose an option for that is ok, as long as it's not on by default. People can use -Syyf to update on a regular basis, and this seems less dangerous than that.

Xyne · 2009-04-12 07:08:03

Ok, I've moved the refactored powerpill from xyne-any-testing to xyne-any. Please report any bugs that you find. I'll be moving powerpill and a few other packages into the community repo soon.

Recap: powerpill now has a new config file syntax, built-in reflector support, built-in pkgd support, a few option changes and a man page.

Zariel · 2009-04-12 12:01:03

Can you add rebase support please? That would be awesome, see my post in pkgd for errors relating to that.

mickael9 · 2009-04-12 16:04:43

Powerpill always uses /usr/bin/pacman even when passing --pacman-bin or changing the PacmanBin option in /etc/powerpill.conf

Edit : wow, already fixed, it was quick

Last edited by mickael9 (2009-04-12 20:29:20)

Xyne · 2009-04-12 20:54:11

@Zariel
I will. I just need to refactor rebase first but I'm short on time right now and don't know how long it will take me to get to it.

@mickael9
Fixed. Note that the package negotiation dialogue (package selection from groups, etc) is handled directly by powerpill but all non-download operations will be passed to the specified pacman binary now.

LeoSolaris · 2009-04-13 00:40:26

Ok... I was having some issues yesterday with powerpill from your repo, Xyne. I noticed that today there is a new version, so I will do a full uninstall of powerpill, and try again.

By the way, is the -h switch supposed to bring up pacman's help rather than powerpill's help?

Last edited by LeoSolaris (2009-04-13 11:31:13)

Xyne · 2009-04-13 01:08:02

It definitely shouldn't take 2-3 minutes so there's something wrong. It might be a bit slower though because powerpill itself now determines which files are needed but I had hoped it wouldn't make a noticeable difference.

Have you removed your old configuration file?
Are you using the built-in reflector function? If you are, don't use the rank-mirrors options or else it will have to wait for that, which could take ages if you're using 50+ mirrors from the site. Just use the "--hours" and "--latest" options. If you're concerned about per-server speeds, just use the minimum speed options in the aria2 section of the powerpill configuration file.

If you're not using the reflector functionality and have removed your old configs, try this and let me know if it still seems to take a while create the metalink:

pacman-optimize
powerpill -S base --get-metalink

If it really is an issue I'll try to revert to wrapping pacman for the download queue but I don't think it should make much of a difference.

Last edited by Xyne (2009-04-13 01:09:23)

mickael9 · 2009-04-13 03:22:58

Are package conflicts handled by powerpill ? I remember having to confirm twice the "X is in conflict with Y. Replace X with Y ?" prompt because pacman was called twice.
Is it still the case in the new version ?
It's not clear when powerpill invokes pacman (and when it used to in the previous versions)

Last edited by mickael9 (2009-04-13 03:25:38)

Arch Linux

#126 2009-03-06 23:27:37

Re: Powerpill - A wrapper and download accelerator for Pacman

#127 2009-03-07 01:04:35

Re: Powerpill - A wrapper and download accelerator for Pacman

#128 2009-03-07 15:07:40

Re: Powerpill - A wrapper and download accelerator for Pacman

#129 2009-03-08 02:13:21

Re: Powerpill - A wrapper and download accelerator for Pacman

#130 2009-03-08 12:12:29

Re: Powerpill - A wrapper and download accelerator for Pacman

#131 2009-03-08 14:43:28

Re: Powerpill - A wrapper and download accelerator for Pacman

#132 2009-03-26 17:59:31

Re: Powerpill - A wrapper and download accelerator for Pacman

#133 2009-03-26 18:17:40

Re: Powerpill - A wrapper and download accelerator for Pacman

#134 2009-03-26 19:46:01

Re: Powerpill - A wrapper and download accelerator for Pacman

#135 2009-03-27 09:33:56

Re: Powerpill - A wrapper and download accelerator for Pacman

#136 2009-03-27 10:17:57

Re: Powerpill - A wrapper and download accelerator for Pacman

#137 2009-03-29 16:01:42

Re: Powerpill - A wrapper and download accelerator for Pacman

#138 2009-03-29 18:23:38

Re: Powerpill - A wrapper and download accelerator for Pacman

#139 2009-03-30 09:06:39

Re: Powerpill - A wrapper and download accelerator for Pacman

#140 2009-03-30 17:25:55

Re: Powerpill - A wrapper and download accelerator for Pacman

#141 2009-03-31 12:15:25

Re: Powerpill - A wrapper and download accelerator for Pacman

#142 2009-04-08 06:03:55

Re: Powerpill - A wrapper and download accelerator for Pacman

#143 2009-04-08 18:21:55

Re: Powerpill - A wrapper and download accelerator for Pacman

#144 2009-04-12 07:08:03

Re: Powerpill - A wrapper and download accelerator for Pacman

#145 2009-04-12 12:01:03

Re: Powerpill - A wrapper and download accelerator for Pacman

#146 2009-04-12 16:04:43

Re: Powerpill - A wrapper and download accelerator for Pacman

#147 2009-04-12 20:54:11

Re: Powerpill - A wrapper and download accelerator for Pacman

#148 2009-04-13 00:40:26

Re: Powerpill - A wrapper and download accelerator for Pacman

#149 2009-04-13 01:08:02

Re: Powerpill - A wrapper and download accelerator for Pacman

#150 2009-04-13 03:22:58

Re: Powerpill - A wrapper and download accelerator for Pacman

Board footer