Powerpill - A wrapper and download accelerator for Pacman

Xyne · 2009-04-13 03:43:18

The only difference is that powerpill now checks the databases itself to determine what it should download if a download operation has been requested. It still leaves everything else to the pacman binary so it shouldn't have anything to do with package conflicts (which don't matter for the download).

Pacman is invoked either immediately if there is nothing to download or after the download completes (just after the verification messages).

I took out the message about passing control to pacman to make powerpill's output a drop-in replacement for most pacman output, e.g. "powerpill -Qqdt" (which is just "pp -Qqdt" on my system).

mickael9 · 2009-04-13 03:52:02

In older versions, powerpill first ran pacman to gather the list of packages that must be downloaded (problems with group prompts / replace X with Y prompts) and another time to install the downloaded packages (one more replace X with Y prompt)
According to your message, it is now fixed, right ?

Xyne · 2009-04-13 04:34:08

The group prompts weren't included in older versions (it used --noconfirm to get the package URLs) and no version has ever dealt with the replace X with Y prompts. Pacman doesn't do that either until the downloads have finished so in that regard powerpill doesnt' change default behavior (download first and ask questions later). The never version does provide a selection dialogue though when dealing with groups which the old version didn't.

Post the specific output you're getting. I suspect that it's not powerpill related but I can't be sure until I see it (although the word "replace" isn't even in the pacman module that I wrote )

Zariel · 2009-04-13 10:40:21

Working great with pkgd!

drag0nl0rd · 2009-04-13 11:08:25

LeoSolaris wrote:

Powerpill used to grab the repo updates, and have the "Do you want to download these files" pretty swiftly after I entered the command. Now it seems to take 2-3 minutes to get the lists before asking me if I want to download/install anything.
What happened? Is powerpill downloading the files in that interval, is that why is doesn't seem to take very long to install them after I say yes to that question?

Which mirror are you using to update system/download files? Tryied another one?

LeoSolaris · 2009-04-13 12:02:29

Ok!

As I was typing this, I was trying different things. I figured out that the delay wasn't on my end, it was the main server's bandwidth cap that slowed down the transmission of the database files.

Geesh... alright alright, I will stop using the main repo. I still feel a day behind in Syu'ing.

hmm... apparently I just need to learn to read! After playing with the mirror lists and actually using reflector it pepped right up and takes just a couple of seconds. I am still not exactly sure what I was doing wrong. I did have to manually copy the /etc/powerpill.conf over to ~/.config/powerpill/powerpill.conf since the gen-conf flag was simply not working properly. Perhaps that was the cause of the slowdown? It was searching for the conf file?

I got it working now.

Thank you, Xyne.

Last edited by LeoSolaris (2009-04-13 12:24:43)

Xyne · 2009-04-13 13:16:54

"--gen-conf" no longer exists. As the install message states, the old configuration file syntax has completely changed and all old files should be removed.

Another reason not to use the main server is the very fact that it is half a day ahead of the average server. That means that only a small portion of the mirrors will be in sync with it when you update which in turn means that you'll get bombarded with "file not found" errors on top of only having a handful of servers that have the new files. Just use a local mirror and add "Reflect = -l 45" to the powerpill options section of powerpill.conf.

Anyway, I'm happy that it's working as it should... I got worried that it the code itself was slowing it down.

mickael9 · 2009-04-13 15:14:24

powerpill.conf seems to be overriden when upgrading

Xyne · 2009-04-13 18:51:32

mickael9 wrote:

powerpill.conf seems to be overriden when upgrading

Sorry about that. I had "backup=(/etc/powerpill.conf)" in the PKGBUILD instead of "backup=(etc/powerpill.conf)"... it confused the **** out of me for about 10 minutes.

*walks off to update the PKGBUILD wiki page*

LeoSolaris · 2009-04-16 11:41:13

Wow... ok now that I got it off of Arch's repo, and added in Reflector... why didn't I do this before? A day or so wait is not THAT big of a deal.

Xyne · 2009-04-16 13:20:31

By "Arch's repo" do you mean the AUR?

What do you mean by "a day or so wait"? I usually upload the sources, update the repo on my site and upload the AUR package all at the same time.

LeoSolaris · 2009-04-16 13:51:57

I meant ftp.archlinux.org

I meant the wait for the mirrors to sync, I was trying for humor. I think I missed.

Xyne · 2009-04-16 14:51:11

No, that was my fault... I had forgotten about your previous post and completely missed the context (I took "got if off of Arch's repo" as you downloading the package).

Anyway, if the wait really bothers you, use the Australian servers... they're about 15 hours* ahead on average so you don't have to wait as long.

*No, I didn't geolocate all the Aussie servers and calculate the average timezone difference (with corrections for summer time) to the main server... I used the highly esteemed RIAA method: use whatever sounds good to your context audience (e.g. to politicians: we're losing billions of dollars due to piracy and can't survive without draconian legislation; to shareholders: we're raking it in, even with the economic downturn... here's some money, we stopped paying the artists; to the artists: your album is selling like shit so we can't pay you).

*crowns himself king of parenthetical comments*

Last edited by Xyne (2009-04-16 14:54:52)

keerthi · 2009-04-16 17:51:03

powerpill 16.0-11

With /etc/yaourtrc having 'PacmanBin /usr/bin/powerpill', I get this:

keerthi ~/Temp  $  yaourt mp3info                                                                            
1 community/mp3info 0.8.5a-2                                                                                 
     An MP3 technical info viewer and ID3 1.x tag editor                                                     
2 aur/ruby-mp3info 0.6.12-1 (12)
    a pure ruby library for access to mp3 files (internal infos and tags)
==>  Enter n° (separated by blanks, or a range) of packages to be installed
==>   ----------------------------------------------
==>1
community/mp3info package not found, searching for group...
error: 'community/mp3info': not found in sync db
If you like this package, please install aurvote
and vote for its inclusion/keeping in [community]

I commented out PacmanBin in /etc/yaourtrc, then I get this

keerthi ~/Temp  $  yaourt mp3info
1 community/mp3info 0.8.5a-2
     An MP3 technical info viewer and ID3 1.x tag editor
2 aur/ruby-mp3info 0.6.12-1 (12)
    a pure ruby library for access to mp3 files (internal infos and tags)
==>  Enter n° (separated by blanks, or a range) of packages to be installed
==>   ----------------------------------------------
==>1
resolving dependencies...
looking for inter-conflicts...

Targets (1): mp3info-0.8.5a-2 [0.03 MB]

Total Download Size:    0.00 MB
Total Installed Size:   0.08 MB

Proceed with installation? [Y/n]

My powerpill.conf

keerthi ~  $  cat .config/powerpill/powerpill.conf                                                           
[options]                                                                                                    
                                                                                                             
# Set the path to the aria2c binary.                                                                         
Aria2Bin = /usr/bin/aria2c                                                                                   
                                                                                                             
# Clean up leftover aria2 control files.                                                                     
Aria2Clean                                                                                                   
                                                                                                             
# Suppress aria2c output.                                                                                    
Aria2Silent                                                                                                  
                                                                                                             
# Powerpill will log output to this file if this                                                             
# option is set.                                                                                             
Log = /var/log/powerpill.log                                                                                 
                                                                                                             
# Suppress colored output.                                                                                   
#NoColor                                                                                                     
                                                                                                             
# Set the path to the pacman binary.                                                                         
# Change this to your preferred pacman binary,                                                               
# e.g. pacman-color                                                                                          
PacmanBin = /usr/bin/pacman-color                                                                            
                                                                                                             
# Set the path to the pacman configuration file.                                                             
# Change this if you want to use a custom configuration                                                      
# with powerpill.                                                                                            
PacmanConf = /etc/pacman.conf                                                                                
                                                                                                             
# Use reflector at runtime to get the latest mirrors.                                                        
# Depending on the reflector options that you use,                                                           
# this may slow down the overall process while retrieving                                                    
# the latest list, so use it wisely.                                                                         
#Reflect = <reflector options>                                                                               
                                                                                                             
# Set the number of times that powerpill will retry a                                                        
# download if it fails. This only has an effect when                                                         
# run with pacman's noconfirm option.                                                                        
#RetryLimit = 3                                                                                              
                                                                                                             
# The minimum size a file must be for powerpill to                                                           
# split the file for segmented downloading (in MB).                                                          
#SplitSize = 5                                                                                               
                                                                                                             
                                                                                                             
                                                                                                             
                                                                                                             
                                                                                                             
# aria2 options                                                                                              
# These follow the same format as the aria2 config file.                                                     
# Please refer to the aria2 documentation for more details                                                   
# including a complete set of options (proxy settings etc).                                                  
# Only the most relevant options have been included here.                                                    
[aria2_options]                                                                                              
                                                                                                             
# maximum number of concurrent downloads                                                                     
# This must be in the range 1-45. Powerpill sets it to 45                                                    
# and configures the metalink to adjust the connections                                                      
# per file.                                                                                                  
#max-concurrent-downloads=45                                                                                 
                                                                                                             
# number of servers to connect to simultaneously                                                             
# This only has a limiting effect. Leave it unset to                                                         
# connect to the maximum number of servers allowed                                                           
# by the settings in the metalink.                                                                           
#metalink-servers=20                                                                                         
                                                                                                             
# Increasing these settings will lose more time while                                                        
# waiting for unresponsive servers but decreasing them                                                       
# will miss active servers and lose time while aria2                                                         
# tries other servers.                                                                                       
#timeout=5                                                                                                   
#max-tries=2                                                                                                 
#retry-wait=2                                                                                                
                                                                                                             
                                                                                                             
# lowest acceptable connection speed (0 = disabled)                                                          
# Downloads below this speed will be aborted.                                                                
# The speed is given in bytes by default. Append "K"                                                         
# or "M" for kilobytes or metabytes, resp.                                                                   
# (1M = 1024K = 1048576)                                                                                     
#lowest-speed-limit=0                                                                                        
                                                                                                             
# maximum total download speed (0 = disabled)                                                                
# Use this to throttle the download.                                                                         
# The speed is given as for lowest-speed-limit above.                                                        
#max-download-limit=0                                                                                        
                                                                                                             
                                                                                                             
# Be careful with this. Aria2 can quickly create huge                                                        
# log files, depending on the log-level option.                                                              
#log=/var/log/aria2.log                                                                                      
                                                                                                             
                                                                                                             
                                                                                                             
                                                                                                             
                                                                                                             
                                                                                                             
# list pkgd-servers on your LAN here                                                                         
# they will be queried before remote servers                                                                 
#[PkgD]                                                                                                      
#Server = http://xxx.xxx.xxx.xxx:yyyyy/$repo                                                                 
                                                                                                             
                                                                                                             





# Here you can specify additional mirrors for powerpill.
# The mirrors specified here will not be used by pacman
# and thus the database will never be updated from them.
# These sections follow the same syntax as pacman.conf.

# Some of you will recognize this section as the heir
# to the additional_mirrors.conf file. :)

# Note that if you use the "reflect" option. there isn't
# any reason to include reflector mirrorlists down here.

#[core]
# Server = http://example.com/arch/$repo/
# Include = /path/to/mirrorlist

#[extra]
# Server = http://example.com/arch/$repo/
# Include = /path/to/mirrorlist

#[community]
# Server = http://example.com/arch/$repo/
# Include = /path/to/mirrorlist

Xyne · 2009-04-17 07:57:07

Ooooh, nice find, thanks keerthi

It should work now. As a consequence of the change, the repo is now prepended to the package name in dialogues. I can easily remove it or make it a configuration if enough people object, but I think it could be useful to be able to see where things are coming from (one example is a certain third-party repo appending packages to the "base" group).

keerthi · 2009-04-17 13:28:51

Great... upgraded to perl-xyne-arch 0.28-1. Works fine now. Thanks.

weltio · 2009-04-17 15:26:41

Ok - I know, stupid questions.... but
a) Why is there no colored output? (doesn't commenting out the color part in /etc/powerpill.conf prevent from this?)
b) How can I search or install in or rather from AUR?

Last edited by weltio (2009-04-17 15:28:02)

Xyne · 2009-04-18 05:58:31

weltio wrote:

Ok - I know, stupid questions.... but
a) Why is there no colored output? (doesn't commenting out the color part in /etc/powerpill.conf prevent from this?)
b) How can I search or install in or rather from AUR?

a) Powerpill is a wrapper around pacman. If there is nothing to download then it just passed control to pacman (or whatever pacman binary you've specified). If you're using standard pacman then you won't get any colored output because pacman just doesn't have it. Take a look at "pacman-color" in the AUR for that. You can use pacman-color with powerpill by specifying it in the powerpill configuration file.

b) Powerpill doesn't support the AUR. It's only a download accelerator for pacman. If you want AUR support, use yaourt and configure yaourt to use powerpill. Many people seem to use yaourt(powerpill(pacman-color)) as a wrapping scheme.

weltio · 2009-04-19 17:48:28

Ah - thanks. How can I configure yaourt to use powerpill?

evr · 2009-04-19 18:04:28

weltio wrote:

Ah - thanks. How can I configure yaourt to use powerpill?

edit /etc/yaourtrc so it points to powerpill:

PacmanBin /usr/bin/powerpill

qubit · 2009-04-19 20:30:02

Hi, I have problem with powerpill downloading packages which should be ignored - IgnorePkg in pacman.conf.
Any idea how to avoid this?

Xiong Chiamiov · 2009-04-19 21:05:21

It didn't download the new version of amarok today, which I have ignored. Since it grabs the packages to download from pacman, it shouldn't have the kind of issue you're describing. Are you sure that you've specified it correctly?

Xyne · 2009-04-19 22:33:22

@qubit
Could you post your IgnorePkg line(s) so I can test something?
I might have introduced a bug yesterday when I changed something but if I did it should be easy to fix.

@Xiong Chiamiov
At the moment powerpill is pacman-independent for the download. I've written a partial implementation of pacman in Perl (Xyne::Arch::Pacman.pm) to gain more freedom and to pave the way for some other ideas that I have.

*edit*
It should be fixed now. Try the latest version and let me know if it works.

Last edited by Xyne (2009-04-19 22:41:49)

qubit · 2009-04-20 09:54:59

It's fixed with the latest upgrade of perl-xyne-arch (0.35-1).
Thanks!

Xyne · 2009-04-20 13:43:41

I think the refactored version is almost stable enough to move it into the community repo. I'd appreciate it if some users could "stress test" the latest version by trying arbitrary downloads (create a cache dir in /tmp and invoke it with "powerpill -Sw --cachedir /tmp/cache ...") with different pacman.conf settings (ignore lines, show size, etc) and report any errors or unexpected behavior (such as downloads being passed on to pacman).

If there are no hiccups I'll move it in at the end of the week.

Arch Linux

#151 2009-04-13 03:43:18

Re: Powerpill - A wrapper and download accelerator for Pacman

#152 2009-04-13 03:52:02

Re: Powerpill - A wrapper and download accelerator for Pacman

#153 2009-04-13 04:34:08

Re: Powerpill - A wrapper and download accelerator for Pacman

#154 2009-04-13 10:40:21

Re: Powerpill - A wrapper and download accelerator for Pacman

#155 2009-04-13 11:08:25

Re: Powerpill - A wrapper and download accelerator for Pacman

#156 2009-04-13 12:02:29

Re: Powerpill - A wrapper and download accelerator for Pacman

#157 2009-04-13 13:16:54

Re: Powerpill - A wrapper and download accelerator for Pacman

#158 2009-04-13 15:14:24

Re: Powerpill - A wrapper and download accelerator for Pacman

#159 2009-04-13 18:51:32

Re: Powerpill - A wrapper and download accelerator for Pacman

#160 2009-04-16 11:41:13

Re: Powerpill - A wrapper and download accelerator for Pacman

#161 2009-04-16 13:20:31

Re: Powerpill - A wrapper and download accelerator for Pacman

#162 2009-04-16 13:51:57

Re: Powerpill - A wrapper and download accelerator for Pacman

#163 2009-04-16 14:51:11

Re: Powerpill - A wrapper and download accelerator for Pacman

#164 2009-04-16 17:51:03

Re: Powerpill - A wrapper and download accelerator for Pacman

#165 2009-04-17 07:57:07

Re: Powerpill - A wrapper and download accelerator for Pacman

#166 2009-04-17 13:28:51

Re: Powerpill - A wrapper and download accelerator for Pacman

#167 2009-04-17 15:26:41

Re: Powerpill - A wrapper and download accelerator for Pacman

#168 2009-04-18 05:58:31

Re: Powerpill - A wrapper and download accelerator for Pacman

#169 2009-04-19 17:48:28

Re: Powerpill - A wrapper and download accelerator for Pacman

#170 2009-04-19 18:04:28

Re: Powerpill - A wrapper and download accelerator for Pacman

#171 2009-04-19 20:30:02

Re: Powerpill - A wrapper and download accelerator for Pacman

#172 2009-04-19 21:05:21

Re: Powerpill - A wrapper and download accelerator for Pacman

#173 2009-04-19 22:33:22

Re: Powerpill - A wrapper and download accelerator for Pacman

#174 2009-04-20 09:54:59

Re: Powerpill - A wrapper and download accelerator for Pacman

#175 2009-04-20 13:43:41

Re: Powerpill - A wrapper and download accelerator for Pacman

Board footer