Powerpill - A wrapper and download accelerator for Pacman

oddsocks · 2008-12-17 23:41:42

Thanks for the reply Xiong. I tried the change of PacmanBin option to /usr/bin/pacman-color but yaourt refuses to use pacman-color (I have pacman-color installed). I tried setting up a ~/.yaourtrc file, but it also doesn't seem to work either. The problem must be yaourt completely ignoring the two setup files. Do you know if yaourt uses another setting file?

Xiong Chiamiov · 2008-12-18 00:05:48

According to the yaourtrc man page, /etc/yaourtrc and /home/user/.yaourtrc are the only two options. How did you install it?

Also, you might want to move this into another thread, since it's no longer about powerpill.

MindTooth · 2008-12-18 07:16:45

@Xyne: Why not do as yaourt. If you pass a command, and you need root privi, you will be asked to enter password. No need, then no password.
Also found a bug, if you type "$ powerpill -Su" it displays nothing. But when I add sudo to that line. It works.

Birger

Xyne · 2008-12-18 14:23:06

MindTooth wrote:

@Xyne: Why not do as yaourt. If you pass a command, and you need root privi, you will be asked to enter password. No need, then no password.
Also found a bug, if you type "$ powerpill -Su" it displays nothing. But when I add sudo to that line. It works.
Birger

It partially does. If you try "pacman -Su --cachedir /tmp/somedir/", it will detect that pacman needs sudo and prompt for a password in order to run pacman with root privileges. If you use the default cache directory or another one that requires root privileges for write access, then powerpill itself needs root privileges because it directly accesses the cachedir to remove aria2c files and corrupted downloads. I could add a password prompt in this case but it's easier to just rerun powerpill with sudo. I'll take a look at the best way to do that and give it some thought.

I'm reluctant to add it though because it deviates from pacman behavior. If pacman requires root privileges, it doesn't prompt for a password, it just tells you what it needs.

You don't see any powerpill messages because you've added "--nomessages" to your powerpill.conf file (it's skipping the "you don't have permissions" messages). It only makes sense to use that when you're using powerpill inside of other wrappers. In your case, I would create a second powerpill configuration directory, symlink everything in the original one except powerpill.conf, copy the old powerpill.conf to the new directory, then remove "--nomessages" from that one. Create an alias for powerpill to use the alternative configuration (e.g. alias pp='powerpill --conf-dir /path/to/dir/').

I'll update the script to print error messages to STDERR when normal messages are suppressed.

Last edited by Xyne (2008-12-18 14:31:00)

Xyne · 2008-12-18 15:06:54

Ok, I've updated powerpill to include the option "--interactive" that can be used to override "--nomessages" and "--noconfirm" for interactive sessions.

I've also changed the message function to print error and warning messages to STDERR when messages are suppressed.

MoonSwan · 2008-12-19 08:41:57

ah <3 xyne. & your powepill.
damn I wish I could pm you on this forum. Oh well, shall find you on irc some time I guess. Apologies for the thread-jack. *returns to space under a large rock*

Last edited by MoonSwan (2008-12-19 08:42:52)

Xyne · 2008-12-19 08:54:32

(it wouldn't let me post the smiley by itself)

Last edited by Xyne (2008-12-19 08:57:48)

deltaecho · 2008-12-19 22:39:42

It's me again

If you don't want to implement my idea, that's cool; I've been looking for a package manager that would install applications as they were downloaded for probably a year now (since I knew what a package manager was), so I'll probably create my own personal patch/wrapper for PowerPill -- but let me bounce 1 more idea off you:

Algorithm:
1) Parse the list of packages and organize them into linked list groups according to dependencies (each could be located in multiple lists)
2) Begin downloading the metapackage
3) As each package is downloaded, push it into a stack for each group its a member of, and remove its pointer from the lists
4) As each list is emptied,
4.a) Create a new Pacman thread for that group
4.b) Pop each package from the group's stack
4.b.1) If the package hasn't been installed
4.b.1.a) install it
4.b.1.b) flag it as installed
4.b.2) Else, pop the next package
4.c) As the stack is emptied, finish the thread and begin the next one until all the packages are installed

It has room for improvement, but I think it would work.

Xyne · 2008-12-20 01:42:35

Ok, I'll look into this. I would need to restructure powerpill a bit though as this would require threads, plus I need to find the best way to check dependencies (shouldn't be difficult, but I need to be careful there).

Just so you know, I've loosely committed myself mentally to moving powerpill over to python and pyalpmm once that's matured (or to something else if I figure out how to use alpm directly myself), at which point it will be possible to bypass pacman and do all of this much more eloquently.

I'll post my thoughts on it later.

skottish · 2009-01-07 22:57:26

Xyne, I think it's time that this thread gets locked.

I've been using powerpill for a little while now and I have to tell you that it is so cool. It makes things much better. Thanks for sharing.

Xyne · 2009-01-07 23:07:23

skottish wrote:

Xyne, I think it's time that this thread gets locked.

Hehe... wait, um, you are joking, right? *looks around for help*

skottish wrote:

I've been using powerpill for a little while now and I have to tell you that it is so cool. It makes things much better. Thanks for sharing.

Thanks for the feedback, skottish. I was happy to see you support my suggestion to use it in another thread too.

Last edited by Xyne (2009-01-07 23:07:52)

skottish · 2009-01-07 23:09:59

Xyne wrote:

skottish wrote:
Xyne, I think it's time that this thread gets locked.
Hehe... wait, um, you are joking, right? *looks around for help*
skottish wrote:
I've been using powerpill for a little while now and I have to tell you that it is so cool. It makes things much better. Thanks for sharing.
Thanks for the feedback, skottish. I was happy to see you support my suggestion to use it in another thread too.

I'm a fan now. I'll be promoting this whenever I can.

userlander · 2009-01-08 00:00:50

wow - way cool. it even has a cool name. no updates now to test, but it sure seems fast so far. thanks

Xyne · 2009-01-14 12:54:42

In reply to lilsirecho's post:

Your system architecture has no effect on powerpill. From the looks of it, it seems that your database mirror is out of sync with your other mirrors. Powerpill is unable to retrieve a package from a mirror that doesn't have it and time is lost if powerpill needs to check multiple mirrors to find the package. Most of your mirrors should be in sync to obtain the best speed. You can use reflector to achieve this.

If you do not want to see the aria2 messages, you can use the "--silent-aria" flag. Normally though, there are no errors. When there are, it's an indication of what's wrong with your setup.

That said, I've just noticed here that there are multiple "not found" errors. I'm going to check the pacman changelog as a new version was recently release. If the latest pacman update has introduced errors to powerpill. I will fix them.

*EDIT*
The only thing that had changed that I noticed was some of aria2's output, which was causing errors in the reported total download size and average rate. Update to the latest version, try tweaking some settings, and try again. In particular, try changing the following:

TOTAL_CONNECTIONS
MINIMUM_SIZE
CONNECTIONS_PER_FILE
MIN_SPEED
TIMEOUT
MAX_TRIES
RETRY_WAIT

You could try increasing the number of mirrors too, but use rankmirrors or reflector to get fast ones.

You'll notice more of a difference when dealing with multiple packages or larger packages. Of course, if you already have a mirror that maximizes your bandwidth, then there is no room left for improvement and you may as well just skip powerpill. Users with fast internet connections will see the greatest benefits as most mirrors will not offer very fast downloads per connection.

Last edited by Xyne (2009-01-14 15:06:32)

lilsirecho · 2009-01-14 17:04:50

Xyne;

Appreciate your informative reply.

I conclude that powerpill will not be of value in my case due to the limited speeds of my internet connection.

One last comment on the performance of the powerpill package...at the initial .....pacman starting...... the system hangs for over a full minute before initiating the next steps and requires user intervention to continue.

I reused powerpill this date to download four packages and encountered the same slow performance (14kb/sec) . I discovered that my modem (verizon) was out of sync, probably incurred during the powerpill activity.

I assume I can remove powerpill with standard pacman commands.

The "last mile" has me in its clutches with Verizon adsl. They provide a fiber optic interface but bundling and home installation give me a pain!!

Many thanks for the experiment and best of the year 2009 to you!

Xyne · 2009-01-14 17:49:36

@lilsirecho

If you installed the powerpill package from my site through the repo or with "pacman -U powerpill-x.xx-x-any.pkg.tar.gz", then you can remove it with "pacman -Rsn powerpill".

The part where it "hangs" after displaying "pacman: starting" is not actually powerpill hanging but rather pacman. Most likely pacman is updating the database but your main server is timing out. Try a simple "pacman -Sy" to see what's happening.

I don't know what you mean by your modem being out of sync. As for bandwidth, 14kb/s is very slow. Have you run any online speed tests? What is your stated bandwidth from your ISP? I suspect that there's something wrong with your network setup.

Xyne · 2009-01-14 18:24:59

I've changed a bit of code to display the database sync output as it's returned. This should prevent powerpill from appearing to be idle while waiting for pacman.

Last edited by Xyne (2009-01-14 18:25:25)

lilsirecho · 2009-01-14 18:55:27

Normal downloads occur at 88KB/sec.

If, sync is lost, the download speed is lost. The initial report I gave reported speeds of 40 KB/s with two downloading packages.

Today, with four packages downloading, the speed indicated for the downloada via aria2 registered 14Kb/sec. After resyncing the modem, the speed indicated was back to normal 88KB/s or more.

I suspect that the sync problem is inherent in the aria2 disconnects and timeouts in my Verizon interface.

I have to give the system another check while monitoring the sync which is in another room.

The time delay I posted is real time lost in waiting for... pacman startup..........(the step is posted near the beginning).

At this juncture, I feel that Verizon's adsl is incompatible with aria2.

The Verizon modem has three lights, all should be continuously lit during internet activity. If sync is lost (probable if downloads are interrupted by timeouts or disconnects) the modem flashes one of the leds to indicate loss. Most of the time this reduces download speed to one half of normal. In the case of throttled sources, this can be a disaster!!

My system is x86-64 on an intel core 2 duo Pentium 4 800mhz FSB.

I appreciate your efforts and will post again after verifying the sync loss or not............

lilsirecho · 2009-01-14 21:02:43

Repeated the experiment with powerpill after running pacman -Sy which gave no errors and verified all repo items as up to date.

Then ran pacman -Syu and did not enable the download of sqlite. This took about 20 secs.

Then I ran ...powerpill -Syu.... to investigate itas performance.

Since there was only one item, the pacman delay of a minute was reduced to a few seconds.

I monitored the sync of the modem while aria2 completed its activities. There was no loss of sync.

I note that average download speed was 9+KB/s and the maximum around 40MB/s.

The aria2 elements did not include any timeouts or disconnects which I suspect cause sync problems. I intend to try again with more packages involved.

The results of this experiment are below:

bash-3.2# pacman -Syu
:: Synchronizing package databases...
core is up to date
extra 380.4K 53.1K/s 00:00:07 [##################################################################] 100%
community is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for inter-conflicts...

Targets (1): sqlite3-3.6.9-1

Total Download Size: 0.39 MB
Total Installed Size: 1.00 MB

Proceed with installation? [Y/n] n
bash-3.2# powerpill -Syu
==> /root/.config/powerpill/powerpill.conf loaded
==> Checking permissions for given cache directories.
==> Invoking pacman to get URIs...
PACMAN: Starting...
PACMAN: :: Synchronizing package databases...
PACMAN: core is up to date
PACMAN: extra is up to date
PACMAN: community is up to date
PACMAN: :: Starting full system upgrade...
PACMAN: resolving dependencies...
PACMAN: Done
==> Added to download queue: sqlite3-3.6.9-1 (402.64 kiB)
==> About to download 1 file
==> Total download size: 402.64 kiB

Proceed? [Y/n] y
ARIA2: Starting...
ARIA2: 2009-01-14 12:45:45.457635 ERROR - CUID#5 - Download aborted. URI=http://archlinux.freeside.ru/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: Resource not found
ARIA2: 2009-01-14 12:45:48.753550 ERROR - CUID#6 - Download aborted. URI=ftp://ftp-stud.hs-esslingen.de/pub/Mirrors/archlinux/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: The response status is not successful. status=550
ARIA2: 2009-01-14 12:45:49.586697 ERROR - CUID#7 - Download aborted. URI=http://mirror.isoc.org.il/pub/archlinux/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: Resource not found
ARIA2: 2009-01-14 12:45:50.107964 ERROR - CUID#8 - Download aborted. URI=http://ftp.uni-kl.de/pub/linux/archlinux/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: Resource not found
ARIA2: 2009-01-14 12:45:53.444314 ERROR - CUID#9 - Download aborted. URI=ftp://ftp.gigabit.nu/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: The response status is not successful. status=550
ARIA2: 2009-01-14 12:45:57.587009 ERROR - CUID#10 - Download aborted. URI=ftp://mirror.pacific.net.au/linux/archlinux/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: The response status is not successful. status=550
ARIA2: 2009-01-14 12:46:00.228210 ERROR - CUID#11 - Download aborted. URI=ftp://ftp.hosteurope.de/mirror/ftp.archlinux.org/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: The response status is not successful. status=550
ARIA2: 2009-01-14 12:46:00.978214 ERROR - CUID#12 - Download aborted. URI=http://schlunix.org/archlinux/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: Resource not found
ARIA2: 2009-01-14 12:46:05.009709 ERROR - CUID#13 - Download aborted. URI=ftp://archlinux.c3sl.ufpr.br/archlinux/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: The response status is not successful. status=550
ARIA2: 2009-01-14 12:46:05.478904 ERROR - CUID#14 - Download aborted. URI=http://archlinux.mirrors.uk2.net/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: Resource not found
ARIA2: 2009-01-14 12:46:18.552475 ERROR - CUID#15 - Download aborted. URI=ftp://ftp.piotrkosoft.net/pub/mirrors/ftp.archlinux.org/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: The response status is not successful. status=550
ARIA2: 2009-01-14 12:46:18.705898 ERROR - CUID#16 - Download aborted. URI=http://mirrors.easynews.com/linux/archlinux/extra/os/x86_64/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Exception: Resource not found
ARIA2: 2009-01-14 12:46:27.983707 NOTICE - Verification finished successfully. file=/var/cache/pacman/pkg/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: 2009-01-14 12:46:27.983754 NOTICE - Download complete: /var/cache/pacman/pkg/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Download Results:
ARIA2: gid|stat|avg speed |path/URI
ARIA2: ===+====+===========+==========================================================
ARIA2: 1| OK| 9.3KiB/s|/var/cache/pacman/pkg/sqlite3-3.6.9-1-x86_64.pkg.tar.gz
ARIA2: Status Legend:
ARIA2: (OK):download completed.
ARIA2: Done.
==> SUCCESS: Checksum verified: sqlite3-3.6.9-1
==> 0.00 B downloaded and verified in 43 s.
==> Average rate: 0.00 B/s
==> Maximum rate: 44.52 kiB/s
==> Done.
==> Exiting powerpill and passing control to pacman...
PACMAN: Starting...
:: Starting full system upgrade...
resolving dependencies...
looking for inter-conflicts...

Targets (1): sqlite3-3.6.9-1

Total Download Size: 0.00 MB
Total Installed Size: 1.00 MB

Proceed with installation? [Y/n] y
checking package integrity...
(1/1) checking for file conflicts [####################################################] 100%
(1/1) upgrading sqlite3

It is probable that the download was from the throttled source arch linux.

lilsirecho · 2009-01-14 21:16:41

Continuing the experiment, I performed a download of three packages as shown in the following sequences:

bash-3.2# powerpill -S midori memtest86 openoffice-base
==> /root/.config/powerpill/powerpill.conf loaded
==> Checking permissions for given cache directories.
==> Invoking pacman to get URIs...
PACMAN: Starting...
PACMAN: This operation requires superusees... restarting with sudo...
PACMAN ERROR: /bin/bash: line 3: sudo: c found
PACMAN: Unable to perform the requested
==> Exiting...
bash-3.2# powerpill -Sy midori memtest86 openoffice-base
==> /root/.config/powerpill/powerpill.conf loaded
==> Checking permissions for given cache directories.
==> Invoking pacman to get URIs...
PACMAN: Starting...
PACMAN: This operation requires superuser privileges... restarting with sudo...
PACMAN ERROR: /bin/bash: line 3: sudo: command not found
PACMAN: Unable to perform the requestedoperation.
==> Exiting...
bash-3.2#
Operation in this computer is all root...no other users.

I assume I need to change some permissions...dunno where?

lilsirecho · 2009-01-14 23:26:33

What edit is required to permit root powerpill?

Xyne · 2009-01-14 23:39:05

I checked some of the URIs in your output and I was able to retrieve the resource. This may be an aria2 error introduced in the last update, or it may depend on your settings. If you've changed powerpill.conf, could you please post it?

Please update to the latest verion of powerpill. I've updated powerpill again to include a check for "sudo". Your last post indicates that you ran powerpill on a system without sudo installed. The error is not due to permissions settings per se, although running powerpill directly as root will circumvent the need to use sudo.

I understand what you were referring to now when you mentioned modem sync status. Normally though, you would have one led to indicate sync status, one to indicate a connection, and one to indicate data transfer. The data transfer led should blink during transfer while the others remain on. "Resource not found" errors should have absolutely no effect whatsoever on your adsl sync status, but the other way around is not true.

I thought of something else as well. Which mirror are you using as your primary mirror? If you're using archlinux.org, then that would explain the problem. archlinux.org is throttled to 50 kb/s to begin with, and because it's the main server off of which others sync, it will always be out of sync with the other servers by several hours on average. That's because a package may be added to the main server at any time but all mirrors usually only sync at a different set times during the day.

*edit*
Please use the code tags when you post code. This will make your code and your entire post more legible.

example

Last edited by Xyne (2009-01-14 23:43:51)

lilsirecho · 2009-01-15 00:00:01

As one of my posts reports I have upgraded powerpill to 1-15 I recall.

Do I need to perform gen-conf again?

I cannot run powerpill now as is indicated on previous post.

I probably do have archlinux server as first mirror, but haven't checked it as yet.

I surmise that the slowdown could occur any time I run into a throttled mirror. Thus I may need to realign the mirror status at every instance of powerpill utilization. I have no clue as to which mirrors are synched to the new releases nor the throttled status as well.

Why at this time after performing the pacman -Sy and installing the latest upgrade via yaourt I am not able to perform a package download as root? I was able to do so before the upgrade.

I was performing a test of multiple packages for the purpose of determining if the download rate decreased with number of packages. This test has not responded because root is not accepted.

My download speed as reported by speedtest is 768kb/s. Upload is 124kb/s.

Do I need to download powerpill again this day via yaourt?

Edit: The download indicators on my computer do blink as you mention, however the Verizon adsl modem has three LEDS and all are green when everything is in sync...the sync led blinks once a second when sync is lost but is steady at all times when sync is maintained.

Last edited by lilsirecho (2009-01-15 00:03:21)

Xiong Chiamiov · 2009-01-15 00:29:19

lilsirecho wrote:

I surmise that the slowdown could occur any time I run into a throttled mirror. Thus I may need to realign the mirror status at every instance of powerpill utilization. I have no clue as to which mirrors are synched to the new releases nor the throttled status as well.

AFAIK, the official archlinux.org mirror is the only one that is capped, at least as far as it will affect you.

You can check the sync status of the mirrors via this page. In addition, Xyne has created an application called Reflector that creates a mirrorlist of the fastest up-to-date mirrors, which works well with powerpill.

In addition, you may find the --silent-aria flag for powerpill useful, as it suppresses all of the "Resource not found" errors.

lilsirecho · 2009-01-15 00:31:29

Rearranged the mirrors as you suggested...performed --gen-conf and overwrote old conf.

Retried to download three packages and received the sudo error again.

My powerpill is 15-1 from yaourt.

Arch Linux

#76 2008-12-17 23:41:42

Re: Powerpill - A wrapper and download accelerator for Pacman

#77 2008-12-18 00:05:48

Re: Powerpill - A wrapper and download accelerator for Pacman

#78 2008-12-18 07:16:45

Re: Powerpill - A wrapper and download accelerator for Pacman

#79 2008-12-18 14:23:06

Re: Powerpill - A wrapper and download accelerator for Pacman

#80 2008-12-18 15:06:54

Re: Powerpill - A wrapper and download accelerator for Pacman

#81 2008-12-19 08:41:57

Re: Powerpill - A wrapper and download accelerator for Pacman

#82 2008-12-19 08:54:32

Re: Powerpill - A wrapper and download accelerator for Pacman

#83 2008-12-19 22:39:42

Re: Powerpill - A wrapper and download accelerator for Pacman

#84 2008-12-20 01:42:35

Re: Powerpill - A wrapper and download accelerator for Pacman

#85 2009-01-07 22:57:26

Re: Powerpill - A wrapper and download accelerator for Pacman

#86 2009-01-07 23:07:23

Re: Powerpill - A wrapper and download accelerator for Pacman

#87 2009-01-07 23:09:59

Re: Powerpill - A wrapper and download accelerator for Pacman

#88 2009-01-08 00:00:50

Re: Powerpill - A wrapper and download accelerator for Pacman

#89 2009-01-14 12:54:42

Re: Powerpill - A wrapper and download accelerator for Pacman

#90 2009-01-14 17:04:50

Re: Powerpill - A wrapper and download accelerator for Pacman

#91 2009-01-14 17:49:36

Re: Powerpill - A wrapper and download accelerator for Pacman

#92 2009-01-14 18:24:59

Re: Powerpill - A wrapper and download accelerator for Pacman

#93 2009-01-14 18:55:27

Re: Powerpill - A wrapper and download accelerator for Pacman

#94 2009-01-14 21:02:43

Re: Powerpill - A wrapper and download accelerator for Pacman

#95 2009-01-14 21:16:41

Re: Powerpill - A wrapper and download accelerator for Pacman

#96 2009-01-14 23:26:33

Re: Powerpill - A wrapper and download accelerator for Pacman

#97 2009-01-14 23:39:05

Re: Powerpill - A wrapper and download accelerator for Pacman

#98 2009-01-15 00:00:01

Re: Powerpill - A wrapper and download accelerator for Pacman

#99 2009-01-15 00:29:19

Re: Powerpill - A wrapper and download accelerator for Pacman

#100 2009-01-15 00:31:29

Re: Powerpill - A wrapper and download accelerator for Pacman

Board footer