[SOLVED] Powerpill fails to download multiple packages (aria2c)

hso · 2010-04-29 03:59:12

Hey, everyone!

I'm having a hard time trying to install archlinux in my laptop: I installed the base system just fine, but when installed powerpill to make things faster, it failed to download a lot of packages. Every time the failed packages were the bigger ones. Installing using pacman alone wasn't an option 'cause I have a flaky connection at home and it's very slow.

I tried:

- Using different mirrors
- Connecting from elsewhere
- At home, switching between LAN and WLAN

But nope. It downloaded just fine, but when verifying the checksums it always failed in the very last bit of the file. I had to re-download the whole package.

Finally, I checked powerpill's config file and, in the newer versions there's an argument passed to aria2c:

file-allocation=none

So i changed it to

file-allocation=prealloc

just for fun. And now it just fails less often, but still fails both checksumming and sometimes just in the middle of a download (maybe the latter is just my fscking internet connection anyway ).

Very odd.

Anybody had the same problem?

My hardware:
- Dell XPS M1530
- Intel Core 2 Duo T5750
- Ethernet controller: Marvell Technology Group Ltd. 88E8040 PCI-E Fast Ethernet Controller (rev 12)
- Network controller: Intel Corporation PRO/Wireless 3945ABG [Golan] Network Connection (rev 02)

Last edited by hso (2010-04-29 13:15:25)

Xyne · 2010-04-29 04:40:23

Without seeing the output I don't know exactly what is causing the problem. When you say that the verification step failed, which one do you mean? aria2c has its own built-in verification, powerpill double-checks the files as well and then pacman checks them before actually installing them. Does it tell you that a download has failed?

If you try to install one of the failed downloads with "pacman -S <pkgname>", what happens?

Without knowing more I recommend that you try the following:

*) make sure you're using an up-to-date mirror by checking the Mirror Status page
*) force a database refresh with "pacman -Syy" (or "powerpill -Syy") to make sure you have the correct files.
*) use Powerpill's Aria2Args option to enable aria2c logging ("log=/path/to/log", "log-level=debug") and use the "Log" option to enable Powerpill logging
*) try the download again

*) check the logs to see if you notice anything strange and post them if you can (e.g. save them to a usb stick or a shared disk|partition)

*) play around with aria2c arguments (e.g. "ftp-pasv=false") to see what happens

*) save the download metalink using "bauerbill [options & arguments] --save-metalink /path/to/metalink" and then try the download directly in aria2c with "aria2c -M /path/to/metalink" and report what happens

If it sometimes fails in the middle of a download it's almost certainly a problem with a particular mirror or a bug in aria2c. The aria2c logs should indicate which.

hso · 2010-04-29 05:51:21

Thank you for your response.

If I try to install a failing download this happens:

[root@overlord ~]# powerpill --log /home/hso/powerpill.log -S neverball

:: The following packages will be downloaded:
   [extra]
     neverball-1.5.4-2

:: Number of Packages:               1
:: Total Download Size:      51.52 MiB
:: Total Installed Size:    187.84 MiB

:: Proceed with download? [Y/n] 
--> Reflecting... retrieved 0 mirrors.

[#1 SIZE:51.5MiB/51.5MiB(100%) CN:0] [Checksum:#1 23.0MiB/51.5MiB(44%)]                               
:: 1 unsuccessful download(s). Would you like to retry? [Y/n] n

If I choose to retry, it downloads the entire file again and fails. Notice that it fails verifying the checksum. From the debugging log of aria2c:

2010-04-29 01:27:02.765522 INFO - Se ha completado la descarga.
2010-04-29 01:27:02.765660 INFO - CUID#3 - Dispatching CheckIntegrityCommand CUID#89.
2010-04-29 01:27:03.272498 DEBUG - CUID#7 - socket: read:0, write:0, hup:0, err:0
2010-04-29 01:27:03.272603 DEBUG - CUID#19 - socket: read:0, write:0, hup:0, err:0
2010-04-29 01:27:03.272637 DEBUG - CUID#23 - socket: read:0, write:0, hup:0, err:0
2010-04-29 01:27:03.272669 DEBUG - CUID#88 - socket: read:0, write:0, hup:0, err:0
2010-04-29 01:27:03.850749 ERROR - Se ha detectado un error en la suma de comprobación. fichero=/var/cache/pacman/pkg/neverball-1.5.4-2-i686.pkg.tar.gz

I've posted the full log on http://pastebin.com/eGemsQfv.

If I try to install a failed download with "pacman -S <pkgname>" it normally succeeds. From time to time, the download is interrupted. But if I try again, pacman resumes the download and there is never a checksum error.

If I try to install a failed download with "powepill -S <pkgname>" it normally fails, offering to retry the download, which fails again and so on. Sometimes, the download succeeds. But this is very rare!

I've followed most of your recommendations:
*) The mirror I'm using is up to date (I've tried many different mirrors also)
*) I've forced a database refresh
*) Enabled Aria2c debug logging and powerpill logging (powerpill doesn't seem to write the log, though)
*) Tried the download again and again
*) I'm not familiar with aria2c logs, so I didn't notice something strange. Maybe this is strange: Just before the download finished, this begun to show a lot:

2010-04-29 01:27:02.268634 INFO - CUID#31 - No segment available.
2010-04-29 01:27:02.268649 DEBUG - CUID#31 - Reusing CreateRequestCommand

*) I've not played with aria2c arguments yet (mostly because i'm using http mirrors exclusively)
*) I've not tried to download directly in aria2c using the metalink (It's too late and I just want to get some sleep ). I'll do it tomorrow if I can.

Again, thank you very much for taking the time to read this!

Last edited by hso (2010-04-29 06:00:00)

stryder · 2010-04-29 05:53:12

For me recently it kept on asking to retry but upon retrying it started from scratch for 1 file (go-openoffice). I had to cancel after two rounds. Pacman found the file to be corrupt and downloaded a fresh copy and it was OK. Another time it kept on asking to retry but downloaded nothing more. Again pacman found a couple of corrupt files and downloaded fresh copies with no problems.

hso · 2010-04-29 06:03:01

Yes. pacman downloads the packages with no issues but it is very, very slow and I need to download 400MiB+ for yesterday

Last edited by hso (2010-04-29 06:04:03)

Xyne · 2010-04-29 07:15:24

Install "perl-crypt-ssleay" if you haven't done so already (pacman -S perl-crypt-ssleay). From the aria2c log it seems that you're only trying to use a single mirror which may be causing the problem. When you run powerpill you should see a message about retrieving mirrors with reflector and then how many it retrieved (45 by default). Try the download again with "perl-crypt-ssleay" installed.

Make sure that you have the latest versions of "aria2" and "perl-xyne-arch" too (pacman -S perl-xyne-arch aria2).

Also try adding the following aria2c arguments to Aria2Args if it still doesn't work: "max-tries=15", "always-resume=false" (separately and|or together)

The problem is with aria2c (as shown by the log) so beyond argument and configuration settings the issue is out of my hands. Sorry that I can't be more useful. I know how frustrating things like this can be when you just want to get something done.

Last edited by Xyne (2010-04-29 07:16:04)

hso · 2010-04-29 12:43:07

Hi!

I tried everything you told me, but it still doesn't work.

I have an arch machine at work wich I haven't updated in a while. Just tested downloading some packages in it and it downloaded just fine.

Btw, aria2-1.9.2-1 is out. I'll try it. If it still doesn't work, I'll downgrade it to 1.8.1 (the old package in my other machine) and tell you how it goes.

Note: I'm behind a squid proxy and reflector doesn't fetch anything from https://www.archlinux.de/?page=MirrorStatusReflector. So, I couldn't try reflector.

hso · 2010-04-29 13:12:36

Update: It does work now!

I just downgraded aria2 from 1.9.2-1 to 1.8.2-1 and tested it a bit and it seems that everything is fine (upgrading aria2 didn't solved my problem).

The only thing that bothers me is running old software. This kind of problems shouldn't happen. But I couldn't give up on arch, I love it so much! I'm using it since 2007 for getting things done and *never* had a real problem. Thank you all!

jdarnold · 2010-04-29 13:26:09

I too have had similar problems with aria2c, esp. on one machine. Maybe I'll see if downgrading helps me.

Xyne · 2010-04-30 02:20:01

If an older version works then maybe a bug was introduced after that version. I would recommend contacting the Aria2 developer about this. From my own experience and what I've seen elsewhere, he's friendly and quick to reply.

It may be something as simple as a change to a default option.

Natanji · 2010-07-22 07:01:25

Sorry if I kinda necro this, but I also have this problem. It basically happens everytime that a download is canceled in between. The package file will then lie half-downloaded in /tmp/bauerbill/build/ (or the pacman cache directory for non-AUR packages), and whenever aria2 resumes the download, it will return an error and fail the download. Only deleting the package manually will fix this. It's very annoying... could you at least add an option that automatically deletes the downloaded package if an error occured?

Xyne · 2010-07-22 14:33:56

Do you want those files to be deleted before it retries the download or when it exits?

Deleting them before retrying would prevent resumption of downloads. Normally the aria2 control files are deleted on exit but I could add code to delete the incomplete files as well, plus there is at least one exit case in bauerbill in which those files are not cleaned up.

What error are you getting? Even if I can work around it by moving files around, this is actually a server error (or aria2c error).

ngoonee · 2010-07-22 23:58:31

I think basically when a fail comes out, you may want to attempt a cleanup of the files. Depends on the fail as well, I guess if the checksum verification fails when the file is supposed to be fully download it would make sense to rm the file and the .aria2. Not for other fails like network connectivity.

Natanji · 2010-07-23 14:36:53

For me what happens is this:
-aria2 half-downloads the file, failing somewhere in the middle
-Next time, aria2 attempts to re-download the file, resuming it where it left off
-The checksum check will fail at the end of download
-Next time, aria2 will attempt to re-download again from the same point where it left off the very first time
-Of course this loops till infinity and beyond until I manually delete the file

So I believe the best idea would be to allow the user to decide if he wants to resume the download when there is a half-downloaded file already. If that is not possible, adding an option to bauerbill that removes all files with a download error on exit would be cool.

Xyne · 2010-07-24 15:35:21

The expected behavior is to attempt resumption of incomplete downloads when retrying and to remove incomplete files on exit. I've updated the cleanup functions (which may have contained a bug and missed some files) and I've also added a hook in bauerbill to run them when choosing to abort the operation after a failed download.

The cleanup should do the following:
*) remove all *.aria2 files in the cachedir
*) remove the matching packages as a leftover aria2 control file indicates an aborted download
*) remove all source files that have not been verified
*) remove all aria2 control files corresponding to the removed source files

The user will be prompted to remove completed downloads that do not pass the integrity check. Even if they remain, re-running the operation should overwrite them.

Last edited by Xyne (2010-07-24 15:36:28)

joetotale · 2010-08-28 16:37:22

Xyne wrote:

Install "perl-crypt-ssleay" if you haven't done so already (pacman -S perl-crypt-ssleay). From the aria2c log it seems that you're only trying to use a single mirror which may be causing the problem. When you run powerpill you should see a message about retrieving mirrors with reflector and then how many it retrieved (45 by default). Try the download again with "perl-crypt-ssleay" installed.

That helped me out. Cheers Xyne.

Arch Linux

#1 2010-04-29 03:59:12

[SOLVED] Powerpill fails to download multiple packages (aria2c)

#2 2010-04-29 04:40:23

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#3 2010-04-29 05:51:21

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#4 2010-04-29 05:53:12

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#5 2010-04-29 06:03:01

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#6 2010-04-29 07:15:24

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#7 2010-04-29 12:43:07

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#8 2010-04-29 13:12:36

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#9 2010-04-29 13:26:09

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#10 2010-04-30 02:20:01

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#11 2010-07-22 07:01:25

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#12 2010-07-22 14:33:56

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#13 2010-07-22 23:58:31

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#14 2010-07-23 14:36:53

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#15 2010-07-24 15:35:21

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

#16 2010-08-28 16:37:22

Re: [SOLVED] Powerpill fails to download multiple packages (aria2c)

Board footer