pacman 3.3.1-1 big part files as hell

Spy · 2009-09-28 16:52:23

Hi. When pacman download some packages (kde-svn) and get file fransfer error then *.part files are extremly big even 900Mb. So I get 0mb free space... Somebody have similar issue ?

svoufff · 2009-09-28 17:34:33

Yes, the same happens here since yesterday.On packages from extra or kdemod repos, and with different mirrors.It takes a few seconds to fill up 6GB of hd space then kde crashes.I'm on arch64.

milomouse · 2009-09-28 17:41:05

Not sure if I had the same exact problem, but since I upgraded pacman (on x86_64), I went ahead and did a "Syu" and it had about 6 packages to upgrade, so I committed by typing "Y", and after a couple packages downloaded I suddenly had 0% left on my root partition (which had 1.5GB free before the "Syu") and nothing else was being downloaded so I went ahead and canceled the upgrade by typing Ctrl+C. After that, I cleaned my pacman database ("yaourt -Scc") and I got my 1.5GB back. I thought this was weird and did the "Syu" again, but starting doing the ugprades one at a time instead of all at once, and had no problem then. This never happened before.

rson451 · 2009-09-28 18:09:36

Are all of you using yaourt by chance? I've not had any issues with pacman.

svoufff · 2009-09-28 18:46:49

This is with pacman for me.

Spy · 2009-09-28 19:44:57

shit I can not do full system update because kde crash when disk is full. Hope this will be fixed soon....

rson451 · 2009-09-28 19:57:42

Has anyone filed a bug report? If not, and you're convinced it's a pacman issue, I'd suggest you do that.

shining · 2009-09-28 20:04:03

http://bugs.archlinux.org/task/16359 ?

Why did so many people start having this issue now ? According to the comments in the bug report, the problem happens since 3.3.0. But many users are only reporting it now. Strange..

If you could all tell me, which mirror you are using, and if you can reproduce the problem with 3.3.0, it would be helpful.

ngoonee · 2009-09-28 22:57:44

I had it with 3.3.0. The problem goes away when I change mirrors. Just got 3.3.1 so I don't know whether it persists.

I'll keep this in mind and post the time/mirror which causes an error if I get it again.

schivmeister · 2009-09-28 23:13:49

shining: Perhaps one of the previous libfetch updates? Because those who are on stable cable are not facing this.

It's not that you change mirrors and you don't see this again - it's just that when you changed mirrors you fortunately ended up with a more stable connection than the previous. As for me, my connection is inherently unstable so changing mirrors does not help. It also does not depend on the file size, it depends on whether the download is within the period of flaw. If you have a very slow unstable connection, if downloading a 10MB file takes long enough, and an interruption occurs, you're doomed.

Anyway, use wget in pacman.conf and see if that does not solve this. Cross-posting from flyspray:

This must be an issue with the default fetching utility, because using wget as the downloader proves to be fine and dandy. Some debunking though:
"In my case the problem is not every X packages, seems to occur only with the big ones (kernel from testing today or go-openoffice days ago). I mean, not always happens, most of the times all is OK.
I guess (I'm not sure) the problem has something to do with losing server connection or so. Pacman continues filling the file without receive and EOF."
Nope. Any package, small or big (depending on your connection speed). Your network is unstable, that is yet another cause of this problem. Packets can stop abruptly and you won't notice it. Usually, download utilities will resume as soon as the packets start flowing again. However, in this case, that's not happening.
What's happening is once the network is interrupted, libfetch (or pacman's downloading function around it) is not aware and continues downloading bogus data. The file.part size grows in the magnitude of around 1-100MB every second (a guess) regardless of your connection speed. Even if it struck at 1KB, you'd end up with a file of 1G and still growing! That is, of course, until you ^C the process.
When this happens with wget, the download speed slowly decreases to 0 and then NIL (--:--). Upon reconnection, it simply resumes from where it was interrupted.
Workaround: XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u

Say, are you all using ext4?

Last edited by schivmeister (2009-09-28 23:16:53)

descendent87 · 2009-09-29 00:28:57

Just done a fresh install and I've got the same problem, tried to install tomboy (total download size including deps was 50mb) but after a few seconds it had filled / which had 7.8GB free before. Had a look in the cache and apparently tomboy was 8GB.
Using ext4 here

rson451 · 2009-09-29 00:33:03

What mirror? That seems to be an important peice of info here.

descendent87 · 2009-09-29 00:35:24

ftp://mirror.lividpenguin.com/pub/archlinux/

ngoonee · 2009-09-29 00:49:38

I'd agree with schivmeister, his hypotheses seems to support what I've been seeing, except that, once I've failed on one file with a particular mirror, deleting the .part file and restarting the download gives me the same problem unless I change the mirror. This could have to do with my ISP's (rather, my university's connection) caching stuff?

If this comes up again I'll try using wget. Where do I set pacman to use that?

schivmeister · 2009-09-29 05:31:31

ngonee: you mean deleting the file and restarting the download gives you the same problem at the exact location where it happened before? or do you mean it doesn't even start? If the latter, then it's yet another issue. But if it's the former, then it's correct because that means the connection can only remain stable up to that point.

Btw, to help ourselves, you must reproduce this by brute force and simulate an interruption; disconnect. Do anything but CTRL+C; disconnect from the network applet, kill your dialer, kill networkmanager, bring down the interface, or simply plug out your device/cable. Then reconnect asap, after which waste no time in issuing a ^C.

The expected behaviour is that the progress "freezes" but the file should not grow. Upon resuming, pacman should warn about corrupted file.

If you feel up to it, you can repeat this for every official mirror. And then, try the same with wget.

# /etc/pacman.conf:
...

SyncFirst   = pacman
XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u

Last edited by schivmeister (2009-09-29 05:41:55)

lumpy211 · 2009-09-29 13:49:49

I've got the same issue on my XPS M1330. I'm not using ext4, and it seems to have started happening just after I upgraded pacman yesterday. My 7 GB free space on my system partition fills up in a matter of a minute or so.

I'm trying now with the wget workaround, and that seems to be working, though it's not finished yet.

Update: Worked great, and df is only reporting a change of 2% disk usage. Thanks

Last edited by lumpy211 (2009-09-29 14:04:44)

Spy · 2009-09-29 15:31:40

I using http://pkg.eth-os.org/kde/svn/i686 repo for kde-svn updates. Because of this bug/shit cannot download updates.I am pissed....

shining · 2009-09-29 17:33:46

I figured out the bug, it was pretty stupid : http://bugs.archlinux.org/task/16359

But you still have to figure out why your network is so unstable and breaks all the time

pacman 3.3.2 should exit properly and display an error instead of filling up your disk

schivmeister · 2009-09-29 18:11:54

Naah, the network problems don't matter here. What matters is the absurd file size!

Well, patched 3.3.1, all fine and dandy once again

lumpy211 · 2009-09-29 20:31:01

I also added the patch and it compiled fine on Arch64. Now I guess I'll just wait until a bunch of packages are updated to see if it worked since I'm already all up to date otherwise. Thanks for the patch shining

Update: I've uploaded a patched Arch64 version to my webspace temporarily:
http://www.math.purdue.edu/~krotz/pacma … pkg.tar.gz
Just install through pacman.

Last edited by lumpy211 (2009-09-29 20:36:24)

harbind · 2009-09-29 20:57:16

Thanks lumpy221, it worked!

brendan · 2009-09-30 19:24:18

your patch says requires libdownload>=1.3

I cant seem to find that in the repos

shining · 2009-09-30 19:30:36

brendan wrote:

your patch says requires libdownload>=1.3
I cant seem to find that in the repos

Where does it say that ?

We are using libfetch now. The bug was also a consequence of the libdownload -> libfetch move.

brendan · 2009-09-30 19:33:37

shining wrote:

brendan wrote:
your patch says requires libdownload>=1.3
I cant seem to find that in the repos
Where does it say that ?
We are using libfetch now. The bug was also a consequence of the libdownload -> libfetch move.

I intalled libfetch from core but it says that when I do pacman -U thenewpacman.tarball

toofishes · 2009-09-30 23:12:27

brendan wrote:

your patch says requires libdownload>=1.3
I cant seem to find that in the repos

$ pacman -Qip /tmp/pacman-3.3.1-1-x86_64.pkg.tar.gz 
Name           : pacman
Version        : 3.3.1-1
URL            : http://www.archlinux.org/pacman/
Licenses       : GPL  
Groups         : base  
Provides       : None
Depends On     : bash  libarchive>=2.6.0  libdownload>=1.3  pacman-mirrorlist  
...

Actually his patch says nothing about libdownload, this is just a broken package that someone built. You should probably find out what PKGBUILD he used...

Arch Linux

#1 2009-09-28 16:52:23

pacman 3.3.1-1 big part files as hell

#2 2009-09-28 17:34:33

Re: pacman 3.3.1-1 big part files as hell

#3 2009-09-28 17:41:05

Re: pacman 3.3.1-1 big part files as hell

#4 2009-09-28 18:09:36

Re: pacman 3.3.1-1 big part files as hell

#5 2009-09-28 18:46:49

Re: pacman 3.3.1-1 big part files as hell

#6 2009-09-28 19:44:57

Re: pacman 3.3.1-1 big part files as hell

#7 2009-09-28 19:57:42

Re: pacman 3.3.1-1 big part files as hell

#8 2009-09-28 20:04:03

Re: pacman 3.3.1-1 big part files as hell

#9 2009-09-28 22:57:44

Re: pacman 3.3.1-1 big part files as hell

#10 2009-09-28 23:13:49

Re: pacman 3.3.1-1 big part files as hell

#11 2009-09-29 00:28:57

Re: pacman 3.3.1-1 big part files as hell

#12 2009-09-29 00:33:03

Re: pacman 3.3.1-1 big part files as hell

#13 2009-09-29 00:35:24

Re: pacman 3.3.1-1 big part files as hell

#14 2009-09-29 00:49:38

Re: pacman 3.3.1-1 big part files as hell

#15 2009-09-29 05:31:31

Re: pacman 3.3.1-1 big part files as hell

#16 2009-09-29 13:49:49

Re: pacman 3.3.1-1 big part files as hell

#17 2009-09-29 15:31:40

Re: pacman 3.3.1-1 big part files as hell

#18 2009-09-29 17:33:46

Re: pacman 3.3.1-1 big part files as hell

#19 2009-09-29 18:11:54

Re: pacman 3.3.1-1 big part files as hell

#20 2009-09-29 20:31:01

Re: pacman 3.3.1-1 big part files as hell

#21 2009-09-29 20:57:16

Re: pacman 3.3.1-1 big part files as hell

#22 2009-09-30 19:24:18

Re: pacman 3.3.1-1 big part files as hell

#23 2009-09-30 19:30:36

Re: pacman 3.3.1-1 big part files as hell

#24 2009-09-30 19:33:37

Re: pacman 3.3.1-1 big part files as hell

#25 2009-09-30 23:12:27

Re: pacman 3.3.1-1 big part files as hell

Board footer