Full disk during pacman download causes system hang

B-Con · 2010-04-18 23:36:17

Recently I had a lot of media stored on my root drive when I did a pacman -Syu. I only had about 200MB free and the downloaded packages filled it up completely. In what appears to account for some of my crashing woes, upon filling my hard disk space pacman froze and my entire system hung. (For what it's worth, I have no swap partition.)

Is that reasonable behavior? Yes, I know my active partition should not be filled up entirely, but is hanging my system a reasonable behavior? I've accidentally filled it up before while manually copying large media and my system didn't freeze. Why did it freeze when pacman was downloading packages?

Just looking for feedback before I file a pacman bug.

[edit]
Whoops, maybe this belongs in the pacman forum.

Last edited by B-Con (2010-04-18 23:37:15)

demian · 2010-04-19 01:00:22

It's sounds like a hardware thing to me, considering what you write in your "crashing"-thread. You should check the temperatures of your system, the s.m.a.r.t. values of your hard drive and run memtest86+ (again) over night.
Neither disk nor swap space isn't exactly ideal though, especially if you pair it with full memory. If there's no way for the kernel to free memory it should start to randomly kill processes. Memory usage spikes in that situation, i believe, can freeze your system.
Still, I'd thourougly check the hardware.

Regards,
demian

Last edited by demian (2010-04-19 01:01:48)

Allan · 2010-04-19 01:07:26

Did pacman actually freeze? I have never seen that happen before in this case. What I have seen is pacman extracting empty files so you could be missing or have corrupted important libraries. Id "pacman -S" through the list of packages that were in the update (perhaps -Sf will be needed).

There is a bug report about pacman's behaviour during an update when the system runs out of drive space. Unfortunately, there is not a lot that can be done about it as checking "/" is not enough given the variety of partition layouts that are possible.

keenerd · 2010-04-19 03:06:19

Allan wrote:

Unfortunately, there is not a lot that can be done about it as checking "/" is not enough given the variety of partition layouts that are possible.

If only Arch linux had some way of configuring programs. Maybe as text files, placed in a common directory. I know that is asking a lot. Would a voice activated GUI be easier?

Allan · 2010-04-19 03:33:20

Pure genius.... please continue describing how you would solve this issue so we can learn from your wisdom.

B-Con · 2010-04-19 04:30:34

demian wrote:

It's sounds like a hardware thing to me, considering what you write in your "crashing"-thread. You should check the temperatures of your system, the s.m.a.r.t. values of your hard drive and run memtest86+ (again) over night.
Neither disk nor swap space isn't exactly ideal though, especially if you pair it with full memory. If there's no way for the kernel to free memory it should start to randomly kill processes. Memory usage spikes in that situation, i believe, can freeze your system.
Still, I'd thourougly check the hardware.
Regards,
demian

I ran memtest and it reports no errors. It hasn't had any history of overheating. I don't bother with swap space because I have 3GB of RAM and I user 250MB after I've booted and loaded all my applications aside from Firefox. I can't remember the last time I actually used more than 40% of my RAM, don't know that I have. I keep an eye on it through Conky. Lack of memory definitely wasn't the problem.

"smartctl -a /dev/sda" doesn't show any issues.

  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
<snip>
5 Reallocate  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
d_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0

I'd be ready to point the finger at hardware as well if not for the fact that not a single crash hasn't happened under some unusual circumstance, half of which have included -Syu upgrades, not that I think pacman has been the common culprit.

Allan wrote:

Did pacman actually freeze? I have never seen that happen before in this case. What I have seen is pacman extracting empty files so you could be missing or have corrupted important libraries. Id "pacman -S" through the list of packages that were in the update (perhaps -Sf will be needed).
There is a bug report about pacman's behaviour during an update when the system runs out of drive space. Unfortunately, there is not a lot that can be done about it as checking "/" is not enough given the variety of partition layouts that are possible.

Well, the system became completely unresponsive and the screen froze. I couldn't get X to respond to any standard kill shortcuts. I didn't have SSH setup so I couldn't try to start another login session. I suppose it's conceivable that the problem was purely with X and pacman did whatever it's supposed to do when it can't write to the disk, but I wouldn't be able to tell because I could only see what was last output to the terminal when my screen froze.

The bug report thread doesn't seem to report behavior similar to what I observed, but it seems to be a little old. Don't know if anything has changed since then.

[edit]
In the bug report someone mentioned that a symptom was that pacman used up 100% CPU for a while. I don't think this was the case for me because I have a dual core CPU and I think pacman would get 1 full core at best, so I should have still been functional even if it ran haywire with CPU. Second, I waited almost a full minute and nothing improved, which seems like complete deadlock or some sort, not just innocent CPU starvation.

Last edited by B-Con (2010-04-19 04:49:12)

keenerd · 2010-04-19 12:39:42

Allan wrote:

Please continue describing how you would solve this issue so we can learn from your wisdom.

No wisdom here, it is just painful to see really smart people use really dumb reasons.

Personally, I don't know how I would solve it. There are too many ways to choose from, and my knowledge of the code is not solid enough to pick the laziest implementation. (I will patch it eventually, it is just slow now that my time is divided between pacman and pacmatic.) Standing on the shoulders of the bug reports and rehashing others' solutions: Check for free space on a user specified partition. Copy an equivalent amount of zeros, see if it fails, delete the zeros, download/untar for real. Use per-file hashes*. Measure the size of a downloaded/extracted files before starting the next one. Extract files to /full/path/file.temp, then move to /full/path/file. Check the return status of the download/untar call.

* This is my preferred answer, because it is the easiest for an outsider (adding new code usually is), sounds the hardest to do (fun!), fixes four other unrelated problems and gets me working on the makepkg code as well.

Mr.Elendig · 2010-04-19 13:07:38

B-Con wrote:

wall of text

I believe that pacman shouldn't be able to freeze your system if the partion gets full in the download step, but if it happens during the package install step, bad things(tm) might happen:

http://bugs.archlinux.org/task/11639
Edit: ^ is the bug report Allan mentioned.

Last edited by Mr.Elendig (2010-04-19 13:09:11)

Allan · 2010-04-19 13:31:51

keenerd wrote:

Allan wrote:
Please continue describing how you would solve this issue so we can learn from your wisdom.
No wisdom here, it is just painful to see really smart people use really dumb reasons.
Personally, I don't know how I would solve it. There are too many ways to choose from, and my knowledge of the code is not solid enough to pick the laziest implementation. (I will patch it eventually, it is just slow now that my time is divided between pacman and pacmatic.) Standing on the shoulders of the bug reports and rehashing others' solutions: Check for free space on a user specified partition. Copy an equivalent amount of zeros, see if it fails, delete the zeros, download/untar for real. Use per-file hashes*. Measure the size of a downloaded/extracted files before starting the next one. Extract files to /full/path/file.temp, then move to /full/path/file. Check the return status of the download/untar call.
* This is my preferred answer, because it is the easiest for an outsider (adding new code usually is), sounds the hardest to do (fun!), fixes four other unrelated problems and gets me working on the makepkg code as well.

Wow... some actual ideas.

My comment about checking for free space not being simple is directed at the usual response given of "Check for free space on a user specified partition" which happens to be the most suggested solution and was your first suggestion above. As I said, that is near impossible because we know nothing about the filesystem and its mount point layout. There is not consistent configuration file for this across platforms (think mingw32) and sometimes even fstab is meaningless (chroots). I was using pacman on a Hurd system with an fstab file for a while when I manually mounted my drives.

Per file hashes just detects the issue after the fact, but does not prevent it. As does checking the return code (I believe pacman-git does something like this and aborts). Using files of zeros detect the issue before the package is extracted, but aborting in an upgrade situation will result in the package being updated not being on the system (which may be worse than a package partially extracted on a system). It would work find in a pure install situation though so that would be a start. Extracting files and then moving them requires around twice the space so makes the issue worse.

I hope this illustrates that there is not an easy solution to this issue. I'd be interested to see how other package managers handle this situation if someone has time to investigate.

keenerd · 2010-04-19 14:47:06

Allan wrote:

"Check for free space on a user specified partition" ... that is near impossible because we know nothing about the filesystem and its mount point layout.

This just does not jive. The user knows everything about his layout. The user specifies (in pacman.conf) which mountpoint listed in "df" is the relevant one. By default, check /. That covers 95% of installs. If you are feeling like a BOFH, set the default to disable the check, aka "traditional" mode. If the user is smart enough to do an offbeat layout, they are smart enough to update one line in pacman.conf. Trying to automagically figure it out from fstab defeats the whole purpose of a config file.

Allan wrote:

[All suggestions] just detect the issue after the fact, but do not prevent it.

You can not prevent it. Ever. Linux is not perfectly atomic with guaranteed transactions. (There are a few in linux, but not enough.) No matter what system you employ, I could defeat it by creating massive files while you write your files. I am a fool, specifically the "bigger fool" who breaks the foolproof.

100% prevention is impossible. 95% is (comparatively) easy. Why do we not even try half heartily?

Allan wrote:

Extracting files and then moving them requires around twice the space so makes the issue worse. ... I'd be interested to see how other package managers handle this situation if someone has time to investigate.

It does not make the situation worse, just apparent sooner. It is the only method which won't leave half installed (or missing) packages floating around. It is as close to total prevention as we can reach. IIRC, this is what other (debian) package managers do.

B-Con · 2010-04-19 17:15:45

Mr.Elendig wrote:

I believe that pacman shouldn't be able to freeze your system if the partion gets full in the download step, but if it happens during the package install step, bad things(tm) might happen:
http://bugs.archlinux.org/task/11639
Edit: ^ is the bug report Allan mentioned.

It happened during the install stage, which scared me, but it didn't seem to cause any problems as far as I could tell. After I rebooted and freed up some disk space I did a -Su to finish the upgrade and it seemed to finish fine.

Are pacman database updates committed before or after the tarbell is extracted to the root directory?

Mr.Elendig · 2010-04-19 17:31:49

B-Con wrote:

Mr.Elendig wrote:
I believe that pacman shouldn't be able to freeze your system if the partion gets full in the download step, but if it happens during the package install step, bad things(tm) might happen:
http://bugs.archlinux.org/task/11639
Edit: ^ is the bug report Allan mentioned.
It happened during the install stage, which scared me, but it didn't seem to cause any problems as far as I could tell. After I rebooted and freed up some disk space I did a -Su to finish the upgrade and it seemed to finish fine.
Are pacman database updates committed before or after the tarbell is extracted to the root directory?

You might want to take a look at pacman(8) and see what -Qk does, and then run it on your system.

Edit: but it won't catch 0 byte files.

Last edited by Mr.Elendig (2010-04-19 17:33:03)

B-Con · 2010-04-19 23:06:34

Mr.Elendig wrote:

You might want to take a look at pacman(8) and see what -Qk does, and then run it on your system.
Edit: but it won't catch 0 byte files.

Hm, pacman hung when -Qk processed the package "boost". I couldn't kill that pacman process. I checked a few dozen other packages and pacman -Qk worked fine on them, "boost" was the only problem. Shutdown commands once again didn't work so I did a hard poweroff. I rebooted to another OS and did an fsck and it cleared up about a dozen "orphaned inodes". I booted back to Arch and -Qk worked fine. (Except for ttf-ms-fonts, which claimed to own files in /tmp, but that's another story.)

I re-installed all the boost files in case any were corrupted. I suppose the orphaned inode issue was the blame for at least some of the problems? Seems unlikely, but it did fix that one pacman hanging issue. Thanks for the suggestion, Elendig.

Last edited by B-Con (2010-04-19 23:08:09)

Arch Linux

#1 2010-04-18 23:36:17

Full disk during pacman download causes system hang

#2 2010-04-19 01:00:22

Re: Full disk during pacman download causes system hang

#3 2010-04-19 01:07:26

Re: Full disk during pacman download causes system hang

#4 2010-04-19 03:06:19

Re: Full disk during pacman download causes system hang

#5 2010-04-19 03:33:20

Re: Full disk during pacman download causes system hang

#6 2010-04-19 04:30:34

Re: Full disk during pacman download causes system hang

#7 2010-04-19 12:39:42

Re: Full disk during pacman download causes system hang

#8 2010-04-19 13:07:38

Re: Full disk during pacman download causes system hang

#9 2010-04-19 13:31:51

Re: Full disk during pacman download causes system hang

#10 2010-04-19 14:47:06

Re: Full disk during pacman download causes system hang

#11 2010-04-19 17:15:45

Re: Full disk during pacman download causes system hang

#12 2010-04-19 17:31:49

Re: Full disk during pacman download causes system hang

#13 2010-04-19 23:06:34

Re: Full disk during pacman download causes system hang

Board footer