[SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

trusktr · 2010-10-27 00:46:15

I didn't have enough space on my root partition so doing pacman -Syu completely broke my system. Between classes and all my homework i'm still trying to restore my system. I'm logging everything here to help others who might encounter the same problem.

If you have any advice or experience with this, please post it!

What happened:
I attempted to do a system update and received many errors during the install that looked like these:

Error syncing configuration data: Failed to write some configuration data to disk
error: command failed to execute correctly
(148/324) upgrading gedit [######################] 100%

(gconftool-2:10150): GConf-WARNING **: Failed to write "/etc/gconf/gconf.xml.defaults/%gconf-tree.xml": Error writing file "/etc/gconf/gconf.xml.defaults/%gconf-tree.xml.new": No space left on device

Error syncing configuration data: Failed to write some configuration data to disk
(149/324) upgrading vte [######################] 100%
error: could not create temp directory
(150/324) upgrading gucharmap [######################] 100%
error: could not create temp directory
error: could not create temp directory
(151/324) upgrading gedit-plugins [######################] 100%
error: could not create temp directory
error: could not create temp directory
^C52/324) upgrading gimp [###############-------] 71%
Interrupt signal received
(152/324) upgrading gimp [######################] 100%
gtk-update-icon-cache: **
Gtk:ERROR:updateiconcache.c:1102:write_bucket: assertion failed: (*offset == ftell (cache))
/tmp/alpm_z9MBQe/.INSTALL: line 1: 10156 Aborted gtk-update-icon-cache -q -t -f usr/share/icons/hicolor
error: command failed to execute correctly

and now my whole system is broken! According to a couple of the devs, i ran out of space in my root partition. In my case, the system update needed to download 2.8 gigs worth of updates witch was just enough to completely fill the root partiton.

Pacman often needs to create temporary folders and files during installation of packages, but since there was no space left at all, it spewed out errors everywhere saying it couldn't create this or that. Despite these errors, pacman continued to show 100% installation progress for each package, the end result being that pacman was under the impression that every package was successfully installed, when in fact this was not the case.

So now, the package database tells pacman that all those packages are installed, even though they are not.

Well I thought to myself, that's no big deal, i'll just remove the package cache to my home partition and re-install all the packages that got messed up. I opened up the pacman log, tediously made a list of all the packages that pacman attempted to install, and tried a -Syyu operation with all the packages appended to the command.

"Not so fast" pacman seemed to say. It spewed out "file exists in filesystem" errors for every single package it attempted to install, and in the end didn't install anything.

A grand total of about 16,400 files were "existing in the filesystem" at this point!

"So, how the heck do i re-install these packages?" I thought to myself. Well, clearly, and from advice in the Wiki, you just need to delete the files, then attempt to re-install the packages.

So, I deleted ALL 16,400 files after creating a small JAVA app to extract just the filenames from the pacman output...... haha
It turns out I did not need to do this and the solution (below) is much simpler.

OK, At this point the system lost all of it's functionality almost entirely. My command prompt was gone, and replaced by a very basic command-less Bash prompt. Luckily, pacman continued to work just fine, so i re-installed coreutils and regained my prompt as well as basic commands like "ls", "cat", "rm", etc; much needed commands if I'm going to fix anything.

I tried again to reinstall all the packages that got clobbered, only to find endless "could not find or read package" errors. Then I said to myself "you know what? I'm just going to restart and see what happens."

Whaddaya know, the system doesn't boot now, but I bet it's because i need to disable a few non-functional daemons among a few other things so I can boot into my terminal. If not, I'll have to use a Live CD to fix this nightmare.

The following is a guide on how to fix this issue, it's not complete yet, but it will be once I figure out how to restore my system. I have a feeling I need to start re-installing everything starting only with [core] packages, then [extra] packages, and followed last by [community] and [multilib] packages.

Any help is greatly appreciated!

__________

The Final Working Solution:
There's a chance that you can't boot into arch now. In my case, the arch boot process was freezing when trying to start daemons.
You'll need to boot into run-level 1 (single user mode with basic functionality) to perform the recovery. You'll need to add a "1" without the quotes to the kernel line in grub when you boot.
See the following to learn how to change the run-level to boot into: https://bbs.archlinux.org/viewtopic.php?id=8198

Before you do any recovering, you will need to move the pacman cache from the root partition to the home partition so this space issue is no longer a problem in the future:
Create the folder /home/pacman/pkg as root:

# mkdir -p /home/pacman/pkg

Open up /etc/pacman.conf with your desired text editor (i used nano) as root:

# nano /etc/pacman.conf
... change the CacheDir variable to /home/pacman/pkg ...

Uncomment the "CacheDir" line. For the value of CacheDir put "/home/pacman/pkg".
Save the file and exit the editor. From now on, pacman will download packages to that folder in your home partition instead of in the root partition (your home partition is likely to have plenty of space for pacman's cache).

Now, before we continue, let's clean out the root partition. Perform a pacman -Scc to clear all the downloaded packages from pacman's old cache your root partiton. Answer yes to all the questions.

Now you need to obtain a list of all the packages that pacman installed incorrectly by taking a look at the pacman log (/var/log/pacman.log).
You will have to re-install all the packages that pacman *thought* it installed correctly (all the packages for which you received errors).

There's an easy way to do this (as vacant pointed out below): copy the recent activity from /var/log/pacman.log and paste it into a new text file, say "pkgs.txt".
Now you can use the command grep "] upgraded\|] installed" pkgs.txt|cut -f4 -d " " to extract a list of just the names of the packages from "pkgs.txt" and send em to pacman.
Also, you will need to use the -f option when you re-install these packages otherwise you'll get many annoying "file exists in filesystem" errors...

So here's what the final sequence of commands to fix your system might look like, including the previous ones i already gave examples of:

# mkdir -p /home/pacman/pkg
# nano /etc/pacman.conf
... change the CacheDir variable to /home/pacman/pkg ...
# pacman -Scc
... answer yes to everything ...
... make the pkgs.txt file as described above ...
# pkgs=$(grep "] upgraded\|] installed" pkgs.txt|cut -f4 -d " "); pacman -Syuf $pkgs

Badaboom, badabang! You're system should be finally fixed after those commands are done executing.

Last edited by trusktr (2010-12-02 20:15:50)

skunktrader · 2010-10-27 00:51:04

I would have thought that pacman -Scc would clear enough space in /var that the remaining steps wouldn't be needed.

trusktr · 2010-10-27 01:11:05

I thought so too, but after doing that, the total download size for the updates was bigger than the space left. It was 2.8 gigs of downloads, and the root partition was 5.6 gigs full out of 7.56 gigs.

I've never experienced such a huge update before! It's been a few hours and i'm still fixing it. There's literally hundreds of thousands of files that are now giving me "file exists in filesystem" errors so i'm making a script to delete all of those.

To get an idea of how many files there was, i outputted the errors to a file, opened it up with less, then pressed and held page down for almost 2 minutes, didn't reach the bottom, so ended up just pressing end.

Anyways, I'm going to post the scripts so it in case anyone encounters this critical problem, they can follow some simple instructions and be on their way.

ngoonee · 2010-10-27 03:31:16

I'd suggest adding a link to this information on the pacman wiki. Not the scripts itself, but just a warning something like "If you try to -Syu with a full hard disk, you'll mess your system up! If you've already done that, you'll need to reinstall all your packages, the scripts here (put your link here) may help *WARNING, USER-CONTRIBUTED*"

keenerd · 2010-10-27 04:01:19

Also, Clyde will warn you if you are going to download/install an upgrade bigger than your free disk.

trusktr · 2010-10-28 03:07:39

@ngoonee Yeah that would definitely be a very good thing to have in the wiki. Unfortunately I think only the admins have access to edit that page so I couldn't add anything. I did file a bug though because i'm pretty sure i won't be the last user to experience this amazing and awesome scenerio.

@keenard Who's Clyde? Are you applying a metaphorical personification to Pacman? Well i think i had barely enough space to download the packages, but no space left at all for the temporary files and whatever other processes pacman needs.

Last edited by trusktr (2010-10-28 03:08:56)

falconindy · 2010-10-28 03:57:13

https://bugs.archlinux.org/task/11639

trusktr · 2010-10-29 08:35:50

Thanks falconindy. I hope that bug get's squashed.

trusktr · 2010-11-03 09:43:34

I've completely recovered my system already. I'mma post the details as soon as I get some time away from homework. Recovering my computer also fixed a couple bugs HEHE

vacant · 2010-11-03 10:54:01

At your own risk:

Pacman's "-f" flag should get you over the "file exists" problem

cut the recent upgrades from pacman.log and paste into a new file (fred.txt):

grep "] upgraded\|] installed" fred.txt|cut -f4 -d " " | exec pacman -Suf

Last edited by vacant (2010-11-03 10:57:03)

ZekeSulastin · 2010-11-03 13:57:17

trusktr wrote:

@keenard Who's Clyde?

http://aur.archlinux.org/packages.php?ID=34686

It's one of the more popular libalpm/AUR helpers.

trusktr · 2010-12-02 20:11:08

Thanks vacant, knowing the -f option and that useful grep|cut command would've helped me fix my system much quicker. I've updated the solution above using your tips and so users now have a final solution if they ever encounter this problem.

trusktr · 2012-04-01 23:07:52

I wanted to note that as of pacman 4 the CheckSpace option in /etc/pacman.conf is enabled by default, which could help prevent a nasty scenario like this one.

kstenger · 2012-05-10 18:56:39

trusktr wrote:

I wanted to note that as of pacman 4 the CheckSpace option in /etc/pacman.conf is enabled by default, which could help prevent a nasty scenario like this one.

I guess it was a bit too late for me... found this post cause it happened to me this monday.
It didn't break completely my system, but X won't start and I can't find another solution so far, so I'll try this one.

Thanks for sharing it!

PD: well, it didn't work but it was worth trying.

Last edited by kstenger (2012-05-11 17:34:25)

Arch Linux

#1 2010-10-27 00:46:15

[SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#2 2010-10-27 00:51:04

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#3 2010-10-27 01:11:05

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#4 2010-10-27 03:31:16

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#5 2010-10-27 04:01:19

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#6 2010-10-28 03:07:39

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#7 2010-10-28 03:57:13

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#8 2010-10-29 08:35:50

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#9 2010-11-03 09:43:34

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#10 2010-11-03 10:54:01

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#11 2010-11-03 13:57:17

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#12 2010-12-02 20:11:08

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#13 2012-04-01 23:07:52

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

#14 2012-05-10 18:56:39

Re: [SOLVED] Pacman ran out of space, blew up my system during -Syu! HELP!

Board footer