pacman -Syu bricked my system part way through updating

12o · 2013-01-28 07:58:44

After reading the Arch Linux home page regarding the glibc and filesystem updating together (according to the website, this is done using pacman -Syu). I performed the operation and partway through the glibc update, pacman tanked out and put me to a command prompt. Whenever I ran any command, I kept getting an error "call to execv failed" on every command. I tried rebooting (since nothing else worked) and ran right into a kernel panic. I searched these forums and found this:

https://bbs.archlinux.org/viewtopic.php … 1#p1127251

and followed the steps to the letter. But when I go to reboot again, I run into the same kernel panic. I can put in "break = postmount" into the kernel line which drops me to a root prompt with /new_root in readonly state. I changed the /new_root partition to rw with:

mount -n -o remount,rw /new_root

But now that I'm here, I have no idea what to do. From what I can tell, there is now a symbolic link in the /new_root folder that points to usr/lib. The permissions on the link are maxed as can be seen here:

lrwxrwxrwx 1 0    0   7 Jan 28   07:30 lib -> usr/lib

Is there anyone out there who's knowledgeable in this stuff to figure out the problem? And no, I did not use --force. I simply used "pacman -Syu" as explained on the Arch website. Thanks in advance for any advice.

ConnorBehan · 2013-01-28 09:24:09

Were you trying to upgrade from glibc-2.16.0-1 or earlier before the /lib -> /usr/lib switch? Also, take a look at the file list for the glibc package. Can you figure out how, if at all, your system deviates from that list by mounting your hard drive from a recovery CD environment?

12o · 2013-01-28 16:35:42

Once I was finally able to root into the FS after mounting using an Arch boot disk (it's been a while since I've had to do this), I was able to get this list:

root@archiso /mnt # ls /mnt/var/cache/pacman/pkg | grep filesystem
filesystem-2012.10.1-any.pkg.tar.xz
filesystem-2012.10.2-any.pkg.tar.xz
filesystem-2012.11.1-any.pkg.tar.xz
filesystem-2012.12.1-any.pkg.tar.xz
filesystem-2012.10.8-1-any.pkg.tar.xz
filesystem-2013.01-3-x86_64.pkg.tar.xz
root@archiso /mnt # ls /mnt/var/cache/pacman/pkg | grep linux-api-headers
linux-api-headers-3.5.1-1-x86_64.pkg.tar/xz
linux-api-headers-3.5.5-1-x86_64.pkg.tar/xz
linux-api-headers-3.6.3-1-x86_64.pkg.tar/xz
linux-api-headers-3.7.1-1-x86_64.pkg.tar/xz
linux-api-headers-3.7.4-1-x86_64.pkg.tar/xz
root@archiso /mnt # ls /mnt/var/cache/pacman/pkg | grep tzdata
tzdata-2012e-1-any.pkg.tar.xz
tzdata-2012f-1-any.pkg.tar.xz
tzdata-2012g-1-any.pkg.tar.xz
tzdata-2012h-1-any.pkg.tar.xz
tzdata-2012i-1-any.pkg.tar.xz
tzdata-2012j-1-any.pkg.tar.xz
root@archiso /mnt #

From what I can tell, the files are there.

Here's the last few lines of the pacman.log

[2013-01-28 00:17] Running 'pacman -Syu'
[2013-01-28 00:17] synchronizing package lists
[2013-01-28 00:17] starting full system upgrade
[2013-01-28 00:17] upgraded acl (2.2.51-2 -> 2.2.51-3)
[2013-01-28 00:17] upgraded linux-api-headers (3.7.1-1 -> 3.7.4-1)
[2013-01-28 00:17] upgraded filesystem (2012.12-1 -> 2013.01-3)
[2013-01-28 00:17] error: problem occurred while upgrading filesystem
[2013-01-28 00:17] upgraded filesystem (2012.12-1 -> 2013.01-3)
[2013-01-28 00:17] error: problem occurred while upgrading filesystem

At that point, the system stopped responding to any linux commands, and I was forced to reboot. I guess the filesystem upgrade borked it all?

PaulBx1 · 2013-01-28 17:02:10

Doing a disk image backup before a system upgrade is always a good idea...

12o · 2013-01-28 17:12:04

PaulBx1 wrote:

Doing a disk image backup before a system upgrade is always a good idea...

Believe me. If I could afford an external hard drive to do this, I definitely would be doing it every time. College just sucks the time, life and money right out of me. :-)

That does give me an idea, though. Is there a way to do a disk image backup and have the backup CD upload the image to a cloud service like Google or whatever as it performs the backup? That would be extremely helpful.

drcouzelis · 2013-01-28 17:47:31

If your operating system is that broken, can't you just reinstall Arch Linux? It should only take about 30 minutes.

Being a college student is no excuse for not having a backup. In regards to the data on your computer, there's only two options: either you have a backup that you are able to restore from or there is nothing on your computer that you don't mind losing if your hard drive dies while you sleep tonight.

Also, I think your definition of bricked is different from my definition of bricked...

12o · 2013-01-28 18:03:31

drcouzelis wrote:

If your operating system is that broken, can't you just reinstall Arch Linux? It should only take about 30 minutes.
Being a college student is no excuse for not having a backup. In regards to the data on your computer, there's only two options: either you have a backup that you are able to restore from or there is nothing on your computer that you don't mind losing if your hard drive dies while you sleep tonight.
Also, I think your definition of bricked is different from my definition of bricked...

I was hoping someone might know of a workaround before I re-installed everything.

And I have a backup for my important documents on a USB flash drive. I just don't have an external hard drive to back up the entire filesystem. I can probably back up the entire drive image to the same hard drive and upload it to a cloud somewhere. I'll have to encrypt it, naturally. But having an update might not do anything for me as updating again will cause the same problem. I'm just looking for two things:

1) How to get my system back up and running after applying the update
2) Finding the root cause of the problem so I can avoid it while updating in step 1

Since I can't afford an external drive, I have to go with what I can: important files on a USB stick and any help from the forums.

And regarding 'bricked'. Sorry, I'm getting a kernel panic after updating using 'pacman -Syu'. I used the link I posted above to get to root, but ConnorBehan suggested I use a recovery CD environment which I now have.

Last edited by 12o (2013-01-28 18:04:20)

rm_win · 2013-01-29 01:22:37

Could try booting the recovery disk, chroot then run -Syu. It mite finish the update. May have to edit mirror list first. But it may work.

Last edited by rm_win (2013-01-29 01:23:15)

12o · 2013-01-29 02:27:56

rm_win wrote:

Could try booting the recovery disk, chroot then run -Syu. It mite finish the update. May have to edit mirror list first. But it may work.

I've been trying that with recovery CDs all day. Three different Arch CDs, Knoppix, SystemRescueCD... nothing. Every time, I get

chroot: failed to run command #/bin/sh#: Input/Output error

At this point, I doubt even Linus Torvalds could fix it. Just plain unfixable. Going to reinstall the main OS and hope that the /home folder remains. My system was installed with the 08/2011 install CD, so it's probably no longer able to be updated with pacman. I'll have to use the new install CD. Hopefully that will set the whole /lib and /usr/lib up so I don't have to worry about it anymore. I guess I'm just not that skilled in Linux yet.

Thank you, everyone, for what help you could offer. Greatly appreciated.

ConnorBehan · 2013-01-29 02:44:54

Definitely back up /home before you do that. Borrow somebody's external drive for just long enough to reinstall. Surely the "cp" command will work to copy files to it.

rm_win · 2013-01-29 04:28:10

With arch install/recovery the command is arch-chroot, if you did not know that. after mount which you know. But I have one partition. If you have several partitions. But good luck hope you get your system back.

Max-P · 2013-01-29 06:51:41

Ok, your system looks messed up pretty deeply...

One thing you can try, is to *not* chroot, but to mount your broken install root, and setup the live system pacman's database and root to that drive instead.

There are a few options in pacman.conf for that,

RootDir=
DBPath=

The idea here is to run the Arch CD, edit those files on the live system so you can use pacman on the broken install, and finish the update, and force reinstalling filesystem and glibc just to make sure the symlinks are ok. From that point, it might fix your system.

If that doesn't work, then you probably can try to post "ls -l" from the live media of your current messed up root, so we can see if something's wrong there.

/bin/ has moved to /usr/bin, which may explain why you can't even chroot to the system.

Last edited by Max-P (2013-01-29 06:53:49)

czubek · 2013-01-29 12:00:08

Similar problem to 12o, "call to execv failed" errors on almost every package upgrade. After reboot couldn't find boot or home partitions. As always reinstalling Arch jumped ahead of the problem and now all works fine.

BenderRodriguez · 2013-01-29 20:50:42

Hi,

I ran into this problem an posted a solution that worked for me here. You might need to "fix" packages with stuff in /usr/lib64.

Last edited by BenderRodriguez (2013-01-29 20:52:49)

windexh8er · 2013-01-29 22:11:27

So, I have a similarly bricked system (https://bbs.archlinux.org/viewtopic.php?id=157066), although mine doesn't seem to be as bad as yours. And, I feel your pain on a lot of not-helpful comments regarding getting something fixed.

Just wondering, though - have you tried booting the live CD, mounting your root partition (/), doing a `pacman -Sy` and then a `pacman --root /your/root/temporary/mount -Syu`?

I tried that and, while it didn't fix my system it may be a path forward to fixing some things. I'm getting to the point where I'm pretty sure it's due to the 'filesystem' package upgrade. Still working through what's borked. My system fully boots, but after I get logged in nothing works. There's kernel modules that won't load and there's not much good from logging to help showcase the real problem. Glad I'm not the only one though... I mean, not for your sake, but glad it seems to be an actual outlier problem.

Good luck, and post if you make any progress!

cfr · 2013-01-29 23:55:19

12o wrote:

From what I can tell, there is now a symbolic link in the /new_root folder that points to usr/lib. The permissions on the link are maxed as can be seen here:
lrwxrwxrwx 1 0    0   7 Jan 28   07:30 lib -> usr/lib

Are you saying this was new? Because this happened last July (https://www.archlinux.org/news/the-lib- … a-symlink/) but you have:

root@archiso /mnt # ls /mnt/var/cache/pacman/pkg | grep filesystem
filesystem-2012.10.1-any.pkg.tar.xz
filesystem-2012.10.2-any.pkg.tar.xz
filesystem-2012.11.1-any.pkg.tar.xz
filesystem-2012.12.1-any.pkg.tar.xz
filesystem-2012.10.8-1-any.pkg.tar.xz
filesystem-2013.01-3-x86_64.pkg.tar.xz

which suggests that you updated much more recently than that.

Incidentally, symbolic links always have those permissions.

Last edited by cfr (2013-01-29 23:56:11)

gary_tuxhat · 2013-01-30 14:54:50

drcouzelis wrote:

If your operating system is that broken, can't you just reinstall Arch Linux? It should only take about 30 minutes.
Being a college student is no excuse for not having a backup. In regards to the data on your computer, there's only two options: either you have a backup that you are able to restore from or there is nothing on your computer that you don't mind losing if your hard drive dies while you sleep tonight.
Also, I think your definition of bricked is different from my definition of bricked...

Yes I totally agreed with you on that, bricked have something to do with firmware not a "Operating System" , as well I don't understand how going to college , you can't afford to backup your system , when it doesn't cost anything to do that. We even got free cloud storage on the web, why not just use archiso back up your configs in the snapshot of ur system , Redo your partition and reinstall archlinux and just take ur root configs and put it on ur newly hdd archlinux in /mnt/root and tada you have ur system back.

Also I do not understand how people can "bricked" their pc or even mess up a archlinux system.. I've been using arch linux for about 4 years now off and on and , i never had any problems with it, just with "amd/ati" drivers , but that is not archlinux fault , its AMD themself.

I will say one good thing about archlinux, is that from switching to systemd is the bestest thing they ever done. My system works very good and i love it , i love the new setup of archlinux, i love the fact i can make snapshot of my system , i love the traditional setup of archlinux now. i love that they keep it simple!

sysvinit isn't that good and was very buggy, kudos on systemd , keep up the good work arch dev team.

-- Tux Hat

alphaniner · 2013-01-30 15:29:01

FWIW, you don't need an external HDD to protect yourself against upgrade breakage. It's not a back up, but with LVM you can take a snapshot of your system before an update and revert if needed. I use fully provisioned snapshots:

  ACTIVE   Original '/dev/VG0/lv_root' [5.00 GiB] inherit
  ACTIVE   Original '/dev/VG0/lv_home' [10.00 GiB] inherit
  ACTIVE   Original '/dev/VG0/lv_var' [5.71 GiB] contiguous
  ACTIVE   Snapshot '/dev/VG0/ss_root' [5.00 GiB] inherit
  ACTIVE   Snapshot '/dev/VG0/ss_var' [5.71 GiB] inherit
  ACTIVE   Snapshot '/dev/VG0/ss_home' [10.00 GiB] inherit

to be safe, because I keep them around until the next update. But that's really absurdly overkill. Snapshots don't need to be provisioned to the full size of the original, and right now my snapshots are using less than 1GB combined. So if you're smart about it you can use partially provisioned snapshots at the cost of a few GB.

And that's just one of a myriad of possibilities.

Last edited by alphaniner (2013-01-30 15:33:25)

drcouzelis · 2013-01-30 15:59:16

alphaniner wrote:

It's not a back up, but with LVM you can take a snapshot of your system before an update and revert if needed.

Ooh, I just finished learning about logical volume management! That seems like a need application of it. Do you advocate using LVM on personal Arch Linux computers?

I probably shouldn't even be considering converting to LVM. I just finished repartitioning my drives (triple boot, with partitions to spare! ) and, in over three years of Arch Linux usage, have never had a problem when updating.

Also, LVM can be stinkin' complicated!

EDIT: By the way, I didn't mean to chastize, I just wanted to gently "scare" 12o into having some sort of backup, which 12o already has. If just one more person in the world is backing up their data then the world is a better place.

Last edited by drcouzelis (2013-01-30 16:01:41)

alphaniner · 2013-01-30 17:39:17

LVM really isn't very complicated once you get past the initial 'culture shock'. Since I first used it, I've never looked back.

Arch Linux

#1 2013-01-28 07:58:44

pacman -Syu bricked my system part way through updating

#2 2013-01-28 09:24:09

Re: pacman -Syu bricked my system part way through updating

#3 2013-01-28 16:35:42

Re: pacman -Syu bricked my system part way through updating

#4 2013-01-28 17:02:10

Re: pacman -Syu bricked my system part way through updating

#5 2013-01-28 17:12:04

Re: pacman -Syu bricked my system part way through updating

#6 2013-01-28 17:47:31

Re: pacman -Syu bricked my system part way through updating

#7 2013-01-28 18:03:31

Re: pacman -Syu bricked my system part way through updating

#8 2013-01-29 01:22:37

Re: pacman -Syu bricked my system part way through updating

#9 2013-01-29 02:27:56

Re: pacman -Syu bricked my system part way through updating

#10 2013-01-29 02:44:54

Re: pacman -Syu bricked my system part way through updating

#11 2013-01-29 04:28:10

Re: pacman -Syu bricked my system part way through updating

#12 2013-01-29 06:51:41

Re: pacman -Syu bricked my system part way through updating

#13 2013-01-29 12:00:08

Re: pacman -Syu bricked my system part way through updating

#14 2013-01-29 20:50:42

Re: pacman -Syu bricked my system part way through updating

#15 2013-01-29 22:11:27

Re: pacman -Syu bricked my system part way through updating

#16 2013-01-29 23:55:19

Re: pacman -Syu bricked my system part way through updating

#17 2013-01-30 14:54:50

Re: pacman -Syu bricked my system part way through updating

#18 2013-01-30 15:29:01

Re: pacman -Syu bricked my system part way through updating

#19 2013-01-30 15:59:16

Re: pacman -Syu bricked my system part way through updating

#20 2013-01-30 17:39:17

Re: pacman -Syu bricked my system part way through updating

Board footer