[FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

pogeymanz · 2011-01-27 18:11:42

I don't even know what I did.

It's been a few days since I've rebooted. For some reason, every single program I tried to run segfaulted. This includes commands like 'su' and 'ls'.

So, I decided it was time for a hard reboot. Upon rebooting, Grub finds the /boot partition and boots the 2.6.37-ck kernel I have installed. However, it obviously doesn't find / because there is no information about loading modules, or clearing /tmp or daemons, etc. It just starts booting the kernel, then drops me to a login that doesn't know the machine name.

I can't log in because there are no users, including root.

I installed Arch on an LVM (except for /boot) which must be the source of the problem. But I don't know how to recover or even what is actually wrong!

Last edited by pogeymanz (2011-01-28 17:25:49)

Mr Green · 2011-01-27 18:15:59

Possibly check grub config and fstab ... maybe replace custom kernel with stock

Chroot in from Arch live cd and repair

If you cannot save system backup what you can and reinstall

Last edited by Mr Green (2011-01-27 18:16:30)

pogeymanz · 2011-01-27 19:09:04

I'm using a livecd to try to access the root partition, but I'm getting this problem:

[root@localhost root]# mount /dev/VolGroup/lvolroot  /mnt/root
mount: wrong fs type, bad option, bad superblock on /dev/mapper/VolGroup-lvolroot,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

Okay. Today I learned how to run fsck on an LVM volume. So, I can mount the volumes from a LiveCD and I've checked Grub and fstab and rc.local (just in case).

I tried rebooting after fsck'ing that group, but it still didn't work.

Last edited by pogeymanz (2011-01-27 19:39:27)

pogeymanz · 2011-01-27 19:59:36

And rebooting with the stock kernel didn't change things. Does any body have any ideas?

Also, here are my last few lines of Everything.log

Jan 27 12:40:55 Planck kernel: startx[27505]: segfault at 3e9301dc38 ip 0000003e92e02fad sp 00007fff51a1d960 error 7 in ld-2.13.so[3e92e00000+1e000]
Jan 27 12:41:38 Planck shutdown[27513]: shutting down for system reboot
Jan 27 12:41:39 Planck init: Switching to runlevel: 6
Jan 27 12:41:40 Planck shutdown[27519]: shutting down for system reboot
Jan 27 12:41:41 Planck kernel: rc.shutdown[27521]: segfault at 3e9301dc38 ip 0000003e92e02fad sp 00007fff65ff7cf0 error 7 in ld-2.13.so[3e92e00000+1e000]

What package owns ld-2.13.so? Maybe the reason I can't reboot is because this file is messed up.

Last edited by pogeymanz (2011-01-27 20:11:59)

pogeymanz · 2011-01-27 20:45:57

So, I see that I updated glibc today and that glibc owns /lib/ld-2.13.so.

The problem is that I can't even roll back to a previous version because as soon as I chroot, I get a segfault (from trying to execute bash).

So I thought I would be tricky and copy /lib/ld-2.11.so from the LiveCD into my root LVM and remove ld-2.13.so, but that doesn't work either. I just get a "could not find file or directory" error from chroot trying to execute bash.

What the hell can I do here? I really don't feel like reinstalling...

eduardo.eae · 2011-01-27 20:57:36

Just for curiosity: Are you using prelink? I got an error like this using prelink. And undoing the changes breaked a lot of apps/libs. But I was able to reinstall them.

You have to copy all lib and lib64 contents from the old package into your root LVM.

pogeymanz · 2011-01-27 21:03:43

Yes. I am (was) using prelink.

Are you saying that I just have to copy all of the contents of glibc into /lib and lib64, or that I have to do it for all of my packages?

And if/when I get this up and running, what will happen when I try to undo prelink? I'll have to reinstall everything?

ANOKNUSA · 2011-01-27 21:06:12

pogeymanz wrote:

So, I see that I updated glibc today and that glibc owns /lib/ld-2.13.so.
The problem is that I can't even roll back to a previous version because as soon as I chroot, I get a segfault (from trying to execute bash).

This exact same thing happened to me last week, and after I installed prelink. There's apparently some disconnect between the library used by the new glibc and prelink; what, I can't possibly say. Sorry I can't be more helpful, but I just reinstalled. After spending half an hour trying chroot, and going so far as to screw around with new bash libraries and binaries (dumb, I know) I gave up. If prelink is on your system, pogeymanz, we may be witnessing the birth of a report for a nasty bug here.

pogeymanz · 2011-01-27 21:14:43

ANOKNUSA wrote:

If prelink is on your system, pogeymanz, we may be witnessing the birth of a report for a nasty bug here.

I think you mean:

we may be witnessing the birth of a nasty bug report here.

Fixed that for you. ;-)

But yes, this is ridiculous. I'm going to keep trying for a little longer.

pogeymanz · 2011-01-27 21:26:45

Holy CRAP! I'm back in business!

eduardo.eae's advice was what did it. I took glibc-2.13 (the current version) and untarred it. Then I just copied everything under lib and lib64 into the /lib and /lib64 directories on my root partition/LVM and I was able to chroot in! So I unprelinked everything and am currently booting into my system!

I'm probably going to edit the prelink wiki entry and let people know how to deal with this.

Thanks so much. What a relief. I wasted pretty much all day on this, but at least I didn't have to waste a whole other day or two reinstalling.

lucke · 2011-01-27 21:29:59

Had to fix it today. Boot with Arch's install cd, mount the root partition and install glibc/binutils/gcc/gcc-libs adding "-r mount_point_of_your_root" to pacman.

ANOKNUSA · 2011-01-27 23:44:51

pogeymanz wrote:

we may be witnessing the birth of a nasty bug report here.

The bug report is never nasty (unless harshly worded, or just a tirade). Rather, they are the fuel that helps drive development and produces clean code. The bug itself is what's nasty. The bug is the birthing pain of potentially better things in the future; they suck, but can lead to beautiful things . Anyway, good job getting that fixed. I had something like 20 hours to get a paper finished and turned in, so I couldn't spend time waiting for help on the forums, and decided to spend the 45 minutes reinstalling instead. It's good to know that a) I didn't necessarily screw this up and b) there's a fairly straightforward fix for it.

ngoonee · 2011-01-28 03:14:28

From an outsider's point of view, isn't prelink sort of hacky anyway? Do you really feel a big benefit from having it? This sort of issues on library updates seem inevitable over time...

antigerme · 2011-01-28 03:16:56

I had the same problem. I started the system with a livecd and untarred the glibc package and copied the files from lib to the original location.
When can again use prelink? The problem is where?

Allan · 2011-01-28 03:33:45

Had any of you done a major glibc update while using prelink before? glibc-2.11 -> 2.12 would have been about 8 months ago.

pogeymanz · 2011-01-28 03:44:31

Allan wrote:

Had any of you done a major glibc update while using prelink before? glibc-2.11 -> 2.12 would have been about 8 months ago.

Unfortunately, I don't think that I had been using prelink back then. I'm pretty sure I only discovered prelink a few months ago.

antigerme · 2011-01-28 03:45:13

I use prelink about 2 years and never had problems with any updates.

zworK · 2011-01-28 16:38:54

Same thing happend to me yesterday after performing a system update. Everything started to segfault after or during a "prelink -afmR" after the update.

I thought I fried my system completely. Gonna try to recover the glibc package now.

I also used prelink over a year without any problems on arch. And a couple of years on my gentoo box.

Update:

Restoring the glibc files from the original package did the trick. I was able to chroot and unprelink everything. Now my system is up and running again.
Thanks for the hint!

Last edited by zworK (2011-01-28 17:35:30)

ANOKNUSA · 2011-01-28 16:49:47

My issues started immediately after installing prelink. I was just planning on playing around with it. Now I'm not sure it's even necessary, in retrospect.

unikum · 2011-01-29 07:09:31

After upgrade glibc (2.12.2-2 -> 2.13-1) I get same problem with prelink. I not use testing repo.

sand_man · 2011-01-29 07:36:56

I just had the same problem too. I only started using prelink a few days ago so I don't have any experience with it. I just chrooted from a live disk and did a 'prelink -au' to unlink everything. Then I uninstalled prelink and cursed it for *@#%%$#% my system.

Allan · 2011-01-29 07:39:33

Someone should really file a bug report for this... I say it is a prelink bug as my glibc package is perfect until proven otherwise!

ngoonee · 2011-01-29 09:39:42

Allan wrote:

Someone should really file a bug report for this... I say it is a prelink bug as my glibc package is perfect until proven otherwise!

We would not expect otherwise .

I'm very glad now that I never bothered to set prelink up.

jonathan183 · 2011-01-29 16:56:56

Hit the same problem myself ... my solution, boot from live CD, mount root partition, create temporary folder and copy /bin/bash to it, chroot in (using the copied bash) and run prelink -au then reboot the system ... worked for me

Last edited by jonathan183 (2011-01-29 16:57:43)

unikum · 2011-02-02 17:46:56

deleted

Last edited by unikum (2011-02-02 18:12:21)

Arch Linux

#1 2011-01-27 18:11:42

[FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#2 2011-01-27 18:15:59

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#3 2011-01-27 19:09:04

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#4 2011-01-27 19:59:36

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#5 2011-01-27 20:45:57

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#6 2011-01-27 20:57:36

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#7 2011-01-27 21:03:43

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#8 2011-01-27 21:06:12

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#9 2011-01-27 21:14:43

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#10 2011-01-27 21:26:45

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#11 2011-01-27 21:29:59

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#12 2011-01-27 23:44:51

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#13 2011-01-28 03:14:28

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#14 2011-01-28 03:16:56

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#15 2011-01-28 03:33:45

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#16 2011-01-28 03:44:31

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#17 2011-01-28 03:45:13

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#18 2011-01-28 16:38:54

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#19 2011-01-28 16:49:47

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#20 2011-01-29 07:09:31

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#21 2011-01-29 07:36:56

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#22 2011-01-29 07:39:33

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#23 2011-01-29 09:39:42

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#24 2011-01-29 16:56:56

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

#25 2011-02-02 17:46:56

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Board footer