You are not logged in.

#1 2011-01-27 18:11:42

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

[FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

I don't even know what I did.

It's been a few days since I've rebooted. For some reason, every single program I tried to run segfaulted. This includes commands like 'su' and 'ls'.

So, I decided it was time for a hard reboot. Upon rebooting, Grub finds the /boot partition and boots the 2.6.37-ck kernel I have installed. However, it obviously doesn't find / because there is no information about loading modules, or clearing /tmp or daemons, etc. It just starts booting the kernel, then drops me to a login that doesn't know the machine name.

I can't log in because there are no users, including root.

I installed Arch on an LVM (except for /boot) which must be the source of the problem. But I don't know how to recover or even what is actually wrong!

Last edited by pogeymanz (2011-01-28 17:25:49)

Offline

#2 2011-01-27 18:15:59

Mr Green
Forum Fellow
From: U.K.
Registered: 2003-12-21
Posts: 5,893
Website

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Possibly check grub config and fstab ... maybe replace custom kernel with stock

Chroot in from Arch live cd and repair

If you cannot save system backup what you can and reinstall

Last edited by Mr Green (2011-01-27 18:16:30)


Mr Green

Offline

#3 2011-01-27 19:09:04

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

I'm using a livecd to try to access the root partition, but I'm getting this problem:

[root@localhost root]# mount /dev/VolGroup/lvolroot  /mnt/root
mount: wrong fs type, bad option, bad superblock on /dev/mapper/VolGroup-lvolroot,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

Okay. Today I learned how to run fsck on an LVM volume. So, I can mount the volumes from a LiveCD and I've checked Grub and fstab and rc.local (just in case).

I tried rebooting after fsck'ing that group, but it still didn't work.

Last edited by pogeymanz (2011-01-27 19:39:27)

Offline

#4 2011-01-27 19:59:36

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

And rebooting with the stock kernel didn't change things. Does any body have any ideas?


Also, here are my last few lines of Everything.log

Jan 27 12:40:55 Planck kernel: startx[27505]: segfault at 3e9301dc38 ip 0000003e92e02fad sp 00007fff51a1d960 error 7 in ld-2.13.so[3e92e00000+1e000]
Jan 27 12:41:38 Planck shutdown[27513]: shutting down for system reboot
Jan 27 12:41:39 Planck init: Switching to runlevel: 6
Jan 27 12:41:40 Planck shutdown[27519]: shutting down for system reboot
Jan 27 12:41:41 Planck kernel: rc.shutdown[27521]: segfault at 3e9301dc38 ip 0000003e92e02fad sp 00007fff65ff7cf0 error 7 in ld-2.13.so[3e92e00000+1e000]

What package owns ld-2.13.so? Maybe the reason I can't reboot is because this file is messed up.

Last edited by pogeymanz (2011-01-27 20:11:59)

Offline

#5 2011-01-27 20:45:57

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

So, I see that I updated glibc today and that glibc owns /lib/ld-2.13.so.

The problem is that I can't even roll back to a previous version because as soon as I chroot, I get a segfault (from trying to execute bash).

So I thought I would be tricky and copy /lib/ld-2.11.so from the LiveCD into my root LVM and remove ld-2.13.so, but that doesn't work either. I just get a "could not find file or directory" error from chroot trying to execute bash.

What the hell can I do here? I really don't feel like reinstalling...

Offline

#6 2011-01-27 20:57:36

eduardo.eae
Member
From: Reconquista - Argentina
Registered: 2010-01-24
Posts: 68

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Just for curiosity: Are you using prelink? I got an error like this using prelink. And undoing the changes breaked a lot of apps/libs. But I was able to reinstall them.

You have to copy all lib and lib64 contents from the old package into your root LVM.

Offline

#7 2011-01-27 21:03:43

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Yes. I am (was) using prelink.

Are you saying that I just have to copy all of the contents of glibc into /lib and lib64, or that I have to do it for all of my packages?

And if/when I get this up and running, what will happen when I try to undo prelink? I'll have to reinstall everything?

Offline

#8 2011-01-27 21:06:12

ANOKNUSA
Member
Registered: 2010-10-22
Posts: 2,141

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

pogeymanz wrote:

So, I see that I updated glibc today and that glibc owns /lib/ld-2.13.so.
The problem is that I can't even roll back to a previous version because as soon as I chroot, I get a segfault (from trying to execute bash).

This exact same thing happened to me last week, and after I installed prelink.  There's apparently some disconnect between the library used by the new glibc and prelink; what, I can't possibly say.  Sorry I can't be more helpful, but I just reinstalled.  After spending half an hour trying chroot, and going so far as to screw around with new bash libraries and binaries (dumb, I know) I gave up.  If prelink is on your system, pogeymanz, we may be witnessing the birth of a report for a nasty bug here.

Offline

#9 2011-01-27 21:14:43

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

ANOKNUSA wrote:

  If prelink is on your system, pogeymanz, we may be witnessing the birth of a report for a nasty bug here.

I think you mean:

we may be witnessing the birth of a nasty bug report here.

Fixed that for you. ;-)

But yes, this is ridiculous. I'm going to keep trying for a little longer.

Offline

#10 2011-01-27 21:26:45

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Holy CRAP! I'm back in business!

eduardo.eae's advice was what did it. I took glibc-2.13 (the current version) and untarred it. Then I just copied everything under lib and lib64 into the /lib and /lib64 directories on my root partition/LVM and I was able to chroot in! So I unprelinked everything and am currently booting into my system!

I'm probably going to edit the prelink wiki entry and let people know how to deal with this.

Thanks so much. What a relief. I wasted pretty much all day on this, but at least I didn't have to waste a whole other day or two reinstalling.

Offline

#11 2011-01-27 21:29:59

lucke
Member
From: Poland
Registered: 2004-11-30
Posts: 4,018

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Had to fix it today. Boot with Arch's install cd, mount the root partition and install glibc/binutils/gcc/gcc-libs adding "-r mount_point_of_your_root" to pacman.

Offline

#12 2011-01-27 23:44:51

ANOKNUSA
Member
Registered: 2010-10-22
Posts: 2,141

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

pogeymanz wrote:

we may be witnessing the birth of a nasty bug report here.

The bug report is never nasty (unless harshly worded, or just a tirade). Rather, they are the fuel that helps drive development and produces clean code.  The bug itself is what's nasty. The bug is the birthing pain of potentially better things in the future; they suck, but can lead to beautiful things .  wink  Anyway, good job getting that fixed.  I had something like 20 hours to get a paper finished and turned in, so I couldn't spend time waiting for help on the forums, and decided to spend the 45 minutes reinstalling instead.  It's good to know that a) I didn't necessarily screw this up and b) there's a fairly straightforward fix for it.

Offline

#13 2011-01-28 03:14:28

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

From an outsider's point of view, isn't prelink sort of hacky anyway? Do you really feel a big benefit from having it? This sort of issues on library updates seem inevitable over time...


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#14 2011-01-28 03:16:56

antigerme
Member
Registered: 2010-09-17
Posts: 2
Website

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

I had the same problem. I started the system with a livecd and untarred the glibc package and copied the files from lib to the original location.
When can again use prelink? The problem is where?

Offline

#15 2011-01-28 03:33:45

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Had any of you done a major glibc update while using prelink before?   glibc-2.11 -> 2.12 would have been about 8 months ago.

Offline

#16 2011-01-28 03:44:31

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Allan wrote:

Had any of you done a major glibc update while using prelink before?   glibc-2.11 -> 2.12 would have been about 8 months ago.

Unfortunately, I don't think that I had been using prelink back then. I'm pretty sure I only discovered prelink a few months ago.

Offline

#17 2011-01-28 03:45:13

antigerme
Member
Registered: 2010-09-17
Posts: 2
Website

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

I use prelink about 2 years and never had problems with any updates.

Offline

#18 2011-01-28 16:38:54

zworK
Member
From: Germany
Registered: 2011-01-28
Posts: 1

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Same thing happend to me yesterday after performing a system update. Everything started to segfault after or during a "prelink -afmR" after the update.

I thought I fried my system completely. Gonna try to recover the glibc package now.

I also used prelink over a year without any problems on arch. And a couple of years on my gentoo box.

Update:

Restoring the glibc files from the original package did the trick. I was able to chroot and unprelink everything. Now my system is up and running again.
Thanks for the hint!

Last edited by zworK (2011-01-28 17:35:30)

Offline

#19 2011-01-28 16:49:47

ANOKNUSA
Member
Registered: 2010-10-22
Posts: 2,141

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

My issues started immediately after installing prelink.  I was just planning on playing around with it.  Now I'm not sure it's even necessary, in retrospect.

Offline

#20 2011-01-29 07:09:31

unikum
Member
From: Russia
Registered: 2010-09-04
Posts: 151
Website

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

After upgrade glibc (2.12.2-2 -> 2.13-1) I get same problem with prelink. I not use testing repo.

Offline

#21 2011-01-29 07:36:56

sand_man
Member
From: Australia
Registered: 2008-06-10
Posts: 2,164

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

I just had the same problem too. I only started using prelink a few days ago so I don't have any experience with it. I just chrooted from a live disk and did a 'prelink -au' to unlink everything. Then I uninstalled prelink and cursed it for *@#%%$#% my system.


neutral

Offline

#22 2011-01-29 07:39:33

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Someone should really file a bug report for this...  I say it is a prelink bug as my glibc package is perfect until proven otherwise! tongue

Offline

#23 2011-01-29 09:39:42

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Allan wrote:

Someone should really file a bug report for this...  I say it is a prelink bug as my glibc package is perfect until proven otherwise! tongue

We would not expect otherwise smile.

I'm very glad now that I never bothered to set prelink up.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#24 2011-01-29 16:56:56

jonathan183
Member
Registered: 2008-06-11
Posts: 14

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

Hit the same problem myself ... my solution, boot from live CD, mount root partition, create temporary folder and copy /bin/bash to it, chroot in (using the copied bash) and run prelink -au then reboot the system ... worked for me wink

Last edited by jonathan183 (2011-01-29 16:57:43)

Offline

#25 2011-02-02 17:46:56

unikum
Member
From: Russia
Registered: 2010-09-04
Posts: 151
Website

Re: [FIX FOUND] HELP! I screwed up Arch something awful. (Prelink + Glibc)

deleted

Last edited by unikum (2011-02-02 18:12:21)

Offline

Board footer

Powered by FluxBB