[solved] Nothing executes after update (including init)

quantumphaze · 2010-10-11 00:00:34

After updating today I could no longer run commands in the terminal I was using. It complained something about there being too many symlinks (I can't remember or reproduce it). I assumed that there was something wrong with that terminal session so I exit it and it re opens (that's what yakuke does) giving me an error in red text that it can't execute bash.

Again I think that rebooting will probably fix this but then I'm greeted with a kernel panic and an error that it can't execute /sbin/init

Luckily I have Arch installer and Ubuntu on USB sticks.

It's end of semester crunch time, this couldn't have happened at a worse possible time.

Here is the pacman log of what I changed today:

[2010-10-11 09:23] Running 'pacman -Syu'
[2010-10-11 09:23] synchronizing package lists
[2010-10-11 09:23] starting full system upgrade
[2010-10-11 09:26] upgraded glibc (2.12.1-1 -> 2.12.1-2)
[2010-10-11 09:26] upgraded binutils-multilib (2.20.1-4 -> 2.20.1-5)
[2010-10-11 09:26] upgraded blas (3.2.2-1 -> 3.2.2-2)
[2010-10-11 09:26] upgraded doxygen (1.7.1-1 -> 1.7.2-1)
[2010-10-11 09:26] upgraded kdebindings-python (4.5.2-3 -> 4.5.2-5)
[2010-10-11 09:26] upgraded lapack (3.2.2-1 -> 3.2.2-2)
[2010-10-11 09:26] upgraded lib32-glib2 (2.24.1-3 -> 2.26.0-1)
[2010-10-11 09:26] upgraded lib32-atk (1.30.0-3 -> 1.32.0-1)
[2010-10-11 09:26] upgraded lib32-freetype2 (2.4.2-3 -> 2.4.3-1)
[2010-10-11 09:26] upgraded lib32-glibc (2.12.1-5 -> 2.12.1-6)
[2010-10-11 09:26] upgraded lib32-pango (1.28.1-3 -> 1.28.3-1)
[2010-10-11 09:26] upgraded lib32-libpng (1.4.3-3 -> 1.4.4-1)
[2010-10-11 09:26] installed lib32-gdk-pixbuf2 (2.22.0-2)
[2010-10-11 09:26] upgraded lib32-gtk2 (2.20.1-4 -> 2.22.0-3)
[2010-10-11 09:27] upgraded qt (4.7.0-2 -> 4.7.0-3)
[2010-10-11 09:27] upgraded lib32-qt (4.6.3-2 -> 4.7.0-3)
[2010-10-11 09:27] upgraded libmysqlclient (5.1.50-1 -> 5.1.51-1)
[2010-10-11 09:27] upgraded libva (1.0.4-1 -> 1.0.5-1)
[2010-10-11 09:27] upgraded mysql-clients (5.1.50-1 -> 5.1.51-1)
[2010-10-11 09:27] upgraded mysql (5.1.50-1 -> 5.1.51-1)
[2010-10-11 09:27] upgraded perl-xyne-common (2010.10.05.1-1 -> 2010.10.09.1-1)
[2010-10-11 09:27] upgraded perl-xyne-arch (2010.10.06.1-1 -> 2010.10.09.1-1)
[2010-10-11 09:27] upgraded polkit-qt (0.95.1-1 -> 0.96.1-1)
[2010-10-11 09:27] upgraded rasqal (0.9.19-2 -> 0.9.20-1)
[2010-10-11 09:27] upgraded redland (1.0.10-2 -> 1.0.11-1)
[2010-10-11 09:27] upgraded wine (1.3.3-2 -> 1.3.4-1)

I attempted to downgrade the binutils package and it said error: script failed to complete or something like that for the .install file. The .install file just fixes up the info docs.

Last edited by quantumphaze (2010-10-12 01:25:16)

quantumphaze · 2010-10-11 02:33:01

The error line is:

switch_root: can't execute '/sbin/init': Too many levels of symbolic links
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: switch_root Not tainted 2.6.36-ARCH #1
Call Trace:

And then there is 5 lines of trace that I'll type up from the photo on my phone if its needed on request.

Allan · 2010-10-11 03:11:37

How full is your disk?

quantumphaze · 2010-10-11 03:17:55

3.8G free on /
about 20G free on /home

All research on the "Too many levels of symbolic links" problem says that there may be a symbolic link loop somewhere as it only happens when there are 32 link to a link in a row. Since there are no links on the way to /sbin/init it may be something that init calls that contains the loop.

Last edited by quantumphaze (2010-10-11 03:49:33)

quantumphaze · 2010-10-11 04:38:51

GAH!! Got it working for now.

I installed Pacman onto my Ubuntu USB stick and tediously downgraded all packages that I updated today.

I still have no idea what package was the culprit but I guess that I will have to install each of them one at a time.

quantumphaze · 2010-10-11 04:45:07

It's glibc-2.12.1-2
As soon as I install it no new command run.

Allan · 2010-10-11 05:21:08

Strange... no-one else seems to have been affected by this.

I notice you are using a 2.6.36 prerelease kernel. Have you tried with the stock Arch one? Maybe glibc is interacting in strange ways with your kernel... (long shot).

quantumphaze · 2010-10-11 11:40:28

Whoops. I'm running the stock .35 kernel. I must have made the mistake when reading off the blurry picture I took using my phone since I can't copy/paste it.

My suspicion is that there are two packages that have symlinks, the new glibc and something else, that when installed together make a link loop. I'll investigate it once my work load lessens.

quantumphaze · 2010-10-11 13:44:59

I discovered what I did to my system. For some reason that I can't remember I had /lib64 symlinked to /lib resulting in the symlinks in /lib64 overwriting the real files in /lib. I have no idea why I did that but I have learned the valuable lesson of not messing with my 'production' system's internals (for at least the next 10 minutes).

karol · 2010-10-11 13:48:48

quantumphaze wrote:

I discovered what I did to my system. For some reason that I can't remember I had /lib64 symlinked to /lib resulting in the symlinks in /lib64 overwriting the real files in /lib. I have no idea why I did that but I have learned the valuable lesson of not messing with my 'production' system's internals (for at least the next 10 minutes).

If I symlink things I always make a note about what and why have I done - less headscratching months later.

djo0012 · 2010-10-14 19:03:38

I got the problem as well for the same reason so probably a few people will have it, would be good to know why it's doing it on this package

quantumphaze · 2010-10-15 08:19:49

The package contains:

/lib/ld-2.12.1.so
/lib/ld-linux-x86-64.so.2 -> ld-2.12.1.so
/lib64/ld-2.12.1.so -> ../lib/ld-2.12.1.so
/lib64/ld-linux-x86-64.so.2 -> ../lib/ld-linux-x86-64.so.2

but if /lib64/ is symlinked to /lib/ (/lib64/ -> ../lib/)then when the package is installed it will install the later two files that go in /lib64/ in /lib/ and overwriting the ones written before them and end up linking to themselves. Resulting in:

/lib/ld-2.12.1.so -> ../lib/ld-2.12.1.so
/lib/ld-linux-x86-64.so.2 -> ../lib/ld-linux-x86-64.so.2

Using the colours that you get with ls -l --color=auto

I think the reason I symlinked it was because some app was stupid and wanted something in /lib64/

Last edited by quantumphaze (2010-10-15 08:21:05)

Arch Linux

#1 2010-10-11 00:00:34

[solved] Nothing executes after update (including init)

#2 2010-10-11 02:33:01

Re: [solved] Nothing executes after update (including init)

#3 2010-10-11 03:11:37

Re: [solved] Nothing executes after update (including init)

#4 2010-10-11 03:17:55

Re: [solved] Nothing executes after update (including init)

#5 2010-10-11 04:38:51

Re: [solved] Nothing executes after update (including init)

#6 2010-10-11 04:45:07

Re: [solved] Nothing executes after update (including init)

#7 2010-10-11 05:21:08

Re: [solved] Nothing executes after update (including init)

#8 2010-10-11 11:40:28

Re: [solved] Nothing executes after update (including init)

#9 2010-10-11 13:44:59

Re: [solved] Nothing executes after update (including init)

#10 2010-10-11 13:48:48

Re: [solved] Nothing executes after update (including init)

#11 2010-10-14 19:03:38

Re: [solved] Nothing executes after update (including init)

#12 2010-10-15 08:19:49

Re: [solved] Nothing executes after update (including init)

Board footer