Arch Linux

naraesk

Member

Registered: 2016-10-11

Posts: 25

[SOLVED] Cannot boot after kernel 4.10 update

On friday I did an "pacman -Syu" and already during the update (includig kernel 4.10) I got strange error messages in the same console window:

/bin/bash/: Error 1242211132

After" /bin/bash" some strange non-ascii characters are displayed.
The system wasn´t working anymore. Mouse and keyboard input was recognized, but I couldn´t start any program. Therefore I did a reboot. But the system wasn´t able to boot anymore. The only output was and still is:

Warning: /lib/modules/4.10.1-1-ARCH/modules.devname not found - ignoring
ERROR: Unable to find root device ´UUID=xyz´
You are being dropped to a recovery shell

I startet a arch linux with kernel 4.9.11 from usb, mounted my system and tried arch-chroot, but again:

/bin/bash/: Error 1234987234

When mounting my system I can access all files including the modules.devname, this file does exist. When running the systems bash with

/mnt/bin/bash -c "echo TEST"

it is working as well. But still no idea how to fix my system. Found somd posts that suggests regenerating initramfs, but this requires arch-chroot, doesn´t it?

Thanks in advance

Last edited by naraesk (2017-03-14 12:30:51)

berbae

Member

From: France

Registered: 2007-02-12

Posts: 1,302

Re: [SOLVED] Cannot boot after kernel 4.10 update

Did you post the real messages?

UUID=xyz

Error 1242211132
Error 1234987234 (1234 987 234)

This seems weird.

When mounting my system I can access all files

So you can look into pacman.log file to see what happened during your last system update.

I startet a arch linux with kernel 4.9.11 from usb, mounted my system and tried arch-chroot

Give precisely what commands you used and exact messages output.

In this state give the output of 'mount'.

naraesk

Member

Registered: 2016-10-11

Posts: 25

Re: [SOLVED] Cannot boot after kernel 4.10 update

Well, I skipped the UUID because I had to type everything by hand. The number after error changes from time to time and looks very random.

I just figured out, that /usr/lib/libc-2.25 is the problem. After copying the libc (and libdl, because they have to be the same version) from the live cd to my system arch-chroot was possible. But this is an older version of libc (2.24) and mkinitcpio complains about libc having the wrong version number. I tried pacman -S glibc, but instantly after the pacman message "(1/1) reinstalling glibc" the same error messages as described in the first post appear again. I am now going to verify the checksums of the package.

Update: md5sum is correct

Last edited by naraesk (2017-03-13 17:48:12)

naraesk

Member

Registered: 2016-10-11

Posts: 25

Re: [SOLVED] Cannot boot after kernel 4.10 update

And here is the pacman.log for the reinstall of glibc as mentioned before. It looks the same for the first update of glibc to version 2.25

[2017-03-13 17:13] [PACMAN] Running 'pacman -S glibc'
[2017-03-13 17:14] [ALPM] transaction started
[2017-03-13 17:14] [ALPM] reinstalled glibc (2.25-1)
[2017-03-13 17:14] [ALPM-SCRIPTLET] /usr/bin/bash: p\F0\97\FD: `\F9LU\90: Error 1421992764
[2017-03-13 17:14] [ALPM] transaction completed
[2017-03-13 17:14] [ALPM] running 'systemd-tmpfiles.hook'...
[2017-03-13 17:14] [ALPM-SCRIPTLET] /bin/sh: 0u\80\96\FD: `\E9*n: Error 1839190844
[2017-03-13 17:14] [ALPM] running 'systemd-update.hook'...
[2017-03-13 17:14] [ALPM-SCRIPTLET] /usr/bin/touch: @\DAe0\FE: `i4\FA\AB: Error 18446744073607738172
[2017-03-13 17:14] [ALPM] running 'texinfo-install.hook'...
[2017-03-13 17:14] [ALPM-SCRIPTLET] /bin/sh: \F0\B19\FD: `INQ\84: Error 1354969916

loqs

Member

Registered: 2014-03-06

Posts: 17,323

Re: [SOLVED] Cannot boot after kernel 4.10 update

Setup the arch-chroot in /mnt but do not run arch-chroot instead please run and post the output of

$ pacman -Qkkr /mnt glibc bash coreutils

Edit:
missing run
Edit2:
Also the outputs of

$ file /mnt/usr/lib/libc-2.25.so
$ sha256sum /mnt/usr/lib/libc-2.25.so

Last edited by loqs (2017-03-13 17:12:26)

naraesk

Member

Registered: 2016-10-11

Posts: 25

Re: [SOLVED] Cannot boot after kernel 4.10 update

backup file: glibc: /mnt/etc/locale.gen (Modification time mismatch)
backup file: glibc: /mnt/etc/locale.gen (Size mismatch)
glibc: 1553 total files, 0 altered files
bash: 235 total files, 0 altered files
coreutils: 429 total files, 0 altered files

naraesk

Member

Registered: 2016-10-11

Posts: 25

Re: [SOLVED] Cannot boot after kernel 4.10 update

$ file /mnt/usr/lib/libc-2.25.so
/mnt/usr/lib/libc-2.25.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /usr/lib/ld-linux-x86-64.so.2, BuildID[sha1]=58c735bc7b19b0aeb395cce70cf63bd62ac75e4a, for GNU/Linux 2.6.32, not stripped, with debug_info

$ sha256sum /mnt/usr/lib/libc-2.25.so
89e7758cdb37c6de4745a777e753bba2061da6f63e695275cd5bb36b6cadaeab  /mnt/usr/lib/libc-2.25.so

And to make sure my earlier edit is recognized: I also had to overwrite libdl, since glibc 2.24 and libdl 2.25 are not compatible. Therefore, same output for libdl:

$ file /mnt/usr/lib/libdl-2.25.so
/mnt/usr/lib/libdl-2.25.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=04790e3832076b45cb28f60635baa69f920a8a5c, for GNU/Linux 2.6.32, stripped, with debug_info

$ sha256sum /mnt/usr/lib/libdl-2.25.so
a390436d5bfd96f2456efe4e5748705a5ab4d15e1f72d8eda72e9767f2f854ca  /mnt/usr/lib/libdl-2.25.so

Last edited by naraesk (2017-03-13 17:53:40)

mich41

Member

Registered: 2012-06-22

Posts: 796

Re: [SOLVED] Cannot boot after kernel 4.10 update

That's some issue with the package. I just downloaded glibc-2.25, unpacked it to /tmp and ran bash with LD_LIBRARY_PATH pointing there. Same effect.

Does this package work for anyone at all?

No crap in dmesg. AMD K10 CPU. Same checksums.

Last edited by mich41 (2017-03-13 17:50:52)

Scimmia

Fellow

Registered: 2012-09-01

Posts: 11,544

Re: [SOLVED] Cannot boot after kernel 4.10 update

Does this package work for anyone at all?

Yes. It works fine for me, and I haven't had a torrent of bug reports, so I would say that it works fine for the vast majority of people.

naraesk

Member

Registered: 2016-10-11

Posts: 25

Re: [SOLVED] Cannot boot after kernel 4.10 update

So I guess the temporary solution is to downgrade everything to 2.24 isn´t it?

Or does it help someone if I stay with the broken system for now to be able to post more information if necessary?

mich41

Member

Registered: 2012-06-22

Posts: 796

Re: [SOLVED] Cannot boot after kernel 4.10 update

OK, good to know it works.

I got something interesting: I no longer get this error if I use the proper v2.25 ld-linux from the new package, but I get it if I use the v2.24 from my system. That was a bit of fail on my side I guess

So naraesk, can you post

ls -ld /usr/lib/ld-*
md5sum /usr/lib/ld-*

edit:

Also

realpath /lib64/ld-linux-x86-64.so.2
md5sum /lib64/ld-linux-x86-64.so.2

as this seems to be the exact path hardcoded in Arch binaries.

Last edited by mich41 (2017-03-13 18:20:56)

naraesk

Member

Registered: 2016-10-11

Posts: 25

Re: [SOLVED] Cannot boot after kernel 4.10 update

Here it comes:

$ ls -ld /mnt/usr/lib/ld-*
-rwxr-xr-x 1 root root 168680 Mar  6 14:06 /mnt/usr/lib/ld-2.25.so
drwxr-xr-x 1 root root     48 Dec 23 05:20 /mnt/usr/lib/ldb
lrwxrwxrwx 1 root root     22 Mar  8 15:12 /mnt/usr/lib/ld-linux.so.2 -> ../lib32/ld-linux.so.2
lrwxrwxrwx 1 root root     10 Mar  6 14:06 /mnt/usr/lib/ld-linux-x86-64.so.2 -> ld-2.25.so
drwxr-xr-x 1 root root   2066 Mar 11 11:12 /mnt/usr/lib/ldscripts

$ md5sum /mnt/usr/lib/ld-*
0ec54f062228168b8f00908eaf97a95b  /mnt/usr/lib/ld-2.25.so
7ce89157f361ed1c56271044cf5c4ad7  /mnt/usr/lib/ld-linux.so.2
0ec54f062228168b8f00908eaf97a95b  /mnt/usr/lib/ld-linux-x86-64.so.2

$ realpath /mnt/lib64/ld-linux-x86-64.so.2
/mnt/lib64/ld-2.24.so

$ md5sum /mnt/lib64/ld-linux-x86-64.so.2
307ed7436fa76b16cf55c8a0841cc22b  /mnt/lib64/ld-linux-x86-64.so.2

The last checksum seems wrong. It is the same checksum as version 2.24 from the live cd.

Last edited by naraesk (2017-03-13 19:25:27)

naraesk

Member

Registered: 2016-10-11

Posts: 25

Re: [SOLVED] Cannot boot after kernel 4.10 update

Yay, it is working! I just found out, that in the lib64 folder the old ld-2.24 versions existed. I copied 2.25 to lib64 and thats it.
Still not sure what has happened in the first place, but thanks for all your help! Maybe the first install of glibc was somehow interrupted or so. Will investigate this tomorrow.

mich41

Member

Registered: 2012-06-22

Posts: 796

Re: [SOLVED] Cannot boot after kernel 4.10 update

You shouldn't have this lib64, it's supposed to be a symlink to /usr/lib. See what's inside - maybe some proprietary application's installer created it or you have very ancient packages which keep files in there.

Anyway, you need to ensure nothing of value is there, delete this directory (well, actually rename would be safer ) and make it a symlink. Use busybox shell (which is statically linked) or livecd because you won't be able to run any dynamically linked executable (including ln -s) after you delete /lib64. Possibly reinstalling the filesystem package would achieve the same. Actually, do try to reinstall it afterwards anyway and see if it reports any conflicts indicating that something else could be broken too.

Last edited by mich41 (2017-03-13 20:06:01)

naraesk

Member

Registered: 2016-10-11

Posts: 25

Re: [SOLVED] Cannot boot after kernel 4.10 update

Hm, sounds like more trouble incoming. :-)

My /lib64 is a directory, not a symlink. It contains 6211 files. How can I found out, how these files were copied to /lib64? the ld-2.24 package was updated in june 2016 and it was in this folder. So something has copied it to /lib64 within the last months, can't be this ancient.

Last edited by naraesk (2017-03-13 21:49:18)

mich41

Member

Registered: 2012-06-22

Posts: 796

Re: [SOLVED] Cannot boot after kernel 4.10 update

6211 is quite a lot - maybe it's just a copy of /usr/lib? No idea how such a thing could come to exist - did you by any chance move the system from one disk to another using cp -r?

Anyway, all files belonging to packages should exist in /usr/lib, unless this one is a symlink to somewhere else. So if you rename /lib64 to /lib64backup and symlink /lib64 to usr/lib things should work.

Also, check this:

$ ls -ld /lib* /usr/lib*
lrwxrwxrwx   1 root root      7 2015-09-30  /lib -> usr/lib
lrwxrwxrwx   1 root root      7 2015-09-30  /lib64 -> usr/lib
drwxr-xr-x 221 root root 163840 12-09 13:07 /usr/lib
drwxr-xr-x  33 root root  24576 10-27 12:24 /usr/lib32
lrwxrwxrwx   1 root root      3 2015-09-30  /usr/lib64 -> lib
$ ls -ld /*bin* /usr/*bin*
lrwxrwxrwx 1 root root      7 2015-09-30  /bin -> usr/bin
lrwxrwxrwx 1 root root      7 2015-09-30  /sbin -> usr/bin
drwxr-xr-x 5 root root 118784 03-13 19:37 /usr/bin
lrwxrwxrwx 1 root root      3 2015-09-30  /usr/sbin -> bin

Only /usr/lib, /usr/lib32 (for multilib packages, may not exist) and /usr/bin are real directories - everything else is symlinks.

naraesk

Member

Registered: 2016-10-11

Posts: 25

Re: [SOLVED] Cannot boot after kernel 4.10 update

Indeed, I did move the system some months ago. But I followed some guide from the wiki. I'll have a look if I can find the page again. Maybe I did something wrong. Obviously, after all.

ls -ld /lib* /usr/lib*
lrwxrwxrwx 1 root root      7  6. Dez 00:43 /lib -> usr/lib
drwxr-xr-x 1 root root 225646 13. Mär 20:09 /lib64
drwxr-xr-x 1 root root 219860 14. Mär 12:27 /usr/lib
drwxr-xr-x 1 root root  27356 13. Mär 19:31 /usr/lib32
lrwxrwxrwx 1 root root      3  6. Dez 00:43 /usr/lib64 -> lib
drwxr-xr-x 1 root root     18  6. Dez 2015  /usr/libexec

$ ls -ld /*bin* /usr/*bin*
drwxr-xr-x 1 root root 107564 13. Mär 14:31 /bin
lrwxrwxrwx 1 root root      7  6. Dez 00:43 /sbin -> usr/bin
drwxr-xr-x 1 root root 109160 14. Mär 12:27 /usr/bin
lrwxrwxrwx 1 root root      3  6. Dez 00:43 /usr/sbin -> bin

So, same thing to do for /bin, I guess? Are there anymore symlinks that might be broken?

Last edited by naraesk (2017-03-14 11:55:08)

naraesk

Member

Registered: 2016-10-11

Posts: 25

Re: [SOLVED] Cannot boot after kernel 4.10 update

Ok, I created the symlink for /lib64 and /bin and everything still works. I also checked all symlinks created in the filesystem-package (and did a reinstall): everything seems fine.
So this problem is solved, thanks again.