[solved] mkinitcpio encrypt hook misses libraries -> system unbootable

jarda-wien · 2015-02-11 11:55:40

Hello, I've been debugging this issue for quite some time now.

I have an encrypted root setup that has been working flawlessly for the past year or so. About a month ago I couldn't boot any more as I didn't get the password prompt during boot to open the root partition. The system jumps directly to "Kernel panic - not syncing: No working init found".

After some debugging, I found out that the initrd image is probably missing something so I recreated the same setup in virtualbox on another machine and transfered the initrd to the unbootable machine and it worked!

Comparing the two images, I see some differences. I suspect that the following might be causing the issue (from mkinitcpio -p linux -v on the virtual):

adding file: /usr/sbin/cryptsetup
adding file: /usr/lib/libcryptsetup.so.4     
adding file: /usr/lib/libpopt.so.0           
adding file: /usr/lib/libdevmapper.so.1.02   
adding file: /usr/lib/libgcrypt.so.20        
adding file: /usr/lib/libgpg-error.so.0      
adding file: /usr/lib/libudev.so.1           
adding file: /usr/lib/libdl.so.2  
adding file: /usr/sbin/dmsetup

Compare it to the output of the same command on the real computer:

adding file: /usr/sbin/cryptsetup
adding file: /usr/sbin/dmsetup

So i figured out and confirmed by lsinitcpio output that some libraries that seem important are indeed missing in the initrd image created on the real machine.

I can't seem to figure out why this happens... The mkinicpio.conf setup is the same and the real machine stopped working without any help from my side (apart from some pacman -Syu maybe...).

I've reinstalled mkinicpio and cryptsetup just to be sure, but that didn't help.

Now I am down to studying the bash sources of mkinitcpio to figure out how the libraries are identified and why they are omitted on the real machine.

Does anybody know why this might be happening? I couldn't find any similar issue in the forums.

Last edited by jarda-wien (2015-02-11 20:41:32)

frostschutz · 2015-02-11 12:53:00

Strange. It works fine for me (in a fresh ArchLinux install). What do you get for `ldd /usr/sbin/cryptsetup`?

jarda-wien · 2015-02-11 14:07:36

On the real system:

sudo ldd /usr/sbin/cryptsetup  
	not a dynamic executable

In the virtual system:

ldd /usr/sbin/cryptsetup
	linux-vdso.so.1 (0x00007fffa9baa000)
	libcryptsetup.so.4 => /usr/lib/libcryptsetup.so.4 (0x00007fb281972000)
	libpopt.so.0 => /usr/lib/libpopt.so.0 (0x00007fb281766000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007fb2813c3000)
	libuuid.so.1 => /usr/lib/libuuid.so.1 (0x00007fb2811be000)
	libdevmapper.so.1.02 => /usr/lib/libdevmapper.so.1.02 (0x00007fb280f75000)
	libgcrypt.so.20 => /usr/lib/libgcrypt.so.20 (0x00007fb280c94000)
	libgpg-error.so.0 => /usr/lib/libgpg-error.so.0 (0x00007fb280a82000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fb281b9a000)
	libudev.so.1 => /usr/lib/libudev.so.1 (0x00007fb281d9f000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007fb280865000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007fb28065d000)
	libcap.so.2 => /usr/lib/libcap.so.2 (0x00007fb280459000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007fb280255000)
	libattr.so.1 => /usr/lib/libattr.so.1 (0x00007fb280050000)

I've also tried reinstalling cryptsetup again just now and I have the same behaviour...

jarda-wien · 2015-02-11 14:09:09

BTW I also just tried doing a md5sum on /usr/sbin/cryptsetup in the virtual system and in the real one -> they are the same!

alphaniner · 2015-02-11 14:37:25

Though I can't imagine how it could have happened, one reason ldd would return "not a dynamic executable" is if the executable is for the wrong architecture.

frostschutz · 2015-02-11 14:55:52

Are you sure the cryptsetup binary is identical on both systems?! What about ldd /usr/bin/* - does it say static binary for all of them?! I never had a broken ldd before, very strange.

jarda-wien · 2015-02-11 15:05:42

ldd says not a dynamic executable for everything it seems... and yet everything is working as long as I don't shut down.

frostschutz · 2015-02-11 15:40:41

Something is odd here. `which ldd` is /usr/bin/ldd? Is the ldd identical with your other system too?

ldd is a shell script; you could create a copy of it with -x option added to the first line to see what it does in more detail, maybe that would show something

#! /usr/bin/bash -x

# x_ldd cryptsetup
+ TEXTDOMAIN=libc
+ TEXTDOMAINDIR=/usr/share/locale
+ RTLDLIST='/usr/lib/ld-linux.so.2 /usr/lib64/ld-linux-x86-64.so.2 /usr/libx32/ld-linux-x32.so.2'
+ warn=
+ bind_now=
+ verbose=
+ test 1 -gt 0
+ case "$1" in
+ break
+ add_env='LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW='
+ add_env='LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out'
+ add_env='LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out LD_VERBOSE='
+ test '' = yes
+ case $# in
+ single_file=t
+ result=0
+ for file in '"$@"'
+ test t = t
+ case $file in
+ file=./cryptsetup
+ test '!' -e ./cryptsetup
+ test '!' -f ./cryptsetup
+ test -r ./cryptsetup
+ test -x ./cryptsetup
+ RTLD=
+ ret=1
+ for rtld in '${RTLDLIST}'
+ test -x /usr/lib/ld-linux.so.2
+ for rtld in '${RTLDLIST}'
+ test -x /usr/lib64/ld-linux-x86-64.so.2
++ /usr/lib64/ld-linux-x86-64.so.2 --verify ./cryptsetup
+ verify_out=
+ ret=0
+ case $ret in
+ RTLD=/usr/lib64/ld-linux-x86-64.so.2
+ break
+ case $ret in
+ try_trace ./cryptsetup
++ eval LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= 'LD_LIBRARY_VERSION=$verify_out' LD_VERBOSE= '"$@"'
++ rc=0
++ printf x
++ exit 0
+ output='+++ LD_TRACE_LOADED_OBJECTS=1
+++ LD_WARN=
+++ LD_BIND_NOW=
+++ LD_LIBRARY_VERSION=
+++ LD_VERBOSE=
+++ ./cryptsetup
	linux-vdso.so.1 (0x00007fffe2334000)
	libcryptsetup.so.4 => /usr/lib/libcryptsetup.so.4 (0x00007f63b6fe3000)
	libpopt.so.0 => /usr/lib/libpopt.so.0 (0x00007f63b6dd7000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f63b6a34000)
	libuuid.so.1 => /usr/lib/libuuid.so.1 (0x00007f63b682f000)
	libdevmapper.so.1.02 => /usr/lib/libdevmapper.so.1.02 (0x00007f63b65e6000)
	libgcrypt.so.20 => /usr/lib/libgcrypt.so.20 (0x00007f63b6305000)
	libgpg-error.so.0 => /usr/lib/libgpg-error.so.0 (0x00007f63b60f3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f63b720b000)
	libudev.so.1 => /usr/lib/libudev.so.1 (0x00007f63b7410000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f63b5ed6000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007f63b5cce000)
	libcap.so.2 => /usr/lib/libcap.so.2 (0x00007f63b5aca000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f63b58c6000)
	libattr.so.1 => /usr/lib/libattr.so.1 (0x00007f63b56c1000)
x'
+ rc=0
+ printf %s '+++ LD_TRACE_LOADED_OBJECTS=1
+++ LD_WARN=
+++ LD_BIND_NOW=
+++ LD_LIBRARY_VERSION=
+++ LD_VERBOSE=
+++ ./cryptsetup
	linux-vdso.so.1 (0x00007fffe2334000)
	libcryptsetup.so.4 => /usr/lib/libcryptsetup.so.4 (0x00007f63b6fe3000)
	libpopt.so.0 => /usr/lib/libpopt.so.0 (0x00007f63b6dd7000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f63b6a34000)
	libuuid.so.1 => /usr/lib/libuuid.so.1 (0x00007f63b682f000)
	libdevmapper.so.1.02 => /usr/lib/libdevmapper.so.1.02 (0x00007f63b65e6000)
	libgcrypt.so.20 => /usr/lib/libgcrypt.so.20 (0x00007f63b6305000)
	libgpg-error.so.0 => /usr/lib/libgpg-error.so.0 (0x00007f63b60f3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f63b720b000)
	libudev.so.1 => /usr/lib/libudev.so.1 (0x00007f63b7410000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f63b5ed6000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007f63b5cce000)
	libcap.so.2 => /usr/lib/libcap.so.2 (0x00007f63b5aca000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f63b58c6000)
	libattr.so.1 => /usr/lib/libattr.so.1 (0x00007f63b56c1000)
'
+++ LD_TRACE_LOADED_OBJECTS=1
+++ LD_WARN=
+++ LD_BIND_NOW=
+++ LD_LIBRARY_VERSION=
+++ LD_VERBOSE=
+++ ./cryptsetup
	linux-vdso.so.1 (0x00007fffe2334000)
	libcryptsetup.so.4 => /usr/lib/libcryptsetup.so.4 (0x00007f63b6fe3000)
	libpopt.so.0 => /usr/lib/libpopt.so.0 (0x00007f63b6dd7000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f63b6a34000)
	libuuid.so.1 => /usr/lib/libuuid.so.1 (0x00007f63b682f000)
	libdevmapper.so.1.02 => /usr/lib/libdevmapper.so.1.02 (0x00007f63b65e6000)
	libgcrypt.so.20 => /usr/lib/libgcrypt.so.20 (0x00007f63b6305000)
	libgpg-error.so.0 => /usr/lib/libgpg-error.so.0 (0x00007f63b60f3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f63b720b000)
	libudev.so.1 => /usr/lib/libudev.so.1 (0x00007f63b7410000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f63b5ed6000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007f63b5cce000)
	libcap.so.2 => /usr/lib/libcap.so.2 (0x00007f63b5aca000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f63b58c6000)
	libattr.so.1 => /usr/lib/libattr.so.1 (0x00007f63b56c1000)
+ return 0
+ rc=0
+ '[' 0 = 5 ']'
+ '[' 0 = 0 ']'
+ exit 0

Last edited by frostschutz (2015-02-11 15:46:54)

jarda-wien · 2015-02-11 17:06:52

➜  ~  which ldd
/usr/bin/ldd
➜  ~  md5sum /usr/bin/ldd
9e22c4aedf1e2713fc6256931c4f139f  /usr/bin/ldd

➜  ~  bash -x /usr/bin/ldd /usr/sbin/cryptsetup
+ TEXTDOMAIN=libc
+ TEXTDOMAINDIR=/usr/share/locale
+ RTLDLIST='/usr/lib/ld-linux.so.2 /usr/lib64/ld-linux-x86-64.so.2 /usr/libx32/ld-linux-x32.so.2'
+ warn=
+ bind_now=
+ verbose=
+ test 1 -gt 0
+ case "$1" in
+ break
+ add_env='LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW='
+ add_env='LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out'
+ add_env='LD_TRACE_LOADED_OBJECTS=1 LD_WARN= LD_BIND_NOW= LD_LIBRARY_VERSION=$verify_out LD_VERBOSE='
+ test '' = yes
+ case $# in
+ single_file=t
+ result=0
+ for file in '"$@"'
+ test t = t
+ case $file in
+ :
+ test '!' -e /usr/sbin/cryptsetup
+ test '!' -f /usr/sbin/cryptsetup
+ test -r /usr/sbin/cryptsetup
+ test -x /usr/sbin/cryptsetup
+ RTLD=
+ ret=1
+ for rtld in '${RTLDLIST}'
+ test -x /usr/lib/ld-linux.so.2
++ /usr/lib/ld-linux.so.2 --verify /usr/sbin/cryptsetup
+ verify_out=
+ ret=1
+ case $ret in
+ for rtld in '${RTLDLIST}'
+ test -x /usr/lib64/ld-linux-x86-64.so.2
+ for rtld in '${RTLDLIST}'
+ test -x /usr/libx32/ld-linux-x32.so.2
+ case $ret in
+ nonelf /usr/sbin/cryptsetup
+ return 1
+ echo '	not a dynamic executable'
	not a dynamic executable
+ result=1
+ exit 1

frostschutz · 2015-02-11 17:18:33

Ummm. Okay, it goes down the nonelf path, like @alphaniner's predicted.

/usr/lib64/ld-linux-x86-64.so.2 --verify /usr/sbin/cryptsetup returns an error, but why?

# file -L /usr/lib64/ld-linux-x86-64.so.2 /usr/sbin/cryptsetup
/usr/lib64/ld-linux-x86-64.so.2: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fc0759b71584d2513b04f1b530965a61b5e499e2, not stripped
/usr/sbin/cryptsetup:            ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=ca89fedc8eae34c0fe7a8551ef3e86c6e8ff08d1, stripped

jarda-wien · 2015-02-11 20:40:53

No wonder it wasn't working. I was missing the /usr/lib64 link!!!

I've reinstalled the filesystem package and it seems to be working ok now.

I will have to find out why /usr/lib64 got deleted... I'll report back tomorow.

Anyway, should there be some kind of check in mkinitcpio for that as it resulted in a non-booting system? What is your opinion?

frostschutz · 2015-02-11 20:54:10

mkinitcpio checks various things but this error condition was already hidden by ldd, so it's the fault of ldd really - it should not claim "not a shared executable" when it really just had an error

jarda-wien · 2015-02-12 07:41:46

I see that ldd is part of glibc. I guess they didn't think that /usr/lib64/ could be a symlink. Looking at old archlinux news, I see that the switch was made by arch more than two years ago.

I'll look into the sources and try to propose something. That will put my virtual encrypted arch that used to generate my initrd to good use;-) I just hope that I can find time to do so.

Anyway, thank you frostschutz and alphaniner for you help. I was able to fix the issue in no time after you've pointed me to the right direction!

Arch Linux

#1 2015-02-11 11:55:40

[solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#2 2015-02-11 12:53:00

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#3 2015-02-11 14:07:36

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#4 2015-02-11 14:09:09

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#5 2015-02-11 14:37:25

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#6 2015-02-11 14:55:52

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#7 2015-02-11 15:05:42

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#8 2015-02-11 15:40:41

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#9 2015-02-11 17:06:52

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#10 2015-02-11 17:18:33

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#11 2015-02-11 20:40:53

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#12 2015-02-11 20:54:10

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

#13 2015-02-12 07:41:46

Re: [solved] mkinitcpio encrypt hook misses libraries -> system unbootable

Board footer