Arch Linux

inquisitivepenguin

Member

Registered: 2020-06-03

Posts: 4

[SOLVED] OS fails to boot properly due to a missing shared object file

Hello Arch Linux forum users! I've been trying to fix this issue for hours with no success.

A few days ago, I ran

sudo pacman -Syu

on my Arch operating system. However, I think the computer either went to sleep or was restarted before the installation process was finished. I didn't try to boot the OS until this morning. When I did, I was greeted by a kernel panic:

systemd-tmpfiles: error while loading shared libraries: libjson-c.so.5: cannot open shared object file: No such file or directory

I noticed this looked similar to another issue posted earlier on the forums (https://bbs.archlinux.org/viewtopic.php?id=255588), involving mismatches between package versions. So I booted a live USB, chrooted into my system, and ran

pacman -Syu

which seemed to work perfectly. Now that all packages had been upgraded to their proper versions, I thought my problems were over.

However, when I booted up my system, I was faced with not only the same errors regarding the missing libjson-c shared object file, but additional errors too:

[FAILED] Failed to start Load Kernel Modules.
[FAILED] Failed to mount /boot.
[DEPEND] Dependency failed for Local File Systems.

However, the kernel didn't panic in this case, and I was taken to a login prompt. However, my keyboard was completely unresponsive.

I've tried numerous things to fix the issue: refreshing the initramfs, regenerating etc/fstab, verifying `uname -a` and `pacman -Q linux` match, reinstalling the linux package, and so on.

However, there is one thing that makes me curious: the fact that the login prompt says I'm using an older version of Linux (the version I was using before my full system upgrade):

Arch Linux 5.4.14-arch1-1 (tty1)

My guess is that I'm booting from an older system image... but how?

I would appreciate any help!

Last edited by inquisitivepenguin (2020-06-04 02:27:57)

loqs

Member

Registered: 2014-03-06

Posts: 18,055

Re: [SOLVED] OS fails to boot properly due to a missing shared object file

Boot the live media,  chroot in what is the output of the following:

pacman -Q linux
file /boot/vmlinuz-linux
pacman -Qikk json-c

inquisitivepenguin

Member

Registered: 2020-06-03

Posts: 4

Re: [SOLVED] OS fails to boot properly due to a missing shared object file

Boot the live media,  chroot in what is the output of the following:

pacman -Q linux
file /boot/vmlinuz-linux
pacman -Qikk json-c

$ pacman -Q linux
linux 5.6.15.arch1-1
$ file /boot/vmlinuz-linux
/boot/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 5.6.15-arch1-1 (linux@archlinux) #1 SMP PREEMPT Wed, 27 May 2020 23:42:26 +0000, RO-rootFS, swap_dev 0x6, Normal VGA
$ pacman -Qikk json-c
Name            : json-c
Version         : 0.14-4
Description     : A JSON implementation in C
Architecture    : x86_64
URL             : https://github.com/json-c/json-c/wiki
Licenses        : MIT
Groups          : None
Provides        : libjson-c.so=5-64
Depends On      : glibc
Optional Deps   : None
Required By     : bluez  cryptsetup  gdal  libmypaint
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 168.96 KiB
Packager        : Jan Alexander Steffens (heftig) <jan.steffens@gmail.com>
Build Date      : Tue 05 May 2020 06:12:11 AM CDT
Install Date    : Wed 03 Jun 2020 09:48:21 AM CDT
Install Reason  : Installed as a dependency for another package
Install Script  : No
Validated By    : Signature

json-c: 34 total files, 0 altered files

loqs

Member

Registered: 2014-03-06

Posts: 18,055

Re: [SOLVED] OS fails to boot properly due to a missing shared object file

Exit the chroot,  unmount /mbt/boot but leaving the root filesystem mounted to /mnt then what is the output of:

file /mnt/boot/vmlinuz-linux

Edit:
Or if /mnt/boot was not mounted,  mount the boot filesystem to /mnt/boot then post the commands output.

Last edited by loqs (2020-06-03 22:06:42)

inquisitivepenguin

Member

Registered: 2020-06-03

Posts: 4

Re: [SOLVED] OS fails to boot properly due to a missing shared object file

Exit the chroot,  unmount /mbt/boot but leaving the root filesystem mounted to /mnt then what is the output of:

file /mnt/boot/vmlinuz-linux

Edit:
Or if /mnt/boot was not mounted,  mount the boot filesystem to /mnt/boot then post the commands output.

Sure enough, when I unmounted the boot partition I get this:

$ file /mnt/boot/vmlinuz-linux
/mnt/boot/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 5.4.14-arch1-1 (linux@archlinux) #1 SMP PREEMPT Thu, 23 Jan 2020 10:07:05 +0000, RO-rootFS, swap_dev 0x6, Normal VGA

That explains everything. My old /etc/fstab had the boot partition being mounted to the wrong directory, so my root partition had its own boot folder that was being loaded by GRUB. But when I was trying to fix my system, I was only updating the unused boot partition. What steps should I take to ensure that the proper boot partition is used by GRUB instead of the root partition's boot directory?

loqs

Member

Registered: 2014-03-06

Posts: 18,055

Re: [SOLVED] OS fails to boot properly due to a missing shared object file

I would first try GRUB#Generated_grub.cfg.  If that fails is the system booting using EFI or BIOS?

eschwartz

Fellow

Registered: 2014-08-08

Posts: 4,097

Re: [SOLVED] OS fails to boot properly due to a missing shared object file

One of the major features of grub is that it can detect and boot a kernel that isn't in the EFI system partition.

So you could simply stop mounting the ESP to /boot, personally I mounted it to /tmp/efi during the installation process. Reinstall the linux package to get the kernel/initramfs installed at /boot on your rootfs (ext4, btrfs, whatever).

Rerun grub-install to generate the grubx64.efi executable with your new ESP and /boot layout.

Now you'll never need to touch the ESP again, it doesn't need to be mounted and your kernel is not installed to it. It can never get out of sync. Your grub.cfg will be in /boot, as normal, and grubx64.efi will mount / in order to read /boot/grub/grub.cfg

The fact that people mount /boot as the ESP is entirely due to the undeserved popularity of the inferior systemd-boot, which is so "simple" it doesn't support anything, including proper filesystems. So you have no CHOICE but to install the kernel into the fat32 ESP.

Both grub and refind are perfectly fine with proper filesystems.

---

Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

inquisitivepenguin

Member

Registered: 2020-06-03

Posts: 4

Re: [SOLVED] OS fails to boot properly due to a missing shared object file

I would first try GRUB#Generated_grub.cfg.  If that fails is the system booting using EFI or BIOS?

Thanks, everything is working fine now! Here's what I did in case anyone else runs into this problem:
1. Unmounted the boot partition
2. Copied boot directory in root partition into a temporary directory, then removed the original boot directory
2. Mounted the boot partition to a newly created boot directory
3. Copy files from the temporary boot directory into the boot partition's directory.
4. Re-generated the config for GRUB
5. Used `grub-install` to update the boot loader entry for GRUB

Thanks for your help!