Arch Linux

Captain Rage

Member

Registered: 2012-02-02

Posts: 49

[Solved] Unable to boot / start the system properly after upgrading it

Hi. This Arch Linux system has been running virtually without any upgrade-related issues since it was build in 2018 and of course now, just before Christmas and the holiday season, the first substantial trouble arose. Thanks for your attention and merry Christmas in advance. :-)

The problem:
After upgrading the system yesterday ('sudo pacman -Ssyu') from a terminal within KDE Plasma the computer refuses to start normally. It simply reaches:

Starting systemd-udevd version 252.3-1-arch
/dev/sdc2: clean, 1037310/30507008 files, 74204424/122019590 blocks 

... and nothing else happens! Normally all partitions get mounted, which merely takes seconds, and KDE fires up.

Also, after the GRUB entry is selected prior to this screen, the computer shows the ASUS bios logo again for a moment before proceeding. I am not entirely sure if it was doing this before.

It is almost as if systemd can't access or mount /boot/, or something, although it is still possible to switch TTY.. and /boot/ and the other partitions show up as mounted.

Background:

The computer might have been a tiny bit sluggish before running the upgrade, since it had not been rebooted for a few days, only resumed from and put into hibernation, although all desktop programs were closed prior to running the upgrade, which is how I usually carry out upgrades and normally never ever have any problems.

During and after the upgrade, no errors to upgrading the packages or post--transactional hooks were visible, but the computer was noticeably slow while executing 'sync' in the terminal.

After rebooting initially, something seemed borked up with GRUB akin to this thread: https://bbs.archlinux.org/viewtopic.php?id=269588
Where

[FAILED] Failed to mount /boot.
[DEPEND] Dependency failed for Local File Systems.
You are in emergency mode. After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" or "exit" to boot into default mode.
Give root password for maintenance
(or press Control-D to continue): _

(copied from the other thread, but the error message was identical)

I tried to resolve it by following the steps in the thread, and commented out the boot partition in /etc/fstab/, rebooted, and got to the state mentioned above. Inbetween I ran 'grub-install' and 'grub-mkconfig -o /boot/grub/grub.cfg' more than once, which were executed without any errors, since when changing TTY it actually looks like all partitions are mounted properly.

Before doing this however, I tried to reinstall the kernel with 'pacman -S linux'. At first, this message appeared:

Total Installed Size: 163.44 MiB
Net Upgrade Size: 0.00 MiB

:: Proceed with installation? [Y/n]
(1/1) checking keys in keyring
(1/1) checking package integrity
(1/1) loading package files
(1/1) checking for file conflicts
(1/1) checking available disk space
:: Processing package changes...
(1/1) reinstalling linux
:: Running post-transaction hooks...
(1/3) Arming ConditionNeedsUpdate...
(2/3) Updating module dependencies...
(3/3) Updating linux initcpios...
==> WARNING: Preset file `/etc/mkinitcpio.d/linux.preset' is empty or does not contain any presets.

I deleted the preset file as suggested in another thread and was able to reinstall the kernel otherwise.

Running 'uname -a' suggests the kernel matches the bootloader files.

As mentioned, the partitions seem to get mounted. In worst case I will try to copy all files from /home/ to an external drive that is usually connected (it also shows up as mounted).

According to systemctl the ssdm service which starts up KDE is running.

When the computer halts indefinitely at the state mentioned above, you can hear that the mechanical drives spin for a moment, and it is still possible to change TTY or turn it off using the power button (in the latter case two messages about watchdog will briefly flash by before it powers down).

Troubleshooting:

What can be done to troubleshoot this issue further? What might have gone awry? Neither 'dmesg' nor the pacman log nor 'journalctl -xb' show any helpful clues.

Thank you!

Last edited by Captain Rage (2022-12-18 09:42:59)

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 76,411

Re: [Solved] Unable to boot / start the system properly after upgrading it

Can you still boot the multi-user.target (2nd link below), in doubt along "nomodeset"?

Running 'uname -a' suggests the kernel matches the bootloader files.

The question is whether it matches the on-disk installed modules, because

[FAILED] Failed to mount /boot.

sounds like you forgot to mount the /boot partition before the update and are booting a dated kernel, compare

uname -a
pacman -Qs linux

I deleted the preset file as suggested in another thread and was able to reinstall the kernel otherwise.

Did the "is empty or does not contain any presets" warning go away afterwards?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Captain Rage

Member

Registered: 2012-02-02

Posts: 49

Re: [Solved] Unable to boot / start the system properly after upgrading it

Thanks for your reply! It led me to the source of the problem as well as the solution, even though it was wrong of me to assume the issue pertained to a faulty boot partition or files. To answer your questions: 'uname -a' corresponded to 'pacman -Qs linux', and yes, adding the multi-target parameter to the GRUB command line made the system fire up TTY1 instead of halting indefinitely at the 'Startin systemd-udevd' line, hence it made a difference (and the error message with the empty preset file went away after deleting the preset file and reinstalling the linux package).

The problem turned however to be lots of wrongly written files post upgrade (like the xorg-server package and plenty of other). The linux package seemed borked up at first, but since partitions got mounted properly it must have meant that /boot/ was doing its work.

While trying to reinstall all packages from the previous upgrade there was a slew (> 70) of similar error messages to this one:

xorg-server: /usr/bin/X exists on filesystem

Eventually forcing overwrites in the package manager and simply rebooting the system fixed everything:

pacman -S --overwrite "*"

followed by all the packages.

I suspect I did a bad move, running a system upgrade after an extended period of not turning off the computer (which has good specifications), only suspending and resuming it, while running programs that required lots of RAM which eventually led to the computer writing these changes in a sluggish fashion, while I didn't let it finish before powering it off (even though pacman reported the package upgrades as successful along with the linux package post-transaction hooks, but just after upgrading running 'sync' took ages so I simply rebooted the computer).

It is also amazing that the system booted at all and that nothing more important got corrupted.

Thanks for your input, seth. It helped to confirm and refute some of the leads I found when troubleshooting the initial problem and averted getting into more trouble or further distress. Plus, now I will know how to approach the problem with an inaccessible /boot/ or a mismatched kernel and related modules, if that ever happens in the future. Much appreciated! Everything is good now and the system works as intended. Merry Christmas!

Last edited by Captain Rage (2022-12-18 09:48:12)