Arch Linux

ZSmith

Member

Registered: 2017-02-25

Posts: 19

[SOLVED] Debug Failed Systemd Units During Startup

I sat down at my computer this afternoon and it no longer is able to boot to a graphical environment.

The system gets as far as the "Welcome to Arch Linux!" prompt, but then a flurry of messages announcing failed systemd services appear. I suspect that some service early in the boot process is failing, and virtually every other subsequent service fails as a consequence. Unfortunately the messages fly off the screen instantly, and it's not clear which service is the first to fail. After a few minutes the last few services timeout and the system is left in a hung state where there's no way to input any kind of commands. The one thing I am able to do is press ctrl+alt+delete to trigger a reboot.

I tried loading the system in an arch-chroot and checking the output of journalctl, but there is no record in the journal of the boot attempt. I suspected filesystem problems, but the root filesystem mounts fine from a live disk, and fsck completes without incident. The one thing that is worth noting is that the root filesystem is an unusual bcachefs multi-drive setup, but it has worked without issue for the past year.

I suspect that if I could could see the journal logs for the failed boot, the source of the problem would be obvious.

Is there some way I can step through the service initialization process one service at the time, or at least write out the journal to a file so I can view it in a live disk?

Last edited by ZSmith (2025-03-31 00:47:31)

twelveeighty

Member

Registered: 2011-09-04

Posts: 1,295

Re: [SOLVED] Debug Failed Systemd Units During Startup

Step 1: Boot into the Arch ISO, mount your data/home partition(s) and back up your data.

Always read the Wiki first before posting: Specify a different journal to view

To fix your system, use the Arch ISO and change your bootloader to add `systemd.unit=multi-user.target` or perhaps even `systemd.unit=rescue.target` (start with multi-user) as a kernel parameter. If you're lucky, the hanging after the failed boot is due to graphical issues only and you should be able to log into the console, read the journal and check/repair the issue(s). If that doesn't work, read the Pacman crashes during an upgrade instructions and go from there.

Last edited by twelveeighty (2025-03-26 14:34:28)

ZSmith

Member

Registered: 2017-02-25

Posts: 19

Re: [SOLVED] Debug Failed Systemd Units During Startup

Thank you for the suggestions.

I tried booting with systemd.unit=multi-user.target and systemd.unit=rescue.target has no effect by modifying the kernel options in rEFInd, but in both cases I ended up with the system stuck without a shell.

I double checked that I was looking at the Journal for the mounted system, and not the live installation environment and confirmed that nothing is being added to the Journal file during the failed boot attempts.

Finally, I tried going through the steps in "Pacman crashes during an upgrade", but this also had no effect. I even tried reinstalling all packages with `pacman -Qqn | pacman -S -`, but this too didn't change anything.

seth

Member

Registered: 2012-09-03

Posts: 62,236

Re: [SOLVED] Debug Failed Systemd Units During Startup

Unfortunately the messages fly off the screen instantly, and it's not clear which service is the first to fail.

I bet you have a phone with a camera that can read faster than you.
If you don't have a tripod, makeshift one by wedging the phone between two books -\_ angling the camera at the moniotor (handheld will be too shaky and blurry to read)

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

twelveeighty

Member

Registered: 2011-09-04

Posts: 1,295

Re: [SOLVED] Debug Failed Systemd Units During Startup

When you perform seth's suggestion with the video recording, make sure you do not have `quiet` and `splash` as your kernel parameter(s).

While we wait for you to post the video, I'll caution again: if booting into multi-user.target doesn't result in any journal messages being captured, it *could* be a disk failure, so make sure you get valuable data off that disk backed up first.

ZSmith

Member

Registered: 2017-02-25

Posts: 19

Re: [SOLVED] Debug Failed Systemd Units During Startup

Taking a video of the boot process and advancing frame-by-frame showed that an error message reading `Failed to mount Huge File System` appeared early in the boot process.

A quick google search suggested that having `glibc-eac` installed might to be blame . Removing it and installing `glibc` resolved all issues.

Thank you seth for the low-tech advice.