Systemd dependency fail for a UUID that is not listed in grb.cfg

foki · 2020-06-14 21:47:18

I did a regular system update for the first time since the pandemic has emerged.

In the boot time my systems prints:

Starting version 245.6.6-arch
/dev/nvme0n1p1: clean, ...
[TIME] Time out waiting for device /dev/disk/by-uuid/SOME_UUID
[DEPENDS] Dependency failed for /dev/disk/by-uuid/SAME_UUID
[DEPENDS] Dependency failed for Swap.

I cannot enter any other tty, because Linux did not even boot, I guess.

When I check my grub.cfg from GRUB mode, that SOME_UUID does not even match what is in grub.cfg in the line:

linux /boot/vmlinuz-linux root=SOME_OTHER_UUID

My first question is; why does it even search for SOME_UUID when it is not even listed in grub.cfg?

You may be asking for my pacman log, but I cannot enter tty and it's unreachable to me. I don't even have my Arch ISO to arch-chroot into it.

Does anyone have any wild guesses on what might have gone wrong here?

Meanwhile, I am running back home to get an USB stick and make an Arch ISO.

Last edited by foki (2020-06-14 21:50:12)

Wild Penguin · 2020-06-14 22:12:51

That boot message (systemd) snippet says it can not find your SWAP partition, and it's dependencies.

Grub will (typically) only contain the UUID of the root filesystem (one exception is the resume partition; the Kernel command line does not care about others). Other filesystems are configured in /etc/fstab, which will be in the root file system (usually).

You need to chroot into that installation and check what is wrong from there (well, don't necessarily need to, but that would make sense to figure out what is wrong and to prevent it from happening again). Might make sense to fsck the root before chrooting.

Does that system have more than one HDD? If yes, are you sure all of them work? If one disk would have failed and/or a cable loose / failed, that would explain the error.

I can not see how pacman could have caused that error. However, the system should still boot without without SWAP, so there is still something else wrong, too (perhaps other file systems have failed?).

Before chrooting into that installation, post the output of these commands here after booting from the USB:

lsblk
blkid

And after chrooting:

cat /etc/fstab

(redirect the output somewhere you have write access and use another computer or OS to read the files).

Cheers!

foki · 2020-06-15 00:43:06

Thank you @Wild Penguin!

Actually, the UUID of my swap partition in /etc/fstab was wrong. I took the right one from blkid, updated /etc/fstab accordingly, and now it boots nicely.

Could you tell us, newbies, more about why you thought that it should boot even if swap UUID is wrong?

Last edited by foki (2020-06-15 00:43:27)

Trilby · 2020-06-15 01:32:29

foki wrote:

Could you tell us, newbies, more about why you thought that it should boot even if swap UUID is wrong?

Because swap is not necessary for the system to boot or run properly. It's an added convenience resource, but absolutely nothing should depend on it. No sanely designed system would completely fail to boot and function in any way due simply to a missing swap partition, or to a failed network connection, or to any number of other bits that are not required for a system to function.

No sane system would behave this way. But what happened to you is - while insane - the expected result of a default arch linux configuration.

Last edited by Trilby (2020-06-15 01:33:05)

seth · 2020-06-15 05:46:51

"man fstab"; man "systemd.swap" - add "nofail" to the expendable entries.

@Trilby, do you happen to know how to undefault this beauty out of systemd?

Head_on_a_Stick · 2020-06-15 08:08:52

foki wrote:

Could you tell us, newbies, more about why you thought that it should boot even if swap UUID is wrong?

I've just tested by incorrecting my fstab swap entry and my Arch system does boot but only after a delay. You should have waited longer.

seth wrote:

do you happen to know how to undefault this beauty out of systemd?

Remove the swap line from /etc/fstab and let systemd automount it.

See also https://systemd.io/DISCOVERABLE_PARTITIONS/

seth · 2020-06-15 10:05:27

Or, as mentioned, you could set it nofail and retain your other options…
But that's besides the point which was (conceding that your findings suggest that this is not indeed the case) to prevent systemd from failing the boot on spurious fstab entries by some *default* configuration.

Trilby · 2020-06-15 12:00:33

Sure, use an init system that isn't contiually falling on it's face.

I'd speculate that modifying the genfstab script could provide a good workaround. But it's the very concept of unit dependencies rendering a system unbootable that is repeatedly resulting in silly breakage. I don't know if there is any good way to appropruately patch a system that has a fundamentally flawed design.

Of course I don't know how welcome these sentiments will be here. We should not, and are not allowed to insult various bits of software, but in a couple particular cases that has resulted in untouchable software for which legitimate critiques based on plainly visible evidence are frowned upon. Which if that's the case, I'll concede that the emperors new software is lovely.

Last edited by Trilby (2020-06-15 12:03:43)

Head_on_a_Stick · 2020-06-15 13:45:15

Just tried completely removing my swap partition and Arch still boots, albeit with the standard[0] 90 second timeout:

Jun 15 14:15:48 archie kernel: audit: type=1131 audit(1592226948.059:60): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-rfkill comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 15 14:17:12 archie systemd[1]: dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.device: Job dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.device/start timed out.
Jun 15 14:17:12 archie systemd[1]: Timed out waiting for device /dev/disk/by-uuid/e24fe3fe-34d1-49a7-8ae2-aca08d5a8887.
Jun 15 14:17:12 archie systemd[1]: Dependency failed for /dev/disk/by-uuid/e24fe3fe-34d1-49a7-8ae2-aca08d5a8887.
Jun 15 14:17:12 archie systemd[1]: Dependency failed for Swap.
Jun 15 14:17:12 archie systemd[1]: swap.target: Job swap.target/start failed with result 'dependency'.
Jun 15 14:17:12 archie systemd[1]: dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.swap: Job dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.swap/start failed with result 'dependency'.
Jun 15 14:17:12 archie systemd[1]: dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.device: Job dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.device/start failed with result 'timeout'.

In fairness to the OP there was also a 90 second delay when I tried to log in, which can be reproduced afterwards by switching to a console and using <ctrl>-d. No idea what that's about, perhaps it is a bug.

EDIT: journal:

Jun 15 14:47:49 archie systemd[1]: Stopped Getty on tty3.
Jun 15 14:47:49 archie systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
Jun 15 14:47:49 archie systemd[1]: Condition check resulted in Store a System Token in an EFI Variable being skipped.
Jun 15 14:47:49 archie systemd[1]: Condition check resulted in First Boot Wizard being skipped.
Jun 15 14:47:49 archie systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
Jun 15 14:47:49 archie kernel: audit: type=1130 audit(1592228869.819:166): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=getty@tty3 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 15 14:47:49 archie kernel: audit: type=1131 audit(1592228869.819:167): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=getty@tty3 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Jun 15 14:47:49 archie systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
Jun 15 14:48:14 archie systemd[1]: Condition check resulted in FUSE Control File System being skipped.
Jun 15 14:48:14 archie systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
Jun 15 14:48:14 archie systemd[1]: Condition check resulted in Store a System Token in an EFI Variable being skipped.
Jun 15 14:48:14 archie systemd[1]: Condition check resulted in First Boot Wizard being skipped.
Jun 15 14:48:14 archie systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
Jun 15 14:48:14 archie systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
Jun 15 14:48:16 archie systemd[1]: Condition check resulted in FUSE Control File System being skipped.
Jun 15 14:48:16 archie systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
Jun 15 14:48:16 archie systemd[1]: Condition check resulted in Store a System Token in an EFI Variable being skipped.
Jun 15 14:48:16 archie systemd[1]: Condition check resulted in First Boot Wizard being skipped.
Jun 15 14:48:16 archie systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
Jun 15 14:48:16 archie systemd[1]: Condition check resulted in Commit a transient machine-id on disk being skipped.
Jun 15 14:49:20 archie systemd[1]: dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.device: Job dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.device/start timed out.
Jun 15 14:49:20 archie systemd[1]: Timed out waiting for device /dev/disk/by-uuid/e24fe3fe-34d1-49a7-8ae2-aca08d5a8887.
Jun 15 14:49:20 archie systemd[1]: Dependency failed for /dev/disk/by-uuid/e24fe3fe-34d1-49a7-8ae2-aca08d5a8887.
Jun 15 14:49:20 archie systemd[1]: Dependency failed for Swap.
Jun 15 14:49:20 archie systemd[1]: swap.target: Job swap.target/start failed with result 'dependency'.
Jun 15 14:49:20 archie systemd[1]: dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.swap: Job dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.swap/start failed with result 'dependency'.
Jun 15 14:49:20 archie systemd[1]: dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.device: Job dev-disk-by\x2duuid-e24fe3fe\x2d34d1\x2d49a7\x2d8ae2\x2daca08d5a8887.device/start failed with result 'timeout'.
Jun 15 14:49:20 archie systemd[1]: Started Getty on tty3.

Trilby wrote:

The very concept of unit dependencies rendering a system unbootable

That is not the case here, we can put the pitchforks and torches away and stop trying to find out where Mr Poettering lives

[0] As defined by DefaultTimeoutStartSec in /etc/systemd/system.conf

Last edited by Head_on_a_Stick (2020-06-15 13:57:24)

foki · 2020-06-15 14:20:59

Head_on_a_Stick wrote:

In fairness to the OP there was also a 90 second delay when I tried to log in, which can be reproduced afterwards by switching to a console and using <ctrl>-d. No idea what that's about, perhaps it is a bug.

Well, it's not what was happening in my case. It prints the message, then waits 90 seconds, then again prints the message, and so on. It was never booting with the incorrect swap partition UUID.

This was my incorrect fstab:

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p1
UUID=480749ce-4304-46ec-8cde-dab3f40a9f35	/         	ext4      	rw,relatime	0 1

# /dev/nvme0n1p3
UUID=fc73a45f-5e0b-4dd5-a93a-60046f2aae9c	/home     	ext4      	rw,relatime	0 2

# /dev/nvme0n1p2
UUID=d368dc83-db79-4e33-99f6-403b6ca2b036	none      	swap      	defaults  	0 0

Trilby · 2020-06-15 14:42:03

Head_on_a_Stick wrote:

That is not the case here

Yes, it clearly is the case here. Read the thread you replied to. If you have some *different* problem, so be it. I was not replying to your different problem that had not yet been presented. I was responding to the problem the OP of this thread had. The fact that you can't replicate the problem doesn't mean it doesn't exist. But welcome to stage 2, please continue on in the process.

As for "pictchforks and torches" that's just as I suspected, evidence-based critiques are turned into strawmen to tear down. This is quite intellectually disengenuous and inflammatory.

Last edited by Trilby (2020-06-15 14:59:06)

seth · 2020-06-15 14:57:12

The boundaries are acutally set, https://wiki.archlinux.org/index.php/Co … d_projects

Views, experiences and opinions are always welcome, but unproductive slander is not.

It's the very concept of unit dependencies rendering a system unbootable that is … a fundamentally flawed design

This is a reasoned opinion on a narrow topic, miles away from any kind of "slander"… unlike tokens in more recent posts.

Also I guess the suggests the pitchfork thing wasn't a take on the current thread but just a reference that - for some unknown reason than nobody can understand - Lennart tends to get some ;-)

What worries me about the topic at hand is that the OP doesn't even get dropped into a rescue shell.
@HoaS, do you mind to post your "broken " fstab?

Edit: the hoodie looks fake, though.

Last edited by seth (2020-06-15 14:58:48)

Head_on_a_Stick · 2020-06-15 15:12:33

foki wrote:

Well, it's not what was happening in my case. It prints the message, then waits 90 seconds, then again prints the message, and so on. It was never booting with the incorrect swap partition UUID.

Ah, I see, my apologies. That was not immediately clear in your OP.

It is normal for systemd to alert the user to failing mountpoints by delaying the boot for DefaultTimeOut seconds and printing a warning, the system should boot afterwards.

Do you happen to have the journal contents for the problematic boots? Or if you could reproduce the problem to generate some more that would be fantastic, TIA.

Trilby wrote:

As for "pictchforks and torches" that's just as I suspected, evidence-based critiques are turned into strawmen to tear down. This is quite intellectually disengenuous and inflammatory.

Calm down d00d, that was just a joke.

seth wrote:

do you mind to post your "broken " fstab?

I don't usually run with an fstab so I made one with genfstab(8), here it is:

# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/nvme0n1p3
UUID=5949b48e-d74b-48ca-9bbe-8201e521f1f5       /               btrfs           rw,relatime,ssd,space_cache,subvolid=336,subvol=/arch,subvol=arch       0 0

# /dev/nvme0n1p2
UUID=e24fe3fe-34d1-49a7-8ae2-aca08d5a8887       none            swap            defaults        0 0

I created that file then deleted /dev/nvme0n1p2 and rebooted to generate the posted logs.

seth · 2020-06-15 15:40:09

Could be because "/" has fs_passno 0 instead of recommended 1 (the value bleeding over)

@foki, if you can still reproduce this with the broken fstab, try to make every last columns "… 0 0"

Scimmia · 2020-06-15 15:44:11

the root partition generally gets fscked in the initramfs, why would you give it a pass number to fsck it again?

Head_on_a_Stick · 2020-06-15 15:52:47

seth wrote:

Could be because "/" has fs_passno 0 instead of recommended 1

Btrfs doesn't fsck. See fsck.btrfs(8).

seth · 2020-06-15 15:54:26

"recommended by the fstab manpage", initramfs fsck depends on the hook, does it?
Edit: (idk how genfstab determines those values)
Off topic: genfstab defaults to 2, sets the root to 1 and filesystems w/o available fsck to 0 - the condition of any initramfs isn't considered.

Edit

Btrfs doesn't fsck.

Yes, but that's irrelevant for the theoretic bug (systemd using the value to determine the hard dependency of the FS and accidentally carries it over or just uses a global increment or so)

Last edited by seth (2020-06-15 16:36:19)

Arch Linux

#1 2020-06-14 21:47:18

Systemd dependency fail for a UUID that is not listed in grb.cfg

#2 2020-06-14 22:12:51

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#3 2020-06-15 00:43:06

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#4 2020-06-15 01:32:29

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#5 2020-06-15 05:46:51

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#6 2020-06-15 08:08:52

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#7 2020-06-15 10:05:27

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#8 2020-06-15 12:00:33

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#9 2020-06-15 13:45:15

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#10 2020-06-15 14:20:59

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#11 2020-06-15 14:42:03

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#12 2020-06-15 14:57:12

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#13 2020-06-15 15:12:33

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#14 2020-06-15 15:40:09

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#15 2020-06-15 15:44:11

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#16 2020-06-15 15:52:47

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

#17 2020-06-15 15:54:26

Re: Systemd dependency fail for a UUID that is not listed in grb.cfg

Board footer