Arch Linux

alex.forencich

Member

Registered: 2011-05-29

Posts: 96

Systemd times out mounting multi-device, encrypted btrfs parition

I have a system (well, two actually) that have four-drive LUKS-encrypted btrfs arrays.  However, systemd has some issue with mounting them (and has had problems for many years, seems like nobody cares about actually fixing systemd).  If I remove the entry from fstab, I can manually mount it just fine.  But with the entry in fstab, systemd will automatically (and silently) unmount it. 

It seems that the issue has something to do with the device mapper layer, though it's unclear exactly what is going on. 

The array consists of four disks that are LUKS encrypted.  /etc/crypttab looks like so:

am0          UUID=bc...a9    /path/to/file    keyfile-timeout=30,x-systemd.device-timeout=30
am1          UUID=ec...ae    /path/to/file    keyfile-timeout=30,x-systemd.device-timeout=30
am2          UUID=ab...3c    /path/to/file    keyfile-timeout=30,x-systemd.device-timeout=30
am3          UUID=65...f3    /path/to/file    keyfile-timeout=30,x-systemd.device-timeout=30

The entry in /etc/fstab looks like so:

/dev/mapper/am0 /mnt/array btrfs nofail,x-systemd.mount-timeout=600,x-systemd.device-timeout=60,x-systemd.requires=/dev/mapper/am1,x-systemd.requires=/dev/mapper/am2,x-systemd.requires=/dev/mapper/am3 0 0

Without all of the devices manually added with x-systemd.requires, it seems like systemd only detects the dependency on /dev/mapper/am0. 

$ systemctl list-dependencies mnt-array.mount -all
mnt-array.mount
● ├─-.mount
● │ └─-.slice
● ├─dev-mapper-am0.device
● │ └─systemd-cryptsetup@am0.service
● │   ├─-.mount
● │   │ └─-.slice
● │   ├─dev-disk-by\x2duuid-bc...a9.device
● │   ├─system-systemd\x2dcryptsetup.slice
● │   │ └─system.slice
● │   │   └─-.slice
● │   └─blockdev@dev-mapper-am0.target
● ├─dev-mapper-am1.device
● │ └─systemd-cryptsetup@am1.service
● │   ├─-.mount
● │   │ └─-.slice
● │   ├─dev-disk-by\x2duuid-ec...ae.device
● │   ├─system-systemd\x2dcryptsetup.slice
● │   │ └─system.slice
● │   │   └─-.slice
● │   └─blockdev@dev-mapper-am1.target
● ├─dev-mapper-am2.device
● │ └─systemd-cryptsetup@am2.service
● │   ├─-.mount
● │   │ └─-.slice
● │   ├─dev-disk-by\x2duuid-ab...3c.device
● │   ├─system-systemd\x2dcryptsetup.slice
● │   │ └─system.slice
● │   │   └─-.slice
● │   └─blockdev@dev-mapper-am2.target
● ├─dev-mapper-am3.device
● │ └─systemd-cryptsetup@am3.service
● │   ├─-.mount
● │   │ └─-.slice
● │   ├─dev-disk-by\x2duuid-65...f3.device
● │   ├─system-systemd\x2dcryptsetup.slice
● │   │ └─system.slice
● │   │   └─-.slice
● │   └─blockdev@dev-mapper-am3.target
● └─system.slice
●   └─-.slice

In this case, all are "green" except dev-mapper-am1.device, dev-mapper-am2.device, and dev-mapper-am3.device, which are gray.  Rebooting results in systemd reporting various combinations of good and dead devices.  Now, they all show up just fine in /dev:

$ ls /dev/mapper
am0  am1  am2  am3  control

And since when I remove the entry from fstab, everything mounts (manually) just fine, so systemd is obviously not tracking the state of these devices correctly. 

This is what systemd reports for the status of those devices:

$ systemctl status --all "dev-mapper-am*.device"
● dev-mapper-am3.device - /dev/mapper/am3
     Loaded: loaded
    Drop-In: /run/systemd/generator/dev-mapper-am3.device.d
             └─40-device-timeout.conf
     Active: inactive (dead)

● dev-mapper-am1.device - /dev/mapper/am1
     Loaded: loaded
    Drop-In: /run/systemd/generator/dev-mapper-am1.device.d
             └─40-device-timeout.conf
     Active: inactive (dead)

● dev-mapper-am2.device - /dev/mapper/am2
     Loaded: loaded
    Drop-In: /run/systemd/generator/dev-mapper-am2.device.d
             └─40-device-timeout.conf
     Active: inactive (dead)

● dev-mapper-am0.device - /dev/mapper/am0
   Follow: unit currently follows state of sys-devices-virtual-block-dm\x2d3.device
     Loaded: loaded
    Drop-In: /run/systemd/generator/dev-mapper-am0.device.d
             └─40-device-timeout.conf, 50-device-timeout.conf
     Active: active (plugged) since Thu 2021-01-07 17:24:39 PST; 16min ago
     Device: /sys/devices/virtual/block/dm-3

Jan 07 17:24:39 atomic systemd[1]: Found device /dev/mapper/am0.

Sometimes the failed devices include logs, but not always.  For example:

● dev-mapper-am0.device - /dev/mapper/am0
     Loaded: loaded
    Drop-In: /run/systemd/generator/dev-mapper-am0.device.d
             └─40-device-timeout.conf, 50-device-timeout.conf
     Active: inactive (dead)

Jan 07 17:08:48 atomic systemd[1]: dev-mapper-am0.device: Job dev-mapper-am0.device/start timed out.
Jan 07 17:08:48 atomic systemd[1]: Timed out waiting for device /dev/mapper/am0.
Jan 07 17:08:48 atomic systemd[1]: dev-mapper-am0.device: Job dev-mapper-am0.device/start failed with result 'timeout'.

and:

● dev-mapper-am1.device - /dev/mapper/am1
     Loaded: loaded
    Drop-In: /run/systemd/generator/dev-mapper-am1.device.d
             └─40-device-timeout.conf
     Active: inactive (dead)

Jan 07 17:08:48 atomic systemd[1]: Unnecessary job for /dev/mapper/am1 was removed.

Basically, the most specific indication I have of any kind of failure is "dev-mapper-am0.device: Job dev-mapper-am0.device/start failed with result 'timeout'." for some combination of devices.  However, the devices always get unlocked and mapped, so I'm assuming this is either a systemd bug or a configuration issue of some sort.  As you can see, I have added a bunch of timeout parameters to fstab and crypttab, but it hasn't made any difference.  Is there anything else that I should be looking at here?

Last edited by alex.forencich (2021-01-08 02:16:05)