You are not logged in.
Alright, I'm having a rather strange issue here. I just ran updates a headless box that's literally in the opposite corner of the country that hadn't been rebooted in about a month and it didn't come back up after rebooting, so I had to log in via IPMI to figure out what happened. Here's the setup:
System has 5 hard drives, one for root and swap and four in a btrfs raid1 for data. The four drives in the btrfs raid1 are encrypted with LUKS, primarily as a safeguard against data recovery in the event a drive fails such that it cannot be non-destructively erased (useful if I want to RMA it). /etc/crypttab is responsible for getting all four drives unlocked and mapped via UUID. Then /etc/fstab is supposed to get the btrfs partition mounted, also by UUID.
However, the second step seems to be failing for some reason. Whenever systemd attempts to mount the partition, it times out. However, as soon as the system boots (well, once I changed 'defaults' to 'nofail' in /etc/fstab), I can just run sudo mount /mnt/array and it mounts just fine. I also tried noauto,x-systemd.automount in /etc/fstab. This causes systemd to attempt to mount the partition many times, but it never succeeds. I also tried mounting by device (/dev/mapper/am0) instead of by UUID in /etc/fstab, but this also failed.
Btrfs is definitely detecting the unlocked and mapped drives:
Aug 22 06:28:05 atomic kernel: BTRFS: device label array devid 1 transid 223737 /dev/dm-0
Aug 22 06:28:05 atomic kernel: BTRFS: device label array devid 3 transid 223737 /dev/dm-1
Aug 22 06:28:05 atomic kernel: BTRFS: device label array devid 4 transid 223737 /dev/dm-2
Aug 22 06:28:05 atomic kernel: BTRFS: device label array devid 2 transid 223737 /dev/dm-3But systemd fails to mount the array for some unknown reason:
Aug 22 06:29:25 atomic systemd[1]: dev-disk-by\x2duuid-5a...92.device: Job dev-disk-by\x2duuid-5a...92.device/start timed out.
Aug 22 06:29:25 atomic systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-5a...92.device.
Aug 22 06:29:25 atomic systemd[1]: Dependency failed for /mnt/array.
Aug 22 06:29:25 atomic systemd[1]: mnt-array.mount: Job mnt-array.mount/start failed with result 'dependency'.
Aug 22 06:29:25 atomic systemd[1]: dev-disk-by\x2duuid-5a...92.device: Job dev-disk-by\x2duuid-5a...92.device/start failed with result 'timeout'./etc/crypttab:
am0 UUID=fd...26 /path/to/keyfile
am1 UUID=21...c7 /path/to/keyfile
am2 UUID=25...f6 /path/to/keyfile
am3 UUID=ac...e8 /path/to/keyfile/etc/fstab:
UUID=ed...95 swap swap defaults 0 0
UUID=2f...df / ext4 defaults 0 1
UUID=5a...92 /mnt/array btrfs defaults 0 0output of blkid:
/dev/sda1: LABEL="arch" UUID="2f...df" TYPE="ext4" PARTUUID="00005f60-01"
/dev/sda2: UUID="ed...95" TYPE="swap" PARTUUID="00005f60-02"
/dev/sdb1: UUID="fd...26" TYPE="crypto_LUKS" PARTLABEL="Linux filesystem" PARTUUID="5e...22"
/dev/sdc1: UUID="21...c7" TYPE="crypto_LUKS" PARTLABEL="Linux filesystem" PARTUUID="68...0b"
/dev/sdd1: UUID="25...f6" TYPE="crypto_LUKS" PARTLABEL="Linux filesystem" PARTUUID="ff...2b"
/dev/sde1: UUID="ac...e8" TYPE="crypto_LUKS" PARTLABEL="Linux filesystem" PARTUUID="c1...e8"
/dev/mapper/am0: LABEL="array" UUID="5a...92" UUID_SUB="1d...a6" TYPE="btrfs"
/dev/mapper/am2: LABEL="array" UUID="5a...92" UUID_SUB="d3...19" TYPE="btrfs"
/dev/mapper/am3: LABEL="array" UUID="5a...92" UUID_SUB="40...66" TYPE="btrfs"
/dev/mapper/am1: LABEL="array" UUID="5a...92" UUID_SUB="37...97" TYPE="btrfs"Any ideas on what might be going on here? Anything in particular I might be missing here?
Offline
It seems that you are affected https://github.com/systemd/systemd/issues/1921 .
Solved similar problem with / on btrfs RAID-1 by creating custom udev rule
# cat /etc/udev/rules.d/66-btrfs-raid1.rules :
# Set raid1 disks as ready to not cause systemd loading loop
SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="array", ENV{SYSTEMD_READY}="1"rebuild initrd
# mkinicpio -p linux
# rebootOffline
Excellent; that seems to have solved the problem. Systemd can now mount the array on boot with that udev rule in place. Thanks!
Offline