You are not logged in.

#1 2016-08-23 00:57:03

alex.forencich
Member
Registered: 2011-05-29
Posts: 96

Systemd times out mounting btrfs parition, but manual mounting works

Alright, I'm having a rather strange issue here.  I just ran updates a headless box that's literally in the opposite corner of the country that hadn't been rebooted in about a month and it didn't come back up after rebooting, so I had to log in via IPMI to figure out what happened.  Here's the setup:

System has 5 hard drives, one for root and swap and four in a btrfs raid1 for data.  The four drives in the btrfs raid1 are encrypted with LUKS, primarily as a safeguard against data recovery in the event a drive fails such that it cannot be non-destructively erased (useful if I want to RMA it).  /etc/crypttab is responsible for getting all four drives unlocked and mapped via UUID.  Then /etc/fstab is supposed to get the btrfs partition mounted, also by UUID. 

However, the second step seems to be failing for some reason.  Whenever systemd attempts to mount the partition, it times out.  However, as soon as the system boots (well, once I changed 'defaults' to 'nofail' in /etc/fstab), I can just run sudo mount /mnt/array and it mounts just fine.  I also tried noauto,x-systemd.automount in /etc/fstab.  This causes systemd to attempt to mount the partition many times, but it never succeeds.  I also tried mounting by device (/dev/mapper/am0) instead of by UUID in /etc/fstab, but this also failed. 

Btrfs is definitely detecting the unlocked and mapped drives:

Aug 22 06:28:05 atomic kernel: BTRFS: device label array devid 1 transid 223737 /dev/dm-0
Aug 22 06:28:05 atomic kernel: BTRFS: device label array devid 3 transid 223737 /dev/dm-1
Aug 22 06:28:05 atomic kernel: BTRFS: device label array devid 4 transid 223737 /dev/dm-2
Aug 22 06:28:05 atomic kernel: BTRFS: device label array devid 2 transid 223737 /dev/dm-3

But systemd fails to mount the array for some unknown reason:

Aug 22 06:29:25 atomic systemd[1]: dev-disk-by\x2duuid-5a...92.device: Job dev-disk-by\x2duuid-5a...92.device/start timed out.
Aug 22 06:29:25 atomic systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-5a...92.device.
Aug 22 06:29:25 atomic systemd[1]: Dependency failed for /mnt/array.
Aug 22 06:29:25 atomic systemd[1]: mnt-array.mount: Job mnt-array.mount/start failed with result 'dependency'.
Aug 22 06:29:25 atomic systemd[1]: dev-disk-by\x2duuid-5a...92.device: Job dev-disk-by\x2duuid-5a...92.device/start failed with result 'timeout'.

/etc/crypttab:

am0          UUID=fd...26    /path/to/keyfile
am1          UUID=21...c7    /path/to/keyfile
am2          UUID=25...f6    /path/to/keyfile
am3          UUID=ac...e8    /path/to/keyfile

/etc/fstab:

UUID=ed...95 swap swap defaults 0 0
UUID=2f...df / ext4 defaults 0 1
UUID=5a...92 /mnt/array btrfs defaults 0 0

output of blkid:

/dev/sda1: LABEL="arch" UUID="2f...df" TYPE="ext4" PARTUUID="00005f60-01"
/dev/sda2: UUID="ed...95" TYPE="swap" PARTUUID="00005f60-02"
/dev/sdb1: UUID="fd...26" TYPE="crypto_LUKS" PARTLABEL="Linux filesystem" PARTUUID="5e...22"
/dev/sdc1: UUID="21...c7" TYPE="crypto_LUKS" PARTLABEL="Linux filesystem" PARTUUID="68...0b"
/dev/sdd1: UUID="25...f6" TYPE="crypto_LUKS" PARTLABEL="Linux filesystem" PARTUUID="ff...2b"
/dev/sde1: UUID="ac...e8" TYPE="crypto_LUKS" PARTLABEL="Linux filesystem" PARTUUID="c1...e8"
/dev/mapper/am0: LABEL="array" UUID="5a...92" UUID_SUB="1d...a6" TYPE="btrfs"
/dev/mapper/am2: LABEL="array" UUID="5a...92" UUID_SUB="d3...19" TYPE="btrfs"
/dev/mapper/am3: LABEL="array" UUID="5a...92" UUID_SUB="40...66" TYPE="btrfs"
/dev/mapper/am1: LABEL="array" UUID="5a...92" UUID_SUB="37...97" TYPE="btrfs"

Any ideas on what might be going on here?  Anything in particular I might be missing here?

Offline

#2 2016-08-23 11:24:52

edacval
Member
From: .LT
Registered: 2008-10-23
Posts: 89

Re: Systemd times out mounting btrfs parition, but manual mounting works

It seems that you are affected  https://github.com/systemd/systemd/issues/1921 .

Solved similar problem with / on  btrfs RAID-1 by creating custom udev rule

# cat /etc/udev/rules.d/66-btrfs-raid1.rules :

# Set raid1 disks as ready to not cause systemd loading loop
SUBSYSTEM=="block", ENV{ID_FS_LABEL}=="array", ENV{SYSTEMD_READY}="1"

rebuild initrd

# mkinicpio -p linux
# reboot

Offline

#3 2016-08-23 15:32:14

alex.forencich
Member
Registered: 2011-05-29
Posts: 96

Re: Systemd times out mounting btrfs parition, but manual mounting works

Excellent; that seems to have solved the problem.  Systemd can now mount the array on boot with that udev rule in place.  Thanks!

Offline

Board footer

Powered by FluxBB