Arch Linux

c3kay

Member

From: New Zealand

Registered: 2015-04-19

Posts: 61

[solved] udev/kernel and drive/device assignment

I have an Arch server running LTS, with a 5 drive 10TB RAID5 array and a 640gb system drive. For a while, it's had a 1TB USB drive that's used for backing up essential stuff. It stays plugged in, sat next to the server, and it's been fine.

It's set up like so:

$ lsblk

NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda         8:0    0 596.2G  0 disk  
├─sda1      8:1    0     1G  0 part  /boot
├─sda2      8:2    0   128G  0 part  /
├─sda3      8:3    0     8G  0 part  [SWAP]
└─sda4      8:4    0 459.2G  0 part  /home
sdb         8:16   0   1.8T  0 disk  
└─sdb1      8:17   0   1.8T  0 part  
  └─md127   9:127  0   7.3T  0 raid5 /media/storage
sdc         8:32   0   1.8T  0 disk  
└─sdc1      8:33   0   1.8T  0 part  
  └─md127   9:127  0   7.3T  0 raid5 /media/storage
sdd         8:48   0   1.8T  0 disk  
└─sdd1      8:49   0   1.8T  0 part  
  └─md127   9:127  0   7.3T  0 raid5 /media/storage
sde         8:64   0   1.8T  0 disk  
└─sde1      8:65   0   1.8T  0 part  
  └─md127   9:127  0   7.3T  0 raid5 /media/storage
sdf         8:80   0   1.8T  0 disk  
└─sdf1      8:81   0   1.8T  0 part  
  └─md127   9:127  0   7.3T  0 raid5 /media/storage
sdg         8:96   0 931.5G  0 disk  
├─sdg1      8:97   0   700G  0 part  /media/bkmusic
└─sdg2      8:98   0 231.5G  0 part  /media/bksys

After the recent update for LTS to kernel 4, whenever I reboot when the USB drive is plugged in, then it'll get to the hooks, call udev, and then fail to boot. If the USB drive is plugged in, it always assigns that drive to /dev/sda, which causes a boot failure. It doesn't even get to the point where it mounts using fstab. Just tells me I'm on my own after it can't find root on /dev/sda2. If I unplug the USB drive, allow the kernel to assign /dev/sda to the system drive and wait until fstab is trying to mount /dev/sdg, then it works as intended, fstab mounts all partitions without issue, and lsblk gets me the result above.

I don't know much about udev, so I spent this afternoon reading about it and how it works. I think it's the kernel suddenly deciding to assign the USB drive as /dev/sda and moving all the other devices up a letter. Once it's assigned I can't do anything about it, because the kernel device nodes can't be changed. Gummiboot obviously finds the /boot partition and works fine, but after that point once the hooks are called, it's all fubar.

Obviously, journal doesn't even get to start when it fails to boot, so I only get information from successful boots, which flag up problems when I introduce them into udev rules deliberately, but otherwise don't show any issues.

Anyone know if this is something simple I'm missing? None of my the persistent naming schemes are persisting, or at least, they're persisting but they're not actually assigning the right nodes any more. How do I force the kernel/udev to recognise a particular piece of hardware and assign it the right kernel node? Can I just botch it by accepting that the USB drive will be a and switching Gummiboot to work on /dev/sdb? But given that, how do I know these device names are going to persist?

Thanks.

Last edited by c3kay (2015-09-30 08:47:07)

mich41

Member

Registered: 2012-06-22

Posts: 796

Re: [solved] udev/kernel and drive/device assignment

You can specify root partition using /dev/disk/by-label, /dev/disk/by-uuid, /dev/disk/by-path and friends instead of /dev/sda2. Alternatively, I bet there is some way to remove usb-storage from initramfs...

c3kay

Member

From: New Zealand

Registered: 2015-04-19

Posts: 61

Re: [solved] udev/kernel and drive/device assignment

Well, UUID in gummiboot fixed it. Thank you. And on the upside I learned all about udev, which wasn't the problem at all. I was super focused on udev and the drive assignment.

It's still kind of annoying how it's just started to do this. I don't think anything else is referenced through kernel device names, so it makes no practical difference except to make me feel slightly awkward that my system drive isn't on /dev/sda.

Last edited by c3kay (2015-09-30 08:48:26)

mich41

Member

Registered: 2012-06-22

Posts: 796

Re: [solved] udev/kernel and drive/device assignment

I think it must have been some random change in the kernel which changed the order in which disks are discovered.

c3kay

Member

From: New Zealand

Registered: 2015-04-19

Posts: 61

Re: [solved] udev/kernel and drive/device assignment

I pretty much thought that, but wasn't entirely sure what to do about it. It's been a long time since I touched anything to do with boot on the server. I went into the .conf earlier to check on what it was doing, but I guess it's just one of those gaps in knowledge that gets filled really quickly when someone else has input. Thanks again.