Arch Linux

severach

Member

Registered: 2015-05-23

Posts: 192

mdadm boot slow to assemble drives with newer installs

I had a problem where my raid1 installs from this year would pause about 5 seconds per mdadm entry yet my old installs from March 2014 would go through all 4 mdadm entries in less than a second. After fussing for a long time on the new installs I finally broke (slowed) one of my old installs which immediately revealed what the problem was.

I didn't forget to pick partition type FD or FD00. I have my install sequence in a text file so I rarely forget things. I install through PuTTY pasting all the way so it's fast and mistakes are rare.

devices of the form /dev/md/* take about 5 seconds to create.

devices of the form /dev/md{0..127} create very fast.

"mdadm -E --scan" always creates devices of the form /dev/md/{0..127}. In the past I was very meticulous about fixing the device names to the more compact form. I never missed so I never found out there was a substantial speed difference. I just like the the more compact look of /dev/md1 in my boot menu. Eventually I progressed to UUID and LABEL but persisted with the compact form. Then I discovered "mdadm --name" so I left mdadm do it's own thing. What could possibly go wrong?

I also noticed that the mdadm instructions changed from using "mdadm" to "mdadm_udev". mdadm_udev seems to assemble faster than mdadm.

To fix, boot once to see if you're slow, fix any usage of /dev/md*/* in your boot scripts or fstab, fix the form in /etc/mdadm.conf, and mkinitcpio...

I still see that the new installs pause at udev more than old installs, but instead of 20s vs >1s, it's now 5s vs >1s, which is much better.

Looking further I compared mkinitcpio.conf and noticed that autodetect is now installed by default. I took that out and though my list of drivers went way down the pause didn't change. Then I compared "lsinitcpio -a /boot/initramfs-linux.img" and I saw that fsck.zfs was loading so ZFS being a data drive only I uninstalled it and remade mkinitcpio. The pause didn't change.

Now "lsinitcpio -a /boot/initramfs-linux.img" and "lsinitcpio /boot/initramfs-linux.img | sort" compare the same yet the pause remains. I know something is lying because the slow pause computer has a SAS controller and the fast pause computer doesn't, and I know that I must remake mkinitcpio when the SAS controller brand changes or it won't boot without Fallback. The mptsas driver is hiding in there somewhere. So then I unpack both initrd files and by-content compare all files with Total Commander.

Only one difference: mdadm.conf

I can't boot without the SAS controller so I can't check to see if that's the source of the pause. I suspect it is because I can see the NumLock blink halfway through the pause and I know the SAS controller isn't concerned with that.

To solve this quandary I made a quickie install on a SATA only computer. I used the fast form of /dev/md* in mdadm.conf and left autodetect in mkinitcpio.

It's fast, but there's still about a half second pause right there at udev. Out goes autodetect. Still about a half second.

Then, back at the slow pause computer, I swapped the Marvell AOC-SASLP-MV8 for the LSI SAS2008 9211-8i. See Zorinaq for what makes this possible. The pause at udev dropped to about 1s which is only a bit slower than SATA. Not really a win since the LSI BIOS boot takes much longer. The good news is that I didn't need to rebuild mkinitcpio to change controllers. autodetect is toast and I get one more degree of freedom!

I guess that's as good as it gets. Fortunately the mdadm --name option doesn't create /dev/md/* entries so I don't need to fix that.

The Internet is strangely silent on this issue. I guess you're all just used to slow boots with mdraid.

This is a bug and needs to be reported. Either mdadm scan needs to stop producing the slow form or needs to make the slow form fast.

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 19,822

Re: mdadm boot slow to assemble drives with newer installs

Is there a question in there?  Or are you telling us you are going to file an upstream bug report?

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

severach

Member

Registered: 2015-05-23

Posts: 192

Re: mdadm boot slow to assemble drives with newer installs

It started as a question but I worked through it so far that it turned into a solution. Now the question is, how many of you are suffering with slow mdraid boots and don't know it.

I can submit the bug report but after a half hour of searching I don't see an obvious place collecting bugs. All I see is the distros collecting bugs which suggests there isn't any place. Besides, I have the workaround, and now everyone else does too.

jasonwryan

Anarchist

From: .nz

Registered: 2009-05-09

Posts: 30,424

Website

Re: mdadm boot slow to assemble drives with newer installs

This belongs on the wiki, not here. Make a brief note on the RAID page with the workaround.

---

Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

frostschutz

Member

Registered: 2013-11-15

Posts: 1,425

Re: mdadm boot slow to assemble drives with newer installs

I didn't forget to pick partition type FD or FD00.

I wouldn't use this partition type anymore. (raid autodetect is deprecated)

Then I discovered "mdadm --name" so I left mdadm do it's own thing.

mdadm --name never worked well for me. I don't see many people using it.

I prefer a minimal mdadm.conf that only specifies the md number and UUID.

ARRAY /dev/md0 UUID=627125a5:abce6b82:6c738e49:50adadae

I don't notice any delays but, in my case the raid is assembled in the background while in the foreground LUKS is waiting for me to type a passphrase.