Intel RST raid1 is resyncing after every boot or reboot

Overbryd · 2018-08-31 20:17:47

Hello everyone,

I have been going through the internet for days now to find a solution to my problem.

I have two disks in a Intel RST raid1 configuration. On top of it, I run my system, that is an ArchLinux.
I regularly shutdown the system, since it is not a server, and is basically my "daily driver" that does not need to be constantly running.

The two disks are consumer disks, so I thought why not put them in a raid1, so if one fails, I can easily replace them.
But since I installed the system, this turned out to be quite the hassle, since it starts a fresh resync *after every boot*.
I tried all kinds of solutions, but I think I need some help on this.

Here is the configuration:

* Intel RST raid1 with two SATA HDD
* Ontop of that I run three partitions, all with LUKS device encryption
* Root also is directly running on these disks
* initramfs is configured with the following HOOKS and BINARIES
* I run 4.18.5-arch1-1-ARCH with mdadm v4.0 2017-01-09

BINARIES=(/sbin/mdon)
HOOKS=(base udev autodetect modconf block mdadm_udev encrypt lvm2 filesystems keyboard fsck shutdown)

So as I understand, mdmon is started with initramfs but will later be taken over by the systemd mdmon.
I cannot not find the reason why the system does not shutdown cleanly, leaving the RAID in a clean state for the next boot.

The main bug reports are going in the direction that it is apparently asking for trouble, having mdmon running in userspace, taking over mdmon from initramfs and later probably not being killed probably by systemd.
But these bugs ought to be fixed, and I am running older versions that those reported in the bugs I found across the Linux sphere (Gentoo, Debian, Arch, all have seem such behaviours with Intel RST raids with mdadm).

It is rather annoying to either have the system running at all times (the raid stays clean after one lenghty many hour long resync) or to always have degraded performance and disk activity for hours after every reboot.

What can I do to further narrow down the problem and find a solution to it?

R00KIE · 2018-08-31 22:11:57

Which guide did you follow to setup the raid array? It's not clear to me exactly what you are using and that is an important detail.

You mention intel rst and mdadm in the same sentence, the first seems to imply something along the lines of fakeraid, which is best avoided, and the second implies software raid which is usually the preferred way to go.

pmarrapese · 2018-09-01 05:24:45

R00KIE wrote:

Which guide did you follow to setup the raid array? It's not clear to me exactly what you are using and that is an important detail.
You mention intel rst and mdadm in the same sentence, the first seems to imply something along the lines of fakeraid, which is best avoided, and the second implies software raid which is usually the preferred way to go.

Your statement that firmware ("fake") RAID should be avoided and software RAID is preferred does not make sense. The reason is that Intel RST is managed by the firmware outside of the operating system, until the OS boots at which point control is passed to the OS. At this point, it is effectively software RAID, which is why it is managed by mdadm. The only reasons it should be "best avoided" are identical to those of any software RAID (e.g. a requirement of battery-backed write cache).

I authored this tutorial last year in an effort to deobfuscate the process of getting Intel RST working with mdadm. It elaborates some on the interplay between firmware RAID and mdadm.

Overbryd wrote:

It is rather annoying to either have the system running at all times (the raid stays clean after one lenghty many hour long resync) or to always have degraded performance and disk activity for hours after every reboot.
What can I do to further narrow down the problem and find a solution to it?

I'm not sure if I'm able to reliably reproduce this yet. After reviewing my journal, I do see that rebuilds have occasionally been triggered after reboots. My machine (a NAS) sees a lot of writes between infrequent reboots, so it's tough to say what the root cause might be.

Can you confirm that the rebuilds are indeed happening on every reboot, and if so whether the system has always done this? Please run the following: last -x | grep system | tac to list your shutdown/startup history, then correlate it against journalctl | grep 'kernel: md'

My example output from January -- now:

shutdown system down  4.12.13-1-ARCH   Fri Jan 26 19:43 - 19:39  (-00:04)
reboot   system boot  4.12.13-1-ARCH   Fri Jan 26 19:39 - 17:18 (26+21:38)
Jan 26 19:39:14 serenity kernel: md/raid1:md126: active with 2 out of 2 mirrors
Jan 26 19:39:14 serenity kernel: md126: detected capacity change from 0 to 3000589352960

shutdown system down  4.12.13-1-ARCH   Thu Feb 22 17:18 - 18:01  (00:43)
reboot   system boot  4.12.13-1-ARCH   Thu Feb 22 18:01 - 02:58 (126+07:56)
Feb 22 18:01:46 serenity kernel: md/raid1:md126: active with 2 out of 2 mirrors
Feb 22 18:01:46 serenity kernel: md126: detected capacity change from 0 to 3000589352960

reboot   system boot  4.12.13-1-ARCH   Sat Apr 14 11:48 - 02:58 (75+15:09)
Apr 14 11:48:17 serenity kernel: md/raid1:md126: not clean -- starting background reconstruction
Apr 14 11:48:17 serenity kernel: md/raid1:md126: active with 2 out of 2 mirrors
Apr 14 11:48:17 serenity kernel: md126: detected capacity change from 0 to 3000589352960
Apr 14 11:48:17 serenity kernel: md: resync of RAID array md126
Apr 14 17:48:41 serenity kernel: md: md126: resync done.

shutdown system down  4.12.13-1-ARCH   Fri Jun 29 02:58 - 02:53  (-00:04)
reboot   system boot  4.17.2-1-ARCH    Fri Jun 29 02:53 - 00:29 (7+21:36)
Jun 29 02:53:46 serenity kernel: md/raid1:md126: not clean -- starting background reconstruction
Jun 29 02:53:46 serenity kernel: md/raid1:md126: active with 2 out of 2 mirrors
Jun 29 02:53:46 serenity kernel: md126: detected capacity change from 0 to 3000589352960
Jun 29 02:53:46 serenity kernel: md: resync of RAID array md126
Jun 29 08:49:29 serenity kernel: md: md126: resync done.

shutdown system down  4.17.2-1-ARCH    Sat Jul  7 00:29 - 00:34  (00:05)
reboot   system boot  4.17.2-1-ARCH    Sat Jul  7 00:34 - 19:56 (55+19:21)
Jul 07 00:34:48 serenity kernel: md/raid1:md126: active with 2 out of 2 mirrors
Jul 07 00:34:48 serenity kernel: md126: detected capacity change from 0 to 3000589352960

shutdown system down  4.17.2-1-ARCH    Fri Aug 31 19:56 - 19:53  (-00:03)
reboot   system boot  4.18.5-arch1-1-A Fri Aug 31 19:53 - 21:11  (01:17)
Aug 31 19:53:45 serenity kernel: md/raid1:md126: not clean -- starting background reconstruction
Aug 31 19:53:45 serenity kernel: md/raid1:md126: active with 2 out of 2 mirrors
Aug 31 19:53:45 serenity kernel: md126: detected capacity change from 0 to 3000589352960
Aug 31 19:53:45 serenity kernel: md: resync of RAID array md126

- Jan 26: No rebuild
- Feb 22: No rebuild
- Apr 14: This was an unexpected power outage, so that rebuild seems justified.
- June 29: I'm not sure why there was a rebuild here, but that was after 75 days of uptime, so perhaps it was sensible.
- July 7: No rebuild
- Aug 31: Rebuild. This was after 55 days of uptime.

I'll try again when the rebuild is complete to see if another rebuild is triggered, in which case a recent package update may be to blame. (edit: Just rebooted post-rebuild. Array looks good, it did not trigger a new rebuild.)

Last edited by pmarrapese (2018-09-01 09:42:17)

jasonwryan · 2018-09-01 05:33:43

pmarrapese's blog wrote:

This firmware-based RAID (also known as “fake RAID”) is an attractive option because it permits functionality such as disk mirroring without further monetary investment.

What?

I have used software RAID for over a decade on my home machine and it has never required "further monetary investment", nor does it do stupid shit like rebuild after a reboot. mdadm works, and works excpetionally well. Intel RST may be an improvement on previous fake raid solutions, but even if it is, it is just another layer that has the propensity to fail. Based on the evidence in this thread, there is little to suggest that it offers any advantages over mdadm alone.

Your "tutorial" looks like a pretty standard mdadm install to me: which raises the question of what purported benefit Intel RST is bringing to this table, other than some fuckupedness that makes OP's system unstable.

Overbryd · 2018-09-03 15:03:59

@pmarrapese I actually used *your* Medium article to set it up Nice to meet you, hi.

So I am getting a resync on every boot so far. Does not matter what I do, or how cleanly the system shuts down, it resyncs every time I reboot.
I think my setup is overly complex, and I think I should reduce its complexity before I move on debugging an issue that is deep down the layers I surround myself with.

Adding to my issue, I recently also got a few unclean shutdowns, because the systems hangs at properly unwinding LVM.

So overall my impression is, that my setup is somewhat brittle. I do not account that particularly to Intel RST + mdadm, but more so with the many layers (dmcrypt + LVM + cache all on root) this is surrounded with.

I will simplify it now to the following:

* Separate root from the RAID
* Setup root on its own disk

And see how that goes.
Main reason I want to have RAID1 is to have a reliable storage, but there is no need put root on it.

Kind regards,

Lukas

Overbryd · 2018-09-04 08:34:59

Following up with the current state:

* The machine did not boot anymore.

I was getting I/O errors left and right on the LVM volumes, I fsck -f and smartctl checked them, no errors found.
So I am actually in recovery mode now, salvaging everything out into a separate disk. That worked just fine, everything is intact.

Doing so I move away from a `udev +` initramfs towards a `systemd` initramfs and that works so much better so far.
I will report back as soon as it is back in normal operation and write up the changes that I made compared to the original guide.

Sorry for the let-down to not further investigate this, but I need it to work and could not even bring it back to its original failure state.

Arch Linux

#1 2018-08-31 20:17:47

Intel RST raid1 is resyncing after every boot or reboot

#2 2018-08-31 22:11:57

Re: Intel RST raid1 is resyncing after every boot or reboot

#3 2018-09-01 05:24:45

Re: Intel RST raid1 is resyncing after every boot or reboot

#4 2018-09-01 05:33:43

Re: Intel RST raid1 is resyncing after every boot or reboot

#5 2018-09-03 15:03:59

Re: Intel RST raid1 is resyncing after every boot or reboot

#6 2018-09-04 08:34:59

Re: Intel RST raid1 is resyncing after every boot or reboot

Board footer