[Solved] Mdadm RAID5 array starting resync after every boot

mosetsuki · 2018-01-14 19:33:05

Firstly, a description of my storage setup:

I have a mdadm RAID5 array of 3 HDDs (sda, sdb and sdc), which are used as the backing device in a bcache setup behind an SSD (sdd).

This setup has worked, as far as I've been able to tell, flawlessly for at least a year and a half. A few days ago, I get a notification from mdadm that it's rebuilding (or resyncing, it uses both terms depending on where I look) my array. My first thought was it was some weird confluence of events that just so happened to trigger a resync, but then a couple days later it happened again. Both times I allowed the resync to complete, which it did without any indication of error from mdadm. The last two days it's been resyncing after every boot, even if it completed a resync literally minutes before and the system then rebooted.

I've checked SMART via smartctl (for what it's worth) on every drive (it reports nothing out of the ordinary, even after a self-test) and /proc/mdstat doesn't show any failed drives. I've also checked all physical connections, both power and data. There were 2-3 boots between the then-most-recent update and the first appearance of this issue, and I've since updated again on the off chance that might help. The only remaining cause I can think of is that one of the drives is failing, but as the process of checking each drive would be very time-consuming (and may require I resort to methods that could risk my data) I was hoping there may be some other possible cause I'm unaware of.

Last edited by mosetsuki (2018-01-24 17:26:02)

WorMzy · 2018-01-14 19:41:20

I get a notification from mdadm

Can you elaborate on this? I've never recieved a notification from mdadm. How do you recieve the alert? What does the alert say? is there any other information in the alert? What does your journal have to say about md-related events?

mosetsuki · 2018-01-14 19:55:07

WorMzy wrote:

I get a notification from mdadm
Can you elaborate on this? I've never recieved a notification from mdadm. How do you recieve the alert? What does the alert say? is there any other information in the alert? What does your journal have to say about md-related events?

I should have been more specific - I wrote a script that will pop up a notification in my DE with the parameters passed to it by mdadm using the PROGRAM config line in /etc/mdadm.conf. By the time my DE has started it's missed the RebuildStarted event, but it will subsequently show the rebuild status events ('RebuildXX on /dev/md0' where the XX is the completion percentage - the event name and device are passed by mdadm). Regarding my journal, I've been unable to find anything at all except the lines about executing the aforementioned script - though I believe that's related to my method of displaying the notification and not from mdadm itself. Rather than guessing at search parameters I simply looked line-by-line through the log since the most recent boot (immediately after which mdadm had, of course, started a resync), and I found nothing else related to mdadm or any of my storage devices.

WorMzy · 2018-01-15 19:36:39

If the rebuild is starting at boot, then check the last few messages from the previous session. Perhaps something during the shutdown process is causing the array to need a rebuild the next time it's assembled.

mosetsuki · 2018-01-16 16:37:22

Yesterday I'd finally given up and booted off a livecd to see if a fsck would fix the issue. In the process I did some terrible things to my filesystem that I spent the rest of the day resolving (including restoring more than a few files from backups), but by the end of it when I booted my system it no longer began a resync. The issue remains absent after booting again today. Apparently, something I did resolved whatever was causing mdadm to resync my array. Perhaps something on-disk was corrupted enough to trigger resyncs, but not enough for mdadm (or anything else) to throw an error, and when I ruined my filesystem I either subsequently fixed the corruption or exacerbated it to the point it was detected and then fixed?

Whatever the cause was and whatever happened that fixed it, it's fixed, and I don't think there's any way for me to figure out for sure what the cause (or fix) was. Thanks for your help anyway!

mosetsuki · 2018-01-24 17:25:22

A bit of an addendum, if anyone else should happen to encounter the same issue I did. After another fsck my array began resycing after every boot again. For several days I would just delay it (with echo frozen > /sys/block/md0/md/sync_action), but eventually I decided to let it run. Before it finished however, I switched my cache mode to writethrough (previously it had been writeback) and dropped writeback_percent to 0 from 10. After allowing the resync to complete I restored my previous settings, and today when I booted there was no resync. Something to try if there's anyone else with a storage setup as eccentric as mine.

Nevermind. It started a resync with today's boot. I'm going ahead with my plan of backing everything up, nuking the array, testing each drive thoroughly and starting from scratch.

Last edited by mosetsuki (2018-01-25 17:15:16)

Arch Linux

#1 2018-01-14 19:33:05

[Solved] Mdadm RAID5 array starting resync after every boot

#2 2018-01-14 19:41:20

Re: [Solved] Mdadm RAID5 array starting resync after every boot

#3 2018-01-14 19:55:07

Re: [Solved] Mdadm RAID5 array starting resync after every boot

#4 2018-01-15 19:36:39

Re: [Solved] Mdadm RAID5 array starting resync after every boot

#5 2018-01-16 16:37:22

Re: [Solved] Mdadm RAID5 array starting resync after every boot

#6 2018-01-24 17:25:22

Re: [Solved] Mdadm RAID5 array starting resync after every boot

Board footer