How to Configure a Hot Spare on RAID5

Carlwill · 2010-01-15 20:44:10

I am creating a RAID5 array and have 4 identical S-ATA drives. My question is RAID5 only requires 3 disks so if I have a 4th disk, how does that fit into the equation? Would I need to make the 4th drive a hot spare or can it also be a drive with in the RAID5 volume?

rusty99 · 2010-01-15 20:57:47

Something like http://en.wikipedia.org/wiki/Raid_5#RAID_5

Carlwill · 2010-01-15 21:08:04

rusty99 wrote:

Something like http://en.wikipedia.org/wiki/Raid_5#RAID_5

Yes, I read that. Would I simply just do the following on Arch?

mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2

Is there anything I am missing on that command?

Ultraman · 2010-01-15 21:59:23

With that command you would use all four drives as a RAID5 array.

RAID5 has a minimum requirement of 3 disks, can also be used with more. Result will be more space and a small performance increase can be expected in several situations. RAID5 writes the data to all disks and also smartly distributes parity data for the written data over the disks.
RAID5 usable disk space is calculated as the disk space total of the drives used minus one. (n-1). So with four 1TB drives, you would end up with the total disk space of 4 - 1 drives == 3 drives => 3TB.
The redundancy is just like that. RAID5 can cope with one failed drive, doesn't matter if you used 3, 4 or 12 disks. The data that the failed drive used to hold is calculated from the parity data on the other disks (using a XOR function).

If you want to use four disks in your RAID5 array, issue something like:

# mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

Of course depending on how you partitioned your disks. If you want to use the 2nd partition, like you are doing in your example, that's perfectly alright.
But remember, you are performing RAID5 over those partitions, not the entire disks. That is how mdadm works.

If you want to use 3 drives for RAID5 and use 1 drive as a hot spare, you could do:

# mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sda1 /dev/sdb1 /dev/sdc1
# mdadm --add /dev/md0 /dev/sdd1

This makes the RAID5 array using sda1, sdb1 and sdc1. And you add the spare device after that.
Perhaps there is also an option to directly add a spare device, I cant find that one quickly in man mdadm. And the internet conatins options that seem incorrect to me, as I can not find them in de man page.
But my above methode works fine for me.
And it also starts to rebuild automatically over here when I do it like this.

The Gentoo Wiki - RAID/Software also contains some interesting info.
Skip the intent-bitmap part. Internal bitmaps can cause huge performance drops. Only consider them an options when you can write the bitmap to a drive that is not part of the RAID array the bitmap belongs to.
If you use big disks for this RAID5 array, don't miss the last bit about Data Scrubbing. Today's big disks have a fair chance to develop some bad blocks in their lives. The data can sometimes still be read, but nothing written to the block, but the data can also be calculated using the parity data that RAID5 uses. But when a disk fails, a bad block means trouble!
So make sure to check your disks regularly for those. I intend to do weekly checks here. Simply put the command in a shell script, make it executable and drop it into /etc/cron.weekly. Your cron daemon will pick it up and run it, keeps you out of trouble when a drive really fails and you have to rebuild.

Last edited by Ultraman (2010-01-15 22:13:22)

nogoma · 2010-01-15 22:51:44

Ultraman wrote:

Perhaps there is also an option to directly add a spare device, I cant find that one quickly in man mdadm. And the internet conatins options that seem incorrect to me, as I can not find them in de man page.
But my above methode works fine for me.
And it also starts to rebuild automatically over here when I do it like this.

Indeed:

man mdadm wrote:

For create, build, or grow:
-n, --raid-devices=
Specify the number of active devices in the array. This, plus the number of spare devices (see below) must equal the number of component-devices (including "missing" devices) that are listed
on the command line for --create. Setting a value of 1 is probably a mistake and so requires that --force be specified first. A value of 1 will then be allowed for linear, multipath, raid0
and raid1. It is never allowed for raid4 or raid5.
This number can only be changed using --grow for RAID1, RAID5 and RAID6 arrays, and only on kernels which provide necessary support.
-x, --spare-devices=
Specify the number of spare (eXtra) devices in the initial array. Spares can also be added and removed later. The number of component devices listed on the command line must equal the number
of raid devices plus the number of spare devices.

So, in your case I believe you want:

# mdadm --create /dev/md0 --level=5 --raid-devices=3 --spare-devices=1 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

Ultraman · 2010-01-16 00:19:22

Ah, yes, that could also work.
Maybe you would have to change that a little bit to:

# mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1 --spare-devices=1 /dev/sdd1

But I am unsure. Perhaps mdadm will simply pick one from nogoma's command.
Try and see what it brings you.
You can use "cat /proc/mdstat" to quickly check the status of your RAID, and also use mdadm's monitor functions to check it out in more detail.

By the way, when using RAID5. If you are going to use an ext2/3/4 filesystem, take a look at the stride=n parameter of mke2fs. It helps layout the filesystem in a more efficient way or the array, this helps performance.
Something to help you determine what stride to use: http://busybox.net/~aldot/mkfs_stride.html
For example, a 4 disk RAID array using a 128KiB chunk size and 4KiB block size, should set a stride=32.
And also stripe-width might be a good idea to set.

Last edited by Ultraman (2010-01-16 00:26:31)

Carlwill · 2010-01-26 18:40:54

Well I added a identical drive to my already working 3 disk RAID5 system and it appeared to take the spare...

[root@tuna ~]# mdadm --misc --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Fri Jan 22 10:01:18 2010
     Raid Level : raid5
     Array Size : 480391552 (458.14 GiB 491.92 GB)
  Used Dev Size : 240195776 (229.07 GiB 245.96 GB)
   Raid Devices : 3
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Jan 26 13:30:56 2010
          State : clean
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : d8814bf1:db9353ad:0cebaba2:6736d1f2
         Events : 0.601

    Number   Major   Minor   RaidDevice State
       0       8        2        0      active sync   /dev/sda2
       1       8       18        1      active sync   /dev/sdb2
       2       8       34        2      active sync   /dev/sdc2

       3       8       50        -      spare   /dev/sdd2

I used 'cfdisk' to create two partitions on the new /dev/sdd. I made /dev/sdd2 the same size as all the other 3 partitions in the /dev/md0 array. Then I added it with:

mdadm --add /dev/md0 /dev/sdd2

It shows up as a spare. I rebooted and everything appears to work. I also mounted /opt to /dev/sdd1 to utilize that 4 GB's of dead space.

Now would it have been possible to add that /dev/sdd2 RAID partition to RAID5 array /dev/md0 not as a hot spare but as additional storage space in /dev/md0? Basically just add it and have it start synchronizing? Would that be possible or would something like that break the already working 3 disk mirror?

Arch Linux

#1 2010-01-15 20:44:10

How to Configure a Hot Spare on RAID5

#2 2010-01-15 20:57:47

Re: How to Configure a Hot Spare on RAID5

#3 2010-01-15 21:08:04

Re: How to Configure a Hot Spare on RAID5

#4 2010-01-15 21:59:23

Re: How to Configure a Hot Spare on RAID5

#5 2010-01-15 22:51:44

Re: How to Configure a Hot Spare on RAID5

#6 2010-01-16 00:19:22

Re: How to Configure a Hot Spare on RAID5

#7 2010-01-26 18:40:54

Re: How to Configure a Hot Spare on RAID5

Board footer