Help With RAID Configuration

Carlwill · 2011-01-25 14:55:04

I've got 4 identical 1 TB drives and would like to use them in a software RAID configuration on my home server. I'm running Debian Linux using 'mdadm' utility to manage the software RAID. I don't know how much I've read is fact or dated or even false so I decided I would ask here to get help from people who know more about this than I do. This is essentially just a file server machine to store all my data so being that I've got four identical SATA hard drives, I was thinking about doing RAID level 5. I guess I'll start here and ask if that is the recommended level of RAID. I think RAID level 5 will be fine for my general server usage. My second issue is partitioning the four individual drives to get maximum performance / space from them. Basically just asking here how would you or you recommend I partition the drives? I was thinking about doing three seperate partitions per drive:

/dev/sda1 = 4 GB (swap)
/dev/sda2 = 1 GB (/boot)
/dev/sda3 = 995 GB (/)

Now from that partition schema above, obviously all the types will be 'fd' for RAID and the partition for /boot is going to be bootable. My confusion is that I read Grub doesn't support booting from RAID 5 since Grub can't handle disk assembly. If /dev/sdx2 (sda2, sdb2, sdc2, sdd2) are partitioned for /boot (bootable), how would you guys configure this RAID to match up equally? I don't think I do a RAID level 1 on 4 identical partitions, right? Can anyone please help and tell me how I should configure these 4 identical drives to work on my system? How would you set this up?

otterfox · 2011-01-26 11:36:14

If you're thinking about using RAID5 then you probably want to read this: https://secure.wikimedia.org/wikipedia/ … ilure_rate
I'm using RAID1 at the moment but if I had 4 disks I'd be using RAID10. RAID 10 also has better performance than RAID5 and if you were going to have a spare disk you may as well use it for a RAID10 array.

Just a small correction, where you have /dev/sdaX it should be /dev/mdX because they're logically in a RAID array now.
What you have there looks fine, I would personally be going for an arrangement with separate partitions for swap, /boot, /, /data or something similar, most sysadmins I know will also have /var/log by itself so that if an application logs too much it doesn't fill up your disks an cause an issue and is easy to fix. For maximum flexibility everything (except for /boot) should be in a LVM group.

With regards to your /boot partition, because /boot only needs to be very small (<500MB) I would just go with a small 300MB RAID1 setup at the start of each disk.

You should end up with something like...
http://i.imgur.com/pWN4b.jpg

--mod edit: use thumbnails, see https://wiki.archlinux.org/index.php/Fo … s_and_Code

Last edited by litemotiv (2011-01-26 11:49:52)

zenlord · 2011-01-26 18:41:21

otterfox wrote:

http://i.imgur.com/pWN4b.jpg

Thx for that - it is really useful!

I read up on BTRFS and it should make LVM redundant in a setup like this one, but the question is if you are willing to let 4TB of data depend on a FS that is not yet deemed stable by the kernel people themselves .

I have been using 3x320GB in RAID-5 for 2 years now, and everything has been smooth sailing. Now I'd like to add an extra 320GB, but since I didn't know how to set up an LVM group 2 years ago (and basically, still don't have real experience doing that), I'll have to reformat the whole lot

Since I have to do a complete reinstall, I'll be installing / and /boot on a separate disc, outside any RAID-setup.

Zl.

Gnarl · 2011-01-26 20:39:47

I have a similar setup as you:

4 x 1 TB drives set up with RAID 5 with 3 partitions on each

Partition 1: /var - 5G on each drive on RAID 5 - This partition size is 15G
Partition 2: /home - 5G on each drive on RAID 5 - This partition size is 15G
Partition 3: /media/data - RAID 5 - Remainder of space

boot and root are on an SSD.

I will probably end up moving the var and home partitions to RAID 10 as I don't need the space and more speed is always awesome

Just a tip: you may want to revise your partition sizes, If you use what you specified above in RAID 5 you will end up with 12G swap partition and a 3G boot.

Also I don't believe you can use RAID on your boot partition unless it is RAID 1, your boot manager won't be able to read it. RAID modules get loaded when the kernel image read off the boot partition.

Last edited by Gnarl (2011-01-26 20:47:43)

Gnarl · 2011-01-27 00:24:29

So I had some time to make the changes above and the result is a little disappointing and confusing.Both drives below are on the same 4 drives and both have a 512k chunk size. I tied 32k chunk on the RAID 10 but it made it about 25MB/sec slower.

RAID 10
/dev/md126:
Timing buffered disk reads: 782 MB in 3.00 seconds = 260.63 MB/sec

RAID 5
/dev/md127:
Timing buffered disk reads: 1112 MB in 3.01 seconds = 369.32 MB/sec

Edit: I changed the RAID 10 back to a RAID 5 with 64k chunks and my speed is back

/dev/md126:
Timing buffered disk reads: 1082 MB in 3.00 seconds = 360.10 MB/sec

Last edited by Gnarl (2011-01-27 01:02:21)

otterfox · 2011-01-27 05:25:47

zenlord wrote:

I read up on BTRFS and it should make LVM redundant in a setup like this one, but the question is if you are willing to let 4TB of data depend on a FS that is not yet deemed stable by the kernel people themselves .

I'd agree with that, I would definitely not touch BTRFS in any environment other than a spare box for testing.

zenlord wrote:

Since I have to do a complete reinstall, I'll be installing / and /boot on a separate disc, outside any RAID-setup.

I guess that is one of the easier things to do, but keep in mind, if what is basically your OS disk fails, then you have to setup everything all over again, which in some cases can be many many hours of work and making sure you get the RAID config right again when you attach your disk array. You only need maybe 20GB for the OS anyway, it's not much.

Gnarl wrote:

RAID 10
/dev/md126:
Timing buffered disk reads: 782 MB in 3.00 seconds = 260.63 MB/sec
RAID 5
/dev/md127:
Timing buffered disk reads: 1112 MB in 3.01 seconds = 369.32 MB/sec
Edit: I changed the RAID 10 back to a RAID 5 with 64k chunks and my speed is back
/dev/md126:
Timing buffered disk reads: 1082 MB in 3.00 seconds = 360.10 MB/sec

What sort of disks are you using to get that sort of performance, that's pretty sweet.
I think RAID5 is supposed to slower for writing to disk than RAID10. How have you setup your RAID10? Have you done [RAID0[RAID1 + RAID1]] ?

anrxc · 2011-01-27 05:53:44

Here's a nice article on RAID5/RAID10 topic: http://www.yonahruss.com/architecture/r … nd-ha.html I read it just yesterday.

Last edited by anrxc (2011-01-27 05:55:16)

Gnarl · 2011-01-27 14:16:40

otterfox wrote:

What sort of disks are you using to get that sort of performance, that's pretty sweet.
I think RAID5 is supposed to slower for writing to disk than RAID10. How have you setup your RAID10? Have you done [RAID0[RAID1 + RAID1]] ?

4 x the following SATA 3 drives with 64M cache

Device Model: WDC WD1002FAEX-00Z3A0
User Capacity: 1,000,204,886,016 bytes

I had setup the raid with --level=10. I hadn't thought of trying 2 separate RAID 0's mirrored.

Edit:

Tried 1+0 and got the same result as 10

Last edited by Gnarl (2011-01-27 18:18:29)

Gnarl · 2011-01-27 15:01:46

Informative article http://webapp5.rrz.uni-hamburg.de/SuSe- … 10cpx.html on RAID 10 AND 1+0

Previous page is good also.

Arch Linux

#1 2011-01-25 14:55:04

Help With RAID Configuration

#2 2011-01-26 11:36:14

Re: Help With RAID Configuration

#3 2011-01-26 18:41:21

Re: Help With RAID Configuration

#4 2011-01-26 20:39:47

Re: Help With RAID Configuration

#5 2011-01-27 00:24:29

Re: Help With RAID Configuration

#6 2011-01-27 05:25:47

Re: Help With RAID Configuration

#7 2011-01-27 05:53:44

Re: Help With RAID Configuration

#8 2011-01-27 14:16:40

Re: Help With RAID Configuration

#9 2011-01-27 15:01:46

Re: Help With RAID Configuration

Board footer