RAID5 and setting up server questions

colbert · 2011-04-06 18:33:27

I am going to set up a server primarily for media storage, but also for backing up business data as well.

I currently have and will be using this mobo: http://www.asus.com/product.aspx?P_ID=LCK1LS8DWojB38IJ It has 6 SATA ports and 2 eSATA ports.

I plan to begin with 4x2TB in RAID5, however, depending on the following, I may start with 6 or 8x2TB:

1. I have been reading quite a bit about RAID and it seems to me that for my needs RAID5 makes the most sense. I am far from an expert so I ask if anyone can inform me of a more suitable option and if so why it would be more appropriate than RAID5? (Hate to set it all up and then find out I should've done it differently hehe).

2. I am thinking of getting this controller card: http://www.newegg.ca/Product/Product.as … 6816115029 Is this a good solution? Or is it more advantageous to use my onboard RAID?

#2 is kind of dependent also on whether or not I can use more than one of these controller cards. The issue I'm facing is that I would be limited to 4 drives in RAID if I use the controller card, and 6 at most if I use my onboard RAID (assuming I connect my OS drive to an eSATA port to keep the 6 SATA ports free for the RAID drives). I have emailed HighPoint web support asking if I can use more than one of the above linked controller cards simultaneously which would allow me to use 8 drives in my array, awaiting their response.

Thanks a lot for any help folks, hope I was somewhat clear in my questions but if not please ask for any info needed as I greatly appreciate the wisdom!

mcsilva · 2011-04-06 19:57:21

I have several servers with RAID-1 and RAID-5.
Some of them with Linux software RAID and others with Hardware RAID.
Of course the best one is Hardware RAID:
Pros:
1 - It is reliable and fast
2 - Easy to administer.
3 - No need to install drivers
4 - If sata controllers breaks, buy any other hardware RAID to substitute.

Cons: - Very Expensible!

But in your case, as I read the links you send, both RAID's seems to be "Software RAID".
So, in this case:

Pros: - Cheapest solution.
Cons:
1 - Need drivers to install.
2 - Not so easy to administer
3 - If sata controllers breaks and you need to buy a new one, they must be from the same manufacturer and have the same drivers. If not, you lose your data!

So, if it was me to chose, I would install "Linux Software RAID" instead of using their drivers.

Pros:
1 - It is reliable and fast
2 - If sata controllers break, you can buy them from other manufacturers, and RAID will work again.
3 - It's cheap.

Cons:
1 - It's not an easy task to administer. You must have some trainning.
2 - Only works in Linux systems. - But... this is not a problem for you ;-)

If you want to use Linux Raid you can test it easily in a VirtualBox ArchLinux and make your trainning.
Create 4 virtual disks, run a Linux Live CD (it can be anyone: Arch, Knoppix, Fedora, etc.), and create the RAID array in command mode.
Then you can install a minimal Archlinux system on this array and test several problems you can think:
- simulate a disk fail and substitute with a new disk.
- simulate a situation of expanding or shrinking partitions
- create new partitions, etc.

You must be able to do it in this simulation system. You can remember later if you have a real situation where you disk broke, or you must enlarge your partition and you don't know how.
I do it sometimes.

Talking about RAID-5.
It is a good option if you have sensible data in you disks.
But if you need a lot of disk space, this can be a problem, because if you have 8 TB disks, you can only use 2 or 3 TB to save your data.

I use a lot RAID-1.
It is a simple mirroring, but if you combine it with frequent backups, you can have more space to save data.
And if some hardware problem occurs, you can reinstall easily a new disk.
It's only a matter of planning:
1- Use RAID-5 if the disk space is enough, and you must give preference to safety.
2- Use RAID-1 if you need more disk space. This is also safe. But is your choice.

Nevertheless, test it in Virtualbox or other virtual system, to get some knowledge first!

Of course, this is only my opinion!

Hope this helps.

mcsilva

Fackamato · 2011-04-06 22:01:05

Go for RAID6. With drives these large, a rebuild is is going to take a LONG time, during which any failure on another HDD is going to cause your data to be gone. 2TB drives are cheap (€65 ish), just put in an extra one and feel safe. :-)

I've a 6x 2TB RAID5 setup with mdadm > lvm > ext4 on it which works fine (knock on wood). It's about a year old now, running checks every week.

edit: And RAID is not a backup system! Make backups on tapes or HDDs then move them to another physical location.

Last edited by Fackamato (2011-04-06 22:01:57)

colbert · 2011-04-07 05:21:14

mcsilva wrote:

I have several servers with RAID-1 and RAID-5.
Some of them with Linux software RAID and others with Hardware RAID.
Of course the best one is Hardware RAID:
Pros:
1 - It is reliable and fast
2 - Easy to administer.
3 - No need to install drivers
4 - If sata controllers breaks, buy any other hardware RAID to substitute.

Cons: - Very Expensible!
But in your case, as I read the links you send, both RAID's seems to be "Software RAID".
So, in this case:
Pros: - Cheapest solution.
Cons:
1 - Need drivers to install.
2 - Not so easy to administer
3 - If sata controllers breaks and you need to buy a new one, they must be from the same manufacturer and have the same drivers. If not, you lose your data!
So, if it was me to chose, I would install "Linux Software RAID" instead of using their drivers.
Pros:
1 - It is reliable and fast
2 - If sata controllers break, you can buy them from other manufacturers, and RAID will work again.
3 - It's cheap.
Cons:
1 - It's not an easy task to administer. You must have some trainning.
2 - Only works in Linux systems. - But... this is not a problem for you ;-)
If you want to use Linux Raid you can test it easily in a VirtualBox ArchLinux and make your trainning.
Create 4 virtual disks, run a Linux Live CD (it can be anyone: Arch, Knoppix, Fedora, etc.), and create the RAID array in command mode.
Then you can install a minimal Archlinux system on this array and test several problems you can think:
- simulate a disk fail and substitute with a new disk.
- simulate a situation of expanding or shrinking partitions
- create new partitions, etc.
You must be able to do it in this simulation system. You can remember later if you have a real situation where you disk broke, or you must enlarge your partition and you don't know how.
I do it sometimes.
Talking about RAID-5.
It is a good option if you have sensible data in you disks.
But if you need a lot of disk space, this can be a problem, because if you have 8 TB disks, you can only use 2 or 3 TB to save your data.
I use a lot RAID-1.
It is a simple mirroring, but if you combine it with frequent backups, you can have more space to save data.
And if some hardware problem occurs, you can reinstall easily a new disk.
It's only a matter of planning:
1- Use RAID-5 if the disk space is enough, and you must give preference to safety.
2- Use RAID-1 if you need more disk space. This is also safe. But is your choice.
Nevertheless, test it in Virtualbox or other virtual system, to get some knowledge first!
Of course, this is only my opinion!
Hope this helps.
mcsilva

Wow, thanks sooo much mcsilva! Fantastic post! I have been using Virtualbox for ages so I can use company software in Windows and your idea to simulate a RAID array never occurred to me, that's an excellent suggestion! I'm going to set Sunday afternoon aside and fiddle around with it so I can be prepared like you advise, for when the actual system goes up

You lay it out very clearly, linux software RAID makes the most sense. A dedicated RAID card will be far too expensive and unnecessary for my needs.

Now as I posted, again my only concern is the hardware issue. My mobo supports 6 SATA + 2 eSATA, so I can have 7 SATA for my RAID (1 for OS). However, what if I want to have 8 or 10 or 12 drives for the RAID? Would a card like this one that adds 8 SATA ports work?

Fackamato wrote:

Go for RAID6. With drives these large, a rebuild is is going to take a LONG time, during which any failure on another HDD is going to cause your data to be gone. 2TB drives are cheap (€65 ish), just put in an extra one and feel safe. :-)
I've a 6x 2TB RAID5 setup with mdadm > lvm > ext4 on it which works fine (knock on wood). It's about a year old now, running checks every week.
edit: And RAID is not a backup system! Make backups on tapes or HDDs then move them to another physical location.

Oh for sure as I mentioned I will be using 2TB drives, and my array will likely be just like yours. Why do you recommend RAID6 over RAID5?

And I've read that elsewhere as well that RAID isn't considered backup, so it is mainly just to "keep things going" in the off event of a drive failure? The data I need to backup will be quite small and selective, mostly my server will be for media storage.

Fackamato · 2011-04-07 08:34:17

The chance of a raid5 array with these large drives failing during rebuild is quite large. When you have 2 parity devices, ie raid6, you can be sure that a rebuild will most likely not fail.

mcsilva · 2011-04-07 11:51:20

colbert wrote:

Now as I posted, again my only concern is the hardware issue. My mobo supports 6 SATA + 2 eSATA, so I can have 7 SATA for my RAID (1 for OS). However, what if I want to have 8 or 10 or 12 drives for the RAID? Would a card like this one that adds 8 SATA ports work?

Linux Raid can create arrays with mixed devices.
You can have one SATA disk, one IDE disk and one USB disk, and create a RAID5 array with them.
Of course this is not a good idea in terms of performance, but sometimes it could be a necessity.
This is just to say that you can do RAID arrays with all your SATA disks, if they are detected by your mobo.

I don't have experience with RAID6 but is clearly good providing fault tolerance and Fackamato seems to have a good experience with it.

You can test them in VirtualBox as I suggested, but don't forget one important thing:
- RAID is good preventing data loss, but sometimes it happens... So, test it in virtual environments and see with your eyes, what is the best for you when a disk gets broken. What is the best system when you have to change a disk, redimension a partition, or reinstalling a system when more then one disk goes to the trash.
All RAID systems can be good, but if you don't know what to do when you loose a disk, or if you screw up everything because you did something wrong, then RAID worth nothing..

Last edited by mcsilva (2011-04-07 11:52:17)

gorky · 2011-04-08 23:12:12

Also make sure you've done some research about the 4kb sector size problems with Linux - here are two links from my bookmarks: (nothing very difficult, but you have to take care)
http://www.osnews.com/story/22872/Linux … ard_Drives
http://www.ibm.com/developerworks/linux … tor-disks/

Fackamato · 2011-04-08 23:13:53

gorky wrote:

Also make sure you've done some research about the 4kb sector size problems with Linux - here are two links from my bookmarks: (nothing very difficult, but you have to take care)
http://www.osnews.com/story/22872/Linux … ard_Drives
http://www.ibm.com/developerworks/linux … tor-disks/

When using mdadm you don't have to worry, just use the whole HDD in the array (without partitions) and you're golden.

gorky · 2011-04-08 23:59:17

Fackamato wrote:

When using mdadm you don't have to worry, just use the whole HDD in the array (without partitions) and you're golden.

Are you sure? Some disks (like mine Samsung F4 2GB) lie about their physical sector size
# hdparm -I /dev/sda | grep Sector
Logical Sector size: 512 bytes
Physical Sector size: 512 bytes

While it really has 4kb PSS.

I don't know how mdadm does the alignment, so if you could explain more I'd be thankful

Fackamato · 2011-04-09 00:17:04

gorky wrote:

Fackamato wrote:
When using mdadm you don't have to worry, just use the whole HDD in the array (without partitions) and you're golden.
Are you sure? Some disks (like mine Samsung F4 2GB) lie about their physical sector size
# hdparm -I /dev/sda | grep Sector
Logical Sector size: 512 bytes
Physical Sector size: 512 bytes
While it really has 4kb PSS.
I don't know how mdadm does the alignment, so if you could explain more I'd be thankful

Hm. My stuff:

~ $ sudo fdisk -l /dev/sdb

Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0e5b3a7a

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  3907029167  1953513560   fd  Linux raid autodetect


=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green (Adv. Format) family
Device Model:     WDC WD20EARS-00MVWB0

So I'm using partitions with a 1MB alignment (to make sure). I've seen on the md mailing list that it supports 4k drives. But, like mine and yours, it lies about the sector size so perhaps it's required to align a partition on it.

gorky · 2011-04-09 00:28:12

Fackamato wrote:

But, like mine and yours, it lies about the sector size so perhaps it's required to align a partition on it.

That's the point - with these drives lying about their real geometry, mdadm may be fooled (but as well may not - i just don't know), and additional care may need to be taken.

But I'm not sure, because I know almost nothing about the internals of mdadm and LVM and how they arrange partitions, hold metadata and so on... Just wanted to show another problem that I also don't know how to solve with 100% certainity.

Last edited by gorky (2011-04-09 00:29:18)

colbert · 2011-04-13 01:51:46

You guys are awesome, thanks all for your info! I finally got around to setting up an Arch install in Virtualbox and I created various RAID arrays, first a quick 8x250MB to get the hang of it, then an 8x1GB. That /proc/mdstat has the spotlight on it I haven't gotten around to removing drives from it and fiddling with that yet, plan to tomorrow though. It's surprisingly straight forward, seems the hardware choices and issues are where the confusion can arise compared to actually putting it together in software

I suppose a couple things I haven't tried yet I'm still curious as to, namely filesystem. Is ext4 fine? Been using it on all my drives so far and probably would continue, unless another FS makes more sense (I haven't tried any others yet).

And also, I don't quite understand the 4kb sector issue. When Fackamoto says:

When using mdadm you don't have to worry, just use the whole HDD in the array (without partitions) and you're golden.

What exactly does this mean? Say I have 2 drives to build an array with, /dev/sda and /dev/sdb. I format them to be /dev/sda1 and /dev/sdb1, and build the array? Do you mean to avoid partitioning them before the array (/dev/sda1, /dev/sda2, etc.)?

Thanks a lot

gorky · 2011-04-13 08:52:57

colbert wrote:

And also, I don't quite understand the 4kb sector issue. When Fackamoto says:

Have you looked at the links I posted above?

Fackamato · 2011-04-13 09:16:51

colbert wrote:

I suppose a couple things I haven't tried yet I'm still curious as to, namely filesystem. Is ext4 fine? Been using it on all my drives so far and probably would continue, unless another FS makes more sense (I haven't tried any others yet).

ext4 is fine. If you're building a very large array you could try xfs, as it's been used in RAID arrays for many years and have good optimizations for it. It's also faster than ext4 in some cases.

colbert wrote:

And also, I don't quite understand the 4kb sector issue. When Fackamoto says:
When using mdadm you don't have to worry, just use the whole HDD in the array (without partitions) and you're golden.
What exactly does this mean? Say I have 2 drives to build an array with, /dev/sda and /dev/sdb. I format them to be /dev/sda1 and /dev/sdb1, and build the array? Do you mean to avoid partitioning them before the array (/dev/sda1, /dev/sda2, etc.)?
Thanks a lot

Fackamato <- spelling

You do not have to create partitions when using the HDDs with mdadm, regardless of the 4KB sector format. The only reason you'd want partitions for use in an array would be when you'd be moving the HDDs to another computer. You wouldn't know that the HDD is used in an array, because it's got no partition table on it. To many OSes it would look like a "new, unformatted drive".

If your sole intention is to use them in RAID and not move them between PCs I'd skip the partitioning.

Remember that xfs can't shrink the filesystems (not sure why you'd want that in a RAID though, for other than troubleshooting purposes/re-doing the RAID without losing data etc), but it can grow them. If you want multiple "partitions" on the MD RAID device I'd look into LVM2, it's great for that purpose.

Arch Linux

#1 2011-04-06 18:33:27

RAID5 and setting up server questions

#2 2011-04-06 19:57:21

Re: RAID5 and setting up server questions

#3 2011-04-06 22:01:05

Re: RAID5 and setting up server questions

#4 2011-04-07 05:21:14

Re: RAID5 and setting up server questions

#5 2011-04-07 08:34:17

Re: RAID5 and setting up server questions

#6 2011-04-07 11:51:20

Re: RAID5 and setting up server questions

#7 2011-04-08 23:12:12

Re: RAID5 and setting up server questions

#8 2011-04-08 23:13:53

Re: RAID5 and setting up server questions

#9 2011-04-08 23:59:17

Re: RAID5 and setting up server questions

#10 2011-04-09 00:17:04

Re: RAID5 and setting up server questions

#11 2011-04-09 00:28:12

Re: RAID5 and setting up server questions

#12 2011-04-13 01:51:46

Re: RAID5 and setting up server questions

#13 2011-04-13 08:52:57

Re: RAID5 and setting up server questions

#14 2011-04-13 09:16:51

Re: RAID5 and setting up server questions

Board footer