Arch Linux

davidjosepha

Member

Registered: 2013-02-21

Posts: 54

Filesystem for RAID 5 / 6 media array

I'm setting up RAID 5 with 3x 14tb drives and planning to add a 4th drive in the near future to switch to RAID 6. So, collectively, I'll have ~28tb of storage. This is to replace my current 3x 4tb drive RAID 5 array in my home server, primarily used for media. System files are all on a separate SSD. About a terabyte of the data is FLAC tracks, mostly in the 10-100M range, and I have a small amount of other small files (ebooks, uncompressed home directory backups from my other computers, some other stuff), but the vast majority of the drive is large files, mostly video, most of the files being multiple gigabytes.

I am undecided on the filesystem for the new array. The three types I'd narrowed it down to from my research are ext4, btrfs, and xfs.

My current array uses btrfs, but I mostly just picked that because I was interested in learning something new and playing around with the new features. That was kind of fun; however, 6 years later, I'm not taking advantage of any of the features it offers over ext4. I'm not opposed to using btrfs again, but for this array, I'm primarily interested in stability and performance, not fancy features. I've seen conflicting info on how the performance of the three options compare in different use cases. As I mentioned above, most reads are large files, so that's the access pattern I want to optimize around. Which of these is likely to be most performant when accessing large files?

Apart from speed, I'm wondering whether there are any concerns around any of these filesystems with RAID 5 or 6. The RAID page on the wiki says to pick a filesystem that can do the following:

- Due to the large volume size not all filesystems are suited.
- The filesystem should support growing and shrinking while online.
- One should calculate the correct stride and stripe-width for optimal performance.

The first isn't an issue for any of them. The second, from the wikipedia comparison of filesystems page, it looks like all three can do online grows, which is good, and while ext4 doesn't support online shrinks and xfs doesn't support shrinks at all, I don't foresee a case where I'd shrink the array. (However, I saw somewhere else that resizing xfs arrays is a bad idea -- not sure if they were just referring to shrinking or whether there's an issue with growing not apparent from the wikipedia comparison.) The third, it looks like all three filesystems support setting stride and stripe-width.

Are any of these filesystems more/less stable with RAID 5/6? Or more/less stable in general?

Are there any other things I might not have thought to ask that would be relevant to this decision?

-thc

Member

Registered: 2017-03-15

Posts: 734

Re: Filesystem for RAID 5 / 6 media array

I have some limited experience with all three file systems in back-end servers with arrays (hardware/software). In the end, as long as the performance "penalty" of the respective RAID/file system doesn't "reach into" the true bottleneck (1 GBit network ~115 MB/s) it's kinda academic - all three of them worked for me without issues for years.

I used XFS for a while on a 6 TB hardware RAID and realized that the integrity check (xfs_check) needs a lot of memory - but maybe this was an issue with 32/64-bit versions of this utility - I honestly can't remember. It was annoying.

I use RAID5 arrays with up to 6 drives maximum - above that I would use RAID6. The latter with only four drives is IMHO overkill.
I had two RAID5 arrays with four drivers each that over a time span of 11 years lost around 10 members due to firmware/hardware failures. Yes there always was a microscopic statistical chance for a second drive failure in the time window needed for replacing the faulty drive and the rebuild - but I never had the feeling that RAID6 would have been justified. Have you considered RAID10 instead? Same amount of storage, fixed member count and member size but drastically simpler (less CPU load) and can handle the loss of two drives in 2/3 of all statistical cases.

davidjosepha

Member

Registered: 2013-02-21

Posts: 54

Re: Filesystem for RAID 5 / 6 media array

Thanks for the reply, this is helpful. I think given all this, I might just go with btrfs again -- it's only become more stable over the years and seemed to work fine for my use cases. It would also give me the option of using some of its fancier features if I ever wanted to in the future.

The latter with only four drives is IMHO overkill.

Basically, the reason I'm thinking of RAID 6 is because I got 4 drives on a good deal and 28tb is plenty of space for me for the foreseeable future. So I could set up RAID 6 and then add more drives in a few years if I needed more than 28tb of storage -- presumably drives will continue to get cheaper for the same amount of capacity as time goes on. I could just have RAID 5 with 3 drives and try to re-sell the 4th drive, but having a bit more redundancy wouldn't hurt, even if it's overkill. I also only have a small amount of the media backed up externally (basically my music collection, some documents, and a regular printout of the full directory structure) because there's no point spending tons of money each month for backups of several terabytes of data when most of it is video that I can easily redownload if something happened. But it's also far more convenient if I don't have to do that, which is where RAID 6 would be handy -- the only practical way I could lose the data is if someone stole the server (unlikely) or my house burned down (in which case I'd have much bigger problems than losing my video collection, like losing my irreplaceable book collection).

Have you considered RAID10 instead?

I'd thought about it, but in the long term, I don't want to have 2 drives for every drive temporary worth of capacity. RAID 6 is just a way to make my 4th drive not completely useless, but the idea is that eventually I'd have 5 or 6 drives in total, with 2 acting as redundancy. Maybe that's stupid though and I should just re-sell the 4th drive and then add another one in the future if I need it. I think I was just falling for some of the "RAID 5 is virtually guaranteed to fail on rebuild if you lose a drive due to UREs" myths, even though I've read refutations of this myth that make logical sense to me.