Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

brando56894 · 2014-05-09 19:46:18

I started my foray into RAID about 6 months ago while looking for an easy way to store all of my massive amounts of media. I bought two off the shelf NAS enclosures but didn't like them. I took the existing software RAID 5 array out of the last NAS and put in my PC which runs Arch. It runs well and I haven't had any problems with it, and I get about 100-150 MB/sec out of it, but I like to tweak things and squeeze the most performance out as possible. I would like to use a much space as possible, so nothing like RAID 1, I like the idea of RAID 10 but I don't have the money to buy enough drives to get the space I want. I know that ext4 is a good choice for all-around storage, that xfs is great for large files but sucks with small files, that btrfs supports compression and snapshots among other things, and that ZFS is a completely different animal than a typical filesystem and that it's more akin to RAID+LVM. From what I understand about COW/snapshots, it wouldn't really benefit me considering I'm only storing multi-gigabyte videos and not system files, correct? I'm leaning towards XFS since I know it's great with huge files but would it be really worth it considering I'm already on ext4? Would zfs be worth it?

I use nzbget to download things, and also use the array for storing large things like hard disk images for virtualbox, and maybe a small mysql database if I ever get around to messing with newznab again.

System Specs
Intel Core i7 @ 3.7 GHz
12 GB DDR3 Triple Channel
128 GB Corsair M4 SSD for root and home, formatted with btrfs
4x 3 TB WD Green in RAID 5 currently formatted with ext4, 8.1 TB useable

WorMzy · 2014-05-09 19:55:42

https://bbs.archlinux.org/viewtopic.php?id=117168

I'm still using XFS for my media storage, it's not let me down yet. I'm using it on a RAID5 currently.

chaonaut · 2014-05-10 09:43:17

brando56894 wrote:

I use nzbget to download things, and also use the array for storing large things like hard disk images for virtualbox, and maybe a small mysql database if I ever get around to messing with newznab again.

so, definitely not btrfs.

well…
zfs is cool. really. but too slow on linux, compared with btrfs / ext4 / xfs. so, after playing with it for a couple of weeks, i went back to ext4.
ext4 is the best tradeoff between everything for the moment.
xfs is good too, but still a bit dangerous in case of crash/power loss (well known empty/zeroed files problem), so i stopped using it, despite it's the fastest for most applications.

frostschutz · 2014-05-10 10:26:19

Power loss is a problem that no filesystem has managed to solve yet. I've lost data with ext* that way, and had lots of luck with XFS in that regard.

XFS is my main file system; I originally started using it because of its tools (dump, restore, defrag, ...). I'm using it today simply because it served me well for so many years.

One thing I dislike about ext* is that by default, it expects an average file size of 16KiB or above. If you have many small files, you run out of inodes before the filesystem is full, and the only way around it is to reformat or add more space you don't need. It happens too often for comfort, and it's unacceptable in a standard filesystem for Joe Average. If you know beforehand that you will be having lots of small files, for example because you like to compile your own kernel and have several source trees in /usr/src/linux, you should make sure to make mkfs.ext* give you plenty of inodes beforehand (-T small or whatever).

Last edited by frostschutz (2014-05-10 10:26:40)

brando56894 · 2014-05-10 14:53:14

Thanks for the suggestions, I used to use xfs on my storage drives before I built the array. I guess I'll switch back to it!

chaonaut · 2014-05-10 15:56:37

frostschutz wrote:

One thing I dislike about ext* is that by default, it expects an average file size of 16KiB or above. If you have many small files, you run out of inodes before the filesystem is full, and the only way around it is to reformat or add more space you don't need. It happens too often for comfort, and it's unacceptable in a standard filesystem for Joe Average. If you know beforehand that you will be having lots of small files, for example because you like to compile your own kernel and have several source trees in /usr/src/linux, you should make sure to make mkfs.ext* give you plenty of inodes beforehand (-T small or whatever).

for lots of small files, i'd prefer btrfs.
moreover, if one like to compile own kernels, packages, etc, btrfs has snapshot capabilities, so it's easy to roll back if something goes wrong.
my arch system partitions are mainly btrfs due to intensive usage of customized packages.

Last edited by chaonaut (2014-05-10 16:01:26)

vostok4 · 2014-05-10 17:18:12

Why not btrfs for large media storage? I've been using it here for a few years with no issue, and I'm about to convert the array from raid1 to raid10, happy to use it. Although I don't use any advanced feature other than raid...

theking2 · 2014-05-23 07:51:37

Been using XFS on a soft-raid 5 with lvm for over 10 years now in my NAS. I decided right from the start that OS and media databases sit on an SSD and the Media files themselves on the RAID5 set. It has survived a harddisk failure. And XFS should keep fragmentation low. I've not lost any data over a power failure but media data are read primarily anyway, aren't they?. The NAS is also in use a backup server for the various laptops around the house using a redmundish file history

I actually am looking for a ZFS solution but read now that it is slow. (Chaonaut?) Maybe I should stick to the XFS solution for the time being.

xfs_db -c frag -r /dev/vg/media
actual 25965, ideal 25844, fragmentation factor 0.47%

Last edited by theking2 (2014-05-23 07:51:46)

chaonaut · 2014-05-23 08:55:14

theking2 wrote:

I actually am looking for a ZFS solution but read now that it is slow. (Chaonaut?)

well, i must mention that my test installation was single-hdd.
maybe on multi-hdd setups it behaves differently, e.g. better.

a quick googling shows me this, for example:
http://openbenchmarking.org/result/1304 … ERGE786390

single-hdd setup may show absolutely different result, for example:
http://www.phoronix.com/scan.php?page=n … px=MTM1NTA

Last edited by chaonaut (2014-05-23 08:59:04)

graysky · 2014-05-23 09:01:53

ZFS. Not slow and very stable and powerful.

theking2 · 2014-05-25 15:16:38

Actually the reason for me looking at this thread was that I was in the (now nearly finalized) process of moving from md+lvm+xfs to a ZFS system. It all went like a breeze until one of the external disks I used as backup (xfs) reported all kind of odd errors and generated several kernel panics. (ouch). I don't think a storage device should do that. So far ZFS appeared unimpressed by this rather harsh treatment. Although it did four files with permanent errors (where actually I could open the files without a problem).

Anyway, I found out that where originally I could get transfers of max 20MB/s I now could (restoring the backups) reach 25-27MB/s. So it is quit a bit faster.

Scrubbing is not. For a 2.5T set it reports 30h ...

Arch Linux

#1 2014-05-09 19:46:18

Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

#2 2014-05-09 19:55:42

Re: Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

#3 2014-05-10 09:43:17

Re: Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

#4 2014-05-10 10:26:19

Re: Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

#5 2014-05-10 14:53:14

Re: Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

#6 2014-05-10 15:56:37

Re: Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

#7 2014-05-10 17:18:12

Re: Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

#8 2014-05-23 07:51:37

Re: Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

#9 2014-05-23 08:55:14

Re: Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

#10 2014-05-23 09:01:53

Re: Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

#11 2014-05-25 15:16:38

Re: Media Storage: RAID 5 & LVM? ZFS? BTRFS? XFS?

Board footer