EXAMPLE: RAID-5 + LUKS media array with SSD Cache

gshearer · 2016-01-29 15:11:02

Hey All,

I'm building a new server and one of the many hats it will have to wear is a media NAS for several TV's I have in my household. I'm betting there are others out there who have similar needs, so I thought I'd share my setup.

Requirements:

1. Lots of single-file system storage (I like one place for all my videos/music, easy to export, etc)
2. Low power consumption (spinning down drives, etc. more on this later)
3. SSD cache for fast access
4. Performance (speed) is not an issue because we're just streaming usually 1080p

HOW:

1. Raid-5 a bunch of very large spinning SATA drives (in my case, 4 WD RED 6TB) using Linux MD
2. SSD cache out in front using dm-cache
3. Just for fun we're gonna encrypt the whole thing because I'm paranoid like that.

Notes:

1. Make sure you have haveged running prior to setting up LUKS: systemctl start haveged
2. I tried writeback cache mode just to evaluate how it worked. Anytime I tried to remove the cache with lvconvert I'd get a kernel panic. This was as of 1/29/2016 on 4.3.3-3. No problems with writethrough however.

Here's the procedure I used. I'm happy to hear your opinions if this can be improved. I'll also make updates to this post as necessary.

So, in my case, my 4 spinning drives present themselves to Linux as sdk through sdn. The SSD I'm using in this example is sdj. And I'm using /dev/md2 for my Linux MD device.

First, prepare drives for RAID:

for x in sdk sdl sdm sdn
do
  wipefs -a /dev/$x
  sgdisk -Z /dev/$x
  sgdisk -n 1:0:0 -t 0:fd00 /dev/$x
done

Enable RAID-5 - This will start a background rebuild -- Safe to use the volume (though it'll be slow) during rebuild

mdadm --create --verbose --level=5 --chunk=512 --raid-devices=4 /dev/md2 /dev/sd[k-n]1

Encrypt with LUKS

dd if=/dev/urandom of=/etc/luks/mediatank.luks bs=512 count=4
chmod 0400 /etc/luks/mediatank.luks
cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random --key-file /etc/luks/mediatank.luks luksFormat /dev/md2
cryptsetup open --type luks --key-file /etc/luks/mediatank.luks /dev/md2 mediatank

Layer on LVM - If you noticed above, I'm using "mediatank" as my LUKS device mapped name. Going forward, vgmt will be "volumegroup for media tank" and lvmt will be "logical volume for mediatank".

pvcreate /dev/mapper/mediatank
vgcreate vgmt /dev/mapper/mediatank
lvcreate -l +100%FREE -n lvmt vgmt

Partition and encrypt the SSD

wipefs -a /dev/sdj
sgdisk -n 1:0:0 -t 0:8e00 /dev/sdj
dd if=/dev/urandom of=/etc/luks/mediatank-cache.luks bs=512 count=4
chmod 0400 /etc/luks/mediatank-cache.luks
cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random --key-file /etc/luks/mediatank-cache.luks luksFormat /dev/sdj1
cryptsetup open --type luks --key-file /etc/luks/mediatank-cache.luks --allow-discards /dev/sdj1 mediatank-cache

Extend the LVM volume group to the now encrypted SSD:

pvcreate /dev/mapper/mediatank-cache
vgextend vgmt /dev/mapper/mediatank-cache

Convert the SSD LV into a cache. This one command does multiple things, which includes setup several hidden volumes for cache metadata storage,etc. For more details read the lvmcache man page.
Note: I am using writethrough caching because I want 100% of my SSD to be used for read caching. My SSD is 500-GB. I hope to allow my spinning drives to spin down (power save) when not in use. I use a different volume for "current" media. So, this volume is really meant for archived media that is rarely accessed. So, this SSD cache isn't for performance, it's so that the spinning array can spend most of it's life NOT spinning and thus also not generating heat. Without this SSD cache, often the simple act of mounting an NFS export from this array or performing directory lookups will cause all drives to wake up.

lvcreate -l +100%free --type cache -n mt-cache --cachemode writethrough --chunksize 1M vgmt/lvmt /dev/mapper/mediatank-cache

Confirm that logical volumes are cached and also confirm cache mode

lvs -a vgmt
lvs -o+cache_policy,cache_settings,cache_mode vgmt/lvmt

Finally, put down a file system. I'm using ext4fs RAID enhancements here using the following calculations:
chunk size == 512
stride == chunk (512) / block size (4) == 128
stripe-width == stride (128) * number of data-bearing disks (3) == 384

mke2fs -v -t ext4 -T largefile -E stride=128 -E stripe-width=384 -E discard -L mediatank /dev/vgmt/lvmt

Ready to use!

mount -t ext4 -o discard,relatime /dev/vgmt/lvmt /mnt/mediatank

Tada!

Last edited by gshearer (2016-01-29 22:18:12)

Xyne · 2016-01-29 20:31:28

This looks like a nice walkthrough (clear, thorough, to the point) that is sure to help users looking to achieve a similar setup, but the forum isn't really the right place for it. Please consider reposting it on the wiki.

gshearer · 2016-01-29 22:14:40

Sure, I just wanted to give people a chance to converse about it - I'm not really sure about the RAID optimizations to mke2fs when an SSD cache is involved, for instance.

jasonwryan · 2016-01-29 22:55:58

The Talk page for the RAID wiki entry would be the best place to canvas opinion, corrections and enhancements.

Arch Linux

#1 2016-01-29 15:11:02

EXAMPLE: RAID-5 + LUKS media array with SSD Cache

#2 2016-01-29 20:31:28

Re: EXAMPLE: RAID-5 + LUKS media array with SSD Cache

#3 2016-01-29 22:14:40

Re: EXAMPLE: RAID-5 + LUKS media array with SSD Cache

#4 2016-01-29 22:55:58

Re: EXAMPLE: RAID-5 + LUKS media array with SSD Cache

Board footer