Arch Linux

ectospasm

Member

Registered: 2015-08-28

Posts: 292

Exploring RAID options (Marvell MV64460/64461/64462|mdadm|lvmraid...)

I have an old Lenovo D20 ThinkStation, with a Marvell RAID controller onboard.  Here are the details of the RAID controller (out of lspci):

01:00.0 RAID bus controller: Marvell Technology Group Ltd. MV64460/64461/64462 System Controller, Revision B (rev 01)
	Subsystem: Lenovo MV64460/64461/64462 System Controller, Revision B
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	Region 2: I/O ports at 2000 [size=128]
	Region 4: Memory at f0200000 (64-bit, non-prefetchable) [size=64K]
	[virtual] Expansion ROM at f0240000 [disabled] [size=256K]
	Capabilities: [48] Power Management version 2
		Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [e0] Express (v1) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 2048 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x4, ASPM L0s, Exit Latency L0s <256ns
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s (ok), Width x4 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Kernel driver in use: mvsas
	Kernel modules: mvsas

I downloaded mrvl_utl_msu-4.1.10.2038_linux_x86-64.zip (software for Marvell RAID controllers) off a Lenovo website. but the mvcli utility contained within it doesn't recognize the adapter.  Marvell no longer lists this device on their website, and Lenovo has a dearth of resources pertaining to Linux for this ThinkStation.  If you know of a Linux utility that can manage this hardware RAID, I wouldn't mind exploring options there.

Until I can find a utility that can manage this particular hardware RAID device, I'm considering constructing a JBOD physical array with a set of five physical disks, as large as I can afford (anywhere from 3T through 12T per disk when I'm ready to purchase).  My options then are software RAID, with mdadm or lvmraid.  The main reason for software RAID in lieu of OS-managed hardware RAID is I suspect it will be easier to monitor the health of the individual disks with OS-level tools (e.g., smartctl, etc.).

mdadm seems the most straightforward, but lvmraid seems to be the most flexible.  I'm seeking comments here:  which would you prefer, and why?  I plan on purchasing six identical SATA or SAS disks, and use RAID10 on four disks with one hot spare, and one cold (disconnected) spare in case of future disk failure.  I'm open to other RAID levels (I have an old QNAP NAS with five 2T disks in software RAID6, and I plan on migrating the data off of that into this new array), but I choose RAID10 as a balance between performance and redundancy (RAID6 reduced my usable capacity from 10T to less than 6T, and I ultimately regretted that decision).

I could also go with an advanced filesystem (such as ZFS or Btrfs).  My experience with Btrfs is limited (I've only used it once on a single physical NVMe disk), and I have no experience with ZFS.  I had trouble with Btrfs, I don't remember the details but one of the internal structures became unreadable, such that my laptop wouldn't boot.  I ended up invalidating the entire filesystem (and losing my home partition) when I tried to recover it.  I had been tracking Btrfs stability via the Btrfs mailing list a while ago, and it never appeared to be totally stable for RAID.  I know little to nothing about ZFS.  My fear is that either of these options aren't stable enough for extended use (I anticipate this RAID array to last 5-6 years).

I may not be building this array for several months.  I haven't even figured out my budget for it yet, but I figure I can start planning now.

frostschutz

Member

Registered: 2013-11-15

Posts: 1,472

Re: Exploring RAID options (Marvell MV64460/64461/64462|mdadm|lvmraid...)

I choose RAID10 as a balance between performance and redundancy (RAID6 reduced my usable capacity from 10T to less than 6T, and I ultimately regretted that decision

RAID 10 reduces the usable capacity even further while not providing true 2 way redundancy at all (it depends on luck). Worse if you add spares.

RAID 6 only really starts to shine when you have more drives (much better usable capacity ratio).

Pillgar

Member

Registered: 2017-07-05

Posts: 41

Re: Exploring RAID options (Marvell MV64460/64461/64462|mdadm|lvmraid...)

I've used mdadm for raid 0, 1, 5 with out any issue's.   Worked great for my fakeraid installs too.  Never used lvmraid before.  My brother swears by UnRaid for his nas and FTP backups..

ectospasm

Member

Registered: 2015-08-28

Posts: 292

Re: Exploring RAID options (Marvell MV64460/64461/64462|mdadm|lvmraid...)

I've used mdadm for raid 0, 1, 5 with out any issue's.   Worked great for my fakeraid installs too.  Never used lvmraid before.  My brother swears by UnRaid for his nas and FTP backups..

My old QNAP NAS uses mdadm, so maybe I should use that.  lvmraid looks awfully complicated (and I don't see much benefit over mdadm for that added complexity).  My Marvell RAID controller only has five SATA/SAS ports on it, and the ThinkStation case only has room for five drives in its HDD cage. 

If I go with RAID6 that means my usable space will be 60 percent of the total capacity of the raw drives.  If I go with 6T drives the raw capacity will be 30T, with 18T usable space in RAID6.  If I can afford 12T drives that will give the raw capacity of 60T, with 36T usable space.  I guess it's the tradeoff you get when you have the redundancy of RAID6.  Really the only benefit this gets me over my QNAP NAS is increased usable space.  I'll need it to transfer the data over, so I can put a real OS on the QNAP when I'm done.

frostschutz

Member

Registered: 2013-11-15

Posts: 1,472

Re: Exploring RAID options (Marvell MV64460/64461/64462|mdadm|lvmraid...)

I guess it's the tradeoff you get when you have the redundancy of RAID6.

Yeah... RAID 5 is more practical if you have fewer than ~8 drives.

Double failures are extremely rare, provided you set up drive monitoring (mdadm, smartd automated checks, email notification) and actually take reallocated/pending sectors seriously instead of just ignoring such issues.

And even with double redundancy, you still need a backup; RAID never replaces that. And with a good backup strategy, RAID failures are no longer a huge concern.

ectospasm

Member

Registered: 2015-08-28

Posts: 292

Re: Exploring RAID options (Marvell MV64460/64461/64462|mdadm|lvmraid...)

And even with double redundancy, you still need a backup; RAID never replaces that. And with a good backup strategy, RAID failures are no longer a huge concern.

I have offsite backups through SpiderOakONE.  Their UI is rather clunky, but I'm grandfathered into unlimited storage at $125/yr (they had a special on World Backup Day about five years ago).

Pillgar

Member

Registered: 2017-07-05

Posts: 41

Re: Exploring RAID options (Marvell MV64460/64461/64462|mdadm|lvmraid...)

36TB of storage?  You ripping movies at 4k?

ectospasm

Member

Registered: 2015-08-28

Posts: 292

Re: Exploring RAID options (Marvell MV64460/64461/64462|mdadm|lvmraid...)

36TB of storage?  You ripping movies at 4k?

Not only that, but I'm using rsnapshot to keep rotating backups of most of my files. I usually backup /etc, /home, /root, and sometimes /var on some of my systems.  Besides, a 4K movie compressed with H.265 compresses down pretty well.  I have a few Blu-Ray discs I've ripped that lead to an MKV of less than 2G.  But you need 50-80G of free space just to rip the files with MakeMKV.  HandBrakeCLI does the magic with H.265.

I have a few scripts that I developed years ago, maybe if I get the gumption I will share them.

cmurf

Member

Registered: 2019-09-06

Posts: 27

Re: Exploring RAID options (Marvell MV64460/64461/64462|mdadm|lvmraid...)

Old thread but, public service announcement:
Timeout Mismatch
https://raid.wiki.kernel.org/index.php/Timeout_Mismatch

This comes up all the time on the linux-raid@ list, applies to mdadm, LVM, and Btrfs raid. And in fact you might actually prefer it for single drive use, better to be warned sooner than later. You can always increase the SCT ERC value to intentionally drive a deep recovery if you can't otherwise replace the missing data. But in particular with HDDs, you really want the drive to give up as soon as possible, and let the underlying raid repair mechanism kick in.

The gist is that SCT ERC needs to be a shorter value than the SCSI command timer. The former is a drive setting. The latter is a kernel setting *per whole block device*.