[SOLVED] RAID 5 smaller than expected

pisomojado · 2020-04-05 19:12:16

Hi, after setting up my new system, I tried to configure a RAID 5 array on 4 disks, each 3TB big, installed in my computer. I expected, since they are ~2.7TiB each, to have an array sized ~8.1TiB, but instead have one of size ~5.5TiB.

I think I followed the wiki on this pretty closely, but I must have made a mistake somewhere?

First, my disks in fdisk:

Disk /dev/sda: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: E9DBE261-F7AD-AE42-A859-D688A46A6705

Device     Start        End    Sectors  Size Type
/dev/sda1   2048 5860533134 5860531087  2.7T Linux RAID


Disk /dev/sdb: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 979968EB-1145-9E49-85F2-993F0EE9F0E3

Device     Start        End    Sectors  Size Type
/dev/sdb1   2048 5860533134 5860531087  2.7T Linux RAID


Disk /dev/sdc: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: AF3A9EC0-2BFE-CB40-9421-83857979F978

Device     Start        End    Sectors  Size Type
/dev/sdc1   2048 5860533134 5860531087  2.7T Linux RAID


Disk /dev/sdd: 2.75 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: WDC WD30EFRX-68E
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: D1E55264-2C89-A248-B33D-4A2026F6FD98

Device     Start        End    Sectors  Size Type
/dev/sdd1   2048 5860533134 5860531087  2.7T Linux RAID


Disk /dev/md127: 5.47 TiB, 6000912367616 bytes, 11720531968 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 1048576 bytes

I see that each has a single partition of type Linux RAID, with the exact same starting and ending points, and sectors as the rest. All disks have a gpt. All have the same units. Each disk is the exact same model as the rest.

The last device is the RAID array, and I don't know why it's on /dev/md127, because I set it up on /dev/md/mdd1. You'll see the latter device in my command history.

This is what I see in mdadm:

michael@icarus ~ [2]> sudo mdadm --detail /dev/md/mdd1
/dev/md/mdd1:
           Version : 1.2
     Creation Time : Sat Apr  4 17:34:58 2020
        Raid Level : raid5
        Array Size : 5860265984 (5588.79 GiB 6000.91 GB)
     Used Dev Size : 2930132992 (2794.39 GiB 3000.46 GB)
      Raid Devices : 3
     Total Devices : 4
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Sun Apr  5 13:09:12 2020
             State : clean 
    Active Devices : 3
   Working Devices : 4
    Failed Devices : 0
     Spare Devices : 1

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : icarus:mdd1  (local to host icarus)
              UUID : 53d9166e:58c4ccb0:9b50f503:4979f30d
            Events : 6403

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       4       8       33        2      active sync   /dev/sdc1

       3       8       49        -      spare   /dev/sdd1

I see the array is a raid5, with 4 total devices, 1 spare and 3 active. The used device size looks right as a per-device statistic, and yet the array size is only half the total device capacity. Chunk size and other details seem fine, but I'm pretty new to RAID.

I used the following command to build the array:

sudo mdadm --create --verbose --level=5 --chunk=512 --raid-devices=3 /dev/md/mdd1 /dev/sda1 /dev/sdb1 /dev/sdc1 --spare-devices=1 /dev/sdd1

I don't think it's relevant, but I later made a filesystem on the array using:

sudo mkfs.ext4 -v -L data -m 0.01 -b 4096 -E stride=128,stripe-width=384 /dev/md/mdd1

I performed the disk partitioning in fdisk, and really all I did was add a gpt, add a single partition on each disk with "Linux RAID" as the type, and later ran cgdisk to add a label, e.g. "data-a" in case it helped in a future array rebuild.

Can anyone spot what I did wrong, or: is there some other constraint on array sizes that I don't understand?

Thanks!

Last edited by pisomojado (2020-04-05 22:46:43)

Lone_Wolf · 2020-04-05 19:34:52

Spare drives are unused until/unless a raid drive fails, so you're missing that space.

4 devices total ~ 12 TiB .
subtract 1 device , ~ 9 TiB left for raid array .
subtract 1/3rd for parity , ~ 6 Tib left for data .

With a raid 5 you usally put spares in storage, not in the raid enclosure. (works great with hotswappable removable drives)

Last edited by Lone_Wolf (2020-04-05 19:36:52)

cmurf · 2020-04-05 22:04:09

pisomojado wrote:

I used the following command to build the array:

sudo mdadm --create --verbose --level=5 --chunk=512 --raid-devices=3 /dev/md/mdd1 /dev/sda1 /dev/sdb1 /dev/sdc1 --spare-devices=1 /dev/sdd1

Chunk/strip/stripe element size of 512 KiB is the mdadm default, suited for an array with files typically 1MiB in size or larger: images, music, and video files. For more generic use, it'll perform better with a smaller chunk size, e.g. 64KiB.

I don't think it's relevant, but I later made a filesystem on the array using:
sudo mkfs.ext4 -v -L data -m 0.01 -b 4096 -E stride=128,stripe-width=384 /dev/md/mdd1

Both mkfs.xfs and mkfs.ext4 automatically detect mdadm/lvm array geometry so you do not need to specify stride or stripe-width.

The specified stripe-width you used is incorrect. It should be 128*2=256. The parity drive does not count. You counting the parity is consistent with your expectation that the array size to be 3*2.7GiB rather than 2*2.7GiB.

I can't estimate the performance impact of this, but I figure it's probably not terrible.

cmurf · 2020-04-05 22:05:23

Also, don't forget timeout mismatches. Common problem.
https://raid.wiki.kernel.org/index.php/Timeout_Mismatch

pisomojado · 2020-04-05 22:45:46

Thanks, Lone_Wolf, I did not correctly understand the nature of spare drives in mdadm. I've added my spare to the array, and resyncing now.

Cmurf, thanks for the tips about ext4. I will let it configure itself, when I reformat the filesystem.

Thanks! I am marking my problem solved.

Arch Linux

#1 2020-04-05 19:12:16

[SOLVED] RAID 5 smaller than expected

#2 2020-04-05 19:34:52

Re: [SOLVED] RAID 5 smaller than expected

#3 2020-04-05 22:04:09

Re: [SOLVED] RAID 5 smaller than expected

#4 2020-04-05 22:05:23

Re: [SOLVED] RAID 5 smaller than expected

#5 2020-04-05 22:45:46

Re: [SOLVED] RAID 5 smaller than expected

Board footer