Arch Linux

Megamannen

Member

Registered: 2008-07-08

Posts: 7

Ridiculously slow mdadm reshape / grow

I had a software RAID5 setup with mdadm and my onboard RAID-controller. It was a 4x1TB RAID5 giving me a total of 3TB. Yesterday I added one more 1TB disk with these commands

mdadm --add /dev/md0 /dev/sdf1
mdadm --grow /dev/md0 --raid-devices=5 --backupfile=/mnt/temp/backupfileraid5

In the begining the speed was around 13MB/s but today when I checked the progress, this is what I saw:

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf1[4] sde1[0] sdc1[3] sdb1[2] sdd1[1]
      2930279424 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [>....................]  reshape =  3.6% (35387136/976759808) finish=58349.5min speed=267K/sec

unused devices: <none>

# cat /proc/sys/dev/raid/speed_limit_min
30000

And I don't read or write to the array, so that is not the problem.

Does anyone have any creative ideas how to increase the reshape speed? :S

Megamannen

Member

Registered: 2008-07-08

Posts: 7

Re: Ridiculously slow mdadm reshape / grow

Bumbelibump! Anyone?

$ cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdf1[4] sde1[0] sdc1[3] sdb1[2] sdd1[1]
      2930279424 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [====>................]  reshape = 22.9% (224112064/976759808) finish=117215.6min speed=105K/sec
      
unused devices: <none>

2g33k4u

Member

From: Glendale AZ

Registered: 2011-09-24

Posts: 2

Re: Ridiculously slow mdadm reshape / grow

I am using
5 2 TB drives
24gb ddr3
x58m board
Xeon 5550 CPU

This is what I am getting with nearly the same storage setup

root@xxxxxxxf(/etc) $ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf1[5] sdb1[0] sde1[4] sdd1[2] sdc1[1]
      5589036288 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [========>............]  reshape = 44.7% (832884860/1863012096) finish=706.3min speed=24306K/sec

Here is the settings I used to do it. You might want to use slightly smaller or larger stripe cache_size depending how much ram you have.

echo 6144 >> /sys/block/md3/md/stripe_cache_size 
echo 75000 >> /sys/block/md3/md/sync_speed_min
echo 225000 >> /sys/block/md3/md/sync_speed_max

* The tweak will go away after a reboot unles syou add it to a boot up file. My reshape was going to take 3600 minutes I made the change and it dropped to 1100 minutes *
The speeds are pretty safe min/max
Adjust the cache in 1024 increments start at 3072 and go up  use top and watch for sillyness and back it off a bit if it is getting hammered

It has already gone up while typing this.

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf1[5] sdb1[0] sde1[4] sdd1[2] sdc1[1]
      5589036288 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      [=========>...........]  reshape = 45.2% (843672896/1863012096) finish=728.1min speed=23331K/sec

Hope this helps

2G33K4U

2g33k4u

Member

From: Glendale AZ

Registered: 2011-09-24

Posts: 2

Re: Ridiculously slow mdadm reshape / grow

Just realized the post is from 2009

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 19,804

Re: Ridiculously slow mdadm reshape / grow

Just realized the post is from 2009

Indeed it is.  Don't worry, it happens to all of us.

Closing.

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way