You are not logged in.

#1 2012-08-23 19:57:27

javex
Member
Registered: 2011-12-30
Posts: 76

Crazy long resize2fs duration & cannot stop it

Hi,

first of all: all sizes provided during this article will be to the power of 1024. I will still call them GB, TB, etc. because TiB is stupid smile

Okay right to the point: I have an LVM logical volume that consisted of 2 * 2.73 TB = 5.46 TB drive space. I also had two "old" 1.82 TB drives (2 * 1.82 TB = 3.64 TB). I also installed those into the logical volume which gave me a nice 9.10 TB storage,

From the old (5.46 TB) logcial volume I create an ext3 parition. After growing my logcial volum with those two other drives, I wanted to grow my partition, too. Since it was mounted (and my kernel supports online resizing), I launched

 resize2fs /dev/vgname/lvname

That was 72h ago. If I checked my drive size before it gave me 5.46 TB (expected), if I check it now it gives 5.8 TB. Boy, that does not escalated quickly. Calculating based on the time it already took and the size it still has to take this puts me at an amazing 770 hours or 32 days.

Why does this take this long? I knew it would not be fast, but I thought 2 days max.

Furthermore, though risky, I tried to end the process. It did not react to SIGINT, SIGHUP, SIGTERM, so I finally tried SIGKILL. Not even that could stop it. My research shows that a kernel operation can block SIGKILL so I assume it hangs in there. This means only a reboot would stop it (but also possibly damage my drive).

Note: I cannot backup the data so backup & format is not an option (who hase 4 TB of storage lying around unused anyway?).

Also, every drive is individually encrypted with a keyfile and merged together as an lvm after decryption (if this may hurt the speed).

I am willing to unmount the drive (for a day or so) if this would speed up the process. My most urgent task is to stop the process (as it slows down all affected drives) and to find a solution that is quicker. I also would prefer not loosing the data, as you can imagine.

Finally, I noticed that most of the work goes into CPU usage. One core is always as maximum 100% and the drive is barely touched (check with HDD led & iotop).

Last edited by javex (2012-08-25 09:13:34)

Offline

#2 2012-08-23 22:47:36

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: Crazy long resize2fs duration & cannot stop it

You're trying to resize a mounted partition?


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#3 2012-08-24 18:32:48

javex
Member
Registered: 2011-12-30
Posts: 76

Re: Crazy long resize2fs duration & cannot stop it

Yes, as resize2fs supports online resizing as long as my kernel does (which it does), I decided to use it without unmounting the partition.

Offline

#4 2012-08-24 22:38:25

nomorewindows
Member
Registered: 2010-04-03
Posts: 3,362

Re: Crazy long resize2fs duration & cannot stop it

A LVM is usually several partitions on a RAID like setup.  Perhaps, you are trying to resize the whole volume group, instead of the individual partititons that have to be the members of the LVM have to be the same size.  Maybe it would be better advised to resize the partitions individually, but since they are part of the volume group. I'm not sure how that all works.  You'd have to rebuild your LVM.


I may have to CONSOLE you about your usage of ridiculously easy graphical interfaces...
Look ma, no mouse.

Offline

#5 2012-08-25 09:10:36

javex
Member
Registered: 2011-12-30
Posts: 76

Re: Crazy long resize2fs duration & cannot stop it

Yeah well, I can't rebuild them, as I noted above.
Also: I added my new PVs, then added them to the VG and then extended by LV so that all was already done. It now behaves like one hard drive with one partition that has excess space left that should be added to the current space.

Offline

#6 2012-10-10 07:36:59

dreww
Member
Registered: 2012-10-10
Posts: 1

Re: Crazy long resize2fs duration & cannot stop it

did you find any resolution? is the process still running?

Offline

#7 2012-10-11 11:22:17

javex
Member
Registered: 2011-12-30
Posts: 76

Re: Crazy long resize2fs duration & cannot stop it

I finally managed to kill it. I was very cautious, but after 800 (!) hours, I had to move, so I had to kill it to shut down the server. Initially, I just did a reboot, but the process prevented it.

The final solution then was to do a lazy unmount of the partition being resized and then do a reboot. The process will take some time (in the range of minutes) and then stop and let the system reboot. I only noticed issues with files I added while resize was running and there it was like 10% of the files are corrupt. Anything that was already on it did not cause any issues yet. I did not run fsck yet, since this will also take forever, but at least you can kill that one wink

The problem still remains, though and my drive is not fully resized. My next step will be to try to resize it while unmounted and if that also has such speed issues, I will just let it continue online as it should not take more than a few weeks... At least, I am happy that no courrption of my data happened and everything went better than I expected. I love Linux!

Offline

Board footer

Powered by FluxBB