Arch Linux

MaddyBoo

Member

From: Portland, OR

Registered: 2017-05-02

Posts: 10

Website

[SOLVED] Poor disk performance after resizing LVM volumes

I'm using a LVM-on-LUKS setup, with the following structure:

/ (root volume, ext4)
/home (home volume, ext4)
(swap volume)

A few weeks ago, my root volume was running out of free space, so I booted into a live image and shrank my home volume/expanded my root volume according to the instructions on the wiki. The LUKS partition size remained the same.

I made sure to run e2fsck after each operation, and it always reported everything to be OK.

After rebooting into my main OS, things started acting funny. For example, once at the TTY login screen, upon typing in my username and pressing enter, there was a delay of 5-10 seconds before the password prompt appeared. Again, after entering my password, it took 5-10 seconds for my MOTD/CDM to appear. After launching X via xinit, it took over a minute for X to start. This is very unusual, as login times were normally 1-3 seconds total.

I immidiately suspected some type of disk/fs corruption, so I booted back into a live image and did a full system backup just in case. I then started running any type of diagnostic tool I could find, scouring logs, etc. I could not find any unusual error messages and all diagnostics reported everything to be OK.

After I couldn't find any exact cause, I thought perhaps it was some type of caching issue. I decided to just try using my system normally for a while, to see if things would return to normal. I've been using it as such for the past 3 weeks, and most things function normally, but certain things are extremely slow.

I then began to realize that the problem seemed to be very slow disk access speeds for certain operations on the my root volume. However, not all operations seemed slow - using disk speed testing techniques such as using dd to read/write 1GB of data from /dev/zero, both my home and root volumes reported read/write speeds exceeding 600MB/s, which seems normal to me.

I recently installed docker as a development tool, primarily to contain a Postgres server. Starting the docker service was extremely slow, taking over 2 minutes. Then, creating and starting new images was slow as well. SQL queries were extremely slow as well, taking several seconds for simple SELECT statements returning only 1 row, which should have been nearly instantaneous. I tried running pg_test_fsync from within my postgres docker container, which reported speeds of less than 1op/sec, which is extremely low.

I then tried running pg_test_fsync outside of a docker container, suspecting docker was the culprit. I used the -f flag to point to /var/lib/postgres/data, the default postgres data directory. The results were the same as within the docker container, less than 1op/sec.

Finally, I tried running pg_test_fsync, using -f to point to a file on my home volume. Completely different results - over 800ops/sec, which is relatively fast!

So, I think I've narrowed my problem down to my LVM root volume, but I'm not sure where to go from here. I am tempted to just wipe my SSD clean and remake all of my partitions/LVM volumes, but that's gonna be a big hassle. I thought before I did that, I would ask to see if anyone can help me troubleshoot this further?

Last edited by MaddyBoo (2017-09-06 12:29:22)

epicepee

Member

Registered: 2014-11-02

Posts: 54

Re: [SOLVED] Poor disk performance after resizing LVM volumes

Just a shot in the dark, but have you checked the filesystem alignment?

MaddyBoo

Member

From: Portland, OR

Registered: 2017-05-02

Posts: 10

Website

Re: [SOLVED] Poor disk performance after resizing LVM volumes

Just a shot in the dark, but have you checked the filesystem alignment?

I couldn't really figure out how to do this with my setup.

However, I ended up just reformatting my SSD and using a new setup: Btrfs on LUKS.

I copied my old root/home directories over to my new Btrfs subvolumes, et voilà - everything works! Not only that, but it's blazingly fast!

At least now I've truly confirmed my suspicion that something was screwy with my filesystem/LVM setup, not my configuration or another part of my system...

Anyway, my original problem wasn't really solved, but it's not a problem anymore... do I mark this thread solved?

Last edited by MaddyBoo (2017-09-05 20:47:54)

circleface

Member

Registered: 2012-05-26

Posts: 639

Re: [SOLVED] Poor disk performance after resizing LVM volumes

It's really up to you.  If you consider reformatting and starting over as solved, then go ahead and mark it as solved.  Also, I'm glad you got something working because I read your post yesterday and was really trying to figure out what could cause such horrible performance.  1op/sec is a bit slow isn't it....