Arch Linux

stefguy

Member

Registered: 2018-07-13

Posts: 5

usb-disk + raid + dm-crypt + btrfs - performance issues since 5.0

Howdy,

have the following setup here running flawlessly for nearly six month: a USB 3.0 JBOD case with two disks, those are mirrored (RAID1) with md, the md device is dm-crypted and there's a btrfs with subvolumes running on top of that. Even with encryption I get write rates of 100 to 120 Megabytes per second, the disks are the limiting factor here (two 5 year old Seagate's, 500GB each (yes, old, but no production data :-)). Everything was fine, but ..

Since upgrading to kernel 5.0 this afternoon, I see the following
- cryptsetup luksopen runs significantly longer, 20 seconds vs. 3 seconds
- mounting/unmounting the fs takes seconds vs. almost instantaneous
- when these operations run, cpu wait goes up (around 40% on one of two cores)
- when I do file I/O on the fs, eg. writing 1GB of zeroes with dd, and then do a sync, sync hangs .. forever .. and cpu wait on both cores goes up to 100%,  but the system stays responsive

I see NOTHING in the journal / via dmesg .. everything just is very very sloooooow ..

I downgraded to 4.20 and got my old performance back .. so I assume something related to the 5.0 upgrade here.

The system itself runs from a 128GB SATA SSD which also has a small encrypted partition (no LUKS / plain cryptsetup, no md, ext4) which runs fine even on 5.0. So it might boil down to an issue with usb-storage or md/RAID.

Anyone seen something similar? Any ideas where to go from here?

Greetings, Stefan

Last edited by stefguy (2019-03-08 21:32:57)

Buddlespit

Member

From: Chesapeake, Va.

Registered: 2014-02-07

Posts: 501

Re: usb-disk + raid + dm-crypt + btrfs - performance issues since 5.0

I don't think it's an md/raid issue, my small md raid 0 array appears to be working fine. No crypt, just ext4.

---

Hw-Probe

frostschutz

Member

Registered: 2013-11-15

Posts: 1,417

Re: usb-disk + raid + dm-crypt + btrfs - performance issues since 5.0

Anyone seen something similar?

Only this https://marc.info/?l=linux-raid&m=155207909531831&w=2

below 1MB/s transfer rate, yikes

loqs

Member

Registered: 2014-03-06

Posts: 17,309

Re: usb-disk + raid + dm-crypt + btrfs - performance issues since 5.0

Possibly related https://bugs.archlinux.org/task/61925

mxfm

Member

Registered: 2015-10-23

Posts: 163

Re: usb-disk + raid + dm-crypt + btrfs - performance issues since 5.0

Are you sure you were able to write large files (>500 MiB) flawlessly on dm-crypt+lvm setup before update?

Because I have for many years nasty IO write bug for large files (IO write hungs for 3-5 minutes, disk is busy but other parts of system are responsive). It was present with ext4 and btrfs filesystems, with and without lvm.

stefguy

Member

Registered: 2018-07-13

Posts: 5

Re: usb-disk + raid + dm-crypt + btrfs - performance issues since 5.0

Are you sure you were able to write large files (>500 MiB) flawlessly on dm-crypt+lvm setup before update?

Yes. Absolutely flawlessly ..

Because I have for many years nasty IO write bug for large files (IO write hungs for 3-5 minutes, disk is busy but other parts of system are responsive). It was present with ext4 and btrfs filesystems, with and without lvm.

I have a few (lvm+)dm-cryprt setups around here .. not all on arch, some on Ubuntu, but I have never seen the issues you describe.

If I find the time later this week I will add another disk to the JBOD case, bring the machine back to 5.0 and test the write performance with dd on the raw disk, then add layer by layer (dm-cryt, fs ..) to see with wich item causes the trouble ...

Stefan

kes31

Member

Registered: 2019-03-12

Posts: 1

Re: usb-disk + raid + dm-crypt + btrfs - performance issues since 5.0

I have the same issue with raid + dm-crypt + xfs.