Arch Linux

Markus.N2

Member

From: Germany

Registered: 2013-08-22

Posts: 106

Swap performance on mdraid/dm-crypt/LVM horribly bad, how to tune ?

As soon as my PC starts swapping more than a few MB, it looks like it is frozen for quite some time. In fact, I always thought it was frozen (and hit the reset button) up to a few weeks ago, when it surprisingly recovered. I have 32 GB RAM and 64 GB swap, the latter one sitting on a RAID-5 (mdraid), with the whole space encrypted as one volume by dm-crypt (LUKS), and LVM on top of it.

Having so much RAM, it looks like I'm often too optimistic about what I can do with it ...

free -h

most times shows that swap is (nearly) unused.

So let me describe the symptom that occurs when my RAM is used up:
Suddenly, the PC becomes completely unresponsive to mouse and keyboard, ping or ssh login via network is also impossible (100% packet loss / host unreachable). Even the clock of my Desktop (XFCE) stops. Up to this moment, there is no notable difference in performance or responsiveness.
After some time it continues to work as if nothing happened. "Some time" means: at least a few minutes, mostly about an hour, in one case over night was still not enough and I needed to reset. After recovery,

free -h

shows that swap is used. During the freezing, no system log is written.

A few days ago, it took nearly 2 hours until it recovered, and swap was at 19 GB afterwards.
While the PC seemed frozen, the HDD LED on the case blinked sometimes. It was not "continuously on, just a bit flickering" like it is when I write files to the data volume on the RAID. There really was only very short blinks with a long dark time between them. This time, I started watching the LED right at the moment when the PC became unresponsive. And the first short bling occurred after nearly 3 minutes. The next one about one minute later. The intervals changed. After the second blink, I started doing something else, and looked at the PC every now and then. A few times, I saw 3 to 5 short blinks in a row, but the most time it was dark.

Why am I describing this in so much detail ?
Well, to me it looks like the system does not even try to write to the swap volume continuously. It looks more like it's thinking about whatever for a long time, then write a little bit, and think again.
The RAID does not have general I/O performance issues. It holds 2 volumes, swap and data. Doing normal file I/O on the data volume is as really fast: copying 19 GB of files takes less than 3 minutes, compared to the 2 hours of creating swap output. I would call this quite a difference.

Thinking about it again, I'm not even sure if the underlying setup of the swap volume is the root cause of problem, maybe only making it worse. I don't constantly check my memory situation (in fact, I bought the big RAM in order to not having to so), but when I check during normal operation, swap is almost unused most of the times. It looks a bit like the system is deciding too late that it's time to start swapping and then it has no internal memory left to do it in an efficient manner.

I've read about tuning vm.swappiness, but I did not dare to touch it. Right now, it's at the default value of 60.

cat /proc/sys/vm/swappiness
60

Any ideas what I can do for tuning or at least troubleshooting ?

twelveeighty

Member

Registered: 2011-09-04

Posts: 1,456

Re: Swap performance on mdraid/dm-crypt/LVM horribly bad, how to tune ?

In my opinion, you are troubleshooting the wrong problem. What is this machine used for and why are you running out of 32 GB of RAM in the first place? Are you running virtual machine guests on that box that balloon RAM out of control? Do you have a memory leak or runaway process? There are situations possible where massive disk operations get cached and that can utilize all your available RAM, but that should be freed up when needed for other purposes. IIRC, that can be tuned with vm.vfs_cache_pressure (see https://wiki.archlinux.org/index.php/Sw … _pressure). But it sounds fishy that you are having these massive swap operations.

Markus.N2

Member

From: Germany

Registered: 2013-08-22

Posts: 106

Re: Swap performance on mdraid/dm-crypt/LVM horribly bad, how to tune ?

Fishy or not ... I think it can always happen that swap is needed. Often unexpectedly. And then the system configuration should just be prepared to swap without introducing additional problems.
Also, my future plans include using the swap space for suspend-to-disk. Therefore I don't think I'm troubleshooting the wrong problem.

This machine is used for a lot of different stuff: Video editing, some development, lots of web browsing, running virtual machines, ...

When I'm doing video editing, about half of the RAM is used for the video files on /tmp. Yes, it just runs so much smoother on tmpfs compared to a HDD or even SSD that I don't want to have the intermediate files somewhere else. Not so long ago, when I had "only" 16 GB, video editing on /tmp was impossible.

Additionally, I run multiple instances of Firefox. I have created separate profiles for my most important tasks. One is for Arch Linux related stuff, one for Arduino development, one for Android development, one for CAD / 3D printing stuff, one for eBay, ... As there are currently two very annoying bugs in Firefox that affect session restore, I prefer to leave the FF instances open as long as I can (i.e. until I need to reboot because of system updates).

Most times, my RAM is sufficient. On the "incident" a few days ago, I thought I could start a VM while having nearly 12 GB video files on /tmp, an open AndroidStudio and device emulator, and all those open FF instances. Although this combination often worked, this time it was too much ... But that's what I bought that RAM for. Unfortunately, I can not upgrade to 64GB without buying a whole new PC. The 32 GB was a lucky strike on eBay.