How does swap in Linux work?

Lockheed · 2012-02-25 16:30:17

I thought I knew, but I am not so sure anymore.

I have 4GB ram and 2GB swap (I know it works because I even have hibernation), fast SATA HDD and Core 2 Duo.

On occasions I run opera (~1GB), and two virtual machines. As soon as RAM utilization goes in the vicinity of 3.8GB, my system becomes totally unresponsive and to recover it, I try to close opera window, but the first challenge is to actually move a mouse, and the second to wait about a minute before the click takes any effect.

I was thinking of putting swap on a USB flash drive to make things fast, but judging from this, it does not look like swap is being used at all.

mariusmeyer · 2012-02-25 16:37:43

You say you have swap, so how does your fstab look? Normally you need at least your RAM size in swap to be sure hibernate to disk works, but suspending should work

lucke · 2012-02-25 16:43:35

What does "cat /proc/sys/vm/swappiness" return?

Check what "free -m" shows when you run out of RAM.

Lockheed · 2012-02-25 17:07:46

$ cat /proc/sys/vm/swappiness
10

mariusmeyer, hibernation snapshot is being compressed, so at most all you need is 50% of your RAM, while usually my snapshot is well under 1gb

lucke · 2012-02-25 17:25:54

Maybe you'd have better results with higher swappiness. With 10 it might be pretty reluctant to swap out until it runs out of memory, and then you could get an unresponsive system.

meph · 2012-02-25 17:41:47

Your swap size is fine for general use, but if you run multiple VM's and other memory heavy software, it might be a problem.

I'd recommend opening 'top' in a terminal, sorting it by memory usage (just hit M), then trying to provoke the problem, i.e. doing whatever it is you do that makes your system become unresponsive and then checking the terminal with top. That should give you live detailed results on how is your memory used, what processess eat most of it and how is your swap used. Hopefully that should tell you whether you're facing real memory problems or something else is going on.

Lockheed · 2012-02-25 17:53:49

meph wrote:

Your swap size is fine for general use, but if you run multiple VM's and other memory heavy software, it might be a problem.
I'd recommend opening 'top' in a terminal, sorting it by memory usage (just hit M), then trying to provoke the problem, i.e. doing whatever it is you do that makes your system become unresponsive and then checking the terminal with top. That should give you live detailed results on how is your memory used, what processess eat most of it and how is your swap used. Hopefully that should tell you whether you're facing real memory problems or something else is going on.

Great testing method (I used htop, though).
Anyway, here are the findings:
1. I start applications in question.
2. Computer becomes very unresponsive. Luckily, I run htop beforehand, because now I would not be able to type it in the terminal.
3. after about a minute of being locked my swap grows from 0MB to 20MB, then very slowly crawls up to 240MB, while the system still stays unresponsive.
4. I managed to kill the memory-intensive app, and RAM usage instantly drops by 2.5GB, but my swap stays shows 220/2143MB 10 minutes after the app has been killed.

lucke · 2012-02-25 18:05:55

You can use "smem -kt" to see what processes use how much swap. There's also something called swapcache which isn't counted by smem (I guess), but is included in "free" and probably htop - you can see its value in /proc/meminfo. That 220 MB can either be used by processes other than the one killed or be swapcache. You can use dstat to have an overview of what your system is doing, second by second.

Lockheed · 2012-02-25 18:48:17

OK, got it.
Now, if I want to move swap to my USB drive, is there some way to safely remove this USB when I don't need it? Is unmounting SWAP partition while the system is running, safe? Would it move things stored there to free segments of RAM?

.:B:. · 2012-02-25 18:52:17

Lockheed wrote:

I was thinking of putting swap on a USB flash drive to make things fast, but judging from this, it does not look like swap is being used at all.

Are you serious?

Have you ever seen flash read/write speeds, not to mention the speed at which it wears out?

USB flash drives aren't the speed monsters SSDs can be. What you are contemplating is the fastest way to the trash can for your USB stick.

Last edited by .:B:. (2012-02-25 18:53:35)

Lockheed · 2012-02-25 18:54:24

How about the popularity of readboost function in Win7?

USB flash surely is faster than swap on a 5400rmp sata, no?

lucke · 2012-02-25 19:00:34

You can safely "swapoff" while the system is running - the swapcache will be dropped and the rest will be moved to RAM. You might be interested in using zram (formerly compache).

masteryod · 2012-02-25 19:01:08

#swapon /dev/pendrvive_swap_partition
#swapoff /dev/hdd_swap_partition - things from this swap will be moved to enabled earlier swap partition on pendrive

before removing pendrive reverse it - enable hdd_swap_partition and then disable pendrive_swap_partition (of course if swap usage is lower than available RAM space you can just # swapoff -a as lucke suggested - if not it will just give you hint that swap cannot be disabled)

if swap on pendrive will be used and you just pull it off most likely it will crush system/swap write error - it's similar to removing RAM stick while system is working - not very good idea

BTW what's your /etc/fstab? because recently I was working on quite old PC with RAM deficiency and slowdowns wasn't as bad as yours on pretty decent hardware

Last edited by masteryod (2012-02-25 19:06:22)

meph · 2012-02-25 19:19:32

You're working around the issue, not solving it. In a healthy system, when you run out of memory, your hdd should start going crazy and overall responsiveness should go down a lot, but you should still be able to use it. What you're experiencing is not how it should work, it just shouldn't completely die on you. You forgot to consider one thing - you said that swap usage slowly crawls up to 240 MB. Key word - slowly. Which means that it's not a hdd performance issue, setting swap on usb flash disk will most likely make zero difference.

Based on how you described it, the kernel is very reluctant to use swap at all, for whatever reason. Your swappiness value is 10, which means that the kernel will almost always prefer ram to swap. It may improve performance in some cases, but also cause problems, which may be your case. Try setting swappiness to something more swap friendly, say 60 (that's ubuntu default, or so I heard) and see if it helps.

Lockheed · 2012-02-25 19:55:45

meph, the USB thing is a separate matter. I know I first need to sort the regular swap out.

Anyway, I changed swappiness to 50 (also setting vm.vfs_cache_pressure=50) but it did not make a difference. It took me 4 minutes to move the cursor over X closing the window, and then after another 8 minutes of fruitless clicking on it, I just pulled the plug.
Sawp behaviour was very much the same.

My fstab:

# 
# /etc/fstab: static file system information
#
# <file system>        <dir>         <type>    <options>          <dump> <pass>
devpts                 /dev/pts      devpts    defaults            0      0
shm                    /dev/shm      tmpfs     nodev,nosuid        0      0

#/dev/cdrom             /media/cd   auto    ro,user,noauto,unhide   0      0
#/dev/dvd               /media/dvd  auto    ro,user,noauto,unhide   0      0
#/dev/fd0               /media/fl   auto    user,noauto             0      0
#/dev/cdroms/cdrom0     /media/dvd   auto   ro,user,noauto,unhide   0      0
#/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto,exec,utf8 0 0
/dev/sr0     /media/dvd   auto   ro,user,noauto,unhide   0      0

/dev/sda1 /mnt/Disk_D  ext4 defaults,noatime 0 2
/dev/sda2 / ext4 defaults,user_xattr,noatime 0 1
#/dev/sda1 /mnt/Disk_D  ext4 rw,suid,dev,exec,auto,nouser,async,user_xattr,iocharset=utf8 0 2
#/dev/sda2 / ext4 rw,suid,dev,exec,auto,nouser,async,user_xattr,iocharset=utf8 0 1
/dev/sda3 /mnt/win ntfs-3g     defaults,locale=en_US.utf8   0    0
/dev/sda5 none swap defaults 0 0
#/dev/sda6 /boot ext4 defaults 0 1
//ANDROID/sdcard /mnt/HTC/ cifs noauto,noperm,noatime,noperm,username=yyy,password=xxx,workgroup=WORKGROUP,ip=192.168.7.77 0 0
#//ANDROID/sdcard /mnt/HTC/ cifs uid=<localuser>,rw,noperm,noauto,noatime,username=yyy,password=xxx,workgroup=WORKGROUP,ip=192.168.7.77,  0 0



#UUID=7c0aafb8-6925-4ced-9d15-49037096af18 /boot ext4 defaults 0 1
#UUID=31d71e99-8cb9-4ff6-a2ee-fb1f7495e2e2 / ext4 defaults,user_xattr 0 1
#UUID=08e49767-99a0-4ac4-8740-813b82352b58 /mnt/Disk_D  ext4 defaults 0 2

#UUID=42a8786e-dbfe-40c1-91ef-31aeed96420c swap swap defaults 0 0
#UUID=0d464915-a3a2-7770-7a73-4c5af95eb273 /mnt/agora ext3 defaults 0 2
#UUID=54E866BFE8669F4C  /mnt/win  ntfs-3g  rw,users,umask=000 1 0

It might not be tidy, but I can't spot anything wrong with it.

Last edited by Lockheed (2012-02-25 20:00:17)

shaunsingh14 · 2012-02-25 20:18:10

Here's a couple of quotes from the LFS handbook.

LFS-BOOK-7.0 wrote:

Because there is not always enough Random Access Memory (RAM) available for compilation processes, it is a good
idea to use a small disk partition as swap space. This is used by the kernel to store seldom-used data and leave more
memory available for active processes. The swap partition for an LFS system can be the same as the one used by
the host system, in which case it is not necessary to create another one.

LFS-BOOK-7.0 wrote:

Most distributions automatically create a swap partition. Generally the recommended size of the swap partition is
about twice the amount of physical RAM, however this is rarely needed. If disk space is limited, hold the swap
partition to two gigabytes and monitor the amount of disk swapping. Swapping is never good. Generally you can tell if a system is swapping by just listening to disk activity and observing
how the system reacts to commands. The first reaction to swapping should be to check for an unreasonable command
such as trying to edit a five gigabyte file. If swapping becomes a normal occurance, the best solution is to purchase
more RAM for your system.

Last edited by shaunsingh14 (2012-02-25 20:19:52)

masteryod · 2012-02-25 21:34:49

OK so Lockheed, I need more data:
- is this a new installation? if so: are you familiar with Arch so we can be sure that installation was made properly?
- if it's old installation: did you notice this behavior from the start or just recently after months of normal work? when was your last system upgrade (pacman -Syu)?

for quick review you've got old devpts and shm entries, afaik it should be replaced (couple months ago when initscripts was upgraded) with this:

tmpfs                  /tmp          tmpfs     nodev,nosuid        0      0

Last edited by masteryod (2012-02-25 21:36:38)

.:B:. · 2012-02-25 21:39:48

Lockheed wrote:

How about the popularity of readboost function in Win7?
USB flash surely is faster than swap on a 5400rmp sata, no?

Check the USB 2.0 spec (480 Mbps), check the first generation S-ATA spec (1,5 Gbps), and do the math. Seektimes are far lower on flash media than on a mechanical disk, but sustained read times are not. Either way, USB sticks don't last long unless you implement some form of wear-leveling. Readyboost probably does that, but swap does not. Or normal filesystems.

Last edited by .:B:. (2012-02-25 21:40:37)

Lockheed · 2012-02-25 21:42:54

This installation is about a year old. I do full system yaourt updates every few days, and lately - daily.

I only started exceeding ~full potential of 4GB memory 2-3 months ago and this is when I started noticing it. What I do is basically run a VM machine (vbox) with assigned memory of ~2.4GB which makes it go highwire. This is a designed scenario for the purpose of this thread. Real-life scenario is that I run 2 VMs of cumulative assigned memory of similar value.

.:B:. · 2012-02-25 21:45:14

There's nothing that helps like more RAM. Swap will never fix things. Do it on an SSD and it will destroy the SSD; do it on a flash drive and it will destroy the flash drive.

That being said, I agree with meph that there should be another way to fix this kind of misbehaviour.

Last edited by .:B:. (2012-02-25 21:48:16)

Lockheed · 2012-02-25 21:48:16

.:B:. wrote:

Do it on an SSD and it will destroy the SSD;

I thought those things are designed for it.

I am at 4GB RAM laptop from 2008. Do you think I have an option to increase it beyond that?
Accidentally, I do have, as ThinkPads are friggin awesome, and I plan to do it, but not just yet.

Last edited by Lockheed (2012-02-25 21:49:17)

.:B:. · 2012-02-25 21:49:14

SSDs do implement wear-leveling, but it's still more prone to wear than a HDD.

meph · 2012-02-25 23:47:00

This is odd. I'm starting to doubt that it's caused by swapping itself. Just think about it, how does running out of memory present itself? It could hang, it could give you all sorts of error messages, but being unresponsive without any errors is weird. It means that the system is actually running but is being slowed down by something. Misbehaving process, whatever. Anyway, there are a few more diagnostic steps I came up with:

1) Try using swap file instead. Using it will mean that different physical location on your hdd will be used, and also that it will use your root partition, which is known to work well (i guess?). If it still misbehaves after you start using it, you will rule out faulty hdd (however unlikely that already is).

2) Did you ever have this problem without using your VM's? Try and use all your memory without running those, and see what happens. For example opening a dvd image in a text editor could work? Just fill your memory with random garbage and see what happens, and while doing that you may want to watch htop again. This should rule out a problem in whatever virtualization software you're using.

3) Try and turn swap off before trying to induce the problem. Again, this is about knowing how exactly will your system behave when it actually runs out of memory. Expected outcome is flooding you with error messages about killing processess, but the system should remain somewhat responsive. Cluttered and obviously not working well, but responsive. This step could help identify a memory defect, obviously the next step would be running a memtest.

4) This one is rather obvious and I suppose you already did it, but look carefully into your log files, especially kernel.log. Or better yet, induce the problem, take a precise note about when exactly things started to go bad, and then look for errors in your logs around that time.

Result is unknown, I have absolutely no idea whether it will tell you anything, but unless someone has a better idea, you might as well give it a shot. Perhaps it will give you (or us, if you post results) some new ideas.

---
Out of topic comment to the SSD discussion. Lots of these are just rumors and urban legends, currently most SSD's have a lifespan of 1-5 million write cycles. That's more than enough for a desktop computer. Obviously it's not desirable to write large chunks of data unnecessarily (such as defragmenting the drive), but other than that there's no need to worry. Yes, it will eventually get worn out, but even with heavy use it will take quite a few years. I'd even say that over the years it's more likely that the drive will die because of some other malfunction than being worn out.

Last edited by meph (2012-02-25 23:47:55)

silentsnake · 2012-02-25 23:50:12

Have you considered using a browser that is less memory consuming? ^^
I also think, 4 GB are kind of insufficient for running 2 VM besides the host operating system - depending on the their OS's. I assume you gave them 1 GB or less and they probably ran out of memory too. So they use their swap space and access the HDD and if the host system is starting to use swap as well, your HDD access times go down.
The easiest solution would be to buy more RAM. RAM is very cheap - you can get 4x4 GB (16 GB total) for around 80 €.

Lockheed · 2012-02-26 00:29:13

meph wrote:

This is odd. I'm starting to doubt that it's caused by swapping itself. Just think about it, how does running out of memory present itself? It could hang, it could give you all sorts of error messages, but being unresponsive without any errors is weird. It means that the system is actually running but is being slowed down by something. Misbehaving process, whatever.

I don't know. Windows behaves pretty much as my Arch when it runs out of memory.

meph wrote:

1) Try using swap file instead. Using it will mean that different physical location on your hdd will be used, and also that it will use your root partition, which is known to work well (i guess?). If it still misbehaves after you start using it, you will rule out faulty hdd (however unlikely that already is).
2) Did you ever have this problem without using your VM's? Try and use all your memory without running those, and see what happens. For example opening a dvd image in a text editor could work? Just fill your memory with random garbage and see what happens, and while doing that you may want to watch htop again. This should rule out a problem in whatever virtualization software you're using.
3) Try and turn swap off before trying to induce the problem. Again, this is about knowing how exactly will your system behave when it actually runs out of memory. Expected outcome is flooding you with error messages about killing processess, but the system should remain somewhat responsive. Cluttered and obviously not working well, but responsive. This step could help identify a memory defect, obviously the next step would be running a memtest.
4) This one is rather obvious and I suppose you already did it, but look carefully into your log files, especially kernel.log. Or better yet, induce the problem, take a precise note about when exactly things started to go bad, and then look for errors in your logs around that time.

1. That's quite a bother. I gonna have to do it a little bit later, if you think it is absolutely necessary.

2. I opened two ISO files (2GB each) with geany, and the result was pretty much the same, except this time swap usage went up to 700MB, but that could be because htop managed to refres after 2-3 minutes, while previously it did not. But the system was freezing up pretty much the same way.

3. Again, the result is similar. Maybe the system is a bit more responsive, especially in the first minute of running on full RAM, but afterwards everything else locks up, too. Still, the mouse is movable for most of the time, but it could be a coincidence.

silentsnake,
I usually have +20 tabs opened in my browser. I think opera is doing a fairly good job. Certainly better than firefox.

The easiest solution would be to buy more RAM. RAM is very cheap - you can get 4x4 GB (16 GB total) for around 80 €.

Yeah, I wish I was running DDR3. The cheapest 4GB SODIMM DDR2 module on ebay.co.uk is almost £60

Arch Linux

#1 2012-02-25 16:30:17

How does swap in Linux work?

#2 2012-02-25 16:37:43

Re: How does swap in Linux work?

#3 2012-02-25 16:43:35

Re: How does swap in Linux work?

#4 2012-02-25 17:07:46

Re: How does swap in Linux work?

#5 2012-02-25 17:25:54

Re: How does swap in Linux work?

#6 2012-02-25 17:41:47

Re: How does swap in Linux work?

#7 2012-02-25 17:53:49

Re: How does swap in Linux work?

#8 2012-02-25 18:05:55

Re: How does swap in Linux work?

#9 2012-02-25 18:48:17

Re: How does swap in Linux work?

#10 2012-02-25 18:52:17

Re: How does swap in Linux work?

#11 2012-02-25 18:54:24

Re: How does swap in Linux work?

#12 2012-02-25 19:00:34

Re: How does swap in Linux work?

#13 2012-02-25 19:01:08

Re: How does swap in Linux work?

#14 2012-02-25 19:19:32

Re: How does swap in Linux work?

#15 2012-02-25 19:55:45

Re: How does swap in Linux work?

#16 2012-02-25 20:18:10

Re: How does swap in Linux work?

#17 2012-02-25 21:34:49

Re: How does swap in Linux work?

#18 2012-02-25 21:39:48

Re: How does swap in Linux work?

#19 2012-02-25 21:42:54

Re: How does swap in Linux work?

#20 2012-02-25 21:45:14

Re: How does swap in Linux work?

#21 2012-02-25 21:48:16

Re: How does swap in Linux work?

#22 2012-02-25 21:49:14

Re: How does swap in Linux work?

#23 2012-02-25 23:47:00

Re: How does swap in Linux work?

#24 2012-02-25 23:50:12

Re: How does swap in Linux work?

#25 2012-02-26 00:29:13

Re: How does swap in Linux work?

Board footer