Memory usage cleaning/shrinking

Acegi · 2010-02-21 20:34:29

Ok just looking for some explanation on how to repeat this in a command line. I usually have many applications running at the same time (some minimized, others on the 3 other desktops). This brings my memory usage to about 90% (out of 2GB).

Then I ran a program that started trying to use up a lot of memory at once. This slowed down my computer to the point of freezing my mouse but after about a minute or two I could move my mouse again and I ran gnome-system-monitor and killed the application which forced itself to 800MB of memory in size. My other application's memory usage (without closing) shrank in size considerably so that I was left with 75% memory left over, only 25% (~450MB) in use.

How do I force my other application's memory usage to shrink by commandline? I would like to be able to control when my computer "cleans/shrinks" it's memory usage rather than wait for an out of control application to potentially crash my computer.

fukawi2 · 2010-02-21 22:30:14

Your system either:
a) Was using a lot of cache for the apps that were already running - when the new app started, it dumped the cache and replaced it with the ram required by the new app.
b) Was actually using the RAM, and had to swap it out when the new app started.

I'm guessing since the app was quite slow for a while, I'm guessing it was probably swapping things out. You can make the system swap earlier by increasing your swappiness:
http://kerneltrap.org/node/3000

Acegi · 2010-02-22 02:11:30

So to increase your swappiness I searched the net and found this command:

sudo sysctl -w vm.swappiness=100

Is this the correct way of setting swappiness?

fukawi2 · 2010-02-22 04:55:44

Yes, however this will make your kernel *very* eager to move applications out to swap, which is *much* slower than RAM. The overall result may be that your system is actually slower in general compared to just when this big application is started. You may need to play with the swappiness to find the 'sweet spot' for your system.

Here is a better article than the one I linked to earlier:
http://lwn.net/Articles/83588/

Acegi · 2010-02-22 05:38:03

That's actually quite a good article. Thanks for the link! The comments on that page were quite helpful since I began to wonder if I would have to run the command after every reboot.

One more thing about this (back to the original question actually), according to that page the system has "distress" variable which the kernel will ignore till it is too high. At which point I think the system will do the shrink/clean/swap for all of the processes memory. Is there a command that will forcibly do this? Swappiness is more about management. The question really is if I could minimize current memory usage to it's bare minimum on command.

fukawi2 · 2010-02-22 09:42:28

sync; echo 1 > /proc/sys/vm/drop_caches

http://www.linuxinsight.com/proc_sys_vm … aches.html

Last edited by fukawi2 (2010-02-22 09:43:42)

Acegi · 2010-02-22 15:18:22

Ah thanks again. I'll be sure to bookmark this page too!

broch · 2010-02-22 16:29:11

Acegi,
your presumption that you are running out of memory is incorrect. What I mean is that you do not need to worry. See my example at the end.

Yes, however this will make your kernel *very* eager to move applications out to swap, which is *much* slower than RAM.

that is not how swappiness works. This is common mistake (so called "performance" trick that decreases swapping), there are even users who clain that swappiness=1 speeds up system.
You need to run program that really is memory hungry to see the effect of swappiness modification.

% of swappiness defines how much of swap file can be used if RAM is saturated. Not how eager swapping is going to be. As long as RAM is available (free + cached), swap stays untouched.
Lets imagine that you have swap file 100MB size 1% is 1MB and your memory hungry program requires 70MB of swap. You just made your system super slow because it will take forever to dump/recover 70MB of cache from swap 1MB in size. Lets make swappiness=60
this means that 60MB of 100MB swap can be used. Dumping and recovering of cache is going to be much. much faster.

What you do with swappiness setting is putting limits on a highway: if traffic is low then it does not matter how many lines is available but if traffic is high then it really matters if you have only one line or five lines available.

There is absolutely wrong suggestion in the Arch wiki advising for performance reasons to set swappiness=1
if someone sets parameter as suggested and does not have problem with system slowing down then it means that RAM is not fully used anyway, and one can set swappiness to 0 (zero). Obviously killing swap completlely means that user have no clue about VM as kernel needs swap by definition (even unused)

if you want to keep sysctl parameters permanent, append them to /etc/sysctl.conf no need to re-run these every time.

example:
[~]$free -m
total used free shared buffers cached
Mem: 2025 1925 100 0 0 1346
-/+ buffers/cache: 578 1447
Swap: 4000 0 4000
[~]$ sysctl -a | grep -i swap
vm.swappiness = 60
[~]$ uptime
08:14:43 up 4 days, 22:13, 1 user, load average: 0.02, 0.04, 0.04

as you can see after four days swap is still free even though swappiness is set to 60 (with openoffice, ff, amarok and different software compilations running). This value (swappiness) will matter as soon as I start LightZone which ofter requires more RAM that I have (so it hits swap). with low swappiness settings system will became super slow, with high swappiness value system will recover relatively fast.

The second suggestion (drop caches) is also incorrect.

Last edited by broch (2010-02-22 21:09:04)

fukawi2 · 2010-02-22 22:02:56

broch wrote:

that is not how swappiness works
....
The second suggestion (drop caches) is also incorrect.

Did you even look at the linked articles?

broch · 2010-02-23 03:16:59

fukawi2 wrote:

broch wrote:
that is not how swappiness works
....
The second suggestion (drop caches) is also incorrect.
Did you even look at the linked articles?

first link I have read six years ago. Second is also old (I read it in 2006). If you are fond of lkml, search for drop cache over there.
I would suggest to remember that this is 2010 and kernel 2.6.32 not 2004 and kernel 2.6.15
Not to mention that if you can set swappiness=0 then obviously you do have more RAM that your app need and in fact swap is not a problem for you at all.

Yup, keep swappiness=60 or more for really good performance (see, Morton and Piggin to better understand swappiness, but this is: 1) old paper that does not consider last six (6) years of changes, 2) it only tackles one aspect of swap and VM)

It would be amazing to think that these articles aren't well known. Do you really believe that nobody knows about these links?
I would think that you never got to the point when program single requires all RAM and more, othewise you would know how bad is setting swappiness below default.

Last edited by broch (2010-02-23 03:20:08)

fukawi2 · 2010-02-23 03:41:52

Feel free to post links to your sources.................

broch · 2010-02-23 05:12:24

You evidently never got to the point when swap is needed. I would strongly suggest some real life experiments instead of citing quite old papers.

fukawi2 · 2010-02-23 07:42:12

If I'm doing it wrong, I'd much prefer to read about it from someone who does know and learn where I've got it wrong, rather than perform my own arbitrary tests and draw my own conclusions

broch · 2010-02-23 15:38:54

fukawi2 wrote:

If I'm doing it wrong, I'd much prefer to read about it from someone who does know and learn where I've got it wrong, rather than perform my own arbitrary tests and draw my own conclusions

no problem:
regarding drop cache:
http://rackerhacker.com/2008/12/03/redu … er-at-bay/

drop cache is used for completely different purpose e.g benchmarking (when you are repeating same benchmark with modified parameters and you need system "naive").

OP question was:

How do I force my other application's memory usage to shrink by commandline?

Modifying these parameters can affect system responsiveness, even if swap is in use.
vm.vfs_cache_pressure
vm.overcommit_memory
vm.overcommit_ratio
other parameters:
vm.dirty_background_ratio
vm.dirty_ratio
vm.dirty_expire_centisecs
vm.dirty_writeback_centisecs
vm.max_map_count

and also set nice for specific app

references:
www.google.com (and discard swappiness)
I am not going to list references for all these (and still more parameters to tackle)

I just run test that dumped data on swap. Guess what, swappiness is set to 60 and OS responsiveness is still unchanged.

end note:

rather than perform my own arbitrary tests and draw my own conclusions

actually shrinking swappiness is arbitrary
if you are serious about performance, then you would consider published papers as hints and run tests on your system to get right parameters for your specific tasks.
OP should check for app memory leak and understand that he has limited resources of RAM/memory to play with (64-bit may help if not used).
I can invoke only one program that will eat all RAM available in no time. So obviously, in some instances irrelevant of OS one needs to deal with specific memory requirements.

I don't know if this will help, if not then sorry for taking your time.

Last edited by broch (2010-02-23 15:39:28)

Acegi · 2010-02-23 17:22:23

Well I don't consider this time wasted! Anyone can post a guide/webpage on the net. It's when there's people to debate these things that real answers come out so I consider this a learning experience!

Arch Linux

#1 2010-02-21 20:34:29

Memory usage cleaning/shrinking

#2 2010-02-21 22:30:14

Re: Memory usage cleaning/shrinking

#3 2010-02-22 02:11:30

Re: Memory usage cleaning/shrinking

#4 2010-02-22 04:55:44

Re: Memory usage cleaning/shrinking

#5 2010-02-22 05:38:03

Re: Memory usage cleaning/shrinking

#6 2010-02-22 09:42:28

Re: Memory usage cleaning/shrinking

#7 2010-02-22 15:18:22

Re: Memory usage cleaning/shrinking

#8 2010-02-22 16:29:11

Re: Memory usage cleaning/shrinking

#9 2010-02-22 22:02:56

Re: Memory usage cleaning/shrinking

#10 2010-02-23 03:16:59

Re: Memory usage cleaning/shrinking

#11 2010-02-23 03:41:52

Re: Memory usage cleaning/shrinking

#12 2010-02-23 05:12:24

Re: Memory usage cleaning/shrinking

#13 2010-02-23 07:42:12

Re: Memory usage cleaning/shrinking

#14 2010-02-23 15:38:54

Re: Memory usage cleaning/shrinking

#15 2010-02-23 17:22:23

Re: Memory usage cleaning/shrinking

Board footer