What is using my memory?

perost · 2008-02-27 11:38:12

Hi!

I've started having an odd problem lately, which is that Linux has started using swap memory. Or rather, I can't understand why it's using swap. Right now it's using 694 MiB of 2.0 GiB RAM and 521 MiB of 1.9 GiB swap according to Gnomes system monitor. So obviously there's no need to use swap right now, but there probably was a need a while ago, and it's still using the swap. Sometimes when I leave the computer on for a while and then come back to it, it's extremely slow for a couple of seconds, but then speeds up to normal. I suspect this has to do with the fact that swap is used. So now I have two questions:

1) Is there some way I can monitor how much each process is using for a long time? I suspect that some process uses an enormous amount of memory when I'm not looking, so I would like to find this process. Gnomes system monitor only shows how much each process is using at the moment, but I would like to log the memory usage over a period.

2) What is using all my memory? 694 MiB of RAM and 521 MiB of swap = 1215 MiB. However, according to the system monitor, the biggest memory user is java (=Azureus) with 119 MiB. Listen is using 81 MiB and firefox 63 MiB. Then there's Xorg using 64 MiB, and various gnome processes. But it does not add up to anywhere near 1215 MiB. It's not used for buffers from what I know, since free shows that 1984 MB of 2010 MB is used, and 1229 MB of those are used for buffers. So somewhere someone is using alot of memory, and I would really like to know what.

floke · 2008-02-27 21:40:50

You could try the 'top' command in the terminal - might give a clue?

perost · 2008-02-27 22:24:50

floke wrote:

You could try the 'top' command in the terminal - might give a clue?

Thanks for the suggestion, but I don't think top can help much. top only shows how much memory the processes use at the moment, but I would like to log the memory usage during a time interval. I basically shows the same information as Gnomes system monitor. I guess it would be possible to somehow call top periodically and gather memory usage information, but it feels like there should be an easier way.

nogoma · 2008-02-28 14:48:05

You could try out atop, it's in AUR. Does logging and such, as you suggest. http://www.atcomputing.nl/Tools/atop/whyatop.html

perost · 2008-02-28 20:34:17

nogoma wrote:

You could try out atop, it's in AUR. Does logging and such, as you suggest. http://www.atcomputing.nl/Tools/atop/whyatop.html

Thanks! atop seems to be just what I was looking for.

Last edited by perost (2008-02-28 20:36:38)

awagner · 2008-02-28 20:53:00

more tools:

- smaps and ps_mem from http://bmaurer.blogspot.com/2006/03/mem … smaps.html
- collectl from http://collectl.sf.net/ (on AUR)

with these, I have a script called monitorme.sh:

#!/bin/sh
# ps -A | grep $1 || $1 &

PID=`pgrep $1`
echo $PID

collectl.pl -sZ -i10:120 -F 0 -Zp$PID -P -f $1 &

while true ; do
    echo " ------------------------------ "
    echo -n `date -R`
    echo "      Monitoring $1"
    echo " private   + shared   = total RAM used by program"
    ps_mem.py 2>/dev/null | grep $1
    echo " "
    echo "Details (smaps):"
    smaps.pl $PID
    echo " "
    echo " "
    echo " "
    sleep 120
done

this is then called by f.ex.

monitorme.sh pinot-dbus-daem > pinot-dbus-daemon.log

(as root)

Note: This is for watching space useage of a single process, but surely the tools can be used to monitor all of them, too.
I wasn't aware of atop yet, going to have a look at it, too...

HTH,
Andreas

markseger · 2008-03-01 14:28:36

I'm the author of collectl and I just wanted to say it's always nice to see new ways that people are using it. Clearly the way suggested about will show specific types of memory for a specific pid. By default collectl will show ALL memory for ALL pids and if you choose to it will log it to a file that you can then play back later - this is a reasonable low overhead operation (<0.1% if you only look at the processes once/minute).

Here's an example of the default process monitoring output which looks pretty much like 'ps', noting if you're running on a kernel that supports process I/O stats you'll see them too:

# PROCESS SUMMARY (faults are /sec)
# PID User PR PPID S VSZ RSS SysT UsrT Pct AccuTime MajF MinF Command
1 root 15 0 S 1M 476K 0.00 0.00 0 0:14.73 0 0 init
2 root RT 0 S 0 0 0.00 0.00 0 0:00.00 0 0 migration/0
3 root RT 0 S 0 0 0.00 0.00 0 0:00.00 0 0 migration/1
4 root 15 1 S 0 0 0.00 0.00 0 0:14.16 0 0 keventd
5 root 34 1 S 0 0 0.00 0.00 0 0:01.94 0 0 ksoftirqd_CPU0

but an alternate format allows you to look at all the types of process memory the kernl knows about like this:

# PID User S VmSize VmLck VmRSS VmData VmStk VmExe VmLib Command
1612 rpcuser S 1524K 0 688K 44K 16K 28K 1408K rpc.statd
1727 root S 1480K 0 604K 32K 8K 20K 1376K /usr/sbin/automount
1740 root S 9296K 0 3520K 1172K 48K 20K 5736K /usr/sbin/snmpd
1749 root S 3508K 0 1516K 96K 20K 268K 2972K /usr/sbin/sshd
1763 root S 2024K 0 864K 248K 16K 132K 1588K xinetd

For those who want to learn more about collectl, and as mentioned in the previous note, you can look on sourceforge at http://collectl.sourceforge.net/. Just keep in mind unlike most tools, collectl is NOT a single function tool in that it is not just for monitoring processes. You can monitor virtually everything all the existing performance monitoring tools do PLUS you can look at Infiniband, Lustre and as of my latest release, you can now look at Interrupts by individual CPU.

There are just too many features for me to try to list them but I did put together a high level summary at http://collectl.sourceforge.net/Features.html so be sure to check it out...

-mark

markseger · 2008-03-01 18:15:50

I forgot to mention, since 'awagner's script is monitoring a specific PID, that's about as efficient as process monitoring can get and you could probably watch that process every couple of seconds with minimal overhead. I talk about that topic in a little more detail in http://collectl.sourceforge.net/Process.html
-mark

Last edited by markseger (2008-03-01 18:16:15)

perost · 2008-03-03 13:20:32

Well, seems like I've managed to solve my swap usage problem at least. I found this article which explains why Linux is using swap when there's free memory. I solved it by settings the swappiness to 10 as proposed, and now Linux doesn't seem to use swap at all as long as I have enough free memory.

Arch Linux

#1 2008-02-27 11:38:12

What is using my memory?

#2 2008-02-27 21:40:50

Re: What is using my memory?

#3 2008-02-27 22:24:50

Re: What is using my memory?

#4 2008-02-28 14:48:05

Re: What is using my memory?

#5 2008-02-28 20:34:17

Re: What is using my memory?

#6 2008-02-28 20:53:00

Re: What is using my memory?

#7 2008-03-01 14:28:36

Re: What is using my memory?

#8 2008-03-01 18:15:50

Re: What is using my memory?

#9 2008-03-03 13:20:32

Re: What is using my memory?

Board footer