On what my memory is spent on?

Kirurgs · 2010-03-12 08:45:03

Hi!

I have strange memory issues on my system. It's x86_64 2.6.32.9 + TuxOnIce patch, 4Gb of ram. I have multilib system and have opened couple of programs in 32 bit mode as well.
Previously I was using x86 system with 2Gb of RAM and now to my surprise on x86_64 4Gb I'm not able to open any more apps than on 32bit with 2Gb. Before decision to use 64bit system I searched forums, LKML etc. and the conclusion was that memory is not used like 2Gb 32bit => 4Gb 64bit = the same amount of apps opened, somehow I came to conclusion that 3Gb 32bit = 4Gb 64 bit, which is - I still have to be able to open more apps on 64 bit with 4Gb than on 2Gb on 32 bit.
It seems I'm missing something here.

See, top header reports the following:

Tasks: 264 total,   2 running, 262 sleeping,   0 stopped,   0 zombie
Cpu(s): 25.0%us,  5.7%sy,  0.0%ni, 68.4%id,  0.8%wa,  0.2%hi,  0.0%si,  0.0%st
Mem:   3949016k total,  3851300k used,    97716k free,    13324k buffers
Swap:  6834200k total,  1796308k used,  5037892k free,  1504244k cached

It says all the memory is gone and we are using swap already for 1.7G.
Gnome system monitor says we are using 2.2G memory (of 3.8G) + 1.7G swap. Poof, 1.6Gb goes to 1.5Gb cache + 100M free mem. I understand that Gnome sysmon might not be the best measure but shows how many memory I'm using for my tasks.

/proc/meminfo says this:

MemTotal:        3949016 kB
MemFree:          118284 kB
Buffers:           10092 kB
Cached:          1496740 kB
SwapCached:       266556 kB
Active:          2551328 kB
Inactive:        1039176 kB
Active(anon):    2441116 kB
Inactive(anon):   933580 kB
Active(file):     110212 kB
Inactive(file):   105596 kB
Unevictable:          72 kB
Mlocked:              72 kB
SwapTotal:       6834200 kB
SwapFree:        5044588 kB
Dirty:               152 kB
Writeback:             0 kB
AnonPages:       1898444 kB
Mapped:           322184 kB
Shmem:           1291020 kB
Slab:             124320 kB
SReclaimable:      36400 kB
SUnreclaim:        87920 kB
KernelStack:        3968 kB
PageTables:        57584 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8808708 kB
Committed_AS:    7269552 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      352404 kB
VmallocChunk:   34359309500 kB
HardwareCorrupted:     0 kB
DirectMap4k:        8192 kB
DirectMap2M:     4081664 kB

And it all comes down to for apps I'm using 2.2G ram, the 1.5G goes to caching and the rest goes somewhere. So, on desktop system I don't want cache in size of half of my RAM. I need memory for apps.
Maybe, I just don't know, cached includes all the libs loaded. But it shouldn't because every program has it's shared memory shown and residential memory, which means memory for just that app. Shared includes libraries and stuff.
I set swappiness to 30 and vfs_cache_pressure to 150, but that doesn't help at all. I didn't expect it to help as they ar just tendencies
Once system reaches 2.3-2.4Gb RAM usage for apps it starts swapping, not that it could shrink caching part, no no - it's swapping, this is just killing me.

I have already searched a lot of places for help, but still there is no definite answer for memory usage, probably I don't understand how linux uses memory.
Can anyone help me understand where is the issue about caching being so big and apps using just half of my mem?
How the memory is spent, where it goes?
Is there anything I can do to improve usage for RAM for apps?

regards
Kirurgs

Last edited by Kirurgs (2010-03-12 08:53:18)

knopwob · 2010-03-12 08:56:45

http://chrisjohnston.org/2009/why-on-li … -ram-usage

In short: Linux frees memory when there is more needed. Because it caches often used files from the harddrive, because the harddrive is MUCH slower. You see in your top header, that 1,5GB are cached. That is used memory, that can be freed if needed, but the other time, it is used to speed up harddrive access and so on.

Read the blog post,i linked, it explains it much better than i could do.

Kirurgs · 2010-03-12 09:23:58

Thanx for the link, but that article is explaining why it seems that there is no free memory while it really is. My situation is different, I have no free memory at all and caching is still high while article says a bit different. That is the thing I said above, I have searched a lot of places and there is no single truth how mem is used.

It seems it prefers to swap all apps out of mem just to keep that cache at 1.5G. What's the logic when there is no more free mem at all...

In 32 bit system I had about 1.8G mem usage, caching was not playing that big role at all and that system was more usable or at the same level than now x86_64 with 4 gigs.
Of course when system is swapping it is slow on both arches, but usage pattern is completely different.

Last edited by Kirurgs (2010-03-12 09:26:13)

KimTjik · 2010-03-12 09:31:39

It looks strange. I don't understand why even SWAP in this case would be used. What activities are said to be used all that space?

Kirurgs · 2010-03-12 10:32:52

Well, I have opened native 64 bit apps, majority, like 70%, I also use wine for couple of apps (32 bit), I use couple native linux apps in 32 bit mode. There are some java apps running as well.
Normal everyday work, a lot of apps open

KimTjik · 2010-03-12 10:57:17

Still it makes no sense to me. 4GB + 1.7GB would demand have RAM use for virtual machines, photo or video editing. Running wine + some applications don't sky rocket memory use. Couldn't you post some lines from top showing the biggest memory hogs? Otherwise we'll just walk about in obscurity.

keenerd · 2010-03-12 13:19:37

http://www.pixelbeat.org/scripts/ps_mem.py

Run that script as root. Great for breaking down ram use.

Kirurgs · 2010-03-12 14:51:14

Ok, here it is, but this post is not about what eats memory, I know I have a lot of apps open, it's about why it uses it in such weird fashion.
Where the memory goes NOT in terms of how many apps are open but how that memory is used for those apps, dependant libraries or caching etc. For instance 1.5G caching - why so much, can it be shrink somehow?

Command: ps -aux --sort=-rss, I took some number of first top rows.

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
user     25466  5.7  9.7 1081388 385508 ?      Sl   Mar11 134:09 /data/.applications/Mozilla/x86_64/firefox-3.7/firefox-bin --no-remote -P
user     28857  0.5  9.7 1269652 383072 ?      Sl   11:13   1:39 /opt/java/bin/java -Xmx640M -Xms128M -Xverify:none -Doracle.ide.util.AddinPolicyUtils.OVERRIDE_FLAG=true -Dsun.java2d
user     17312  0.1  8.0 1517512 318264 ?      Sl   Mar10   3:10 java -jar sqldeveloper.jar
user      9617  0.7  5.2 1132772 208560 ?      Sl   Mar10  25:48 /usr/lib/openoffice/program/soffice.bin -writer /file
user     20111  1.6  4.1 1191292 163532 ?      Sl   Mar10  47:33 firefox --no-remote -P
root      4368  3.8  2.5 261200 101848 tty7    Ss+  Mar09 154:19 /usr/bin/Xorg :0 -br -verbose -auth /var/run/gdm/auth-for-gdm-i9EXmb/database -nolisten tcp
user      9488  0.2  2.4 568088 97288 ?        Sl   Mar10   9:24 /data/.applications/Mozilla/x86_64/firefox-3.0/firefox-bin --no-remote -P
user     28206  1.4  2.1 483444 85592 ?        Sl   11:06   4:44 mono /usr/lib/gnome-do/Do.exe
user      9101  0.2  2.0 520304 81860 ?        Sl   Mar10   8:59 /opt/ibm/lotus/notes/framework/rcp/eclipse/plugins/com.ibm.rcp.base_6.2.1.20090925-1604/linux/x86/notes2 --launcher.s
ora11gr1 29465  0.0  2.0 1082176 81832 ?       Ss   11:31   0:08 oracleXX (LOCAL=NO)
ora11gr1 31218  0.0  1.7 1060676 67572 ?       Ss   12:14   0:01 oracleXX (LOCAL=NO)
ora11gr1 28757  0.0  1.6 1062692 64648 ?       Ss   11:10   0:01 oracleXX (LOCAL=NO)
ora11gr1 19180  0.0  1.5 1056352 62904 ?       Ss   Mar10   0:08 ora_smon_XX
ora11gr1 30015  0.0  1.5 1061648 60048 ?       Ss   11:43   0:01 oracleXX (LOCAL=NO)
user      9087  0.5  1.5 336884 59288 ?        Sl   Mar10  18:13 /opt/ibm/lotus/notes/framework/../notes /authenticate

keenerd - thanx, good script actually.

Script mentioned in keenerd's post reports 2.5Gb use in total. I see the same amount in Gnome sysmon. And if I calculate correctly I could get that number from /proc/meminfo as well.
BUT, why only 2.5Gb, where does 1.5G physical ram goes and 2.4G swap? These are current values now.
Something is really not right, but what.

Last edited by Kirurgs (2010-03-12 15:02:50)

anonymous_user · 2010-03-12 17:51:10

Afaik won't Linux decrease cache when an app needs memory? Also you can tweak it so Linux will always try to use RAM first:

http://wiki.archlinux.org/index.php/Max … Swappiness

ddaedalus · 2010-03-12 18:06:15

Rather than asking yourself where your memory is spent, which is a _hard_ question since memory usage is not a tree but a graph, you should be more concerned about hard page faults (you can call it swap flux).
Hard page faults describe how many pages are written to and read from swap (among other things). This is afaik the only metric which matters when measuring system performance. vmstat 1, columns si & so are your friends.

Last edited by ddaedalus (2010-03-12 18:06:34)

Kirurgs · 2010-03-12 22:46:49

Well, I'm askin myself where does memory goes, I don't care about hard page faults, because when I am over my physical RAM limits I know that and I'm not expecting any nice performance at all.
What I'm trying to understand is simple as pie: I have memory usage of 2.5Gb for applications, system reports that I also have used 2.4Gb of swap. In total I have 4Gb of physical memory. So where does those 4 - 2.5 + 2.4 => 1.5 + 2.4 => 3.9Gb are spent, on what else than my stupid applications? Why caching is 1.5G not 500M or 200 or 162Mb? Where those immense amounts of memory are spent if apps are using just 2.5Gb?
It seems it is very hard question and I need to understand what is the reason for this usage...

I'm starting to think I'll try to post in LKML maybe

lagagnon · 2010-03-12 23:31:09

What is not shown in the top, free, ps aux commands is the memory consumed by common libraries used by the numerous applications you have open. I suspect it is the libraries that are consuming all that RAM. And you do have some heavy use apps open : ff, java, oracle, notes etc etc. All these will be using system libraries...

ddaedalus · 2010-03-13 10:31:34

Kirurgs wrote:

Well, I'm askin myself where does memory goes, I don't care about hard page faults, because when I am over my physical RAM limits I know that and I'm not expecting any nice performance at all.
What I'm trying to understand is simple as pie: I have memory usage of 2.5Gb for applications, system reports that I also have used 2.4Gb of swap. In total I have 4Gb of physical memory. So where does those 4 - 2.5 + 2.4 => 1.5 + 2.4 => 3.9Gb are spent, on what else than my stupid applications? Why caching is 1.5G not 500M or 200 or 162Mb? Where those immense amounts of memory are spent if apps are using just 2.5Gb?
It seems it is very hard question and I need to understand what is the reason for this usage...
I'm starting to think I'll try to post in LKML maybe

To my understanding you want to know exactly how your memory is used. Is this correct?

If so look at the following files /proc/*/maps and /proc/*/smaps. Man page here: http://linux.die.net/man/5/proc.

I am not sure, but i think you will hardly find more detailed stats than that. Someone correct me if i am wrong.

Arch Linux

#1 2010-03-12 08:45:03

On what my memory is spent on?

#2 2010-03-12 08:56:45

Re: On what my memory is spent on?

#3 2010-03-12 09:23:58

Re: On what my memory is spent on?

#4 2010-03-12 09:31:39

Re: On what my memory is spent on?

#5 2010-03-12 10:32:52

Re: On what my memory is spent on?

#6 2010-03-12 10:57:17

Re: On what my memory is spent on?

#7 2010-03-12 13:19:37

Re: On what my memory is spent on?

#8 2010-03-12 14:51:14

Re: On what my memory is spent on?

#9 2010-03-12 17:51:10

Re: On what my memory is spent on?

#10 2010-03-12 18:06:15

Re: On what my memory is spent on?

#11 2010-03-12 22:46:49

Re: On what my memory is spent on?

#12 2010-03-12 23:31:09

Re: On what my memory is spent on?

#13 2010-03-13 10:31:34

Re: On what my memory is spent on?

Board footer