Excessive memory use with ZFS [SOLVED sort of]

graysky · 2013-10-18 21:11:40

I have a server w/ 8G of RAM. When using ZFS, I'm finding used memory hovering around the 7G mark. My understanding is that ZFS on Solaris anyway, should not use this much[1]. Perhaps ZFS on Linux is different?

% cat /proc/spl/kstat/zfs/arcstats | grep c_m
c_min                           4    523521280
c_max                           4    4188170240

As I understand it, those values are in bits but convert to GB they are 0.48 - 3.9. Any suggestions are welcomed.

1. http://docs.oracle.com/cd/E26502_01/htm … zfs-3.html

Last edited by graysky (2013-10-26 11:08:45)

xxxspuddy · 2013-10-25 22:13:22

I've never got round to trying out ZFS on Linux, but on Solaris it defaults to use all but the last 1GB of memory if you have over 4GB installed, but it does give it back as required (there is a small performance hit though). You can tune this by adjusting the zfs_arc_min and zfs_arc_max values. The gentoo wiki has some information on how to do this

http://wiki.gentoo.org/wiki/ZFS

Last edited by xxxspuddy (2013-10-25 22:14:22)

fukawi2 · 2013-10-25 23:19:10

Also, AFAIK the Linux implementation isn't focusing on performance too much at the moment, so that probably includes memory optimisation.

graysky · 2013-10-26 11:08:26

I am able to limit it back to 4-5 G used of 8 G total with the following:

% cat /etc/modprobe.d/zfs.conf 
options zfs zfs_arc_max=2147483648

kokoko3k · 2013-10-26 11:23:35

Doesn't the same happens with Ext* ?

The common answer to "linux is eating my ram" was always "this is good; why wasting unused ram?"

Why is this different now?

Last edited by kokoko3k (2013-10-26 11:24:03)

graysky · 2013-10-26 11:31:28

No, you misunderstand. Read up on ZFS.

kokoko3k · 2013-10-26 19:14:10

Well, actually i did and still can't understand why one have to limit ZFS ARC and not the linux PageCache.
In the end the difference seems to be just the algorithm used to choose what to keep in the cache itself:

ZFS - Gentoo Wiki wrote:

ZFSOnLinux uses ARC page replacement algorithm instead of the Last Recently Used page replacement algorithm used by other filesystems. This has a better hit rate (and therefore provides better performance. The implementation of ARC in ZFS differs from the original paper in that the amount of memory used as cache can vary. This permits memory used by ARC to be reclaimed when the system is under memory pressure (via the kernel's shrinker mechanism) and grow when the system has memory to spare. The minimum and maximum amount of memory allocated to ARC varies based on your system memory. The default minimum is 1/32 of all memory, or 64MB, whichever is more. The default maximum is the larger of 1/2 of system memory or 64MB.
The manner in which Linux accounts for memory used by ARC differs from memory used by the page cache. Specifically, memory used by ARC is included under "used" rather than "cached" in the output used by the `free` program. This in no way prevents the memory from being released when the system is low on memory. However, it can give the impression that ARC (and by extension ZFS) will use all of system memory if given the opportunity.

kernelOfTruth · 2014-01-14 00:36:34

if I remember correctly from reading bug entries, code and commits on github; ZfsOnLinux doesn't use the Linux PageCache

thus ARC is the tuning knob

Last edited by kernelOfTruth (2014-01-14 00:36:56)

Arch Linux

#1 2013-10-18 21:11:40

Excessive memory use with ZFS [SOLVED sort of]

#2 2013-10-25 22:13:22

Re: Excessive memory use with ZFS [SOLVED sort of]

#3 2013-10-25 23:19:10

Re: Excessive memory use with ZFS [SOLVED sort of]

#4 2013-10-26 11:08:26

Re: Excessive memory use with ZFS [SOLVED sort of]

#5 2013-10-26 11:23:35

Re: Excessive memory use with ZFS [SOLVED sort of]

#6 2013-10-26 11:31:28

Re: Excessive memory use with ZFS [SOLVED sort of]

#7 2013-10-26 19:14:10

Re: Excessive memory use with ZFS [SOLVED sort of]

#8 2014-01-14 00:36:34

Re: Excessive memory use with ZFS [SOLVED sort of]

Board footer