[WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

seth · 2021-08-03 14:01:20

I tested this on an nvidia system… I had hoped for or sw GL xephyr to sufficiently abstract this, but well.
GEM is i915 specific, it will probably not even happen w/ amdgpu.

Good news would be that this isn't a leak but on purpose.
Gooder news is that the userspace can actually inform the kernel whether to keep the buffer around, https://blog.ffwll.ch/2012/11/i915gem-c … art-3.html

Trilby · 2021-08-03 14:37:18

Awesome thanks. I just came here to post another follow up test that shows that the i915.gem ram usage is actually the lions share of the apparent/suspected leak, and the incremement in heap usage is consistent but trivial in comparison. In case anyone is curious I wrote this script to collect some data from my actual usecase with my browser "weaver":

#!/bin/sh

pid=$(pgrep WeaverServer)

sizeA=$(sed -n 's/VmSize:\s*\([0-9]*\).*/\1/p' /proc/$pid/status)
heapA=$(sed -n '/heap/{n;s/Size:\s*\([0-9]*\).*/\1/p;}' /proc/$pid/smaps)
i915A=$(sed -n '/i915.gem/{n;s/Size:\s*\([0-9]*\).*/\1/p;}' /proc/$pid/smaps | awk '//{sum+=$0;}END{print sum;}')

weaver bbs.archlinux.org //sleep/2 //target/1 //close //sleep/2

sizeB=$(sed -n 's/VmSize:\s*\([0-9]*\).*/\1/p' /proc/$pid/status)
heapB=$(sed -n '/heap/{n;s/Size:\s*\([0-9]*\).*/\1/p;}' /proc/$pid/smaps)
i915B=$(sed -n '/i915.gem/{n;s/Size:\s*\([0-9]*\).*/\1/p;}' /proc/$pid/smaps | awk '//{sum+=$0;}END{print sum;}')

# D=delta or change in value
sizeD=$((sizeB-sizeA))
heapD=$((heapB-heapA))
i915D=$((i915B-i915A))

printf "           Process Name: %s\n" $(cat /proc/$pid/comm)
printf "           Initial Size: %8d KB\n" $sizeA
printf "           Initial Heap: %8d KB\n" $heapA
printf "Initial i915.gem memory: %8d KB\n" $i915A
printf "\n"
printf "               New Size: %8d KB\n" $sizeB
printf "               New Heap: %8d KB\n" $heapB
printf "    New i915.gem memory: %8d KB\n" $i915B
printf "\n"
printf "             Delta Size: %8d KB\n" $sizeD
printf "             Delta Heap: %8d KB  %2d%%\n" $heapD $((100*heapD/sizeD))
printf "             Delta i915: %8d KB  %2d%%\n" $i915D $((100*i915D/sizeD))

A fairly representative output:

           Process Name: WeaverServer
           Initial Size:  3153000 KB
           Initial Heap:    12708 KB
Initial i915.gem memory:   110984 KB

               New Size:  3218384 KB
               New Heap:    14896 KB
    New i915.gem memory:   149592 KB

             Delta Size:    65384 KB
             Delta Heap:     2188 KB   3%
             Delta i915:    38608 KB  59%

The "Delta Heap" or the change in heap use after the opening and closing of one web page, is about 3% of the total increment in memory use. Meanwhile the change in i915.gem usage accoutns for ~60%. There are apparently yet other consumers of ram that are not releasing it. But given the link above I suspect they are all related to this ~~bug~~ feature of the intel driver. I'll dig into that link and try to confirm this momentarily.

EDIT: ok, I read the link a couple times, but it's a bit over my head and I'm not seeing hints on how I can trigger i915 to free these cached buffers (or whatever they are). Whether I should regularly do this or not is a fair question, but trying it out would allow me to test/confirm that this is the true / only source of what looks like a memory leak.

EDIT 2: even if intentional, it is not viable as the buffers are never released even as memory runs out. VmSize of my application went over 12GB and it pretty much froze up the entire system presumably due to these buffers being swapped about (as this system only has 8G ram). Interestingly, it seems there may be a per-process limit of actual ram use from this GEM, because even at a VmSize over 12GB, the process wasn't using all that much of the physical ram (most of which remained available). So this will not result in a system-wide OOM situation, but the swapping of maintained buffers just kills the long-running browser process. The browser will need a way to drop old GEM buffers. This is very much in line with complaints on bug trackers of other qt5webengine browsers: the system ram may not be fully consumed, but the browser itself starts seizing up afer a fair bit of uptime.

Last edited by Trilby (2021-08-03 15:04:56)

seth · 2021-08-03 19:19:31

I can trigger

You can't - userspace here is somwhere in the chromium code when allocating the buffer.
However:
https://github.com/swaywm/sway/issues/5 … -602203168

nb. that you might be using the iris driver instead of i965 (incidentally maybe a meaningful parameter, https://wiki.archlinux.org/title/Intel_ … ver_(i965) )

Trilby · 2021-08-03 19:40:13

The drirc setting doesn't have any effect at all. In contrast MESA_LOADER_DRIVER_OVERRIDE=i965 in contrast has a huge effect ... a bad one. With that environment variable set, the test code uses 11 times as much heap space with a single pass through the loop, over 20x for 10 times, and over 30x as much for 100 times through the loop. (That's 30x more than 100 times through the loop without that variable set).

So it sounds like this could then be considered a bug in the chromium code then right? Maintaining cached buffers while there is ample memory can be called a feature, but maintaining an unneeded cache at the expense of the running program being starved out of memory is certainly a problem.

Last edited by Trilby (2021-08-03 19:41:12)

seth · 2021-08-03 19:52:45

"Opportunistic ressource management"

Did you try the drirc w/ w/o or either the MESA_LOADER_DRIVER_OVERRIDE=i965 ? (If you didn't adapt the driver there, it will only apply w/ MESA_LOADER_DRIVER_OVERRIDE=i965)

Trilby · 2021-08-03 20:17:50

I tested with each one individually before. But I just tried again with both the environment variable and ~/.drirc - the results appeared the same as with just the evironment variable (i.e., very bad).

But thanks to your insights, I've broadened my search to issues with chromium / intel. I've found that qt5webengine-based applications respond to chromium command line flags (of which there are *many*). I've tried with --aggressive-cache-discard and it made a notable difference significantly reducing the i915.gem memory usage, but the total heap usage increased by a slightly larger margin. But that at least means there's potential for other chromium flags to have a useful effect - so I'm going through some others by trial and error.

AmritaBithi · 2023-07-27 22:14:03

I wonder if this line within the code to do garbage collection on closures be the source of the memory leak?

https://github.com/ImageMagick/glib/blo … ure.c#L727

At the top of that file is some information which I have pasted below, maybe it can help to track down this bug that I am experiencing as well.

/**
* SECTION:gclosure
* @short_description: Functions as first-class objects
* @title: Closures
*
* A #GClosure represents a callback supplied by the programmer. It
* will generally comprise a function of some kind and a marshaller
* used to call it. It is the responsibility of the marshaller to
* convert the arguments for the invocation from #GValues into
* a suitable form, perform the callback on the converted arguments,
* and transform the return value back into a #GValue.
*
* In the case of C programs, a closure usually just holds a pointer
* to a function and maybe a data argument, and the marshaller
* converts between #GValue and native C types. The GObject
* library provides the #GCClosure type for this purpose. Bindings for
* other languages need marshallers which convert between #GValues
* and suitable representations in the runtime of the language in
* order to use functions written in that language as callbacks. Use
* g_closure_set_marshal() to set the marshaller on such a custom
* closure implementation.
*
* Within GObject, closures play an important role in the
* implementation of signals. When a signal is registered, the
* @c_marshaller argument to g_signal_new() specifies the default C
* marshaller for any closure which is connected to this
* signal. GObject provides a number of C marshallers for this
* purpose, see the g_cclosure_marshal_*() functions. Additional C
* marshallers can be generated with the [glib-genmarshal][glib-genmarshal]
* utility. Closures can be explicitly connected to signals with
* g_signal_connect_closure(), but it usually more convenient to let
* GObject create a closure automatically by using one of the
* g_signal_connect_*() functions which take a callback function/user
* data pair.
*
* Using closures has a number of important advantages over a simple
* callback function/data pointer combination:
*
* - Closures allow the callee to get the types of the callback parameters,
* which means that language bindings don't have to write individual glue
* for each callback type.
*
* - The reference counting of #GClosure makes it easy to handle reentrancy
* right; if a callback is removed while it is being invoked, the closure
* and its parameters won't be freed until the invocation finishes.
*
* - g_closure_invalidate() and invalidation notifiers allow callbacks to be
* automatically removed when the objects they point to go away.
*/

Trilby · 2023-07-27 22:44:51

I'm struggling to see any connection at all. Can you elborate? You linked to a line in a a test program for glib (i.e., code that isn't even included in the final package, right) and not at all specific to chromium/webengine.

Arch Linux

#26 2021-08-03 14:01:20

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

#27 2021-08-03 14:37:18

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

#28 2021-08-03 19:19:31

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

#29 2021-08-03 19:40:13

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

#30 2021-08-03 19:52:45

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

#31 2021-08-03 20:17:50

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

#32 2023-07-27 22:14:03

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

#33 2023-07-27 22:44:51

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

Board footer