You are not logged in.

#1 2021-07-30 03:56:04

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

[WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

WORKAROUND:

Thanks to Seth for the heroic efforts below.  However it appears this may be an issue between chromium code and intel drivers and far outside my ability to even create a bugreport for that will get any traction upstream.  I've resorted to using a workaround in my own code that restarts the program at opportune moments.  This is a ridiculously ugly hack, but it does cope with the ridiculously ugly bug.

--- original post below ---

If anyone could help by A) confirming and / or B) helping to trace an apparent memory leak in qt5-webengine I'd appreciate it as I've hit a dead end.  I started tracking this down when a project of mine seemed to have a slow but consistent memory leak.  With the following sample code I can confirm that it is not from my project's code, but is infact from qt5 webengine itself. (good news for me, I didn't break it; bad news for me, that means I likely can't fix it).

The minimal example below demonstrates the problem.  The code creates only one object on the heap, and deletes it again before entering the main loop (where it stays indefinitely due to the call to setQuitOnLastWindowClosed(false)).  The program will need to be killed with Ctrl-C.  But while it's still running, you can confirm it's memory use with `pmap`.

qwebtest.cpp

// QWEBTEST.CPP

#include <QApplication>
#include <QWebEngineView>

int main(int argc, char *argv[]) {
	if (argc < 2) return -1;
	QApplication app(argc, argv);
	app.setApplicationName("WebEngineTest");
	app.setQuitOnLastWindowClosed(false);
	auto v = new QWebEngineView();
	v->setContent("");
	for (int i = 0; i < QString(argv[1]).toInt(); i++) {
		v->show();
		v->close();
	}
	delete v;
	return app.exec();
}

This can be compiled as follows:

g++ -o qwebtest qwebtest.cpp $(pkgconf --libs --cflags Qt5WebEngine Qt5WebEngineWidgets) -fPIC

The for loop in the code simply "shows" and "closes" the window a number of times specified by argv[1] (so pass a single integer on the command line).  With this, you can compare the memory use from runs of this program with a single time through the loop or several times through the loop.

Specifically, to see the heap allocations:

pmap $(pgrep qwebtest) | grep heap

The difference is substantial, and highly repeatable.

I confirmed that this was specific to QWebEngineView by replacing it first with QWindow with the code otherwise the same (but also removing the setContent line which doesn't apply to QWindow).  With QWindow, the heap use was identical regardless of how many times through the loop.  I also confirmed there was no such issue with QWidget (which is the base class of QWebEngineView).

This led me to wonder what QWebEngineView was doing differently in it's response to show / close.  The source code for that module is here:

https://code.woboq.org/qt5/qtwebengine/ … w.cpp.html

QWebEngineView does not redefine show or close, but only inherits those functions from QWidget.  It does, however, have it's own showEvent and closeEvent handlers which - in addition to calling the equivalent base class handler, also call the QWebEnginePage function setVisible(true/false).

So I tried again with a subclass of QWebEngineView in which I redefined those event handlers to *only* call the QWidget handlers and not call setVisible on the QWebEnginePage object.  But this didn't seem to change anything: each show/close loop still consumed more memory.

At that point I was stumped.  I don't see how the QWebEngineView object could be doing anything different from QWidget other than those two event handlers - yet the QWebEngineView is leaking memory while a QWidget does not.

Any help would be appreciated - at the simplest, confirmation of the problem would be good by checking heap use after different numbers of times through the loop in the above sample code.

Last edited by Trilby (2021-08-08 21:26:12)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#2 2021-07-30 06:12:58

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

QAccessible::installFactory(&webAccessibleFactory);

?

Offline

#3 2021-07-30 12:56:53

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

Thanks for chiming in.  That line piqued my curiosity too - but I've not yet dug through the QAccessible code nor found an efficient way to prevent that line from running (short of rebuilding qt5webengine) to test if that's the culprit.  I'm skeptical, though, that the issue could be with QAccessible itself due to it's wider use, but perhaps just the webAccessibleFactory.

Although it appears building qt5-webengine with -DQT_NO_ACCESSIBILITY should prevent that code from being included which should provide a test of that ... if I can build it in a reasonable time on my hardware.  I'll soon find out.

EDIT: After reading a bit about QAccessibility, it does seem odd that the QWebEngineView passes a heap allocated object but does not maintain any reference to it.  According to the docs, there is the QAccessibility::installFactory, and a QAccessibility::removeFactory, each of which would be passed a pointer to an object.  This suggests that it is expected the caller maintain a reference to the object and is likely responsible for deleting it, which QWebEngineView most likely can't do (due to not maintaining a pointer to it).

Last edited by Trilby (2021-07-30 13:09:42)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#4 2021-07-30 13:25:32

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

QAccessibility::installFactory takes a function pointer (the static function right above the call in qwebengineview.cpp) but the factory will create new  QWebEngineViewAccessible objects on the heap. They're qobject children of the parenting QWebEngineView
So the interesting part would be to look at the QWebEngineView and it's qobject children - in particular whether it grows a list of QWebEngineViewAccessible

Offline

#5 2021-07-30 13:35:59

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

seth wrote:

So the interesting part would be to look at the QWebEngineView and it's qobject children - in particular whether it grows a list of QWebEngineViewAccessible

I checked that early on in tracing this, and there is no increase in the list count of any of the qt5 maintained lists that I am aware of.  I've checked the QWebEngineView's instance's children() as well as it's page()->children() and the QApplication lists of windows(), widgets(), and children().  None of these vary based on the number of times through the loop.


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#6 2021-07-30 13:59:41

seth
Member
Registered: 2012-09-03
Posts: 59,158

Offline

#7 2021-07-30 14:49:25

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

Tested. There's no memory lost on the heap at all. I also ran on Xephyr to rule out nvidia / mesa GL.
But I need to update this system (qt5-webengine 5.15.3-5, it's because I'm lazy as fuck) - I'll try to get myself there over the weekend.

Offline

#8 2021-07-30 14:59:41

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

seth wrote:

Tested. There's no memory lost on the heap at all.

Was this with the pmap command, or were you using other tools like massif (which I've not yet tried)?

I doubt the qt5webengine version matters that much as I've been trying to trace this issue for quite some time - almost certainly since that version was in the repos.

Note: I don't know how good of a tool pmap is for this.  I consider myself a competent coder and pretty good general trouble-shooter, but I have no knowledge of or experience with real debugging and profiling tools.  But pmap was easy to get a check on memory use and it allowed me to detect and narrow down sympts I was seeing.

Last edited by Trilby (2021-07-30 15:03:44)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#9 2021-07-30 15:07:58

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

pmap and the very nifty https://gitlab.com/thiagomacieira/rssanalyse
Do you get the same issue in Xephyr or w/ "export LIBGL_ALWAYS_SOFTWARE=true" ?
(massif shows quite some heap allocation through /usr/lib/libGLX.so)

Offline

#10 2021-07-30 18:21:09

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

`export LIBGL_ALWAYS_SOFTWARE=true` doesn't make any notable change, there is still an apparent memory leak.  Running under Xephyr changed the numbers a bit showing a smaller increment in memory use between 1 and 10 times through the loop, but there was still a consistent / repeatable increase in heap use with more times through the loop under Xephyr.

I'd gather this would suggest that the same problem exists, but whatever is "leaking" just takes less heap space when run under Xephyr than when run under the native Wayland session I'm running.  I've done similar tests under X11 and see symptoms of an apparent leak there too, but I've not compared numbers side by side yet.  Edit: now I have.

I tested under X11 and Wayland and Xephyr under each of these.  The magnitude of the leak varied widely (except being identical between Xephyr regardless of the parent display protocol).  I also saw that under X11 or Xephyr it was not apparent until the number of loops got a bit higher.  Here's the data for runs under each condition with loop counts of 1, 10, and 100.  The values are in KB of heap use as reported by pmap.  Most of these were replicated a couple times to see that values were pretty consistent (perfectly consistent under xephyr):

N-Loops:              1      10     100   Ea.
X11:               7404    7804   11056    36
X11->Xephyr:      17180   17176   22384   ~57
Wayland:           8276   13784   71440   640
Wayland->Xephyr:  17180   17176   22384   ~57

The per-loop leak stands out under wayland, but is present in all conditions.  The exception being that it is not apparent in Xephyr between 1 and 10 loops.  But the overall calculated per-loop leak should make the difference between 1 and 10 loops sufficiently larger than any rounding error (due to using KB here).

Last edited by Trilby (2021-07-30 18:49:09)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#11 2021-07-30 20:52:12

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

Race condition?
Since the eventloop is not running while you're showcloselooping, it should… only be for multithreading…
What if you "usleep(10000);" after show and close?

Offline

#12 2021-07-30 21:12:02

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

I'm not sure I follow, but I tested with a usleep (either before or after and both before and after the `delete v`) and there was no difference.

Though that gave me another idea of adding a much longer sleep and remove the app.exec() call alltogether, and the same leak is seen.

So you really don't see any memory leak even with hundreds of times through the loop?

EDIT: I reread your post, and if it matters I ran it again with the sleeps within the loop after each show() and close(), but still had the same apparent leak.

Last edited by Trilby (2021-07-30 21:14:07)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#13 2021-07-30 21:35:02

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,863
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)


https://ugjka.net
paru > yay | vesktop > discord
pacman -S spotify-launcher
mount /dev/disk/by-...

Offline

#14 2021-07-30 21:40:40

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

Yes, that is likely the same thing.  Qt5 webengine's bug tracker is absolutely overflowing with reports of memory leaks or memory management issues.  Most of the reports are closed as "wont fix" or there's some odd attempt to explain them away and ignore them.  To be fair, I'm not aware of any good quality specific bug reports on it.

So I'm hoping if we can actually pin down the root cause and offer a solution, it might get a bit more traction upstream.

I suspect most qtwebengine projects are not as concerned as a lot of users will occasionally close all their browser windows which results in a full shut down releasing the memory.  Or perhaps they - like me - assumed the problem was in our own code rather than checking the webengine lib.

In my case, I'm working on my "weaver" browser which runs indefinitely even when all the windows are closed.  A backup plan (which is more a plan F than a plan B) is to actually restart or self-exec the weaver code every time all windows are closed - but this is a really ugly workaround for a problem that should be fixed upstream.

I just joined in the conversation on the Falkon bug tracker - perhaps we can converge on a good solution with more (angry) hackers involved.

Last edited by Trilby (2021-07-30 21:47:57)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#15 2021-07-30 22:23:58

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

I'm tinkering with massif - but I can't make much sense of what I'm looking at yet.  Though I see the following backtrace-style series several times in the massif output from runs that go multiple times through the loop:

        n1: 3357904 0x21669447: ??? (in /usr/lib/qt/plugins/wayland-graphics-integration-client/libqt-plugin-wayland-egl.so)
         n1: 3357904 0x2149315D: QtWaylandClient::QWaylandIntegration::createPlatformOpenGLContext(QOpenGLContext*) const (in /usr/lib/libQt5WaylandClient.so.5.15.2)
          n2: 3357904 0xDC9C02D: QOpenGLContext::create() (in /usr/lib/libQt5Gui.so.5.15.2)
           n2: 2442112 0xEA87616: ??? (in /usr/lib/libQt5QuickWidgets.so.5.15.2)
            n1: 2136848 0xEA87E41: QQuickWidget::showEvent(QShowEvent*) (in /usr/lib/libQt5QuickWidgets.so.5.15.2)
             n1: 2136848 0x4B192E4: ??? (in /usr/lib/libQt5WebEngineWidgets.so.5.15.5)
              n1: 2136848 0xD6050BD: QWidget::event(QEvent*) (in /usr/lib/libQt5Widgets.so.5.15.2)
               n1: 2136848 0xD5C1D61: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib/libQt5Widgets.so.5.15.2)
                n1: 2136848 0xE4AF3A9: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib/libQt5Core.so.5.15.2)
                 n1: 2136848 0xD6018A7: QWidgetPrivate::show_helper() (in /usr/lib/libQt5Widgets.so.5.15.2)
                  n1: 2136848 0xD604BBC: QWidgetPrivate::setVisible(bool) (in /usr/lib/libQt5Widgets.so.5.15.2)
                   n1: 2136848 0x5585BE5: QtWebEngineCore::WebContentsAdapter::undiscard() (in /usr/lib/libQt5WebEngineCore.so.5.15.5)
                    n1: 2136848 0x5585CF4: QtWebEngineCore::WebContentsAdapter::setLifecycleState(QtWebEngineCore::WebContentsAdapterClient::LifecycleState) (in /usr/lib/libQt5WebEngineCore.so.5.15.5)
                     n1: 2136848 0x5585ED2: QtWebEngineCore::WebContentsAdapter::setVisible(bool) (in /usr/lib/libQt5WebEngineCore.so.5.15.5)
                      n1: 2136848 0xD6050BD: QWidget::event(QEvent*) (in /usr/lib/libQt5Widgets.so.5.15.2)
                       n1: 2136848 0xD5C1D61: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib/libQt5Widgets.so.5.15.2)
                        n1: 2136848 0xE4AF3A9: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib/libQt5Core.so.5.15.2)
                         n1: 2136848 0xD6018A7: QWidgetPrivate::show_helper() (in /usr/lib/libQt5Widgets.so.5.15.2)
                          n1: 2136848 0xD604BBC: QWidgetPrivate::setVisible(bool) (in /usr/lib/libQt5Widgets.so.5.15.2)
                           n0: 2136848 0x10925E: main (qwebtest.cpp:16)

This is the series of calls coming from the show / setVisible.  I gather my attempt to override the showEvent handler to prevent the setVisible did not change much because QWidget(Private) also has it's own setVisible.  As seen in the trace above, the series includes three calls back to QWebEngine code but not in the web view or web page, but in QtWebEngineCore.  So those are two new candidates to check.

EDIT: I've ran massif-visualizer on the output which is pretty nifty.  That also makes it look like the out of control heap use is in QtWebEngineCore.

I've not dug into it yet, but for anyone else who wants to join the hunt, the upstream source for QtWebEngineCore::WebContentsAdapter is here:

https://code.woboq.org/qt5/qtwebengine/ … r.cpp.html

EDIT: setVisible and setLifecycleState appear begnin ... I think.  But the undiscard function may to be creating a new help-stored object through m_webContents->GetRenderViewHost() which I'm trying to trace down now.

Last edited by Trilby (2021-07-30 22:46:52)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#16 2021-07-31 12:50:13

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

        n1: 3357904 0x21669447: ??? (in /usr/lib/qt/plugins/wayland-graphics-integration-client/libqt-plugin-wayland-egl.so)
         n1: 3357904 0x2149315D: QtWaylandClient::QWaylandIntegration::createPlatformOpenGLContext(QOpenGLContext*) const (in /usr/lib/libQt5WaylandClient.so.5.15.2)
          n2: 3357904 0xDC9C02D: QOpenGLContext::create() (in /usr/lib/libQt5Gui.so.5.15.2)
           n2: 2442112 0xEA87616: ??? (in /usr/lib/libQt5QuickWidgets.so.5.15.2)
            n1: 2136848 0xEA87E41: QQuickWidget::showEvent(QShowEvent*) (in /usr/lib/libQt5QuickWidgets.so.5.15.2)
             n1: 2136848 0x4B192E4: ??? (in /usr/lib/libQt5WebEngineWidgets.so.5.15.5)
              n1: 2136848 0xD6050BD: QWidget::event(QEvent*) (in /usr/lib/libQt5Widgets.so.5.15.2)

undiscard() somehow™ (maybe flushes the event queue or there's an inline somewhere) sets some QWebEngineWidget vsible, this triggers QQuickWidget::showEvent
The anonymous function next up is very most likely d->createContext() https://code.woboq.org/qt5/qtdeclarativ … eContextEv

Does it leak if you

export QMLSCENE_DEVICE=software
export QT_QUICK_BACKEND=software

(which would hopefully then sidestep the context creation)

Offline

#17 2021-07-31 13:33:00

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

Those exports drastically reduce but do not eliminate the leak.  Under wayland the (calculated / estimated) per-loop leak is just over 40K.  I'll test under X11 in a bit.  Those exports actually seemed to increase the leak ever so slightly under X11 to values very close to those seen under Wayland with those exports to the point that I suspect with those exports whatever code path is eating heap space is shared.  But without those exports, X11 and Wayland each take different paths, the former leaking ever so slightly less than with the "software" exports, and the latter leaking much more.

Last edited by Trilby (2021-07-31 13:41:28)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#18 2021-07-31 14:29:44

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

What does memcheck report about the leak? (just run valgrind, it's the default tool)

Offline

#19 2021-07-31 15:36:22

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

It doesn't look like valgrind sees it.  Here's the leak summary for 1 loop and 10 loops each with the link to the full valgrind output directly below:

==548035== LEAK SUMMARY:
==548035==    definitely lost: 64 bytes in 1 blocks
==548035==    indirectly lost: 0 bytes in 0 blocks
==548035==      possibly lost: 1,648,181 bytes in 5,230 blocks
==548035==    still reachable: 2,527,355 bytes in 7,027 blocks
==548035==         suppressed: 0 bytes in 0 blocks

http://ix.io/3uAJ

==548134== LEAK SUMMARY:
==548134==    definitely lost: 64 bytes in 1 blocks
==548134==    indirectly lost: 0 bytes in 0 blocks
==548134==      possibly lost: 1,648,181 bytes in 5,230 blocks
==548134==    still reachable: 2,527,357 bytes in 7,027 blocks
==548134==         suppressed: 0 bytes in 0 blocks

http://ix.io/3uAK

I also repeated this with --trace-children=yes for both conditions of 1 and 10 loops and didn't see any notable difference.  I'm confused by this as if I'm reading this right valgrind isn't seeing (much of) a leak, and certainly isn't detecting anything different between the two conditions of 1 or 10 loops.  But `free` output shows a marked difference in memory consumption, and pmap shows the processes using much more heap space with 10 loops.

EDIT: oops!  Perhaps I don't understand valgrind output.  What I posted above is the LEAK SUMMARY from the start of it's output.  There's another similar block at the end of it's output that does report more of a leak and a notable difference in the two conditions, though these numbers still don't come close to accounting for the apparent leak based on pmap heap usage.

==548025== LEAK SUMMARY:
==548025==    definitely lost: 9,043 bytes in 291 blocks
==548025==    indirectly lost: 6,971 bytes in 84 blocks
==548025==      possibly lost: 1,325,636 bytes in 3,855 blocks
==548025==    still reachable: 91,029,841 bytes in 1,441,399 blocks
==548025==                       of which reachable via heuristic:
==548025==                         multipleinheritance: 72 bytes in 1 blocks
==548025==         suppressed: 1,704 bytes in 2 blocks
==548122== LEAK SUMMARY:
==548122==    definitely lost: 14,611 bytes in 345 blocks
==548122==    indirectly lost: 6,587 bytes in 84 blocks
==548122==      possibly lost: 1,342,996 bytes in 3,974 blocks
==548122==    still reachable: 91,040,832 bytes in 1,441,421 blocks
==548122==         suppressed: 1,704 bytes in 2 blocks
==548122== Rerun with --leak-check=full to see details of leaked memory

Rerunning these two valgrind tests with the above-listed 'export' settings change the raw numbers a little, gets rid of the 64 byte definite loss in the top LEAK SUMMARY in both conditions, but otherwise doesn't seem to have an impact (i,e, instead of 9K vs 14K definitely lost in the bottom summary I see 7K and 11K).

Last edited by Trilby (2021-07-31 15:47:56)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#20 2021-07-31 15:58:40

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

FWIW, a slight variation on the test code results in a much more obvious memory leak with the creation / deletion of the object in the loop:

	for (int i = 0; i < QString(argv[1]).toInt(); i++) {
		auto v = new QWebEngineView();
		v->setContent("");
		v->show();
		v->close();
		delete v;
	}

This is closer to the early version of the test I had and (a little) closer to the real world issue that I'm trying to fix.  But I found I still saw a leak with only the show/close in the loop, so I focused in on that.  But the above variation may make it more easy to replicate on other systems - though it also casts a much wider net of possible suspects (I'm starting to wonder if there aren't multiple sources of this apparent leak having a cumulative effect, or if it is a single source, the size of the effect seems to vary widely based on the context).  I just tried on another system of mine, and the test code in my first post made it hard to see a leak until I got up to 1000 times through the loop.  But with the creation / deletion of the object in the loop, the leak was apparent at smaller loop counts.

Last edited by Trilby (2021-07-31 16:02:08)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#21 2021-07-31 21:07:02

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

That loop wouldn't (really) undiscard() anything, though - you get a fresh view & page w/ every cycle and show and close and delete it.
Do you get the leak while the eventloop is running?

#include <unistd.h>
#include <QApplication>
#include <QShowEvent>
#include <QTimer>
#include <QWebEngineView>
#include <QWebEnginePage>
#include <QWebEngineSettings>

int count = 0;
void functor() {
    auto v = new QWebEngineView();
    v->setContent("");
//    v->page()->setVisible(false);
    //v->page()->setLifecycleState(QWebEnginePage::LifecycleState::Discarded);
    printf("Starting %d cycles …\n", count);
    for (int i = 0; i < count; i++) {
        v->show();
        //v->hide();
        usleep(10000);
        v->close();
    }
    printf("Done!\n");
    delete v;
}

int main(int argc, char *argv[]) {
    if (argc < 2) return -1;
    count = QString(argv[1]).toInt();
    QApplication app(argc, argv);
    app.setApplicationName("WebEngineTest");
    app.setQuitOnLastWindowClosed(false);
    //functor();
    QTimer::singleShot(10, &functor);
    return app.exec();
}

Offline

#22 2021-07-31 21:59:40

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

seth wrote:

Do you get the leak while the eventloop is running?

Of course, or we wouldn't be here.  It's consistent with any repeated creation / deletion of webviews* which is where the investigation started and why every project based on qtwebengine has bug reports claiming that they gradually consume more and more memory as tabs/windows are opened / closed.  My testing code was just the result of making it as simple as I could to replicate the symptoms.  With your test code I see it as well with heap sizes for 1, 10, and 100 loops of  8M, 13M, and 72M.

*note: so long as there is sufficient action with it while it exists such as showing/closing (which in turn requires a url be loaded or setContent had been called).  I suspect the relevant threshold is just enough to trigger the background render process to do it's thing.

Last edited by Trilby (2021-07-31 22:31:16)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#23 2021-08-01 13:55:22

seth
Member
Registered: 2012-09-03
Posts: 59,158

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

Was still chasing the race condition sad

Is the close() actually relevant?
With the

auto v = new QWebEngineView();
v->setContent("");
v->show();
v->close();
delete v;

I suspect simply

auto v = new QWebEngineView();
v->setContent("");
v->show();
// v->close();
delete v;

Might cause the same leaks and the next thing to narrow down on is the relevance of "v->setContent("");" (ie. whether the leak comes from the web page)
Maybe even the explicit "show()" is useless.

hide/show toglles between discarded and active what causes reloads of the webpage, so likely simply loading a webpage leaks?

Offline

#24 2021-08-01 14:39:56

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

No the close is not necessary, the apparent leak still happens without it, though it's a *slightly* smaller per-loop incremement in heap usage without the close().  So explicitly closing the window does seem to excacerbate the issue a bit.

Without the show() a bulk of the heap usage increment is gone, but not all of it.

Without the setContent(), show(), or close(), so inside the loop there's just the 'new' and 'delete', then there is no increment in heap use at all regardless of the loop count.  With just the show() but no setContent() there is also no increment in heap usage - however, based on previous testing I'm pretty sure "show()" is effectively a no-op if there has not yet been a call to one of the functions that puts content on the view (setContent, load, setHtml, etc).  In the absence of these content setting functions, the show() will never actually show a window.

But this is all why I moved the creation of the view and the setContent call outside the loop prior to the start of this thread to test the memory leak in the absence of any new content being created or placed on the page.

For completeness, sake, with setContent inside the loop (along with the new and delete) but no show(), there is a small leak: 1 loop uses 5636K heap and 10 loops uses 6028K.  But with setContent and show() it's more substantial: 1 loop uses 8272K and 10 loops uses 20404K.

Last edited by Trilby (2021-08-01 14:44:56)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#25 2021-08-03 13:42:59

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,330
Website

Re: [WORKAROUND] memory leak in qt5-webengine (i915 GEM buffer issue?)

As I've been exploring other diagnostics, I've been looking through the /proc/*/smaps data for these processes.  It seems in addition to escalated heap use, there are also a large number of entries for the 'file' anon_inode:i915.gem which must be specific to this being an intel graphics system.  Each entry for anon_inode:i915.gem is listed as having a size from several to several dozen KB.  The number of anon_inode:i915.gem entries in /proc/<pid>/smaps correlates perfectly with the number of times through the loop (from 10-32 entries per loop depending on exactly which lines of code are inside the loop vs out).

Hoping for a parsimonious explanation, I suspect that there is a leak of some heap-allocated object/structure which itself results in these anon_inode:i915.gem entries.  More of these objects are created for each time through the loop, but they fail to be properly freed/released when they should resulting in both the increase in heap use and the increased number of i915.gem entries. Note that these are not the same: the i915.gem memory use is different than the escalated heap use - but they increase simultaneously and are therefore both likely symptoms of the same underlying problem.

The increase in heap use being correlated with an increase in i915.gem entries has also been observed in my browser (i.e., more realistic use cases) and is not an artifact of the artificial nature of the testing code.

Seth, if you are unable to replicate any leak even at high loop counts, are you perhaps not running on intel video hardware?  I've replicated this on all my machines, but they are all intel video.

Last edited by Trilby (2021-08-03 13:49:39)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

Board footer

Powered by FluxBB