You are not logged in.

#76 2021-02-02 17:08:00

schard
Member
From: Hannover
Registered: 2016-05-06
Posts: 1,161
Website

Re: Weaver: a socket controlled web browser

There is a problem with makepkg reevaluating the vars and thusly recalculating the date, which can cause problems.
Otoh, the SCM used appears to need the current date for some reason (I don't know, since I don't use fossil).
There was a discussion on this that was moved to the dust bin by request.

Offline

#77 2021-02-02 17:12:47

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

qinohe wrote:

Or do you mean it is not ready to publish, yet?

To keep it simple - I'll just acknowledge I'm not clear on the standards currently in use on aur.archlinux.org and I've decided to opt out of participation there.  But you can certainly feel free to submit any PKGBUILD for weaver that you'd like to maintain.  However, I would suggest waiting until the code cleanup and memory leak issues are removed from the project page to-do list.  Until that point this is still very much in the 'experimental' zone.

schard wrote:

the SCM used appears to need the current date for some reason

No, that was just a handy pkgver to satisfy makepkg's needs.  The SCM is quite flexible in what download urls it will respond to.

Last edited by Trilby (2021-02-02 17:14:18)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#78 2021-02-02 17:52:09

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,235

Re: Weaver: a socket controlled web browser

Regarding REM usage, I noticed that, with fresh start (server only) weaver uses about 400M, after opening anyone window QtWebEngineProcess consumes another 500M. Even when I close all windows QtWebEngineProcess keeps running. So with no windows open weaver+webengine consumes nearly 1GB.
EDIT: and if I kill QtWebEngineProcess it comes back instantly.

EDIT2: Regarding, defaultzoom. I have 150% default zoom setup in javascript, as suggested before. Everytime I open a new page, for initial one or two second page is displayed with 100% zoom and then it changes to 150% zoom.

Last edited by Docbroke (2021-02-02 18:12:59)


Arch is home!
cwm rofi weaver vifm vim lizzy pass terminator

Offline

#79 2021-02-02 20:20:22

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 4,069

Re: Weaver: a socket controlled web browser

Trilby wrote:
qinohe wrote:

It's just that I tried that to use 'fossil clone repo' both as source

I'm not sure I know what this even means.  The source array is an array, not a function - you cannot run commands in source.  But to use fossil "like git" is awkward because fossil is not git, but to force it into the same pigeonhole you can either add a new DLAGENT for it in makepkg, or do all the retrieval of source code in prepare like we used to have to do with all VCSs:

svn is not git
hg is not git
bzr is not git

All these things are supported in the source array, despite not being git. I don't see how this is awkward?

The only trick here is that for select VCSes makepkg needs to know how they operate. I'd like to see more VCSes supported. I've even added official support in makepkg for flexibly extending makepkg to new VCSes by dropping one file into /usr/share/makepkg, because I'd like this to be easier for people to develop. See below for details.

Trilby wrote:
qinohe wrote:

Or do you mean it is not ready to publish, yet?

To keep it simple - I'll just acknowledge I'm not clear on the standards currently in use on aur.archlinux.org and I've decided to opt out of participation there.  But you can certainly feel free to submit any PKGBUILD for weaver that you'd like to maintain.  However, I would suggest waiting until the code cleanup and memory leak issues are removed from the project page to-do list.  Until that point this is still very much in the 'experimental' zone.

I don't think this has anything to do with the AUR, but rather, it's clear that it is interacting with makepkg in ways that produce build-time parse errors. Specifically the use of a metadata variable that changes every time you source the PKGBUILD, which causes several problems like:

  • makepkg --printsrcinfo produces different output every time you run it

  • makepkg --packagelist produces a different list of built package files every time you run it (and thus can never correctly report the correct file to install via makepkg && makepkg --packagelist | $rootelevationtool pacman -U -) or copy into a directory and run repo-add on

  • makepkg re-runs itself in a subprocess under fakeroot in order to carry out the packaging step, and in the process re-sources and re-evaluates the PKGBUILD in that subprocess; this then fails to match the child process

makepkg does provide the pkgver() function which is intended to communicate changes in the pkgver back to the PKGBUILD, typically in conjunction with a VCS. This is the only robust way to produce a PKGBUILD with a dynamic pkgver that actually "works" by running to completion and exiting successfully, without considering the issue of "standards".

Given the topic at hand is the creation of a package that pulls the latest code from a version control system and builds it, the most likely way to handle it would be to use a VCS source i.e. fossil+https:// which then checks out the source code using fossil, and can derive version numbers by running fossil commands to get the # of revisions.

Unfortunately, makepkg doesn't support this out of the box (it does support git, hg, svn, bzr).
Fortunately, makepkg from git does support this out of the box, thanks to the heroic effort of escondida.
(We hope to release pacman 6.0.0 soon enough. We've tagged an alpha to get more exposure for some nifty pacman download features.)

Fortunately², makepkg is designed to be extensible, and you may package /usr/share/makepkg/source/fossil.sh as its own package ("makepkg-fossil"?) -- it will work essentially standalone (minus special convenience handling of inferred .fossil cloned file extensions I think, solved by file::url format). You may get it from https://git.archlinux.org/pacman.git/co … de4e6effc2

Are there any particular reasons to dislike the idea of using VCS sources? It seems both convenient and dead-simple to me.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#80 2021-02-02 21:30:06

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

This is not the packaging thread.  I have no interest in arch packaging.  Period.  In fact I'm not sure I'll even check back on this thread.  I'll continue my project.  Anyone is free to use it.  Bug reports / tickets about the software will be accepted on the project page.  If anyone takes issue with the PKGBUILD I've put in the project repo, ignore it.  Or perhaps I'll delete it myself and just target other distros.

Last edited by Trilby (2021-02-02 21:31:31)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#81 2021-02-03 02:44:35

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

I may be at a dead end for the memory leak as it is not in my objects.  The following trivial example that uses the default QWebEngineView class shows the same problem: when the window is closed, the object is destroyed, but the memory is not actually freed:

#include <QApplication>
#include <QWebEngineView>

int main(int argc, char *argv[]) {
	QApplication app(argc, argv);
	app.setQuitOnLastWindowClosed(false);

	QWebEngineView *view = new QWebEngineView();
	view->load(QUrl("https://bbs.archlinux.org"));
	view->setAttribute(Qt::WA_DeleteOnClose);
	view->show();
	return app.exec();
}

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#82 2021-02-03 03:05:32

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,235

Re: Weaver: a socket controlled web browser

I tried to compare with falkon ( with my limited understandings)
When falkon is running (with 1 webpage), memory consumption is similar to weaver ( falkon ~450MB + QtWebEngine ~500MB), but when I close the page both disappear. Only difference I can see is weaver being server, it stays on and qtwebengine is not killed so 900MB to 1GB is like baseline. I need to keep check on my expectations, as memory consumption of QtWebEngine is higher compared to my experience with firefox and vimb.

What do you think about killing the server with last window closed, and starting the server again automatically whenever there is request to open a page or find a way to kill webengine when last page is closed ?

Last edited by Docbroke (2021-02-03 03:08:08)


Arch is home!
cwm rofi weaver vifm vim lizzy pass terminator

Offline

#83 2021-02-03 03:13:10

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

Docbroke wrote:

What do you think about killing the server with last window closed, and starting the server again automatically whenever there is request to open a page or find a way to kill webengine when last page is closed ?

The "server" is just the socket server.  If the server is killed, there is nothing listening for subsequent commands.  But as for the second part that's precisely what should happen: all webengine data is deleted - nothing should be left.  And it's the same even without the server: a webengineview doesn't free memory when deleted.

It seems QWebView (the webkit version) also doesn't free it's memory when deleted under a QApplication.  It's amazing how much less memory the webkit version uses ~20 MB for a page rather than ~500 MB, but if it's not properly freed, it'd still be a problem.

EDIT: actually now that I've been testing more - I'm not currently seeing a consistent leak.  The first webview open allocates a lot of memory (~500 MB on my system) for various worker processes.  Each additional web view only adds a small increment of ram.  These small increments seem to be properly freed when the pages are destroyed except that when the last one is destroyed, the worker processes are still there despite the program having no remaining pointers to them.

EDIT: I just confirmed I can't replicate an actual memory leak - webengine is a memory hog at startup, and it's worker threads stick around when the last page is closed (which is annoying) but there is not incremental increase.  I just wrote a script to open and close 100 web pages, and the ram usage at the end was identical to that at the start.  Docbroke, can you elaborate on your ram usage symptoms?  Can you give specific steps that I could try to replicate?

Last edited by Trilby (2021-02-03 04:10:59)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#84 2021-02-03 04:27:56

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,235

Re: Weaver: a socket controlled web browser

I think we are seeing same symptoms. As you already noticed, QtWebEngine consumes plenty of memory and it keeps using about 500MB even when all windows are closed, which made me think that there is memory leak, but probably it is not.


Arch is home!
cwm rofi weaver vifm vim lizzy pass terminator

Offline

#85 2021-02-03 04:40:31

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

Thanks.  That much still annoys me, and I'm going to keep hunting for a way to have all the webengine stuff shut down when the last window closes, but I'll demote that from bug/memory-leak to annoyance/desired-feature.

EDIT: these seems to be a known issue and is reportedly deliberate from chromium:
https://bugreports.qt.io/browse/QTBUG-56774

One approach would be to let the weaver server shutdown whenever the last window closes and change the exit code for calls to "quit".  If this was paired with running the server in a loop (e.g. `while weaver; do :; done`) it would acheive the intended goal.  I'll have to give this some thought.

Last edited by Trilby (2021-02-03 05:15:22)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#86 2021-02-03 05:45:06

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,560
Website

Re: Weaver: a socket controlled web browser


Fix yo shit: journalctl -b -p warning
Github

Offline

#87 2021-02-03 14:31:11

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

Thanks for the link, but can you clarify if you saw anything promising there?  Much of that thread is specifically discussing python which weaver does not use.  It's also no longer clear if there actually is any memory leak at all.  The discussion linked above links to Qt documentation page highlighting that what were seeing is intended behavior: use of QWebengine launches a separate QtWebEngineProcess binary that continues in the background.  Ideally there would be a way of telling QtWebEngineProcess to shut down when it's no longer needed, but it is not controlled by anything in the program that triggered it to start (except for being in the same process group).  What I am looking into is whether there is more graceful way of telling it to exit than just doing something like an execv(pkill, pkill, QtWebEngineProcess).


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#88 2021-02-03 17:11:35

qinohe
Member
From: Netherlands
Registered: 2012-06-20
Posts: 1,276

Re: Weaver: a socket controlled web browser

@eschwartz, thanks for explaining, appreciate it;) I will open a new thread if I'd like to know more about it, thanks.

@Trilby, okay, I get your point and why you don't want anything to do with the packing part, thanks.

About the memory part, I'm not seeing what you guys do, I would have noticed it late anyway for I have lots of it.
Also when I close the last page I use the 'quit' command,I use dmenu for it..
Anyway, my findings on the memory part, simply tested with 'free'
pasted only 'free used':

# weaver not started
Mem:                  521Mi

# weaver proc. started
Mem:                  562Mi

# weaver proc. & 1page opened (bbs.archlinux.org )
Mem:                  606Mi

# weaver proc. & 2pages opened ^^ + forum.opnsense.org)
Mem:                  617Mi

# weaver proc. pages closed!
Mem:                  601Mi

# pages closed & weaver proc. stopped
Mem:                  525Mi

For now weaver seems to rum stable ;-)

I'm busy creating a firejail profile for weaver, it's actually done, the socket is started.
Still I get an error, I open a separate thread about that.

Last edited by qinohe (2021-02-03 17:29:28)

Offline

#89 2021-02-03 17:22:49

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

qinohe, the issue in question is the one test you didn't list.  What is the memory usage after closing the two pages but before weaver exits?  Ideally it should return back to 562M, but if it follows the pattern I see, it'd be around 600M.  You can also follow ps(tree) output and see the same effect that a set of background webengine processes are started when the first page is opened, and these porcesses remain after the last page is closed.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#90 2021-02-03 17:31:52

qinohe
Member
From: Netherlands
Registered: 2012-06-20
Posts: 1,276

Re: Weaver: a socket controlled web browser

Yes, I added '# weaver proc. pages closed!' to the above results, it's around 600M ... get the picture...;-)

Weaver users that also use 'firejail/apparmor' there is a basic working profile here (see)https://bbs.archlinux.org/viewtopic.php … 2#p1953742
I actually had error which I figured out #88

Last edited by qinohe (2021-02-03 19:33:06)

Offline

#91 2021-02-03 20:55:37

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

Docbroke wrote:

One more thing, I would like is ability to get link_url, which may-be url under the mouse...

I'm revisiting this comment - while this has been possible for a bit through js-run, I just pushed an example in WeaverJS.js.  If that userscript is copied to your ~/.config/weaver/scripts folder (and weaver restarted) you can then get the url under the mouse as follows:

weaver js-run 'WeaverJS.urlUnderMouse()'
sleep 0.1
weaver js-result

If you are typing those manually, there'd be no need for any "sleep".  In a script, you should really check / wait for "weaver js-result-ready" to return true, but in this case, the results should be ready nearly instantaneously, so an arbitrarily short sleep should do just fine.  In fact I'd not be surprised if the two commands back to back *might* work fine, but not necessarily reliably.  There are currently other examples in WeaverJS for grabbing the url targets of all links.  I'd love to expand on useful examples in that file, and welcome user contributions for other useful WeaverJS functions.

I'll be adding a command to load a page without displaying the window, so weaver could be used to essentially load/render a page and run arbitrary JS on it and retrieve results, all without showing any GUI.

Last edited by Trilby (2021-02-03 21:01:31)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#92 2021-02-04 03:09:35

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,235

Re: Weaver: a socket controlled web browser

This works great for my use, I just created keybind for this

#!/bin/sh

weaver js-run 'WeaverJS.urlUnderMouse()'
sleep 0.1

link-handler -u $(weaver js-result)

where link-handler is rofi based script which gives all options I need to use with the given url.


Arch is home!
cwm rofi weaver vifm vim lizzy pass terminator

Offline

#93 2021-02-05 08:09:51

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,235

Re: Weaver: a socket controlled web browser

Some sites are not displayed properly. This issue is not there with falkon. For e.g,. news.google.com


Arch is home!
cwm rofi weaver vifm vim lizzy pass terminator

Offline

#94 2021-02-05 08:30:32

ugjka
Member
From: Latvia
Registered: 2014-04-01
Posts: 1,560
Website

Re: Weaver: a socket controlled web browser

Docbroke wrote:

Some sites are not displayed properly. This issue is not there with falkon. For e.g,. news.google.com

Maybe because of this user agent?

Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0

Fix yo shit: journalctl -b -p warning
Github

Offline

#95 2021-02-05 15:31:04

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

Docbroke, in what way is it not displayed properly?  I just tested that site and it looked ok, but I'd never used that site before so I didn't have anything to compare it to.  I opened the same site in another browser, and it had a totally different (dark) theme, but that's just because I was logged in to a google account on the other browser: when I logged out, that page was identical on weaver and the other browser (qutebrowser in this case).

Are you logged in on that site?  Do you have any google-account-based theming for it?


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#96 2021-02-05 16:24:42

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,235

Re: Weaver: a socket controlled web browser

with firefox
firefox-google.png

with weaver
weaver-google.png


Arch is home!
cwm rofi weaver vifm vim lizzy pass terminator

Offline

#97 2021-02-05 16:31:46

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

Thanks.  Unfortunately I can't replicate that at all.  Does it depend on the width of the weaver window?  As I resize my window I can see something that looks like your problematic image instantaneously as the site hides/shows the sidebar depending on the width - but I can't get any width where it freezes like that.

EDIT: ooh, is this with the defaultZoom.js? EDIT2: I can't replicate with defaultzoom.js either.  Can you test with all user scripts disabled (moved or renamed to not end in .js, then restart the weaver server).  I anticipate that injecting much JS into JS-heavy pages will be problematic, and as the hide/unhide of that sidebar is controlled by the page's native JS, that seems a likely cause.  If it is the cause - you can specify a @include to limit the pages on which the userScript is included (this should be common practice) or perhaps use an @exclude if you really want it everywhere except on the page where it causes issues.

Last edited by Trilby (2021-02-05 16:40:10)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#98 2021-02-05 16:47:42

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,235

Re: Weaver: a socket controlled web browser

Ah, thanks. This is indeed one of the userscript. I disabled all scripts and issue was resolved. I think I will enable them one by one to find the culprit.

EDIT: It is defaultZoom.js . After disabling that issue is solved. I guess I have to find some other way to get 150% default zoom.
EDIT2: QT_SCALE_FACTOR is doing fine job.

Last edited by Docbroke (2021-02-05 17:01:07)


Arch is home!
cwm rofi weaver vifm vim lizzy pass terminator

Offline

#99 2021-02-05 17:02:38

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

Glad QT_SCALE_FACTOR is working.  I always thought that was the better approach - but as the other was working I wasn't going to mess with it.  I suspect an actual customCSS specified in a userscript would be pretty reliable to - but the SCALE_FACTOR should be the most robust.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#100 2021-02-06 19:25:33

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,282

Re: Weaver: a socket controlled web browser

Manual intervention advised for the code just pushed.  The changes I just pushed are preparing for a private mode.  To do so, named profile(s) are now used.  The name of the profile is picked up from the WEAVER_PROFILE environment variable if it is set in the environment where the server is started.  If it is unset (e.g., does not exist in the environment) a default value of $USER will be used.  If WEAVER_PROFILE is set to an empty string, a ram-only profile will be used and nothing (no cookies, cache data, etc) will be written to disk.  If you have no interest in multiple profiles or a private mode, then you do not need to use this variable at all - However in order to maintain any existing persistent cookies and cache data, you should run the following command prior to running the newly pushed code:

mv ~/.local/share/weaver/QtWebEngine/Default ~/.local/share/weaver/QtWebEngine/$USER

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

Board footer

Powered by FluxBB