You are not logged in.

#176 2021-07-28 16:33:41

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

weaver seems to respond to the commands one by one ( I guess this is not intended)
For example when I do

weaver google.com //target/0 duckduckgo.com

weaver opens single window with google.com, it opens another blank window when I press ^c in the terminal, and after one more ^c it opens duckduckgo.com, and it takes one more ^c for command to exit.

Even with simple "weaver duckduckgo.com", command in the terminal keeps running after closing the weaver window. This happens when running any weaver command, command doesn't exit after printing output.

Offline

#177 2021-07-28 20:17:55

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

It sounds like netcat is not exiting.  What implementation of 'nc' are you using?

EDIT: I just confirmed this is the case.  At least gnu-netcat needs an explicit -c flag to tell it to close after receiving an EOF from stdin ... which is pretty silly.  You can edit the weaver script and add the -c flag to the 'nc' line near the end.  I'll set up some conditionals in the script to identify which nc is in use.

And openbsd nc needs yet another flag -N.

I just pushed a fix that will work "out of the box" on a majority of arch linux systems (so long as mkinitcpio-busybox is installed).  This commit also includes additions to the config file to use other implementations of netcat if needed.

Last edited by Trilby (2021-07-28 20:32:26)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#178 2021-07-29 02:55:36

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

I end up installing busybox.
Problem with gnu-netcat was, with -c option, I was not getting any output, so commands like //list //info etc were not working.


EDIT: I am updating my keybindings from pure javascript to weaver //action which is more reliable and works when javascript bindings fail. For example, when weaver is loading large pdf and you want to go back, "window.history.back(); break" fails but 'location.href="weaver://action/Back"' works.

document.addEventListener('keydown', function(e) {
	if (hinter.running) return;
	/* Process escape early, to unfocus an input element */
	if (e.key == 'Escape') e.target.blur();
	/* Don't process keypresses on editable elements */
    if (e.target.nodeName.toLowerCase() === "textarea" ||
        e.target.nodeName.toLowerCase() === "input" ||
        document.designMode === "on" ||
        e.target.contentEditable === "true")
		return;
	/* check modifiers: currently just control */
	var key = e.key;
	if (e.ctrlKey) key = '^' + key;
    if (e.altKey) key = 'A-' + key;
	if (e.metaKey) key = 'M-' + key; /* meta = mod4 aka "Super" */
	/* Check passthrough */
	if (key == '^z') passthrough = !passthrough;
	if (passthrough) return;
	/* Key bindings */
	switch (key) {
        /* modifiers must be in order of M-A-^ */
		/* e.g., Ctrl+c = ^c, Mod+Ctrl+c = M-^c, Alt+Mod+r = M-A-r */
//		case 'h': window.history.back(); break;
//		case 'l': window.history.forward(); break;
		case '^[': location.href="weaver://action/Back"; break;
		case 'h': location.href="weaver://action/Back"; break;
		case '[': location.href="weaver://action/Back"; break;
		case '^]': location.href="weaver://action/Forward"; break;
		case ']': location.href="weaver://action/Forward"; break;
		case 'l': location.href="weaver://action/Forward"; break;
		case 'j': window.scrollBy({top: 100, behavior: 'smooth' }); break;
		case 'k': window.scrollBy({top: -100, behavior: 'smooth' }); break;
		case '/': needle = prompt("Search String"); window.find(needle, false, false, true); break;
		case 'n': window.find(needle, false, false, true); break;
        case 'f': hinter.run(); break;
        case 'F': hinter.run(true); break;
		case 'p': location.href="weaver://action/SavePage"; break; //save as mhtml
        case '^p': window.print(); break; //save as pdf
//        case '^c': window.stop(); break;
        case '^c': location.href="weaver://action/Stop"; break;
//		case 'F5': location.reload(); break;
		case 'F5': location.href="weaver://action/Reload"; break;
		case '^r': location.href="weaver://action/ReloadAndBypassCache"; break;
		case 's': location.href="weaver://action/ViewSource"; break;
		case '^s': location.href="weaver://action/InspectElement"; break;
//        case 'F10': weaverCommand("close"); break;
        case 'F10': location.href="weaver://close"; break;
        case 'q': location.href="weaver://close"; break;
		case 'G': window.scrollTo(0, document.body.scrollHeight); break;
		case 'g': window.scrollTo(0, 0); break;
        case '+': document.body.style.zoom = "1.0"; break;
        case '=': document.body.style.zoom = (window.innerWidth * 125 / document.body.clientWidth) + '%'; break;
        case '-': document.body.style.zoom = (window.innerWidth * 80 / document.body.clientWidth) + '%'; break;

        case 'r': simplyread(); break;
	}
});

Last edited by Docbroke (2021-07-29 05:56:18)

Offline

#179 2021-08-04 18:57:36

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

I am trying to have a function in my url-opener script, to open some url using simplyread (javascript that makes page easy to read). Here is what I have come up with ( it works), but it appears hacky, as it is based on assumption that newly opened page becomes page 1 in the list.

simplyread) weaver //target/0 "$url" && sleep 1 && weaver //target/1 //js/simplyread\(\) ;;

Any suggestions ?

Offline

#180 2021-08-04 20:02:54

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

Any newly open page will be #1 on the list - that is not only a safe assumption, but a deliberate design.

Also, only some commands need an explicit //target/1 (as that would otherwise be the default anyways).  The intent / goal is for most commands to simply work on the #1 window except for those commands that could close the page or redirect it to another url or otherwise be "destructive" of existing content.  These "destructive" functions can target the #1 window, but it needs to be made explicit to do so.  This means the commands //close and //open require an explicit //target/1 call right before hand to work on anything other than the currently gui-focused window.  This admittedly needs clarification in the README - but that's because it still needs clarification of which functions really should be in which category.  But in anycase, an extra //target/1 is never harmful, though //js/ doesn't require it.

Also note that weaver has it's own "sleep".  So the below should suffice:

simplyread) weaver //target/0 "$url" //sleep/1 //js/simplyread\(\)

However, I doubt the sleep is necessary (or if it is //sleep would likely be sufficient as the default sleep time is 0.1s).  But I'm pretty sure this would work:

simplyread) weaver //target/0 "$url" //js/simplyread\(\) ;;

Also unless there is the potential of that code being run while a weaver window is the active / focused / front-most GUI window and you'd not want the url opened in that window, then the //target/0 is also not needed.  So the following might meet your needs:

simplyread) weaver "$url" //js/simplyread\(\) ;;

Last edited by Trilby (2021-08-04 20:03:46)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#181 2021-08-05 02:52:06

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

Thanks for explanation. Simplyread works only after page is loaded, therefore sleep is needed. I have now updated command as

simplyread) weaver "$url" //sleep/1 //js/simplyread\(\) ;;

Offline

#182 2021-09-17 05:20:29

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

I was thinking about exitonclose workaround.
Instead of exiting weaverserver can we just exit qtwebengine while keeping server running, so whenever there is need to open a window, weaverserver can start the webengine.
This will need separating server process from webengine, so that starting server will only listen on the socket, but doesn't load webengine unless it needs to.

Offline

#183 2021-09-17 12:22:07

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

That's been in the back on my mind for a while and will likely happen eventually.  It will take a complete rewrite though as the way qt5webengine works requires it to be started before any qApplication or other qObjects.  So there'd be two binaries, one without any qt5 at all to handle the socket and all IPC, and another to actually run all the qt5 code.  But this would limit the interactions that can be done.  For example, several commands, and several core functions of weaver require the tool that the user calls from the command line to be able to get information about existing webengine windows: this requires maintaining a list of said windows, which requires qt5.

I think this means I'd need two separate ipc channels: one for the non-qt5 interactions that can persist after the qt5webengine processes are taken down, and one for all the qt5 interaction.  Of course this would also require code to bridge these two interfaces and to setup and take-down this bridging mechanism each time webengine is restarted.

The whole thing gets pretty ugly - and it's all a workaround for a bug in qt5webengine which shouldn't exist in the first place.  Such complex mechanisms just end up being bug-magnets themselves.  There's usually an aesthetic of simplicity when the right solution to a problem is found - that "Aha" moment - and this multi-IPC-channel approach certainly doesn't inspire that reaction.  This has prevented me from being enthusiastic about pursuing that approach - and so I spend more time trying to learn about firefox's remote control mechanisms hoping progress may come there (but it's not likely).

But - if you're asking about this - does it mean you are facing the memory leak issues?  Previously I thought you said you weren't seeing it.

Last edited by Trilby (2021-09-17 13:15:45)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#184 2021-09-17 13:29:04

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

I needed to adjust my tabby bar script, so that weaverserver and qtwebengine don't keep appearing continuously, defeating the purpose of the script. While doing that I noticed that even when no weaver window is open qtwebengine uses >20% RAM on my system.

Offline

#185 2021-09-17 14:10:03

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

It's a memory hog under the best of conditions - but is it consistently that high, or does it climb gradually as windows continue to be opened / closed? I have 8BG ram, and weaver is currently using just under 5% ram.  It's been running for a little while and has had at least a couple dozen or so windows opened / closed - but as I'm working on Tabby I am regularly restarting the compositor which restarts weaver.  So I'm not currently dealing with multi-day uptimes for weaver.

I really have a love-hate relationship with qt5-webengine.  It's a wonderful API with great documentation.  But a lot of the core internals just feel sloppy.  To be fair, I think a lot of this is actually in the chromium code that is imported.  My impression is that the google folks don't care to much about issues that their libs cause as long as it doesn't affect their end-products (chrome / chromium).  But the qt5 people are in a good position to really push them to cooperate to find solutions ... but I don't get the sense that any of this pushing is happening.  The speed at which these bugs are closed as "wont fix" on the qt5-webengine channels is pretty crazy.

Last edited by Trilby (2021-09-17 14:16:40)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#186 2021-09-17 14:15:02

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

It climbs, so if I keep my session running, opening and closing weaver windows, memory consumptions increases. However this is very inconsistent so I am not sure it is just a memory hog or there is memory leak.

Offline

#187 2021-09-17 14:22:18

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

If it's increasing with more use, that's a leak.  Memory hog symptoms are huge uses of memory when weaver / webengine are actually doing something, like displaying a media-heavy page.  But if no pages are open, memory use should drop back to baseline - a failure to do so is a leak.

However, the relevant baseline is not the ram use right after the server is started, but rather if you open N windows, then close them again, the ram use at that time (with all weaver windows closed) is the reference baseline for any future use - at least as long as the number of windows open at any time is less than N.  This allows for the potential of conserving some allocated memory that'd be reused by the next page to be opened.

But if continued use results in a steady increase, then the ram consumption is not due to preserved re-usable resources, but rather conserved but not reusable memory.  And if that isn't the definition of a memory leak, I don't know what is.  The webengine / chromium library code holds on to memory allocations that will *not* ever be used again.

The symptoms of being a memory hog would be just that if you open the same page(s) in a chromium/webengine-based browser and also in firefox, firefox will use much less memory while displaying those pages.  Chromium/webengine will always be a hog, but in the absence of a leak that would only be apparent while web pages were open.

I really love how efficient firefox is, but there is no reasonable API to use the gecko rendering engine outside of firefox itself, and I'm not a fan of it requiring gtk3 (this latter issue I could accept if there was a decent API).  Mozilla's Servo provides *some* options to use the engine itself - but it is still lagging far behind modern web rendering engines and will really butcher a lot of web content ... it's not useable for a "real" browser.  Sadly Servo development is likely dead in the water as of last year.

Last edited by Trilby (2021-09-17 14:38:15)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#188 2021-09-17 14:31:21

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

ok, based on your definition, what I am seeing is both, webengine is memory hog as well as there is memory leak.

Last edited by Docbroke (2021-09-17 14:31:36)

Offline

#189 2021-12-20 21:44:11

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

Weaver just got a complete rewrite and overhaul while converting from qt5 to qt6.  The new code is much leaner, yet should behave very similarly from the user's point of view.  There have, however, been a few important changes.  Most notably, the crippling memory issues have been resolved / avoided as weaver is no longer an indefinitely running server.  Instead it fires up a local server when first run, and any subsequent invocations connect to that until the last window is closed at which point it shuts down completely (this is similar to the behavior of qutebrowser and firefox).  I've also yet to actually do any well controlled tests, but I'm thinking qt6-webengine might be a bit less of a memory hog than the qt5 implementation (though it still certainly uses it's fair share).

With this there is no longer any reason to connect to the socket directly, nor have different binaries for the server and the client.  All commands go through the single binary "/usr/bin/weaver": commands are simply passed as command line arguments.  You can, technically, still write directly to the unix-domain socket, but there's no reason to, and I'd advise against it (some minimal checks are performed by 'weaver' before sending command line arguments to the socket).  In any case, there is no more need to worry about different netcat implementations and the trouble cause by their differing behaviors.

Another change is that managing downloads has been pulled out of weaver completely.  This is best handled by an external tool.  I've included a simple curl command as a default that will continue running even after weaver has been closed (for large downloads).  This can be replaced by more elaborate download management tools as desired.  This has worked well in preliminary testing, but more testing would be welcomed.

Other minor functional changes:

- Fullscreening has been drastically simplified (and may need to be recomplexified a bit again soon) and may not work properly on other WMs / compositors.  This will need some testing and definitely some patching.

- Handling of authenticaltion requests and proxy authentication requests was brought over directly from the qt5 version and so should work fine, but as I rarely if ever use them, they could use testing and feedback.

- Handling of feature permission requests (to use camera / microphone / geolocation / etc) was also imported from the qt5 version and should just work, but remains currently untested in the qt6 version.

- Info / List format specifiers have been greatly reduced.  The old version had options to break down the url into many components including host, path, fragment id, query string, etc.  I highly doubt these were ever used and so they were removed for cleanliness.  If anyone has a usecase for these, let me know - they'd be pretty easy to add back if there is a valid use for them.

Last edited by Trilby (2021-12-21 00:31:47)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#190 2021-12-22 09:32:46

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

Great, I will try this soon.
EDIT: I think I will have to wait, till there is tagged release. After that I may be able to ask alpine community to add this. At present qt6-webengine is not there in alpine.

Last edited by Docbroke (2021-12-22 10:28:16)

Offline

#191 2021-12-22 13:14:18

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

It looks like they have much of qt6 including a webchannel and websockets, just no webengine ... odd.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#192 2021-12-22 13:30:54

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: Weaver: qt6-webengine browser w/ minimal UI

Trilby wrote:

It looks like they have much of qt6 including a webchannel and websockets, just no webengine ... odd.

webengine is probably annoying to build, so you skip it until you have some other software that requires it.


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#193 2021-12-22 13:46:13

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

Docbroke, are you suggesting that a tagged release of weaver might contribute to getting qt6-webengine included in the alpine repos.  If so, I could make a tagged release soon.  I'd want to add some documentation and make weaver respond sanely to some flags like -h --help -v --version before I tag a release, but that all shouldn't take much.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#194 2021-12-22 14:28:56

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

Offline

#195 2022-01-15 05:45:50

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

I am not able to open youtube with weaver, it gives blank page.

2022-01-15T11:11:14: debug: src/webview.cpp (192) WebView::command: "www.youtube.com"
2022-01-15T11:11:15: js.warning:  (0) unknown: Error with Permissions-Policy header: Unrecognized feature: 'ch-ua-bitness'.

I have also seen this happening with 1 or 2 other sites but I don't exactly remember those. Also I feel there is some high CPU usage on few sites.

Offline

#196 2022-01-15 06:09:34

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

That js warning is normal from youtube.  I'd suspect this is another problem with the alpine webengine build likely lacking other necessary patches.  It'd be useful to check potential issues against another qt6-webengine browser, or a minimal test browser to identify whether the issue is in fact related to weaver or the webengine build.  I'll see if I can get some minimal test code that could be used.

EDIT: here you go:

// g++ -I/usr/include/qt6/ -fPIC -lQt6WebEngineWidgets -lQt6Core -lQt6Widgets -o qt6webtest qt6webtest.cpp

#include <QtWidgets/QApplication>
#include <QtWebEngineWidgets/QWebEngineView>

int main(int argc, char *argv[]) {
	if (argc < 2) return -1;
	QApplication app(argc, argv);
	auto v = new QWebEngineView();
	v->setUrl(QUrl(argv[1]));
	v->show();
	return app.exec();
}

Compile that as in the comment, and test youtube with that.  Note that this must be given a full url as a command line parameter - if will fail to render anything or give any error output if you run `qt6webtest youtube.com` but you must run `qt6webtest https://youtube.com` instead.  If the same pages fail with this as do with weaver, then the issue is with the qt6-webengine build.

EDIT 2: And are you using Carl Chave (Sodface)'s build of qt6-qtwebengine or did you switch to Bart Ribber's build in the alpine testing repository?

Last edited by Trilby (2022-01-15 06:37:30)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#197 2022-01-15 08:26:41

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

I am using sodface build. Now trying to find out which package provides qtwidgets.
EDIT: I think it would be better to ask sodface, if he is able to open youtube. I am not good at compiling things with alpine.

Last edited by Docbroke (2022-01-15 16:41:15)

Offline

#198 2022-01-15 16:40:32

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

You already have qtwidgets installed if you were using anything qt related including sodface's build of weaver.  It is provided by qt6-base.

But the first step would be to try Bart's official build of qt6-qtwebengine.

Last edited by Trilby (2022-01-15 16:41:36)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#199 2022-01-15 16:44:07

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,433

Re: Weaver: qt6-webengine browser w/ minimal UI

When I try to compile it gives me error about include lines, QtWidgets not found, and if I comment out that line it complains about second include line.

engine.c:3:10: fatal error: QtWidgets/QApplication: No such file or directory
    3 | #include <QtWidgets/QApplication>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

Offline

#200 2022-01-15 16:54:55

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Weaver: qt6-webengine browser w/ minimal UI

Ah, so that's just due to the header file being in a different location.  Use `find /usr/include -name QApplication` to see where it is, then adjust the -I flag on the compiler line accordingly.  Ideally this would all be handled with pkgconfig, but at least in arch there is no .pc file yet for qt6-webengine, so the makefile just hardcodes the include paths which might differ between distros.

As for compiling things, you were always compiling weaver weren't you?  You were just using an unofficial build of webengine.  Now there is an official build of webengine in the alpine repos, use that instead of the unofficial one.

If you really still want to use sodface's build of weaver that should be fine but you should still use the official webengine package (note weaver is quick and easy to build, webengine takes a lot of work).

EDIT: if you actually haven't built anything at all but are only using third-party binary builds, then you would need to install the relevant -dev packages as alpine uses split packages for headers.

Last edited by Trilby (2022-01-15 16:57:36)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

Board footer

Powered by FluxBB