You are not logged in.

#176 2021-07-28 16:33:41

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,335

Re: Weaver: a socket controlled web browser

weaver seems to respond to the commands one by one ( I guess this is not intended)
For example when I do

weaver google.com //target/0 duckduckgo.com

weaver opens single window with google.com, it opens another blank window when I press ^c in the terminal, and after one more ^c it opens duckduckgo.com, and it takes one more ^c for command to exit.

Even with simple "weaver duckduckgo.com", command in the terminal keeps running after closing the weaver window. This happens when running any weaver command, command doesn't exit after printing output.


Arch is home!
cwm rofi weaver vifm vis lizzy pass terminator
https://github.com/Docbroke

Online

#177 2021-07-28 20:17:55

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,629
Website

Re: Weaver: a socket controlled web browser

It sounds like netcat is not exiting.  What implementation of 'nc' are you using?

EDIT: I just confirmed this is the case.  At least gnu-netcat needs an explicit -c flag to tell it to close after receiving an EOF from stdin ... which is pretty silly.  You can edit the weaver script and add the -c flag to the 'nc' line near the end.  I'll set up some conditionals in the script to identify which nc is in use.

And openbsd nc needs yet another flag -N.

I just pushed a fix that will work "out of the box" on a majority of arch linux systems (so long as mkinitcpio-busybox is installed).  This commit also includes additions to the config file to use other implementations of netcat if needed.

Last edited by Trilby (2021-07-28 20:32:26)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#178 2021-07-29 02:55:36

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,335

Re: Weaver: a socket controlled web browser

I end up installing busybox.
Problem with gnu-netcat was, with -c option, I was not getting any output, so commands like //list //info etc were not working.


EDIT: I am updating my keybindings from pure javascript to weaver //action which is more reliable and works when javascript bindings fail. For example, when weaver is loading large pdf and you want to go back, "window.history.back(); break" fails but 'location.href="weaver://action/Back"' works.

document.addEventListener('keydown', function(e) {
	if (hinter.running) return;
	/* Process escape early, to unfocus an input element */
	if (e.key == 'Escape') e.target.blur();
	/* Don't process keypresses on editable elements */
    if (e.target.nodeName.toLowerCase() === "textarea" ||
        e.target.nodeName.toLowerCase() === "input" ||
        document.designMode === "on" ||
        e.target.contentEditable === "true")
		return;
	/* check modifiers: currently just control */
	var key = e.key;
	if (e.ctrlKey) key = '^' + key;
    if (e.altKey) key = 'A-' + key;
	if (e.metaKey) key = 'M-' + key; /* meta = mod4 aka "Super" */
	/* Check passthrough */
	if (key == '^z') passthrough = !passthrough;
	if (passthrough) return;
	/* Key bindings */
	switch (key) {
        /* modifiers must be in order of M-A-^ */
		/* e.g., Ctrl+c = ^c, Mod+Ctrl+c = M-^c, Alt+Mod+r = M-A-r */
//		case 'h': window.history.back(); break;
//		case 'l': window.history.forward(); break;
		case '^[': location.href="weaver://action/Back"; break;
		case 'h': location.href="weaver://action/Back"; break;
		case '[': location.href="weaver://action/Back"; break;
		case '^]': location.href="weaver://action/Forward"; break;
		case ']': location.href="weaver://action/Forward"; break;
		case 'l': location.href="weaver://action/Forward"; break;
		case 'j': window.scrollBy({top: 100, behavior: 'smooth' }); break;
		case 'k': window.scrollBy({top: -100, behavior: 'smooth' }); break;
		case '/': needle = prompt("Search String"); window.find(needle, false, false, true); break;
		case 'n': window.find(needle, false, false, true); break;
        case 'f': hinter.run(); break;
        case 'F': hinter.run(true); break;
		case 'p': location.href="weaver://action/SavePage"; break; //save as mhtml
        case '^p': window.print(); break; //save as pdf
//        case '^c': window.stop(); break;
        case '^c': location.href="weaver://action/Stop"; break;
//		case 'F5': location.reload(); break;
		case 'F5': location.href="weaver://action/Reload"; break;
		case '^r': location.href="weaver://action/ReloadAndBypassCache"; break;
		case 's': location.href="weaver://action/ViewSource"; break;
		case '^s': location.href="weaver://action/InspectElement"; break;
//        case 'F10': weaverCommand("close"); break;
        case 'F10': location.href="weaver://close"; break;
        case 'q': location.href="weaver://close"; break;
		case 'G': window.scrollTo(0, document.body.scrollHeight); break;
		case 'g': window.scrollTo(0, 0); break;
        case '+': document.body.style.zoom = "1.0"; break;
        case '=': document.body.style.zoom = (window.innerWidth * 125 / document.body.clientWidth) + '%'; break;
        case '-': document.body.style.zoom = (window.innerWidth * 80 / document.body.clientWidth) + '%'; break;

        case 'r': simplyread(); break;
	}
});

Last edited by Docbroke (2021-07-29 05:56:18)


Arch is home!
cwm rofi weaver vifm vis lizzy pass terminator
https://github.com/Docbroke

Online

#179 2021-08-04 18:57:36

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,335

Re: Weaver: a socket controlled web browser

I am trying to have a function in my url-opener script, to open some url using simplyread (javascript that makes page easy to read). Here is what I have come up with ( it works), but it appears hacky, as it is based on assumption that newly opened page becomes page 1 in the list.

simplyread) weaver //target/0 "$url" && sleep 1 && weaver //target/1 //js/simplyread\(\) ;;

Any suggestions ?


Arch is home!
cwm rofi weaver vifm vis lizzy pass terminator
https://github.com/Docbroke

Online

#180 2021-08-04 20:02:54

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,629
Website

Re: Weaver: a socket controlled web browser

Any newly open page will be #1 on the list - that is not only a safe assumption, but a deliberate design.

Also, only some commands need an explicit //target/1 (as that would otherwise be the default anyways).  The intent / goal is for most commands to simply work on the #1 window except for those commands that could close the page or redirect it to another url or otherwise be "destructive" of existing content.  These "destructive" functions can target the #1 window, but it needs to be made explicit to do so.  This means the commands //close and //open require an explicit //target/1 call right before hand to work on anything other than the currently gui-focused window.  This admittedly needs clarification in the README - but that's because it still needs clarification of which functions really should be in which category.  But in anycase, an extra //target/1 is never harmful, though //js/ doesn't require it.

Also note that weaver has it's own "sleep".  So the below should suffice:

simplyread) weaver //target/0 "$url" //sleep/1 //js/simplyread\(\)

However, I doubt the sleep is necessary (or if it is //sleep would likely be sufficient as the default sleep time is 0.1s).  But I'm pretty sure this would work:

simplyread) weaver //target/0 "$url" //js/simplyread\(\) ;;

Also unless there is the potential of that code being run while a weaver window is the active / focused / front-most GUI window and you'd not want the url opened in that window, then the //target/0 is also not needed.  So the following might meet your needs:

simplyread) weaver "$url" //js/simplyread\(\) ;;

Last edited by Trilby (2021-08-04 20:03:46)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#181 2021-08-05 02:52:06

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,335

Re: Weaver: a socket controlled web browser

Thanks for explanation. Simplyread works only after page is loaded, therefore sleep is needed. I have now updated command as

simplyread) weaver "$url" //sleep/1 //js/simplyread\(\) ;;

Arch is home!
cwm rofi weaver vifm vis lizzy pass terminator
https://github.com/Docbroke

Online

#182 2021-09-17 05:20:29

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,335

Re: Weaver: a socket controlled web browser

I was thinking about exitonclose workaround.
Instead of exiting weaverserver can we just exit qtwebengine while keeping server running, so whenever there is need to open a window, weaverserver can start the webengine.
This will need separating server process from webengine, so that starting server will only listen on the socket, but doesn't load webengine unless it needs to.


Arch is home!
cwm rofi weaver vifm vis lizzy pass terminator
https://github.com/Docbroke

Online

#183 2021-09-17 12:22:07

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,629
Website

Re: Weaver: a socket controlled web browser

That's been in the back on my mind for a while and will likely happen eventually.  It will take a complete rewrite though as the way qt5webengine works requires it to be started before any qApplication or other qObjects.  So there'd be two binaries, one without any qt5 at all to handle the socket and all IPC, and another to actually run all the qt5 code.  But this would limit the interactions that can be done.  For example, several commands, and several core functions of weaver require the tool that the user calls from the command line to be able to get information about existing webengine windows: this requires maintaining a list of said windows, which requires qt5.

I think this means I'd need two separate ipc channels: one for the non-qt5 interactions that can persist after the qt5webengine processes are taken down, and one for all the qt5 interaction.  Of course this would also require code to bridge these two interfaces and to setup and take-down this bridging mechanism each time webengine is restarted.

The whole thing gets pretty ugly - and it's all a workaround for a bug in qt5webengine which shouldn't exist in the first place.  Such complex mechanisms just end up being bug-magnets themselves.  There's usually an aesthetic of simplicity when the right solution to a problem is found - that "Aha" moment - and this multi-IPC-channel approach certainly doesn't inspire that reaction.  This has prevented me from being enthusiastic about pursuing that approach - and so I spend more time trying to learn about firefox's remote control mechanisms hoping progress may come there (but it's not likely).

But - if you're asking about this - does it mean you are facing the memory leak issues?  Previously I thought you said you weren't seeing it.

Last edited by Trilby (2021-09-17 13:15:45)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#184 2021-09-17 13:29:04

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,335

Re: Weaver: a socket controlled web browser

I needed to adjust my tabby bar script, so that weaverserver and qtwebengine don't keep appearing continuously, defeating the purpose of the script. While doing that I noticed that even when no weaver window is open qtwebengine uses >20% RAM on my system.


Arch is home!
cwm rofi weaver vifm vis lizzy pass terminator
https://github.com/Docbroke

Online

#185 2021-09-17 14:10:03

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,629
Website

Re: Weaver: a socket controlled web browser

It's a memory hog under the best of conditions - but is it consistently that high, or does it climb gradually as windows continue to be opened / closed? I have 8BG ram, and weaver is currently using just under 5% ram.  It's been running for a little while and has had at least a couple dozen or so windows opened / closed - but as I'm working on Tabby I am regularly restarting the compositor which restarts weaver.  So I'm not currently dealing with multi-day uptimes for weaver.

I really have a love-hate relationship with qt5-webengine.  It's a wonderful API with great documentation.  But a lot of the core internals just feel sloppy.  To be fair, I think a lot of this is actually in the chromium code that is imported.  My impression is that the google folks don't care to much about issues that their libs cause as long as it doesn't affect their end-products (chrome / chromium).  But the qt5 people are in a good position to really push them to cooperate to find solutions ... but I don't get the sense that any of this pushing is happening.  The speed at which these bugs are closed as "wont fix" on the qt5-webengine channels is pretty crazy.

Last edited by Trilby (2021-09-17 14:16:40)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#186 2021-09-17 14:15:02

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,335

Re: Weaver: a socket controlled web browser

It climbs, so if I keep my session running, opening and closing weaver windows, memory consumptions increases. However this is very inconsistent so I am not sure it is just a memory hog or there is memory leak.


Arch is home!
cwm rofi weaver vifm vis lizzy pass terminator
https://github.com/Docbroke

Online

#187 2021-09-17 14:22:18

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 25,629
Website

Re: Weaver: a socket controlled web browser

If it's increasing with more use, that's a leak.  Memory hog symptoms are huge uses of memory when weaver / webengine are actually doing something, like displaying a media-heavy page.  But if no pages are open, memory use should drop back to baseline - a failure to do so is a leak.

However, the relevant baseline is not the ram use right after the server is started, but rather if you open N windows, then close them again, the ram use at that time (with all weaver windows closed) is the reference baseline for any future use - at least as long as the number of windows open at any time is less than N.  This allows for the potential of conserving some allocated memory that'd be reused by the next page to be opened.

But if continued use results in a steady increase, then the ram consumption is not due to preserved re-usable resources, but rather conserved but not reusable memory.  And if that isn't the definition of a memory leak, I don't know what is.  The webengine / chromium library code holds on to memory allocations that will *not* ever be used again.

The symptoms of being a memory hog would be just that if you open the same page(s) in a chromium/webengine-based browser and also in firefox, firefox will use much less memory while displaying those pages.  Chromium/webengine will always be a hog, but in the absence of a leak that would only be apparent while web pages were open.

I really love how efficient firefox is, but there is no reasonable API to use the gecko rendering engine outside of firefox itself, and I'm not a fan of it requiring gtk3 (this latter issue I could accept if there was a decent API).  Mozilla's Servo provides *some* options to use the engine itself - but it is still lagging far behind modern web rendering engines and will really butcher a lot of web content ... it's not useable for a "real" browser.  Sadly Servo development is likely dead in the water as of last year.

Last edited by Trilby (2021-09-17 14:38:15)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#188 2021-09-17 14:31:21

Docbroke
Member
From: India
Registered: 2015-06-13
Posts: 1,335

Re: Weaver: a socket controlled web browser

ok, based on your definition, what I am seeing is both, webengine is memory hog as well as there is memory leak.

Last edited by Docbroke (2021-09-17 14:31:36)


Arch is home!
cwm rofi weaver vifm vis lizzy pass terminator
https://github.com/Docbroke

Online

Board footer

Powered by FluxBB