[SOLVED] xwininfo fails to extract titles containing non-ASCII chars

kyptin · 2020-12-05 03:18:40

When a window contains only regular characters, I can execute `xwininfo | grep "Window id"`, click the window, and get output that includes the window title. For example, here's the output for an xterm window with title `jeff@lemur:~`:

xwininfo: Window id: 0x400001e "jeff@lemur:~"

However, when I click on a window that includes irregular characters, xwininfo doesn't give me anything useful. For example, Firefox windows end with " ‒ Mozilla Firefox". The dash there is an em-dash, not a hyphen. For those windows, the output (modulo window ID) is:

xwininfo: Window id: 0xe00022 " (failure in conversion from UTF8_STRING to ANSI_X3.4-1968)"

Anybody know why I get this error and how to fix it?

The backstory: my university's campus lets you register your device with their "eduroam" wireless network. The registration process, which is apparently part of the "SecureW2" product, supports Linux, but the registration script didn't work for me because of this issue. I eventually worked around it by hacking the Python script to use `xprop` instead, but it was a pain. I opened a support ticket asking them to fix this, but they told me that only Ubuntu, Fedora, and Oracle Linux are supported. I'm trying to understand whether this might be something weird on my machine, or something likely to affect other users, so that I can decide whether I care to try to push this upstream.

Last edited by kyptin (2020-12-06 19:09:43)

Ropid · 2020-12-05 03:28:25

Can you share an example URL to test the problem? I looked up some random web-pages with Japanese characters and with emojis in the title and that worked fine with "xwininfo", it didn't show that error message you are seeing.

kyptin · 2020-12-05 03:34:26

At least for me, literally any page in Firefox has this problem. The em-dash itself is the thing that xwininfo is stumbling over even if all other characters are regular, and the em-dash is always included in the window title.

Ropid · 2020-12-05 03:49:47

Maybe there's a problem with the "locale" settings and programs think they can't print unicode (UTF-8) characters. Here's the wiki article for the locale settings: https://wiki.archlinux.org/index.php/Locale

kyptin · 2020-12-05 04:55:43

I'm not sure what's expected, but here's what I have:

$ locale -a
C
POSIX
en_US.utf8
$ locale
LANG=en_US.UTF-8
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=C

Furthermore, I can view unicode characters in my terminal just fine. So if there's a problem with my locale, maybe it's not an obvious one?

If you run `xwininfo` and click on a Firefox window, do you see the same thing as me, or do you actually see the real title?

Trilby · 2020-12-05 05:29:16

I also can't replicate the issue, even when I set a non-utf8 locale for the xwininfo command. Let's check these:

type xwininfo
pacman -Qo /bin/xwininfo
pacman -Qkk xorg-xwininfo

seth · 2020-12-05 08:38:41

LC_ALL=C xwininfo| grep "Window id"

actually gets me the same error (tried a window title w/ a lot of chinese glyhs and the terminal itself still was de_DE.UTF-8) and it's kinda strongly suggested to fix your https://wiki.archlinux.org/index.php/Locale anyway, because other clients will fail less gracefully (iir. gnome-terminal just crashes) when facing a non-utf8 locale.

LC_ALL=C xprop _NET_WM_NAME

will print the escaped byte sequence for the UTF-8 range.

No problem for the 7bit parts of the title, though (so C locale on a UTF-8 title that's fully C-compatible yields no error)

Trilby · 2020-12-05 13:38:29

seth wrote:

LC_ALL=C xwininfo| grep "Window id"
actually gets me the same error (tried a window title w/ a lot of chinese glyhs and the terminal itself still was de_DE.UTF-8) and it's kinda strongly suggested to fix your https://wiki.archlinux.org/index.php/Locale anyway

Then why can't I replicate the problem with that command here? Which actual webpage/title was your test with? What in this thread leads you to conclude that the OP's locale is "broken"? Their `locale -a` and `locale` output look valid to me.

EDIT: Nevermind, I just found a page that did replicate the error with LC_ALL=C, ~~but it would seem an m-dash is not sufficient to trigger it here.~~ EDIT 2: my first test was on a title that looked very much like an m-dash, but apparently was a U+002D hyphen-minus, while a page with an actual U+2013 en-dash did trigger the error with LC_ALL=C. Aside: I think I need a new font: a hyphen, en-dash, and m-dash all appear identical in my terminal ... that's horrible.

Last edited by Trilby (2020-12-05 13:46:28)

seth · 2020-12-05 13:57:25

https://zh.wikipedia.org/wiki/%E4%BE%8D

Trilby wrote:

What in this thread leads you to conclude that the OP's locale is "broken"?

seth wrote:

because other clients will fail less gracefully (iir. gnome-terminal just crashes) when facing a non-utf8 locale

It's not technically "broken" (ie. misconfigured), but a lot of stuff meanwhile assumes a UTF-8 locale (no, should™ not be that way… do you need the shouldland video?)

Trilby · 2020-12-05 14:02:05

seth wrote:

(no, should™ not be that way… do you need the shouldland video?)

Nope, I live in shouldland! I only occasionally venture out into whatotherpeopletolerateland. I hand to install xwininfo to test this, xprop handles things much more gracefully with LC_ALL=C.

Last edited by Trilby (2020-12-05 14:02:50)

seth · 2020-12-05 15:46:10

xprop is a different tool that does different things.

xprop's purpose (here) is to print the X11 properties on a window

xwininfo tells you "something" about the window - the identifier being grepped for here is condensed out of various information (eg. the window id and it'll probably also fall back to WM_NAME over _NET_WM_NAME - the former not even being defined as UTF-8 but "whatever encoding you want" STRING…)

Anyway, @kyptin - fix your locale! ;-)
The "LC_ALL=C" looks particularily suspicious, because of https://wiki.archlinux.org/index.php/Lo … leshooting

Trilby · 2020-12-05 16:52:04

seth wrote:

xprop is a different tool that does different things.
xprop's purpose (here) is to print the X11 properties on a window
xwininfo tells you "something" about the window - the identifier being grepped for here is condensed out of various information

I don't see how these are different. You can grep the xprop output for the same information. Xprop prints the properties, true. Xwininfo tells you "something"? What is that something? Oh, yeah, it's a property.

Sure there are differences in output format, and flags/options, but they seem to serve the same functions.

Last edited by Trilby (2020-12-05 16:52:27)

seth · 2020-12-05 16:55:13

It's different in that the xwininfo line is not supposed to be a direct reflection on some specific window data (_NET_WM_NAME) but just a "we're talking about this window here" alloy.

Trilby · 2020-12-05 17:14:34

seth wrote:

It's different in that the xwininfo line is not supposed to be a direct reflection on some specific window data (_NET_WM_NAME) but just a "we're talking about this window here" alloy.

What on earth could you mean by this. Xwininfo is not a sentient AI that is offering a subjective description of the window. It is most definitely printing deterministicly generated and specific window data.

I just checked the source code, and in that field it does indeed print _NET_WM_NAME and falls back on WM_NAME if the former is not set.

Last edited by Trilby (2020-12-05 17:19:41)

seth · 2020-12-05 17:25:41

Yes, this deterministic way: https://gitlab.freedesktop.org/xorg/app … nfo.c#L804

Routine to display a window id in dec/hex with name if window has one

As you can see, it clips a bunch of alternatives to come up w/ some sort of name "if window has one" and then to print that "if possible": https://gitlab.freedesktop.org/xorg/app … fo.c#L1890

It's not an effort reflect the data of a specific field but just some fancy header for the output.

Trilby · 2020-12-05 17:32:42

Not "a bunch", two. And you really seem to be arguing for the sake of argument. I stated, and stand by the statement, that xprop handles UTF-8 content of window names gracefully even with LC_ALL=C while xwininfo handles the same task poorly. Somehow you find this disagreeable because xwininfo actually might fail equally badly on either _NET_WM_NAME or WM_NAME while xprop succeeds on both but certainly that's really not a counterpoint.

If you want the name of a window (whether _NET_WM_NAME or WM_NAME ... either way) then using xprop is more robust than xwininfo as can be seen in this thread. The two tools DO NOT do different things as you claim, except to the extent that one succeeds at the task in conditions where the other fails. This last point is the only claim I made, and the one you somehow disagreed with. But the facts very clearly and obviously support this claim.

Last edited by Trilby (2020-12-05 17:37:45)

seth · 2020-12-06 00:01:55

And you really seem to be arguing for the sake of argument

No. I'll try this one last time and then happily agree to disagree.

"xprop _NET_WM_NAME" will print the escaped byte sequence in a non-UTF-8 environment.

Unlike xprop, xwininfo is not a tool to query specific data points, but to get a summary over several window atributes and "\344\276\215 - 347\273\264\345\237\272\347\231\276\347\247\221\357\274\214\350\207\252\347\224\261\347\232\204\347\231\276\347\247\221\345\205\250\344\271\246 - Chromium" is hardly a useful description of a window the user might be interested in
(if one btw. was only interested in the trailing class, testing WM_CLASS would be far more reliable)

There is no sane way to convert UTF-8 > 127 into ASCII, so anything like "this isn't printable" is a perfectly fine description of the window title that will leave the user less confused than a train of slashes and numbers.

The reason why xprop works for the OP while xwininfo doesn't is that they're facing a special corner case where some pretentious developer figured they needed an em-dash instead of a simple proletarian dash while the rest of the title is the 7bit range of UTF-8 where it's identical w/ ASCII and they're probably grepping for some of that. But I doubt the developers of xwininfo were concerned w/ this scenario when deciding how to handle inconvertible strings.
And if they were interested in the window title of a CJK login portal, the entire thing would fall apart w/ xprop just as much.

kyptin · 2020-12-06 19:08:41

Ah, thanks @seth and @Trilby! Indeed, when I say `LC_ALL=en_US.utf8 xwininfo`, I get the actual title, emdash and all.

Turns out I had `export LC_ALL=C` in my `.bashrc`. (I think I set this years ago when I was sorting IP addresses, because at the time the default sort order effectively stripped out the `.`s and treated them as numbers, which meant that IP addresses from the same subnet were sometimes separated by an IP address from a different subnet. Anyway...) Marking this as solved. Thanks again!

Arch Linux

#1 2020-12-05 03:18:40

[SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#2 2020-12-05 03:28:25

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#3 2020-12-05 03:34:26

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#4 2020-12-05 03:49:47

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#5 2020-12-05 04:55:43

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#6 2020-12-05 05:29:16

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#7 2020-12-05 08:38:41

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#8 2020-12-05 13:38:29

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#9 2020-12-05 13:57:25

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#10 2020-12-05 14:02:05

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#11 2020-12-05 15:46:10

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#12 2020-12-05 16:52:04

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#13 2020-12-05 16:55:13

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#14 2020-12-05 17:14:34

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#15 2020-12-05 17:25:41

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#16 2020-12-05 17:32:42

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#17 2020-12-06 00:01:55

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

#18 2020-12-06 19:08:41

Re: [SOLVED] xwininfo fails to extract titles containing non-ASCII chars

Board footer