[SOLVED] Arch & KDE special characters encoding problem

Celeano · 2023-01-13 10:38:51

Hi

I use Arch and KDE Plasma and I have a weird behavior of the special characters shown e.g in the terminal.

When I open a terminal and type a German accent/umlaut (like ä, ü, ö) these are shown correctly.

But if I create a file with an accent like menü and then do a ls to list the files the name is shown as 'men'$'\303\274'
In the file managers like dolphin or konqueror the filename is shown correctly.

These is what I did:
I added english and german as system languages in the kde plasma settings.
The I adjusted the file /etc/locale.conf and ran locale-gen:

locale:

[phil@arch-2700x tmp]$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME=en_CH.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

cat /etc/locale.conf:

[phil@arch-2700x tmp]$ cat /etc/locale.conf 
LANG=en_US.UTF-8:ISO-8859-1

I also tried without the ISO fallback with no effect.

This is the output of localectl status:

[phil@arch-2700x tmp]$ localectl status
System Locale: LANG=en_US.UTF-8:ISO-8859-1
    VC Keymap: (unset)                    
   X11 Layout: (unset)

Of course I did always logut/login or even reboot after I changes the settings or config files.

I also recognized that some programs are crashing when opening a file dialog where some special characters are present.
Any help is appreciated. Thanks!

Last edited by Celeano (2023-01-13 20:27:42)

seth · 2023-01-13 12:03:18

But if I create a file with an accent like menü""

Create "how" and where and did you mean the quotes to be part of the filename?

"\303\274" is the octal representation of the utf-8 codepoint 252: "ü"

touch /tmp/testä
ls /tmp/test*

Celeano · 2023-01-13 12:17:25

Hi seth

Sorry for the extra quotes. I removed them in my question.
I used touch to create the file menü in the folder /tmp as your example.

your example:

[phil@arch-2700x tmp]$ touch /tmp/testä
[phil@arch-2700x tmp]$ ls /tmp/test*
'/tmp/test'$'\303\244'

It's clear to me that this is an octal representation of ü. I'm wondering about what of my config could be wrong. since the terminal shows the ü when I type it.

Some programs are also struggling with special characters. E.g. when using unrar to extract an archive. if the archive contains a file which has a special character, it truncates the filename at the occurrence of this special character.

dimich · 2023-01-13 12:22:48

Celeano wrote:

The I adjusted the file /etc/locale.conf and ran locale-gen:

What lines are uncommented in your /etc/locale.gen?

$ grep -v "^#" /etc/locale.gen

Celeano wrote:

But if I create a file with an accent like menü and then do a ls to list the files the name is shown as 'men'$'\303\274'

Try

$ LANG=C ls

seth · 2023-01-13 12:24:36

Also let's see the locale of the terminal process (in case the locale is fixed by your shell but the terminal is unable to render it)

tr '\0' '\n' < /proc/$PPID/environ

Celeano · 2023-01-13 12:29:43

dimich wrote:

What lines are uncommented in your /etc/locale.gen?

Here are the lines:

de_CH.UTF-8 UTF-8  
en_US.UTF-8 UTF-8  
en_US ISO-8859-1

This has no effect.

dimich wrote:

$ LANG=C ls

Celeano · 2023-01-13 12:31:22

seth wrote:

Also let's see the locale of the terminal process (in case the locale is fixed by your shell but the terminal is unable to render it)
tr '\0' '\n' < /proc/$PPID/environ

That's the output:

[phil@arch-2700x tmp]$ tr '\0' '\n' < /proc/$PPID/environ
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
DEBUGINFOD_URLS=https://debuginfod.archlinux.org
DESKTOP_SESSION=plasma
DISPLAY=:0
GTK2_RC_FILES=/etc/gtk-2.0/gtkrc:/home/phil/.gtkrc-2.0:/home/phil/.config/gtkrc-2.0
GTK_MODULES=canberra-gtk-module
GTK_RC_FILES=/etc/gtk/gtkrc:/home/phil/.gtkrc:/home/phil/.config/gtkrc
HOME=/home/phil
INVOCATION_ID=360e552eab904ce4962d6975ad32e39b
JOURNAL_STREAM=8:66684
KDE_APPLICATIONS_AS_SCOPE=1
KDE_FULL_SESSION=true
KDE_SESSION_UID=1000
KDE_SESSION_VERSION=5
LANG=en_US.UTF-8
LANGUAGE=en_US:de
LC_TIME=en_CH.UTF-8
LOGNAME=phil
MAIL=/var/spool/mail/phil
MANAGERPID=978
MOTD_SHOWN=pam
PAM_KWALLET5_LOGIN=/run/user/1000/kwallet5.socket
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
PWD=/home/phil
QT_AUTO_SCREEN_SCALE_FACTOR=0
QT_SCREEN_SCALE_FACTORS=DP-1=1.25;DP-2=1.25;HDMI-1=1.25;HDMI-2=1.25;DVI-D-1=1.25;
SHELL=/bin/bash
SHLVL=0
SYSTEMD_EXEC_PID=4925
USER=phil
XAUTHORITY=/home/phil/.Xauthority
XCURSOR_SIZE=24
XCURSOR_THEME=breeze_cursors
XDG_CONFIG_DIRS=/home/phil/.config/kdedefaults:/etc/xdg
XDG_CURRENT_DESKTOP=KDE
XDG_RUNTIME_DIR=/run/user/1000
XDG_SEAT=seat0
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
XDG_SESSION_CLASS=user
XDG_SESSION_DESKTOP=KDE
XDG_SESSION_ID=7
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session5
XDG_SESSION_TYPE=x11
XDG_VTNR=1
DESKTOP_STARTUP_ID=arch-2700x;1673602843;975492;5650_TIME478708

seth · 2023-01-13 12:38:53

Ftr:

LC_TIME=en_CH.UTF-8

this isn't a valid locale (though I don't see how this would break your filenames)

Your interactive shell is bash, too?

echo $SHELL

Do you have the same problems outside a KDE session? (ie. ctrl+alt+F2, log into the console and try the behavior there)

LANGUAGE=en_US:de

is probably added by KDE.
Weird case scenario: KDE picks up that, maps en_US to en_US ISO-8859-1 and somehow uses that somewhere somehow (while this makes no sense, you'd get some "Ã¼" for it)
=> comment en_US ISO-8859-1 and re-generate the locale and re-login.
Though: weird. It'S almost more likely that KDE get's thrown off by the invalid LC_TIME.
Iff this is limited to KDE.

Celeano · 2023-01-13 12:52:13

seth wrote:

Ftr:
LC_TIME=en_CH.UTF-8
this isn't a valid locale (though I don't see how this would break your filenames)

how to change it? or does it make no sense to change it?

seth wrote:

Your interactive shell is bash, too?
echo $SHELL

yes : /bin/bash

seth wrote:

Do you have the same problems outside a KDE session? (ie. ctrl+alt+F2, log into the console and try the behavior there)

Yes the same issue outside the KDE session.

seth wrote:

LANGUAGE=en_US:de
is probably added by KDE.
Weird case scenario: KDE picks up that, maps en_US to en_US ISO-8859-1 and somehow uses that somewhere somehow (while this makes no sense, you'd get some "Ã¼" for it)
=> comment en_US ISO-8859-1 and re-generate the locale and re-login.
Though: weird. It'S almost more likely that KDE get's thrown off by the invalid LC_TIME.
Iff this is limited to KDE.

after regenerating without iso it's still:

LANGUAGE=en_US:de

Shall I try removing German from the country/language settings in KDE?

seth · 2023-01-13 13:13:05

how to change it? or does it make no sense to change it?

Where do you set it? (apparently not /etc/locale.conf)
Does it show up in the "locale" output in the console login?

Yes the same issue outside the KDE session.

So at least we don't have to focus on KDE.

after regenerating without iso it's still:

That's expectable - I just wanted the ISO-8859-1 locale out of sight.

Celeano · 2023-01-13 13:25:55

seth wrote:

Where do you set it? (apparently not /etc/locale.conf)

I have no clue where it's set :-(

seth wrote:

Does it show up in the "locale" output in the console login?

No. this is the output in the console login of locale:

LANG=en_US.UTF-8:ISO-8859-1
LC_CTYPE="en_US.UTF-8:ISO-8859-1"
LC_NUMERIC="en_US.UTF-8:ISO-8859-1"
LC_TIME="en_US.UTF-8:ISO-8859-1"
LC_COLLATE="en_US.UTF-8:ISO-8859-1"
LC_MONETARY="en_US.UTF-8:ISO-8859-1"
LC_MESSAGES="en_US.UTF-8:ISO-8859-1"
LC_PAPER="en_US.UTF-8:ISO-8859-1"
LC_NAME="en_US.UTF-8:ISO-8859-1"
LC_ADDRESS="en_US.UTF-8:ISO-8859-1"
LC_TELEPHONE="en_US.UTF-8:ISO-8859-1"
LC_MEASUREMENT="en_US.UTF-8:ISO-8859-1"
LC_IDENTIFICATION="en_US.UTF-8:ISO-8859-1"
LC_ALL=

seth · 2023-01-13 13:30:45

So probably KDE and probably irrelevant and

LANG=en_US.UTF-8:ISO-8859-1

is garbage.

'kinda missed that

cat /etc/locale.conf 
LANG=en_US.UTF-8:ISO-8859-1

make that

LANG=en_US.UTF-8

Celeano · 2023-01-13 13:40:13

seth wrote:

make that
LANG=en_US.UTF-8

Done, but no difference.

I removed German form the list in the Region & Language setting in KDE but LC-Time is still en_CH.UTF-8

seth · 2023-01-13 13:41:25

Done, but no difference.

Did you reboot or at least re-login?

Celeano · 2023-01-13 13:47:12

seth wrote:

Done, but no difference.
Did you reboot or at least re-login?

I always did at least a re-login. At the last attempt I rebooted.

seth · 2023-01-13 13:49:57

Updated "locale" output (in the console, we ignore KDE for the moment)

Celeano · 2023-01-13 13:54:09

seth wrote:

Updated "locale" output (in the console, we ignore KDE for the moment)

Ah sorry, didn't recheck on the console.
locale is:

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

and now it's working on the console.
so it must be a KDE problem then I guess.

seth · 2023-01-13 14:14:25

and now it's working on the console.

"locale" output in KDE?

Celeano · 2023-01-13 14:20:45

seth wrote:

and now it's working on the console.
"locale" output in KDE?

locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME=en_CH.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

looks like this is the difference:
LC_TIME=en_CH.UTF-8

Celeano · 2023-01-13 14:24:19

FUnny thing. If I comment de de_CH line in /etc/locale.gen

regenrate and re-login it is:

LC_TIME=en_CH.UTF-8

seems the generation is messed up.

V1del · 2023-01-13 14:29:59

The generation of the locale has no effect or relevance to the values exported in the environment (... only to the point of, if you do have a reference to a value in the environment you should have the accompanying locale generated as well).

That it is incorrectly exported is likely a peculiarty of KDEs current locale system and you'll likely find reference to the faulty value in ~/.config/plasma-localerc where you could relevantly correct it.

Afaik Qt "knows" more locale combinations than plain glibc and KDE currently exports/shows those as is and can then write values that do not make sense to glibc to their locale file. AFAIK work is underway to fix this, not sure as to the current state.

Last edited by V1del (2023-01-13 14:32:47)

seth · 2023-01-13 14:32:32

In addition

locale: Cannot set LC_ALL to default locale: No such file or directory

locale -a

?

Celeano · 2023-01-13 14:33:41

seth wrote:

In addition
locale: Cannot set LC_ALL to default locale: No such file or directory
locale -a
?

locale -a
C
C.UTF-8
en_US.utf8
POSIX

Celeano · 2023-01-13 14:36:22

V1del wrote:

The generation of the locale has no effect or relevance to the values exported in the environment (... only to the point of, if you do have a reference to a value in the environment you should have the accompanying locale generated as well).
That it is incorrectly exported is likely a peculiarty of KDEs current locale system and you'll likely find reference to the faulty value in ~/.config/plasma-localerc where you could relevantly correct it.
Afaik Qt "knows" more locale combinations than plain glibc and KDE currently exports/shows those as is and can then write values that do not make sense to glibc to their locale file. AFAIK work is underway to fix this, not sure as to the current state.

I changed the LC_TIME there to en_US.UTF-8 and it works.
That's really unexpected to me. But I don't know KDE very well.

Anyway thansk very much for your help seth and V1del !!!

seth · 2023-01-13 14:50:20

\o/

Please always remember to mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

Arch Linux

#1 2023-01-13 10:38:51

[SOLVED] Arch & KDE special characters encoding problem

#2 2023-01-13 12:03:18

Re: [SOLVED] Arch & KDE special characters encoding problem

#3 2023-01-13 12:17:25

Re: [SOLVED] Arch & KDE special characters encoding problem

#4 2023-01-13 12:22:48

Re: [SOLVED] Arch & KDE special characters encoding problem

#5 2023-01-13 12:24:36

Re: [SOLVED] Arch & KDE special characters encoding problem

#6 2023-01-13 12:29:43

Re: [SOLVED] Arch & KDE special characters encoding problem

#7 2023-01-13 12:31:22

Re: [SOLVED] Arch & KDE special characters encoding problem

#8 2023-01-13 12:38:53

Re: [SOLVED] Arch & KDE special characters encoding problem

#9 2023-01-13 12:52:13

Re: [SOLVED] Arch & KDE special characters encoding problem

#10 2023-01-13 13:13:05

Re: [SOLVED] Arch & KDE special characters encoding problem

#11 2023-01-13 13:25:55

Re: [SOLVED] Arch & KDE special characters encoding problem

#12 2023-01-13 13:30:45

Re: [SOLVED] Arch & KDE special characters encoding problem

#13 2023-01-13 13:40:13

Re: [SOLVED] Arch & KDE special characters encoding problem

#14 2023-01-13 13:41:25

Re: [SOLVED] Arch & KDE special characters encoding problem

#15 2023-01-13 13:47:12

Re: [SOLVED] Arch & KDE special characters encoding problem

#16 2023-01-13 13:49:57

Re: [SOLVED] Arch & KDE special characters encoding problem

#17 2023-01-13 13:54:09

Re: [SOLVED] Arch & KDE special characters encoding problem

#18 2023-01-13 14:14:25

Re: [SOLVED] Arch & KDE special characters encoding problem

#19 2023-01-13 14:20:45

Re: [SOLVED] Arch & KDE special characters encoding problem

#20 2023-01-13 14:24:19

Re: [SOLVED] Arch & KDE special characters encoding problem

#21 2023-01-13 14:29:59

Re: [SOLVED] Arch & KDE special characters encoding problem

#22 2023-01-13 14:32:32

Re: [SOLVED] Arch & KDE special characters encoding problem

#23 2023-01-13 14:33:41

Re: [SOLVED] Arch & KDE special characters encoding problem

#24 2023-01-13 14:36:22

Re: [SOLVED] Arch & KDE special characters encoding problem

#25 2023-01-13 14:50:20

Re: [SOLVED] Arch & KDE special characters encoding problem

Board footer