[SOLVED] System-wide character encoding issue

johan.cb · 2017-08-19 10:04:36

Hello,

I installed Arch on my home and office computers a few days ago and am facing a character encoding issue on both. I am using two different physical keyboard layouts, French and German with following issues:

German characters can be entered withtout any problem with the German keyboard layout, but when I try to enter French characters I only get replacement characters (question mark in a square) for non-ASCII characters
French characters can be entered withtout any problem with the French keyboard layout, but when I try to enter German characters I only get replacement characters (question mark in a square) for non-ASCII characters

In addition to that, I can not access files and folders created under previous Arch installs which contain non-ASCII characters in their path or filename like é,è,ß and so on.

System information:
Up to date Arch Linux using the KDE Plasma desktop environment.

locale

LANG=fr_FR.UTF-8
LC_CTYPE=
LC_NUMERIC=
LC_TIME=
LC_COLLATE=
LC_MONETARY=
LC_MESSAGES=
LC_PAPER=
LC_NAME=
LC_ADDRESS=
LC_TELEPHONE=
LC_MEASUREMENT=
LC_IDENTIFICATION=
LC_ALL=

locale -a

C
fr_FR.utf8
POSIX

/etc/locale.gen

...
#fr_CH.UTF-8 UTF-8
#fr_CH ISO-8859-1
fr_FR.UTF-8 UTF-8
#fr_FR ISO-8859-1
#fr_FR@euro ISO-8859-15
...

/etc/locale.conf

LANG=fr_FR.UTF-8

I am running out of ideas on how to solve this issue.

Johan

Last edited by johan.cb (2017-08-31 19:50:23)

ugjka · 2017-08-19 10:35:28

You are experiencing this bug

https://bugs.archlinux.org/task/54988
https://github.com/systemd/systemd/issues/6407

johan.cb · 2017-08-19 12:59:40

Is there any solution/workaround to this bug? Or do I have to wait for the next systemd release?

I am wondering why there aren't more people affected by the bug.

loqs · 2017-08-19 13:09:09

See https://bbs.archlinux.org/viewtopic.php … 8#p1728478 for a possible work around.

ugjka · 2017-08-19 13:15:43

Or simply set/override all variables with the exact configuration you want

johan.cb · 2017-08-20 11:40:26

My locale now looks like this:

LANG=fr_FR.UTF-8
LC_CTYPE=fr_FR.UTF-8
LC_NUMERIC=
LC_TIME=
LC_COLLATE=
LC_MONETARY=
LC_MESSAGES=
LC_PAPER=
LC_NAME=
LC_ADDRESS=
LC_TELEPHONE=
LC_MEASUREMENT=
LC_IDENTIFICATION=
LC_ALL=

But I still have the encoding issue.

loqs · 2017-08-20 15:03:37

If you are going to only set one of the LC vars set LC_ALL.

johan.cb · 2017-08-20 15:07:27

I just tried that, LC_ALL shows as

LC_ALL=fr_FR.UTF-8

but I am still affected by the issue.

I noticed that I am not experiencing this bug under LibreOffice or Firefox.

Last edited by johan.cb (2017-08-20 15:08:25)

loqs · 2017-08-20 15:15:12

You are setting the vars in /etc/locale.conf then rebooted or at least switched to a new tty?
Also how do you start X/Wayland?

Last edited by loqs (2017-08-20 15:17:16)

johan.cb · 2017-08-20 15:16:46

Yes, exactly.

My actual locale.conf after reboot:

LANG=fr_FR.UTF-8
LC_CTYPE=fr_FR.UTF-8
LC_NUMERIC=fr_FR.UTF-8
LC_TIME=fr_FR.UTF-8
LC_COLLATE=fr_FR.UTF-8
LC_MONETARY=fr_FR.UTF-8
LC_MESSAGES=fr_FR.UTF-8
LC_PAPER=fr_FR.UTF-8
LC_NAME=fr_FR.UTF-8
LC_ADDRESS=fr_FR.UTF-8
LC_TELEPHONE=fr_FR.UTF-8
LC_MEASUREMENT=fr_FR.UTF-8
LC_IDENTIFICATION=fr_FR.UTF-8
LC_ALL=

loqs · 2017-08-20 15:23:25

When you say you can not access files and folders created under previous Arch installs which contain non-ASCII characters in their path or filename like é,è,ß and so on
is that because you can not type the path or the path is not displayed correctly? In what program is this?
Edit:
spelling correctly not correctily
Edit2:
@ugjka
Are you sure this is https://bugs.archlinux.org/task/54988?

Last edited by loqs (2017-08-20 15:31:59)

johan.cb · 2017-08-20 15:40:09

Examples:
-when I donwload a file into my folder /home/johan/Téléchargements/ using Firefox and try to open it, Firefox tells me that the downloaded file does not exist. If I download it, or move it, to a folder named /home/johan/Downloads it works
-using the Dolphin file manager I can not create a new folder using é,è,ß, I only get question marks
-using the Konsole terminal emulator I can list folder with non ASCII characters which display fine, bud can not

cd

to the folder Téléchargements, I get the ? character instead.

Maybe a screenshot is worth more than a thousand words:

http://imgur.com/a/RJMI1

johan.cb · 2017-08-20 15:49:43

What's curious is that I can write the character "é" with a French keyboard but not with the german or the american anymore. Same for german characters that can be entered with the german keyboard but not the french anymore.

ugjka · 2017-08-20 16:41:06

Guessing here, try also uncomenting fr_FR ISO-8859-1 fr_FR@euro ISO-8859-15 in your locale.gen and run locale-gen

johan.cb · 2017-08-20 16:45:06

Helo ugjka,

it did not work.

ugjka · 2017-08-20 16:53:09

post the output of "env", redact anything sensitive if you have something

johan.cb · 2017-08-20 16:56:26

env:

johan@Johan-PC : env
GS_LIB=/home/johan/.fonts
KDE_FULL_SESSION=true
LC_MEASUREMENT=fr_FR.UTF-8
LC_PAPER=fr_FR.UTF-8
LC_MONETARY=fr_FR.UTF-8
LANG=fr_FR.UTF-8
PROFILEHOME=
DISPLAY=:0
SHELL_SESSION_ID=608a2e88948a41a59f6c46a411b8b0e2
INVOCATION_ID=10999ca115a54bd1a8ef402c3eb83482
COLORTERM=truecolor
MOZ_PLUGIN_PATH=/usr/lib/mozilla/plugins
XDG_VTNR=1
QT_QUICK_CONTROLS_STYLE=org.kde.desktop
LC_NAME=fr_FR.UTF-8
XDG_SESSION_ID=c1
USER=johan
LC_COLLATE=fr_FR.UTF-8
PWD=/home/johan
HOME=/home/johan
LC_CTYPE=fr_FR.UTF-8
JOURNAL_STREAM=9:16684
XCURSOR_SIZE=0
XDG_DATA_DIRS=/usr/share:/usr/share:/usr/local/share
KONSOLE_DBUS_SESSION=/Sessions/1
LC_ADDRESS=fr_FR.UTF-8
LC_NUMERIC=fr_FR.UTF-8
KONSOLE_DBUS_WINDOW=/Windows/1
MAIL=/var/spool/mail/johan
WINDOWPATH=1
TERM=xterm-256color
SHELL=/bin/bash
KONSOLE_DBUS_SERVICE=:1.45
LC_MESSAGES=fr_FR.UTF-8
XCURSOR_THEME=breeze_cursors
XDG_CURRENT_DESKTOP=KDE
KONSOLE_PROFILE_NAME=Nouveau profil
XDG_SEAT=seat0
SHLVL=4
COLORFGBG=15;0
LANGUAGE=
WINDOWID=58720262
LC_TELEPHONE=fr_FR.UTF-8
LOGNAME=johan
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
XDG_RUNTIME_DIR=/run/user/1000
XAUTHORITY=/home/johan/.Xauthority
QT_AUTO_SCREEN_SCALE_FACTOR=0
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl
LC_IDENTIFICATION=fr_FR.UTF-8
KDE_SESSION_UID=1000
PS1=\[\e[32m\]\u\[\e[m\]@\[\e[31m\]\h\[\e[m\] :
KDE_SESSION_VERSION=5
SESSION_MANAGER=local/Johan-PC:@/tmp/.ICE-unix/548,unix/Johan-PC:/tmp/.ICE-unix/548
LC_TIME=fr_FR.UTF-8
_=/usr/bin/env

ugjka · 2017-08-20 16:58:14

Ok try setting LANGUAGE= to fr_FR.UTF-8 too in your locale.conf

johan.cb · 2017-08-20 17:06:24

Still no difference.

johan.cb · 2017-08-20 17:08:04

I am wondering if this is not KDE Plasma/Qt related, all GTK applications seem to work properly: Firefox, LibreOffice, Gimp

ugjka · 2017-08-20 17:09:01

Ok just for sanity what does localectl say, you have something weird going on

johan.cb · 2017-08-20 17:09:42

Output of localectl:

System Locale: LANG=fr_FR.UTF-8
VC Keymap: n/a
X11 Layout: n/a

ugjka · 2017-08-20 17:10:13

johan.cb wrote:

I am wondering if this is not KDE Plasma/Qt related, all GTK applications seem to work properly: Firefox, LibreOffice, Gimp

Check KDE's Regional Settings in the kde's settings manager

ugjka · 2017-08-20 17:11:37

johan.cb wrote:

Output of localectl:
System Locale: LANG=fr_FR.UTF-8
VC Keymap: n/a
X11 Layout: n/a

That could be the problem, you need to have all those others there too

mine looks like this

[ugjka@ArchT bin]$ localectl 
   System Locale: LANG=en_GB.UTF-8
                  LC_CTYPE=lv_LV.UTF-8
                  LC_NUMERIC=lv_LV.UTF-8
                  LC_TIME=lv_LV.UTF-8
                  LC_COLLATE=lv_LV.UTF-8
                  LC_MONETARY=lv_LV.UTF-8
                  LC_MESSAGES=lv_LV.UTF-8
                  LC_PAPER=lv_LV.UTF-8
                  LC_NAME=lv_LV.UTF-8
                  LC_ADDRESS=lv_LV.UTF-8
                  LC_TELEPHONE=lv_LV.UTF-8
                  LC_MEASUREMENT=lv_LV.UTF-8
                  LC_IDENTIFICATION=lv_LV.UTF-8
       VC Keymap: dvorak
      X11 Layout: us,lv,lv,ru
     X11 Variant: dvorak,minuskeydvorak,apostrophe,phonetic
     X11 Options: grp:alt_shift_toggle

johan.cb · 2017-08-20 17:17:44

Do you know how to do this? I already appended fr_FR.UTF-8 to all the variables in locale.conf file and ran locale-gen. But localectl shows non of them.

Arch Linux

#1 2017-08-19 10:04:36

[SOLVED] System-wide character encoding issue

#2 2017-08-19 10:35:28

Re: [SOLVED] System-wide character encoding issue

#3 2017-08-19 12:59:40

Re: [SOLVED] System-wide character encoding issue

#4 2017-08-19 13:09:09

Re: [SOLVED] System-wide character encoding issue

#5 2017-08-19 13:15:43

Re: [SOLVED] System-wide character encoding issue

#6 2017-08-20 11:40:26

Re: [SOLVED] System-wide character encoding issue

#7 2017-08-20 15:03:37

Re: [SOLVED] System-wide character encoding issue

#8 2017-08-20 15:07:27

Re: [SOLVED] System-wide character encoding issue

#9 2017-08-20 15:15:12

Re: [SOLVED] System-wide character encoding issue

#10 2017-08-20 15:16:46

Re: [SOLVED] System-wide character encoding issue

#11 2017-08-20 15:23:25

Re: [SOLVED] System-wide character encoding issue

#12 2017-08-20 15:40:09

Re: [SOLVED] System-wide character encoding issue

#13 2017-08-20 15:49:43

Re: [SOLVED] System-wide character encoding issue

#14 2017-08-20 16:41:06

Re: [SOLVED] System-wide character encoding issue

#15 2017-08-20 16:45:06

Re: [SOLVED] System-wide character encoding issue

#16 2017-08-20 16:53:09

Re: [SOLVED] System-wide character encoding issue

#17 2017-08-20 16:56:26

Re: [SOLVED] System-wide character encoding issue

#18 2017-08-20 16:58:14

Re: [SOLVED] System-wide character encoding issue

#19 2017-08-20 17:06:24

Re: [SOLVED] System-wide character encoding issue

#20 2017-08-20 17:08:04

Re: [SOLVED] System-wide character encoding issue

#21 2017-08-20 17:09:01

Re: [SOLVED] System-wide character encoding issue

#22 2017-08-20 17:09:42

Re: [SOLVED] System-wide character encoding issue

#23 2017-08-20 17:10:13

Re: [SOLVED] System-wide character encoding issue

#24 2017-08-20 17:11:37

Re: [SOLVED] System-wide character encoding issue

#25 2017-08-20 17:17:44

Re: [SOLVED] System-wide character encoding issue

Board footer