[SOLVED]Issues with chararter decoding

Panadestein · 2017-05-18 10:34:19

Hello friends.

I'm not sure of the nature of this issue, so I decided to post it in the newbie corner.

I have currently two Arch machines (home and work), the first one with Gnome and i3wn, and the second one only with i3wm as GUI. At home, I don't have any problem with character decoding, but at my work, some applications are presenting issues with this, for example, vim displays this "~D~X" instead of a "smiley face". Also , the bumblebee-status bar, whenever I try to use a theme that have this kind of "special characters" (sorry for the ignorance, I don't know the real name) crashes with this exception:

 
Traceback (most recent call last):
  File "/home/panadestein/.config/i3/bar/bumblebee/theme.py", line 110, in load
    return json.load(data)
  File "/usr/lib/python3.6/json/__init__.py", line 296, in load
    return loads(fp.read(),
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xee in position 33: ordinal not in range(128)

This is some relevant output:

$ locale
LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

$ localectl status
   System Locale: LANG=en_US.UTF-8
       VC Keymap: n/a
      X11 Layout: n/a

$ localectl list-locales
en_US.utf8
fr_FR.utf8

Any idea of what's happening here? Thanks for your time in advance.

Last edited by Panadestein (2017-05-19 07:59:48)

ugjka · 2017-05-18 10:39:04

Idk try something else than "C"

Last edited by ugjka (2017-05-18 10:39:18)

Panadestein · 2017-05-18 12:21:38

ugjka wrote:

Idk try something else than "C"

I have tried to change this "C", but any time I regenerate locale, I got exactly the same result ("C" in all the variables)

seth · 2017-05-18 15:02:49

cat /etc/locale.conf

However, since the system locale is considered to be en_US.UTF-8, the problem is likely that you export (or unset) LC_ALL and probably LANGUAGE and maybe more LC variables in your bashrc, profile or the startup script of i3wm

Panadestein · 2017-05-18 15:08:39

seth wrote:

cat /etc/locale.conf

Here it is:

$ cat /etc/locale.conf 
LANG=en_US.UTF-8

$ cat ~/.bashrc
stty -ixon

export PS1="\[$(tput bold)\]\[$(tput setaf 1)\][\[$(tput setaf 3)\]\u\[$(tput setaf 2)\]@\[$(tput setaf 4)\]\h \[$(tput setaf 5)\]\W\[$(tput setaf 1)\]]\[$(tput setaf 7)\]\\$ \[$(tput sgr0)\]"

#Some alias

$ cat cat ~/.bash_profile
#
# ~/.bash_profile
#

[[ -f ~/.bashrc ]] && . ~/.bashrc

if [ -z "$DISPLAY" ] && [ -n "$XDG_VTNR" ] && [ "$XDG_VTNR" -eq 1 ]; then
  exec startx
fi

seth · 2017-05-18 15:29:55

What "locale" do you get in VT2 - if it's english, check your xinitrc and follow up scripts (i3 autostart)
Otherwise look into the global configs, notably /etc/profile*

And try to "LC_ALL=en_US.UTF-8 some_app_with_utf8_demand"

ugjka · 2017-05-18 15:35:09

what about /etc/locale.gen have you configured it?

Panadestein · 2017-05-18 15:52:03

I'm surprised!!!

In VT2, I'm getting this:

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

This is my .xinitrc

setxkbmap -option caps:escape
xmodmap ~/.config/Scripts/remaps

exec i3

And I have found this file in /etc/profile.d:

$ vi /etc/profile.d/locale.sh
#!/bin/sh

if [ -z "$LANG" ]; then
  if [ -n "$XDG_CONFIG_HOME" ] && [ -r "$XDG_CONFIG_HOME/locale.conf" ]; then
    . "$XDG_CONFIG_HOME/locale.conf"
  elif [ -n "$HOME" ] && [ -r "$HOME/.config/locale.conf" ]; then
    . "$HOME/.config/locale.conf"
  elif [ -r /etc/locale.conf ]; then
    . /etc/locale.conf
  fi
fi

LANG=${LANG:-C}
export LANG
[ -n "$LC_CTYPE" ]          && export LC_CTYPE
[ -n "$LC_NUMERIC" ]        && export LC_NUMERIC
[ -n "$LC_TIME" ]           && export LC_TIME
[ -n "$LC_COLLATE" ]        && export LC_COLLATE
[ -n "$LC_MONETARY" ]       && export LC_MONETARY
[ -n "$LC_MESSAGES" ]       && export LC_MESSAGES
[ -n "$LC_PAPER" ]          && export LC_PAPER
[ -n "$LC_NAME" ]           && export LC_NAME
[ -n "$LC_ADDRESS" ]        && export LC_ADDRESS
[ -n "$LC_TELEPHONE" ]      && export LC_TELEPHONE
[ -n "$LC_MEASUREMENT" ]    && export LC_MEASUREMENT
[ -n "$LC_IDENTIFICATION" ] && export LC_IDENTIFICATION

I think this is the problem here. But, why this script is executed in the first place, if I don't call it in any of my local configuration files?

Edit: I have also found this:

$ cat /etc/profile.d/vte.sh
# Not bash or zsh?
[ -n "$BASH_VERSION" -o -n "$ZSH_VERSION" ] || return 0

# Not an interactive shell?
[[ $- == *i* ]] || return 0

# Not running under vte?
[ "${VTE_VERSION:-0}" -ge 3405 ] || return 0

__vte_urlencode() (
  # This is important to make sure string manipulation is handled
  # byte-by-byte.
  LC_ALL=C
  str="$1"
  while [ -n "$str" ]; do
    safe="${str%%[!a-zA-Z0-9/:_\.\-\!\'\(\)~]*}"
    printf "%s" "$safe"
    str="${str#"$safe"}"
    if [ -n "$str" ]; then
      printf "%%%02X" "'$str"
      str="${str#?}"
    fi
  done
)

# Print a warning so that anyone who's added this manually to his PS1 can adapt.
# The function will be removed in a later version.
__vte_ps1() {
  echo -n "(__vte_ps1 is obsolete)"
}

__vte_osc7 () {
  printf "\033]7;file://%s%s\007" "${HOSTNAME:-}" "$(__vte_urlencode "${PWD}")"
}

__vte_prompt_command() {
  local pwd='~'
  [ "$PWD" != "$HOME" ] && pwd=${PWD/#$HOME\//\~\/}
  printf "\033]0;%s@%s:%s\007%s" "${USER}" "${HOSTNAME%%.*}" "${pwd}" "$(__vte_osc7)"
}

case "$TERM" in
  xterm*|vte*)
    [ -n "$BASH_VERSION" ] && PROMPT_COMMAND="__vte_prompt_command"
    [ -n "$ZSH_VERSION"  ] && precmd_functions+=(__vte_osc7)
    ;;
esac

true

which assigns the LC_ALL variable.

Last edited by Panadestein (2017-05-18 16:03:22)

Panadestein · 2017-05-18 15:56:29

ugjka wrote:

what about /etc/locale.gen have you configured it?

Yes, I uncommented en_US.UTF-8 UTF-8 and fr_FR.UTF-8 UTF-8. Then executed locale-gen.

seth · 2017-05-18 16:27:46

/etc/profile.d/vte.sh is a "gnome" (VTE terminal emulator class, used by some gtk terminals) thing, but the LC_ALL variable isn't exported and except for a shell bug should constrain to the __vte_urlencode function.

Replace "exec i3" with "xterm", then with your preferred terminal emulator and see what happens.

Also, though probably unrelated, see the 2nd note in https://wiki.archlinux.org/index.php/Xinit#xinitrc

Panadestein · 2017-05-19 07:58:53

seth wrote:

Also, though probably unrelated, see the 2nd note in https://wiki.archlinux.org/index.php/Xinit#xinitrc

My friend, this was totally related! With this .xinitrc

if [ -d /etc/X11/xinit/xinitrc.d ]; then
  for f in /etc/X11/xinit/xinitrc.d/*; do
    [ -x "$f" ] && . "$f"
  done
  unset f
fi

exec i3

everything works just perfectly! As always, the Arch Wiki was totally right. But maybe this 2nd note should be expressed with more strength, i.e, if you don't add this 'if' block, then things can misbehave.

Thanks for your help.

Arch Linux

#1 2017-05-18 10:34:19

[SOLVED]Issues with chararter decoding

#2 2017-05-18 10:39:04

Re: [SOLVED]Issues with chararter decoding

#3 2017-05-18 12:21:38

Re: [SOLVED]Issues with chararter decoding

#4 2017-05-18 15:02:49

Re: [SOLVED]Issues with chararter decoding

#5 2017-05-18 15:08:39

Re: [SOLVED]Issues with chararter decoding

#6 2017-05-18 15:29:55

Re: [SOLVED]Issues with chararter decoding

#7 2017-05-18 15:35:09

Re: [SOLVED]Issues with chararter decoding

#8 2017-05-18 15:52:03

Re: [SOLVED]Issues with chararter decoding

#9 2017-05-18 15:56:29

Re: [SOLVED]Issues with chararter decoding

#10 2017-05-18 16:27:46

Re: [SOLVED]Issues with chararter decoding

#11 2017-05-19 07:58:53

Re: [SOLVED]Issues with chararter decoding

Board footer