Arch Linux

smoneck

Member

Registered: 2014-05-14

Posts: 105

Can't type UTF-8 characters in terminal

Non-ascii characters are directly being translated to question marks in terminal and python print fails with nonascii as well.

E.g.

echo -e "\xE2\x98\xA0"
☠

but

echo -n ☠ | hexdump
is being translated to
echo -n ??? | hexdump

python -c "print(\"\xE2\x98\xA0\")"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

cat UTF-8-demo.txt (www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-demo.txt) works fine.

cat /etc/locale.conf

LC_CTYPE=en_US.UTF-8
LC_TIME=de_DE.UTF-8

locale -a

C
POSIX
de_DE.utf8
en_US.utf8

I don't know where to start. Thanks for any help! It's quite annoying to be limited to print(item.encode('ascii')) ...

dockland

Member

From: Sweden

Registered: 2015-06-06

Posts: 861

Re: Can't type UTF-8 characters in terminal

What is your current font and what is stated in vconsole.conf

---

I possess a device, in my pocket, that is capable of accessing the entirety of information known to man.
I use it to look at funny pictures of cats and to argue with strangers.

smoneck

Member

Registered: 2014-05-14

Posts: 105

Re: Can't type UTF-8 characters in terminal

cat /etc/vconsole.conf

KEYMAP=us-acentos

I'm using Termite what I configured to:

cat ~/.config/termite

...
font = DejaVu Sans Mono 16
...

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 78,881

Re: Can't type UTF-8 characters in terminal

https://wiki.archlinux.org/index.php/In … ide#Locale

How did you install archlinux?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

smoneck

Member

Registered: 2014-05-14

Posts: 105

Re: Can't type UTF-8 characters in terminal

The standard wiki-way. Once upon a time there was a good reason to switch to us-acentos. I don't fully remember but it was something with German vowels I think ... Nevertheless, neither with de-latin1 it's working.

loqs

Member

Registered: 2014-03-06

Posts: 19,114

Re: Can't type UTF-8 characters in terminal

$ unset LANG
$ python -c "print(\"\xE2\x98\xA0\")"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
$ LANG=en_US.UTF-8 python -c "print(\"\xE2\x98\xA0\")"
☠

smoneck

Member

Registered: 2014-05-14

Posts: 105

Re: Can't type UTF-8 characters in terminal

python -c "print(\"\xE2\x98\xA0\")"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

LANG=en_US.UTF-8

python -c "print(\"\xE2\x98\xA0\")"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

loqs

Member

Registered: 2014-03-06

Posts: 19,114

Re: Can't type UTF-8 characters in terminal

$ unset LANG
$ printenv LANG
$ LANG=en_US.utf-8
$ printenv LANG
$ export LANG=en_US.utf-8
$ printenv LANG
en_US.utf-8

smoneck

Member

Registered: 2014-05-14

Posts: 105

Re: Can't type UTF-8 characters in terminal

~/ $ unset LANG
~/ $ printenv LANG 
~/ $ LANG=en_US.utf-8           
~/ $ printenv LANG
en_US.utf-8
~/ $ export LANG=en_US.utf-8
~/ $ printenv LANG
en_US.utf-8
~/ $ python -c "print(\"\xE2\x98\xA0\")" 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

Last edited by smoneck (2017-06-27 21:07:58)

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 78,881

Re: Can't type UTF-8 characters in terminal

unset LANG
printenv LANG 
LANG=en_US.utf-8           
printenv LANG

en_US.utf-8

I'm not entirely sure how to read this, but it doesn't seem to make sense. Notably the blank lines are rather worrysome.
Wild guess:

echo $SHELL

And please fix your  /etc/locale.conf regardless.

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

smoneck

Member

Registered: 2014-05-14

Posts: 105

Re: Can't type UTF-8 characters in terminal

I'm sorry, the blank lines were intended to make it more readable. I edited my post above.

In /etc/locale.conf I tried out

LC_CTYPE=en_US.UTF-8

yesterday since I read that it would be preferable over LANG for some reasons I already forgot. I now switched it back to:

LANG=en_US.UTF-8

Anyway

~/ $ echo $SHELL
/usr/bin/zsh

and thank you both for the help!

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 78,881

Re: Can't type UTF-8 characters in terminal

LC_CTYPE is preferably set to C, LANG is kind of "mandatory" and reasonably some UTF-8 variant.

Simply assigning a variable should not get it into the environment, though - do you have something like OMZ installed?
Did you try the behavior on bash? Best change your login shell for this:

chsh -s /bin/bash

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

smoneck

Member

Registered: 2014-05-14

Posts: 105

Re: Can't type UTF-8 characters in terminal

Yes, I have omzsh running. As well as bash-it on bash.

On bash:

○ → locale -a
C
POSIX
de_DE.utf8
en_US.utf8
○ → python -c "print(\"\xE2\x98\xA0\")"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)
○ → printenv LANG
C
○ → echo $LANG
C
○ → export LANG=en_US.utf-8
○ → python -c "print(\"\xE2\x98\xA0\")"
â˜

seth

Member

From: Won't reply 2 private help req

Registered: 2012-09-03

Posts: 78,881

Re: Can't type UTF-8 characters in terminal

Yes, I have omzsh running.

Jason will come around and round you up in
3 .. 2 .. 1 ...

;-)

Ie. get rid of that and try again. It's responsible for about 120% of all zsh problems.

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

jasonwryan

Anarchist

From: .nz

Registered: 2009-05-09

Posts: 30,426

Website

Re: Can't type UTF-8 characters in terminal

I picked it in post #11...

---

Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

smoneck

Member

Registered: 2014-05-14

Posts: 105

Re: Can't type UTF-8 characters in terminal

Indeed, without omzsh it also works:

% echo $LANG
C
% export LANG=en_US.utf-8
% python -c "print(\"\xE2\x98\xA0\")"
â˜

But what can I do against the fallback to LANG=C?

smoneck

Member

Registered: 2014-05-14

Posts: 105

Re: Can't type UTF-8 characters in terminal

Strange. Having omzsh uninstalled and rebooted, LANG is now en_US.utf-8, how it's supposed to be. I rebooted prior to every post and it always fell back to C.

I found that in the syslog:

Locale not supported by C library.
Using the fallback 'C' locale.

I will try everything out again tomorrow. I'm kind of suspicious that uninstalling omzsh did it ...

Last edited by smoneck (2017-06-27 22:48:35)

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 20,722

Re: Can't type UTF-8 characters in terminal

I will try everything out again tomorrow. I'm kind of suspicious that uninstalling omzsh did it ...

I would bet the mortgage on it.

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
The shortest way to ruin a country is to give power to demagogues.— Dionysius of Halicarnassus
---
How to Ask Questions the Smart Way