[Solved] Extended ASCII in Xfce Terminal

beretta · 2012-03-27 15:57:09

I'm sure there's an easy solution to this.. but I'm not finding it. I'm trying to print extended ASCII characters in a bash prompt, and can't get any characters beyond the first 128 to print. It does work from the console, however

I've tried running a script like this:

#!/bin/bash

i=32
while [ $i -lt 150 ]; do
	j=`echo "obase=8; $i" | bc`
	printf "%i : %i\t%b\n" "$i" "$j" "\0$j"
	let "i++"
done

The script prints as decimal : octal ASCII. (requires extra/bc to be installed.)
The characters up to 127 print as they should, but the extended characters from 128 on do not.

One website suggested needing to set the LANG environment variable to en_US... didn't help. Nor does setting to en_US.utf8. Xfce-terminal is listed as supporting utf-8 on the wiki, though. Any ideas?

Last edited by beretta (2012-04-02 00:40:48)

beretta · 2012-03-29 01:09:41

I've been messing around, and after installing urxvt out of frustration, I can't get it to display unicode properly either.

Available locales:

  > locale -a
C
en_US
en_US.iso88591
en_US.utf8
POSIX
  >

Setting LANG="en_US" or LANG="en_US.utf8" gives the same result as it did in Xfce Terminal-- no unicode characters are printed. What am I missing?

karol · 2012-03-29 01:23:13

Terminus font in X doesn't print anything beyond 127 and prints ♦ in the console.

Edit:

One notable way in which ISO character sets differ from code pages is that the character positions 128 to 159, corresponding to ASCII control characters with the high-order bit set, are specifically unused and undefined in the ISO standards, though they had often been used for printable characters in proprietary code pages, a breaking of ISO standards that was almost universal.

http://en.wikipedia.org/wiki/Extended_A … _adaptions

Last edited by karol (2012-03-29 01:27:11)

beretta · 2012-03-29 01:41:22

I'm using Monospace, which as far as I know supports the extended ASCII characters. I did find another thread finally that describes a problem similar to mine, but with no solution. The only suggestion seems to not apply in my system, as the /usr/lib/locale directory has the proper permissions.

Similar thread: https://bbs.archlinux.org/viewtopic.php?id=122497

thisoldman · 2012-03-29 03:56:39

Monospace is an alias, usually mapped to DejaVu Sans Mono. You can see this by running 'fc-match Monospace'. The command shows the font family that fontconfig matches to "Monospace".

I don't know which editor you are using to edit your PS1, but there are several ways to get unicode equivalents of the extended ASCII characters:

Cut and paste.
For most X apps, Ctrl+Shift+u, then type the UTF number – for example, Ctrl+Shift+u then 0 0 c 6 produces Æ.
Vim has digraphs – try the help topics with the commands ':he unicode' and ':he digraphs'
Set up key combinations with xmodmap – this can slow your terminal's responsiveness down, particularly with Xfce.
Change your Xorg config to use the alt-intl or altgr-intl keyboard instead of the US keyboard.

This thread, https://bbs.archlinux.org/viewtopic.php?id=133522 has many of the details.

Then you can have a prompt that includes the unicode equivalents.

I did this with a copy and paste in xterm and it does display in a linux console:

23:39 ~ $ echo $PS1
\[\e[1;34m\]\A \W \$ \[\e[0;37m\]

23:39 ~ $ PS1='\[\e[1;34m\]\A \W çâèÆ \$ \[\e[0;37m\]'
23:42 ~ çâèÆ $

Last edited by thisoldman (2012-03-29 03:57:09)

karol · 2012-03-29 04:07:01

I've tried using vim to get these characters, but I only get e.g. <80> (ASCII 128 = hex 80) for the ASCII 128 - ASCII 159 characters - see the wikipedia quote from my previous post.

ASCII 198 (Æ) and other works fine.

thisoldman · 2012-03-29 11:32:24

Trying to get extended ascii in a terminal in X may be so complicated it's not reasonably possible. We have so many settings telling X to use UTF-8. For example, 'stty -a' in an xterm or urxvt instance shows 'iutf8', whose meaning is that the emulator assumes input is UTF-8.

@karol, for Vim, these work to produce Æ for me in Insert mode, in either of the above two terminals, using DejaVu Sans Mono as the font:
Ctrl+v u 0 0 c 6
Ctrl+k Shift+a Shift+e (shift used for uppercase)
Pasting from the primary buffer using Shift+Insert

karol, you are using a terminus font? I've found terminus to have some peculiar encodings.

In Vim's command mode, with the cursor over the Æ, when I type ga or :ascii the output is,

<Æ> 198, Hex 00c6, Octal 306

karol · 2012-03-29 11:34:58

Yes, I'm using Terminus font. Æ works, ASCII 128 - ASCII 159 characters don't. Do they work for you?

thisoldman · 2012-03-29 11:46:19

Some ASCII characters in the range 128 -159 work for me in Vim, some don't. ' Ctrl+k C , (comma)' produces Ç, 'Ctrl+k A 0 (zero)' does not produce Å, 'Ctrl+v u 2 1 2 b' does produce Å.

karol · 2012-03-29 11:58:49

Which of the ones you mentioned are in ASCII 128-159 range?

thisoldman · 2012-03-29 12:35:32

Ç is ascii-128; Å is ascii 143.

karol · 2012-03-29 13:09:06

'Ctrl+k C ,' prints 'Ç' - ASCII 199, according to vim.
'Ctrl+k c ,' prints 'ç' - ASCII 231.

'Ctrl+k A 0' does not produce Å, 'Ctrl+v u 2 1 2 b' does produce Å - ASCII 8491.

I'm not sure if it's Terminus' fault, vim's fault or I've messed something up, but 'Ç' & 'ç' are definitely not in the right place wrt to their ASCII code.
Even if I paste them into vim, it still shows ASCII 199 and ASCII 231.

I can change my PS1 and the 'çâèÆ' (and even 'Å') characters are printed OK, so maybe we should focus on this, as after all, this is what OP wants.

Thanks for your help, thisoldman :-)

beretta · 2012-03-29 17:51:56

Thanks, both of you-- this has been very helpful. I can use the ctl-shift-u method in Terminal, and Monospace does indeed map to DejaVu Sans Mono. But not all unicode characters are available, unfortunately-- is there a way to get a mapping of the characters available in the font?

Edit: extra/gucharmap is a good way to browse them. DejaVuSans does have things such as box drawing, but pasting ┌─╺ into Terminal yields +-? instead, so something is keeping Terminal from displaying characters that are actually available.

Last edited by beretta (2012-03-29 18:05:13)

thisoldman · 2012-03-29 23:15:12

I use gucharmap to find character codes, but unless you apply "Show only glyphs from this font," located under "View" on the menu bar, there is no guarantee that a character displayed with gucharmap is actually in the font.

The glyphs I think you want are listed under "Box Drawing" in gucharmap.

What gucharmap shows by default is how fontconfig matches a requested character code. If a font does not include a requested character, fontconfig will substitute the glyph from the next closest matching font. For instance, Deja Vu Sans Mono does not include the glyph for U+3057, Hiragana shi, し, but the glyph is displayed in gucharmap when I've asked it to display DVSMono. If I right-click on the glyph, it shows that shi is actually IPAGothic on my installation. Fontconfig's substitution works well for most GUI apps except terminals and code and text editors.

DejaVu font tables are also available as pdfs on their website: http://dejavu-fonts.org/wiki/PDF_samples.

beretta · 2012-03-30 14:57:42

According to the pdf for DejaVu Sans Mono, pg 27, the font has box drawing glyphs. Yet they do not display correctly in Terminal... I'm really uncertain to what's happening here. Does anyone use Xfce Terminal with these kinds of characters? In dabbling with urxvt, I wasn't able to get them to appear correctly there either, so perhaps its a problem in Xfce somewhere.

karol · 2012-03-30 15:09:56

It seems to work in my urxvt using Terminus font:

[karol@black ~]$ echo $TERM
rxvt-256color
[karol@black ~]$ PS1='\[\e[1;34m\]\A \W çâèÆ \$ \[\e[0;37m\]'
17:06 ~ çâèÆ $

It also works with xterm using the default font.

beretta · 2012-03-30 20:56:36

Ok, something has changed, and I can get that to work. But I still can't get the box-drawing type characters to work at all. I'm able to copy-past, print with ctl-shift-u, and echo -e "\uNNNN" and get some unicode characters to work, but not all, including many that in gucharmap are listed as being part of DejaVu Sans Mono.

thisoldman · 2012-03-30 23:45:45

It's a hunch, but there may be something wrong with your LANG setting. I've investigated my hunch with some Google and an incorrect or unset LANG variable seems to be frequently occurring with Xfce4.

Try setting the LANG variable in the terminal to see if the character display problem clears up. An example:

$ export LANG=en_US.UTF-8

If it works, you will have to find a way to set the variable correctly, permanently. Using the above line in your ~/.bashrc would be only a partial fix but may be enough. The wiki article, https://wiki.archlinux.org/index.php/Locale is marked out-of-date, so be sure to check the article's discussion page. Also note that /etc/rc.conf has two lines that affect locale settings – the lines in my rc.conf are set this way:

$ grep -i locale /etc/rc.conf
LOCALE="en_US.UTF-8"
DAEMON_LOCALE="yes"

beretta · 2012-04-02 00:40:21

thisoldman: Thanks, that solved it! I did have the line enabled in rc.conf, but had typed it as en_US.utf8 instead of en_US.UTF-8. A frustrating inconsistency... locales.gen lists it as en_US.UTF-8 (which I assume is why it should be listed that way in rc.conf), but when I run locale -a, it lists it as en_US.utf8. In any case, it appears to be working, now. Thank you so much!

Arch Linux

#1 2012-03-27 15:57:09

[Solved] Extended ASCII in Xfce Terminal

#2 2012-03-29 01:09:41

Re: [Solved] Extended ASCII in Xfce Terminal

#3 2012-03-29 01:23:13

Re: [Solved] Extended ASCII in Xfce Terminal

#4 2012-03-29 01:41:22

Re: [Solved] Extended ASCII in Xfce Terminal

#5 2012-03-29 03:56:39

Re: [Solved] Extended ASCII in Xfce Terminal

#6 2012-03-29 04:07:01

Re: [Solved] Extended ASCII in Xfce Terminal

#7 2012-03-29 11:32:24

Re: [Solved] Extended ASCII in Xfce Terminal

#8 2012-03-29 11:34:58

Re: [Solved] Extended ASCII in Xfce Terminal

#9 2012-03-29 11:46:19

Re: [Solved] Extended ASCII in Xfce Terminal

#10 2012-03-29 11:58:49

Re: [Solved] Extended ASCII in Xfce Terminal

#11 2012-03-29 12:35:32

Re: [Solved] Extended ASCII in Xfce Terminal

#12 2012-03-29 13:09:06

Re: [Solved] Extended ASCII in Xfce Terminal

#13 2012-03-29 17:51:56

Re: [Solved] Extended ASCII in Xfce Terminal

#14 2012-03-29 23:15:12

Re: [Solved] Extended ASCII in Xfce Terminal

#15 2012-03-30 14:57:42

Re: [Solved] Extended ASCII in Xfce Terminal

#16 2012-03-30 15:09:56

Re: [Solved] Extended ASCII in Xfce Terminal

#17 2012-03-30 20:56:36

Re: [Solved] Extended ASCII in Xfce Terminal

#18 2012-03-30 23:45:45

Re: [Solved] Extended ASCII in Xfce Terminal

#19 2012-04-02 00:40:21

Re: [Solved] Extended ASCII in Xfce Terminal

Board footer