creating custom locale → "cannot set LC_CTYPE to default locale"

ivanborodin · 2013-08-30 02:12:36

NEWBIE here: first post after a couple of months tinkering with arch; no previous linux knowledge at all

want to: create a custom locale in order to set system-wide preferences (and a couple of custom keyboard layouts afterwards)

done this: created custom locale with some custom strings (d_fmt and the like), compiled it, set it as default, rebooted

problem: cannot get it working: getting messages like: Cannot set LC_CTYPE to default locale: No such file or directory

researched: similar situations (there are a lot) producing these messages but no clear issue/workaround identified so far

keep-it-simple-stupid: cloned (UN-MODIFIED) en_US, compiled it, set it as default ... problem persists: this is when I started suspecting something else

1. having system-wide en_US.UTF-8 locale working as expected:

[someone@somewhere ~]# localectl list-locales
en_US.utf8
[someone@somewhere ~]# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
... everything in-between set to same: en_US.UTF-8
LC-ALL=

2. creating new custom locale based on (unmodified) en_US named en_US@custom (making it system-wide):

[someone@somewhere ~]# cp /usr/share/i18n/locales/en_US /usr/share/i18n/locales/en_US@custom
[someone@somewhere ~]# vi /etc/locale.gen: only the two following locales are un-commented: [en_US.UTF-8 UTF-8] [en_US@custom.UTF-8 UTF-8]
[someone@somewhere ~]# locale-gen
[someone@somewhere ~]# localectl list-locales
en_US.utf8
en_US.utf8@custom.UTF-8
en_US@custom.UTF-8
[someone@somewhere ~]# localectl set-locale LANG="en_US@custom.UTF-8"
[someone@somewhere ~]# systemctl reboot

3. checking whether the new locale is set correctly:

[someone@somewhere ~]# locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_US@custom.UTF-8
LC_CTYPE="en_US@custom.UTF-8"
... everything in-between set to same: en_US@custom.UTF-8
LC-ALL=

something seems wrong; reverting changes:

[someone@somewhere ~]# localectl set-locale LANG="en_US.UTF-8"
[someone@somewhere ~]# systemctl reboot
[someone@somewhere ~]# localedef --delete-from-archive en_US@custom.UTF-8
[someone@somewhere ~]# localedef --delete-from-archive en_US.utf8@custom.UTF-8
[someone@somewhere ~]# localectl list-locales
en_US.utf8

question#1: since I didn't modify anything within en_US@custom definition file ... is this behavior possibly a bug ? Quite frankly I think I am doing something wrong somewhere ... call it newbie

question#2: when I run locale-gen to compile my new locale why I get two entries instead of only en_US@custom.UTF-8 as I expected ? My default en_US.UTF-8 locale always shows as one entry

I even tried a manual

localedef --inputfile=en_US@custom --charmap=UTF-8 en_US@custom.UTF-8

suspecting something wrong with locale-gen but I'm getting identical results: this is how I learned that locale-gen invokes localedef updating [/usr/lib/locale/locale-archive] to begin with

Xyne · 2013-08-30 04:02:52

Solutions/workarounds to the LC_CTYPE issue can be found here. Related discussion can be found here.

ivanborodin · 2013-08-30 05:18:51

Fist and foremost THANKS for your reply (and your time) I appreciate that !

Reading the links you provided (and it seems authored) and noting LANG=en_XX.UTF-8@POSIX instead of LANG=en_XX@POSIX.UTF-8 I think I have a problem with locale naming convention:

As I stated before locale-gen with [en_US@custom UTF-8] in [/etc/locale.gen] adds the following two entries:

[someone@somewhere ~]# locale-gen
en_US.utf8@custom.UTF-8
en_US@custom.UTF-8

I was using the last one: en_US@custom.UTF-8 with:

[someone@somewhere ~]# localectl set-locale LANG="en_US@custom.UTF-8"

but if I use: en_US.utf8@custom.UTF-8 as following:

[someone@somewhere ~]# localectl set-locale LANG="en_US.utf-8@custom.UTF-8"

everything works as expected and all the LC_* errors reported by the locale command dissapear

so I have to ask: which one is the proper format for localectl set-locale LANG="..." and why I have two similar entries to begin with ?

For a local definition file in /usr/share/i18n/locales/ such as eu_ES@euro, el_GR@euro, etc; I assumed localectl set-locale LANG="xx_XX@whatever.UTF-8" (not-working) instead of LANG="xx_XX.UTF-8@whatever.UTF-8" (working)

If this is the case why entries on /etc/locale.gen are stated as xx_XX@whatever ?

It seems now I know what is happening but I want to know why is happening

Xyne · 2013-08-30 15:52:52

In my en_XX@POSIX package, I install the locale file as "/usr/share/i18n/locales/en_XX@POSIX", add "en_XX.UTF-8@POSIX UTF-8" to /etc/locale.gen, and then run /usr/bin/locale-gen. With that, locale-gen generates "en_XX.UTF-8@POSIX" and

$ localectl list-locales
...
en_XX.utf8@POSIX

At the time that I wrote en_XX@POSIX I was unable to find clear guidelines for naming so I simply mimicked the naming of other locales. If you find clear documentation then I would be interested in reading it, but otherwise I suggest that you just use whatever works.

It was also written before the advent of systemd so I haven't explored the usage of localectl yet. I plan to look into it, but you may find the right way of managing custom locales with it before I do.

ivanborodin · 2013-08-30 20:28:39

I see you are getting one entry only then; I tried many locales in /etc/locale.gen and I am always getting two entries for each:

eg: un-commenting (standard) aa_ER UTF-8 in /etc/locale.gen and then running locale-gen produces:

locatectl list-locales
aa_ER
aa_ER.utf8

The locales are working as expected (when I use the first one) however I wonder why I am getting the other one with the appended .utf8 suffix ...

Please, could you un-comment aa_ER on your system and run locale-gen and then locatectl list-locales to check whether you are getting one or two entries ?

Xyne · 2013-08-30 21:21:29

locale.gen

en_US.UTF-8 UTF-8
aa_ER UTF-8  
en_XX.UTF-8@POSIX UTF-8

$ locale-gen 
Generating locales...
  en_US.UTF-8... done
  aa_ER.UTF-8... done
  en_XX.UTF-8@POSIX... done
Generation complete.

$ localectl list-locales
aa_ER
aa_ER.utf8
en_US.utf8
en_XX.utf8@POSIX

If I add ".UTF-8" to the aa_ER locale

en_US.UTF-8 UTF-8
aa_ER.UTF-8 UTF-8  
en_XX.UTF-8@POSIX UTF-8

I then only get one locale for aa_ER:

$ localectl list-locales
aa_ER.utf8
en_US.utf8
en_XX.utf8@POSIX

ivanborodin · 2013-09-03 04:08:32

back to town; thanks for your examples, I see that both of us have the same results un-commenting en_US.UTF-8 UTF-8 and aa_ER UTF-8 in locale.gen so it seems there's nothing wrong with my installation to begin with:

• with locale.gen=aa_ER UTF-8 → locale-gen → aa_ER and aa_ER.utf8 dual-locales are compiled
• with locale.gen=en_US.UTF-8 UTF-8 → locale-gen → en_US.utf8 locale is compiled

there are some things to highlight here:

• for a locale in the form xx_XX CharMap locale-gen will always produce two compiled locales:

1. xx_XX
2. xx_XX.charmap ... where charmap is a lowercase representation of actual charmap (without -); eg: UTF-8 → utf8

• for a locale in the form xx_XX.CharMap CharMap locale gen will produce just one locale as expected:

1. xx_XX.charmap ... where charmap is a lowercase representation of actual charmap (without -); eg: UTF-8 → utf8

I find this behavior somewhat confusing. If locale-gen (ie: localedef) is supposed to compile to the form xx_XX.charmap (ie: always adding the charmap suffix either in native or translated form) then for some entries we're are getting some more useless ones: in my case switching the locale to xx_XX works but with xx_XX.charmap I get the LC_* messages which I mentioned while starting this post; so I assume the correct one for these cases is just plain xx_XX

However for the en_US.UTF-8 UTF-8 style locales I suppose en_US.UTF-8 is just the name of the locale (meaning UTF-8 here is meaningless to locale-gen) and the following UTF-8 is the actual charmap locale-gen should be expecting. Maybe this locale was named en_US.UTF-8 to make it distinct from the former en_US ISO-8859-1. Maybe a name like en_USunicode8 UTF-8 (or en_USwhatever UTF-8) instead of en_US.UTF-8 UTF-8 should be chosen to avoid the confusion. When you see it for the first time you instinctively assume the first UTF-8 should have some meaning and act accordingly but it seems it means nothing at all (from the locale-gen perspective I mean).

There are lots of examples (installation guides for arch and the like) where you see some saying whatever.UTF-8, other whatever.UTF8, even utf8, and when you come to linux as newbie one of the first things you encounter is that everyone keep reminding you that linux (unlike Windows) is case-sensitive so this is where I guess the confusion starts building up.

Some things I learned:

• LC_* variables are not necessary at all in /etc/locale.conf (following the LANG=* entry) when using custom locales since all the preferences were already specified in the locale definition file before compilation; however, LC_* variables can also be specified in /etc/locale.conf below LANG=* to override settings in the current locale when using standard-ones

• from http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8: The official name and spelling of this encoding is UTF-8, where UTF stands for UCS Transformation Format. Please do not write UTF-8 in any documentation text in other ways (such as utf8 or UTF_8), unless of course you refer to a variable name and not the encoding itself.

• locale definition files follow the following format: <lang>_<territory>[@<modifiers>]; eg: [en_US], [en_GB]

• locale IDs in /etc/locale-gen follow the following format: <lang>_<territory>.<codeset>[@<modifiers>]; eg: [en_US.UTF-8] [UTF-8], [en_GB.UTF-8] [UTF-8]

• official documentation seems to be located at https://wiki.linuxfoundation.org/en/OpenI18N but finding anything within this site is almost impossible; lots of dead links too.

No matter what I have my custom locale up and working without any issues. I really appreciate your support and time on this one. I see you are something like a trusted member so I guess your free time is anything but plenty.

THANK YOU !

Xyne · 2013-09-03 11:46:18

Thanks for the detailed follow-up.

ivanborodin wrote:

• LC_* variables are not necessary at all in /etc/locale.conf (following the LANG=* entry) when using custom locales since all the preferences were already specified in the locale definition file before compilation; however, LC_* variables can also be specified in /etc/locale.conf below LANG=* to override settings in the current locale when using standard-ones

Could you elaborate on this point? How do you specify preferences in the locale definition? I'm assuming that preferences are not the same as the definitions between tags such as "LC_CTYPE" and "END LC_CTYPE"?

I know I should investigate this thoroughly myself, but I won't have time for at least a few days and I may forget about it by then.

Thanks!

ivanborodin · 2013-09-03 17:47:21

I beg your pardon, I used the term preferences for definitions within the definition locale file. What I gathered so far is given any locale you can override (per user or even system-wide) what is defined within the chosen locale, so you don't have to modify (for example) en_US if you only want just a different date representation but everything else as is.

Making a custom locale definition file (should be ?) the proper way to handle locales when you want everything customized from scratch while LC_* overrides are useful for just a couple of overrides (and for common users without the ability/necessity to modify locale definition files which don't make any sense at all).

If you build a custom locale from scratch (based on whatever locale you choose) requiring users to also specify some LC_* variables after setting your locale then maybe you can fix this LC_* override within your locale allowing the users to plainly set your locale and nothing else ... what do you think about it ?

A poor choice in my wording I guess.

Xyne · 2013-09-03 20:54:52

ivanborodin wrote:

If you build a custom locale from scratch (based on whatever locale you choose) requiring users to also specify some LC_* variables after setting your locale then maybe you can fix this LC_* override within your locale allowing the users to plainly set your locale and nothing else ... what do you think about it ?

This is what I have so far not been able to do. The en_XX@POSIX locale's LC_CTYPE section is as follows:

LC_CTYPE
% en_US copies en_GB, which copies i18n, so this should be equivalent to copying
% from en_US
copy "i18n"
%copy "en_US"
END LC_CTYPE

Despite deferring to a recognized locale (either directly by copying i18n or indirectly by copying the more common en_US), the locale's LC_CTYPE remains unrecognized. The discussion in the other thread that I have linked in my first post explains that applications do not necessarily look at the locale's own definitions. This is why I have found it necessary to set LC_CTYPE to the (universally) recognized en_US.UTF-8 locale in /etc/locale.conf.

You seem to know how to "fix this LC_* override within your locale allowing the users to plainly set your locale and nothing else", but it is still unclear to me. I have tried a few things (e.g. removing the unnecessary pre-"@" ".UTF-8" suffix and regenerating the locale), but applications still report that the locale is unsupported unless I set the LC_CTYPE to a standard locale, despite the definition being exactly the same.

Arch Linux

#1 2013-08-30 02:12:36

creating custom locale → "cannot set LC_CTYPE to default locale"

#2 2013-08-30 04:02:52

Re: creating custom locale → "cannot set LC_CTYPE to default locale"

#3 2013-08-30 05:18:51

Re: creating custom locale → "cannot set LC_CTYPE to default locale"

#4 2013-08-30 15:52:52

Re: creating custom locale → "cannot set LC_CTYPE to default locale"

#5 2013-08-30 20:28:39

Re: creating custom locale → "cannot set LC_CTYPE to default locale"

#6 2013-08-30 21:21:29

Re: creating custom locale → "cannot set LC_CTYPE to default locale"

#7 2013-09-03 04:08:32

Re: creating custom locale → "cannot set LC_CTYPE to default locale"

#8 2013-09-03 11:46:18

Re: creating custom locale → "cannot set LC_CTYPE to default locale"

#9 2013-09-03 17:47:21

Re: creating custom locale → "cannot set LC_CTYPE to default locale"

#10 2013-09-03 20:54:52

Re: creating custom locale → "cannot set LC_CTYPE to default locale"

Board footer