[SOLVED] locale disappearing after reboot

kescherArch · 2021-05-02 12:37:03

New thread to avoid necro-bumping https://bbs.archlinux.org/viewtopic.php?id=228776

I am having the exact same issue as OP of the original thread linked above.
It's a fresh Arch install.
I have the following content in my /etc/locale.gen:

en_US.UTF-8 UTF-8
de_AT.UTF-8 UTF-8
en_GB.UTF-8 UTF-8

I run locale-gen, and locale -a outputs:

C
de_AT.utf8
en_GB.utf8
en_US.utf8
POSIX

After a reboot, however, locale -a is suddenly missing an entry (and it's always de_AT):

C
en_GB.utf8
en_US.utf8
POSIX

Now I have tried adding

de_DE.UTF-8 UTF-8

, running locale-gen and rebooting. Curiously, this is the result:

C
de_AT.utf8
de_DE.utf8
en_GB.utf8
en_US.utf8
POSIX

Now, while this kind of fixes the issue, I wonder what causes this and how I can avoid including de_DE, since I only want de_AT as additional locale.

Last edited by kescherArch (2021-05-03 07:23:06)

seth · 2021-05-02 13:14:44

It's because you're not a real contry

less /usr/share/i18n/locales/de_AT

(while the perceived behavior somehow seems buggy, de_AT only augments de_DE - I assume you're not getting en_GB w/o en_US either because it copies LC_MESSAGES from there)

kescherArch · 2021-05-02 13:31:13

seth wrote:

(while the perceived behavior somehow seems buggy, de_AT only augments de_DE - I assume you're not getting en_GB w/o en_US either because it copies LC_MESSAGES from there)

That does sound plausible, but I still have to wonder why the locale only disappears after a reboot in that case? Also, this doesn't happen on another machine with Manjaro instead of Arch on it. There's no de_DE in its locale.gen, either.

seth · 2021-05-02 13:45:21

I'd assume it because some files in a volatile cache or still being opened (sudo lsof | grep DE) which are gone w/ the reboot.
Which glibc version is the manjaro at and what does /usr/share/i18n/locales/de_AT look in comparism? (you could just diff it)

kescherArch · 2021-05-02 19:30:28

seth wrote:

I'd assume it because some files in a volatile cache or still being opened (sudo lsof | grep DE) which are gone w/ the reboot.

No, nothing is different about the lsof output between the state after locale-gen and the state after the reboot.

seth wrote:

Which glibc version is the manjaro at

The exact same package as on Arch, 2.33-4.

seth wrote:

what does /usr/share/i18n/locales/de_AT look in comparism?

No diff.

However, I noticed one thing.
You see, I use LUKS encryption. Very standard setup, first partition as boot with vfat (with Grub), second partition LUKS-encrypted, LVM2 container, one partition. On an SSD.
I'm starting to wonder whether... Something are simply not written correctly. My issue may have become something else entirely.

I've checked /usr/lib/local/locale-archive. This file, oddly enough, is always of the exact same size: 3396672 (according to ls).
On the Manjaro machine, this file remains the same from after locale-gen to after reboot.
On the Arch machine, however, this is a different story... And that's where my understanding of this entire issue fades into non-existence.

 $ diff <(xxd locale-archive-before-reboot) <(xxd locale-archive-after-reboot)
1,4c1,4
< 00000000: 0901 02de 0000 0000 3800 0000 0300 0000  ........8.......
< 00000010: 8b03 0000 bc2a 0000 2100 0000 4c1d 0000  .....*..!...L...
< 00000020: 0848 0000 0300 0000 a401 0000 38f9 0000  .H..........8...
< 00000030: 0600 0000 d307 0000 0000 0000 0000 0000  ................
---
> 00000000: 0901 02de 0000 0000 3800 0000 0000 0000  ........8.......
> 00000010: 8b03 0000 bc2a 0000 0000 0000 4c1d 0000  .....*......L...
> 00000020: 0848 0000 0000 0000 a401 0000 38f9 0000  .H..........8...
> 00000030: 0000 0000 d307 0000 0000 0000 0000 0000  ................
119c119
< 00000760: 0000 0000 4bb4 e362 bc2a 0000 0848 0000  ....K..b.*...H..
---
> 00000760: 0000 0000 0000 0000 0000 0000 0000 0000  ................

What could cause a file to change between reboots in this way? Especially just this file. Other files aren't affected.
Furthermore, if I overwrite the broken file with the "before-reboot" file and reboot again, the locale is finally available.

Anyway, the current workaround for fixing this for every run of locale-gen seems to be:
- cat the file into another before rebooting
- reboot
- cat the other file back into the locale-archive

I'll provide the weird files here in case there's some insight to be had: https://cloud.kescher.at/s/ws7CSp9LELNwFB9

Last edited by kescherArch (2021-05-02 19:35:28)

seth · 2021-05-02 19:59:57

Is the added locale preserved across a subsequent reboot?
Als what are the stats (esp. timestamps) on the file before and after the reboot?

The FS theory does't really fit the part where only de_AT is affected but can be generated along de_DE - it would rather seem the file is replaced/sanitized on reboot.
You could also try to "chattr +i" it (make it immutable)

kescherArch · 2021-05-02 20:17:39

seth wrote:

The FS theory does't really fit the part where only de_AT is affected but can be generated along de_DE - it would rather seem the file is replaced/sanitized on reboot.

If I manually cat the "ok" locale-archive onto the locale-archive and reboot, everything is fine.
However, if I now do another locale-gen and reboot, the issue reappears!
Therefore, something must be going wrong when locale-gen, or rather localedef, modifies the archive.
It may still be the FS.

seth wrote:

Is the added locale preserved across a subsequent reboot?

Only if I do the "workaround", and don't have anything run locale-gen without applying said workaround. Another thing to note is that it doesn't matter how much time or other IO activity passes. I can even play some Tetris and watch a few videos, reboot, and have the exact same effect. It has now become reproducible on my machine.

seth wrote:

Als what are the stats (esp. timestamps) on the file before and after the reboot?

Before reboot:

  File: /usr/lib/locale/locale-archive
  Size: 3396672   	Blocks: 6512       IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 22443928    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-05-02 22:12:27.502493028 +0200
Modify: 2021-05-02 22:12:29.642492998 +0200
Change: 2021-05-02 22:12:29.642492998 +0200
 Birth: 2021-05-02 22:12:27.502493028 +0200

After reboot:

  File: /usr/lib/locale/locale-archive
  Size: 3396672   	Blocks: 6512       IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 22443928    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-05-02 22:13:16.332311658 +0200
Modify: 2021-05-02 22:12:29.642492998 +0200
Change: 2021-05-02 22:12:29.642492998 +0200
 Birth: 2021-05-02 22:12:27.502493028 +0200

The only thing that seems to have changed is the access time.
If I repeat the same thing with chattr +i, the same issue still happens.

Edit: I just realized: OP of the other thread I linked had their first change here, in the "Blocks" number. I don't.

Last edited by kescherArch (2021-05-02 20:26:49)

seth · 2021-05-03 06:13:34

Let's assume the de_AT pattern is a (very weird) red herring - the file seems to alter outside the FS.
a) Do you have a parallel windows installation?
b) Do you mount the SSD "discard" ie. w/ continuous trim?

kescherArch · 2021-05-03 06:21:43

seth wrote:

a) Do you have a parallel windows installation?

No. Only Arch.

seth wrote:

b) Do you mount the SSD "discard" ie. w/ continuous trim?

Nope. The fstab entry for the SSD looks as follows:

UUID=b41c85c2-fe04-441c-88c1-10de3dffe51e / ext4 defaults,rw,noatime,i_version 0 1

...wait a second. noatime.

kescher wrote:

After reboot:

  File: /usr/lib/locale/locale-archive
  Size: 3396672   	Blocks: 6512       IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 22443928    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-05-02 22:13:16.332311658 +0200
Modify: 2021-05-02 22:12:29.642492998 +0200
Change: 2021-05-02 22:12:29.642492998 +0200
 Birth: 2021-05-02 22:12:27.502493028 +0200

Why is the access time changed if I mount the SSD with noatime?

seth · 2021-05-03 06:27:05

On boot?
You could try "relatime,lazytime" instead.

"i_version" should™ be "iversion" - why is it there to begin with?

kescherArch · 2021-05-03 07:14:17

seth wrote:

"i_version" should™ be "iversion" - why is it there to begin with?

see i_version in ext4(5)
It's for 64bit inode version counter support, and has nothing to do with the "iversion" mount option.

seth wrote:

On boot?

Yes, the second stat was taken right after the being able to login after unlocking /dev/nvme0n1p2, my root partition.

seth wrote:

You could try "relatime,lazytime" instead.

Will try.

kescherArch · 2021-05-03 07:22:01

relatime,lazytime

instead of

noatime

fixes the issue. No more changed locale-archive after boot.

seth · 2021-05-03 07:29:06

Afacis i_version is at best an ext4 specific iversion parameter but might just as much be a 2nd tier switch.
Doesn't explain why it's enabled - and from the context of this thread and also https://bugzilla.redhat.com/show_bug.cgi?id=1848084 it looks like a very good contender for the role of the bad guy.
Since you can reproduce the problem, I'd certainly check whether it impacts it.

kescherArch · 2021-05-03 07:44:12

seth wrote:

Afacis i_version is at best an ext4 specific iversion parameter but might just as much be a 2nd tier switch.

iversion enables the counter to begin with. i_version allows it to be a 64 bit number.

seth wrote:

Doesn't explain why it's enabled - and from the context of this thread and also https://bugzilla.redhat.com/show_bug.cgi?id=1848084 it looks like a very good contender for the role of the bad guy.
Since you can reproduce the problem, I'd certainly check whether it impacts it.

I've now tested both noatime and relatime,lazytime with and without i_version. That mount option doesn't seem to affect this particular issue.

Arch Linux

#1 2021-05-02 12:37:03

[SOLVED] locale disappearing after reboot

#2 2021-05-02 13:14:44

Re: [SOLVED] locale disappearing after reboot

#3 2021-05-02 13:31:13

Re: [SOLVED] locale disappearing after reboot

#4 2021-05-02 13:45:21

Re: [SOLVED] locale disappearing after reboot

#5 2021-05-02 19:30:28

Re: [SOLVED] locale disappearing after reboot

#6 2021-05-02 19:59:57

Re: [SOLVED] locale disappearing after reboot

#7 2021-05-02 20:17:39

Re: [SOLVED] locale disappearing after reboot

#8 2021-05-03 06:13:34

Re: [SOLVED] locale disappearing after reboot

#9 2021-05-03 06:21:43

Re: [SOLVED] locale disappearing after reboot

#10 2021-05-03 06:27:05

Re: [SOLVED] locale disappearing after reboot

#11 2021-05-03 07:14:17

Re: [SOLVED] locale disappearing after reboot

#12 2021-05-03 07:22:01

Re: [SOLVED] locale disappearing after reboot

#13 2021-05-03 07:29:06

Re: [SOLVED] locale disappearing after reboot

#14 2021-05-03 07:44:12

Re: [SOLVED] locale disappearing after reboot

Board footer