Arch Linux

Leviticoh

Member

Registered: 2021-12-13

Posts: 9

[SOLVED] ESTALE when mounting overlay filesystem

Hi, it's my first time asking help here, not sure if this is the right section,

I'm trying to set my home directory up so that most of my files are on an external ssd, in order to switch more easily between my desktop and laptop.

the external ssd uses btrfs with subvolumes mounted on the main directories in my home,
it all works well except for the .ssh directory,
since i wanted to keep the authorized_keys, config and id_ed25519 files on the respective computers i mounted the ssh volume on .ssh-extern and mounted .ssh-extern/.ssh as upperdir on .ssh

relevant fstab lines:

/dev/mapper/home                                        /home/luca/Documenti            btrfs     rw,noatime,ssd,subvolid=257      0 8
/dev/mapper/home                                        /home/luca/Immagini             btrfs     rw,noatime,ssd,subvolid=259      0 9
/dev/mapper/home                                        /home/luca/Musica               btrfs     rw,noatime,ssd,subvolid=261      0 10
/dev/mapper/home                                        /home/luca/Progetti             btrfs     rw,noatime,ssd,subvolid=258      0 11
/dev/mapper/home                                        /home/luca/sources              btrfs     rw,noatime,ssd,subvolid=260      0 12
/dev/mapper/home                                        /home/luca/Video/extern         btrfs     rw,noatime,ssd,subvolid=256      0 13
/dev/mapper/home                                        /home/luca/packages             btrfs     rw,noatime,ssd,subvolid=263      0 15
/dev/mapper/home                                        /home/luca/.ssh-extern          btrfs     rw,noatime,ssd,subvolid=262      0 14

overlay                                                 /home/luca/.ssh                 overlay          noauto,x-systemd.automount,lowerdir=/home/luca/.ssh-local,upperdir=/home/luca/.ssh-extern/.ssh,workdir=/home/luca/.ssh-extern/work-home 0 0

on the laptop, where i set it up first, it all works fine and the .ssh directory is correctly populated, but on the desktop it fails

when mounted with fstab the .ssh directory shows up like this:

d?????????  ? ?    ?       ?            ?  .ssh

when i run

ls -lah $HOME

and when i try to list its contents it fails with this output:

ls: cannot access '.ssh': No such device

also each time i try to list it these lines are added to dmesg:

[  493.260719] overlayfs: "xino" feature enabled using 2 upper inode bits.
[  493.260727] overlayfs: failed to verify origin (luca/.ssh-local, ino=13894717, err=-116)
[  493.260730] overlayfs: failed to verify upper root origin

on the other hand,
when it isn't mounted by fstab and i try to mount it manually i get this output:

[luca@luca-camera ~]$ export LC_ALL=C
[luca@luca-camera ~]$ sudo mount -t overlay overlay -o lowerdir=/home/luca/.ssh-local,upperdir=/home/luca/.ssh-extern/.ssh,workdir=/home/luca/.ssh-extern/work .ssh/
[sudo] password for luca:
mount: /home/luca/.ssh: mount(2) system call failed: Stale file handle.
       dmesg(1) may have more information after failed mount system call.
[luca@luca-camera ~]$

and so these same three lines are added to dmesg:

[   93.109125] overlayfs: "xino" feature enabled using 2 upper inode bits.
[   93.109131] overlayfs: failed to verify origin (luca/.ssh-local, ino=13894717, err=-116)
[   93.109133] overlayfs: failed to verify upper root origin

the machines i'm using both run arch linux and were updated in the last 12 hours

i wasn't able to find much about setups similar to mine by googling, and couldn't find warnings about what i'm doing in the overlayfs documentation on kernel.org
does anyone have an idea of what's happening?

Last edited by Leviticoh (2022-04-15 15:19:28)

Leviticoh

Member

Registered: 2021-12-13

Posts: 9

Re: [SOLVED] ESTALE when mounting overlay filesystem

from https://www.kernel.org/doc/html/latest/ … layfs.html

Mounting an overlay using an upper layer path, where the upper layer path was previously used by another mounted overlay in combination with a different lower layer path, is allowed, unless the “inodes index” feature or “metadata only copy up” feature is enabled.

it can be solved by adding

index=off,metacopy=off

to the overlayfs mount options

since the arch kermel has the options

CONFIG_OVERLAY_FS_INDEX=y
CONFIG_OVERLAY_FS_METACOPY=y

set by default

Last edited by Leviticoh (2022-04-15 15:23:13)

TuxAndy

Member

Registered: 2023-03-11

Posts: 1

Re: [SOLVED] ESTALE when mounting overlay filesystem

For the confused souls roaming the internet

overlayfs: failed to verify upper root origin

https://www.kernel.org/doc/html/latest/ … ing-layers

For “metadata only copy up” feature there is no verification mechanism at mount time. So if same upper is mounted with different set of lower, mount probably will succeed but expect the unexpected later on. So don’t do it.

basically means upper directory was used before for a different lower directory. and overlayfs does verification to make sure the specified upper directory was used for the older lower directory.
So to make it usable without playing with mount options I backed up the upper directory then removed the work and upper directories and remade them remade them (same names worked just fine), and put the backed up data back in the upper directory.
I do not know where this information is stored, I assume they are tied to the directory attributes/metadata.
I'm not sure if this is a good solution, maybe there is something deeper linking the upper directory files to the lower directory, but it worked for me.