You are not logged in.

Pages: 1

#1 2025-09-27 08:11:55

tigerjack
Member
Registered: 2017-08-20
Posts: 97

[SOLVED] ls on root hangs the system

I noticed that, despite the system booting correctly, I have issues with the root partitions.
As an example, tools like df hangs forever, and the same happens by launching ls /.
The thing is, this problem is happening only some of the times, not always.

My setup, as per the title, is LVM on LUKS.

LVM info (focus is on vg0)

❯ sudo lvs
  LV         VG     Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data_linux vg0    -wi-ao----  680.99g                                                    
  home       vg0    -wi-ao----   80.00g                                                    
  opt        vg0    -wi-ao----   40.00g                                                    
  root       vg0    -wi-ao----   50.00g                                                    
  tmp        vg0    -wi-ao----   15.00g                                                    
  var        vg0    -wi-ao----   65.00g                                                    
  archive    vgdata -wi-ao---- <931.37g                                                    
❯ sudo pvs
  PV                     VG     Fmt  Attr PSize    PFree
  /dev/mapper/cryptroot  vg0    lvm2 a--   930.99g    0 
  /dev/mapper/data_crypt vgdata lvm2 a--  <931.37g    0 
❯ sudo vgs
  VG     #PV #LV #SN Attr   VSize    VFree
  vg0      1   6   0 wz--n-  930.99g    0 
  vgdata   1   1   0 wz--n- <931.37g    0

block attributes 

❯ sudo blkid
/dev/mapper/vg0-root: UUID="a3eeXXX" BLOCK_SIZE="4096" TYPE="ext4"
/dev/nvme0n1p1: UUID="7BBXXX" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="ESP" PARTUUID="259XXX"
/dev/nvme0n1p2: UUID="989XXX" TYPE="crypto_LUKS" PARTLABEL="primary" PARTUUID="97bXXX"
/dev/mapper/vgdata-archive: UUID="c71XXX" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdb: UUID="8c4XXX" UUID_SUB="c1eXXX" LABEL="hal9000:0" TYPE="linux_raid_member"
/dev/mapper/vg0-data_linux: UUID="ccfXXX" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/vg0-tmp: UUID="bfaXXX" BLOCK_SIZE="4096" TYPE="ext4"
/dev/md0: UUID="ea3XXX" TYPE="crypto_LUKS"
/dev/mapper/vg0-home: UUID="932XXX" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/cryptroot: UUID="w2uXXX" TYPE="LVM2_member"
/dev/mapper/data_crypt: UUID="2gSXXX" TYPE="LVM2_member"
/dev/sda: UUID="8c4XXX" UUID_SUB="104XXX" LABEL="hal9000:0" TYPE="linux_raid_member"
/dev/mapper/vg0-var: UUID="f9aXXX" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/vg0-opt: UUID="67fXXX" BLOCK_SIZE="4096" TYPE="ext4"
>sudo lsblk -o +name,mountpoint,uuid
[sudo] password for simone: 
NAME                 MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS     NAME           MOUNTPOINT      UUID
sda                    8:0    0 931.5G  0 disk                  sda                            8c4XXX
└─md0                  9:0    0 931.4G  0 raid1                 md0                            ea3XXX
  └─data_crypt       253:7    0 931.4G  0 crypt                 data_crypt                     2gSXXX
    └─vgdata-archive 253:8    0 931.4G  0 lvm   /mnt/archive    vgdata-archive /mnt/archive    c71XXX
sdb                    8:16   0 931.5G  0 disk                  sdb                            8c4XXX
└─md0                  9:0    0 931.4G  0 raid1                 md0                            ea3XXX
  └─data_crypt       253:7    0 931.4G  0 crypt                 data_crypt                     2gSXXX
    └─vgdata-archive 253:8    0 931.4G  0 lvm   /mnt/archive    vgdata-archive /mnt/archive    c71XXX
nvme0n1              259:0    0 931.5G  0 disk                  nvme0n1                        
├─nvme0n1p1          259:1    0   512M  0 part  /boot           nvme0n1p1      /boot           7BB5-XXX
└─nvme0n1p2          259:2    0   931G  0 part                  nvme0n1p2                      989XXX
  └─cryptroot        253:0    0   931G  0 crypt                 cryptroot                      w2uXXX
    ├─vg0-root       253:1    0    50G  0 lvm   /               vg0-root       /               a3eXXX
    ├─vg0-home       253:2    0    80G  0 lvm   /home           vg0-home       /home           932XXXX
    ├─vg0-opt        253:3    0    40G  0 lvm   /opt            vg0-opt        /opt            67f9XXX
    ├─vg0-tmp        253:4    0    15G  0 lvm   /tmp            vg0-tmp        /tmp            bfaXXX
    ├─vg0-var        253:5    0    65G  0 lvm   /var            vg0-var        /var            f9aXXX
    └─vg0-data_linux 253:6    0   681G  0 lvm   /mnt/data_linux vg0-data_linux /mnt/data_linux ccfXXXafsudo lsblk -o +name,mountpoint,uuid
[sudo] password for simone: 
NAME                 MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS     NAME           MOUNTPOINT      UUID
sda                    8:0    0 931.5G  0 disk                  sda                            8c4XXX
└─md0                  9:0    0 931.4G  0 raid1                 md0                            ea3XXX
  └─data_crypt       253:7    0 931.4G  0 crypt                 data_crypt                     2gSXXX
    └─vgdata-archive 253:8    0 931.4G  0 lvm   /mnt/archive    vgdata-archive /mnt/archive    c71XXX
sdb                    8:16   0 931.5G  0 disk                  sdb                            8c43XXX
└─md0                  9:0    0 931.4G  0 raid1                 md0                            ea3XXX
  └─data_crypt       253:7    0 931.4G  0 crypt                 data_crypt                     2gSXXX
    └─vgdata-archive 253:8    0 931.4G  0 lvm   /mnt/archive    vgdata-archive /mnt/archive    c71XXX
nvme0n1              259:0    0 931.5G  0 disk                  nvme0n1                        
├─nvme0n1p1          259:1    0   512M  0 part  /boot           nvme0n1p1      /boot           7BB5-XXX
└─nvme0n1p2          259:2    0   931G  0 part                  nvme0n1p2                      989XXX
  └─cryptroot        253:0    0   931G  0 crypt                 cryptroot                      w2uXXX
    ├─vg0-root       253:1    0    50G  0 lvm   /               vg0-root       /               a3eXXX
    ├─vg0-home       253:2    0    80G  0 lvm   /home           vg0-home       /home           932cXXX
    ├─vg0-opt        253:3    0    40G  0 lvm   /opt            vg0-opt        /opt            67fXXX
    ├─vg0-tmp        253:4    0    15G  0 lvm   /tmp            vg0-tmp        /tmp            bfadXXX
    ├─vg0-var        253:5    0    65G  0 lvm   /var            vg0-var        /var            f9aXXX
    └─vg0-data_linux 253:6    0   681G  0 lvm   /mnt/data_linux vg0-data_linux /mnt/data_linux ccfXXX

Tables info 

❯ sudo grep -vE "^(#|$)" /etc/crypttab
data_crypt UUID=ea3XXX none luks
❯ sudo grep -vE "^(#|$)" /etc/fstab
UUID=a3eXXX	/         	ext4      	rw,relatime	0 1
UUID=932XXX	/home     	ext4      	rw,relatime	0 2
UUID=67fXXX	/opt      	ext4      	rw,relatime	0 2
UUID=f9aXXX	/var      	ext4      	rw,relatime	0 2
UUID=bfaXXX	/tmp      	ext4      	rw,relatime	0 2
UUID=ccfXXX	/mnt/data_linux	ext4      	rw,relatime	0 2
UUID=7BBXXX     /boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2
UUID=c71eXXX /mnt/archive  ext4 defaults 0 2

initramfs 

❯ grep -E "^\s*HOOKS" /etc/mkinitcpio.conf
HOOKS=(base systemd autodetect microcode modconf kms keyboard sd-vconsole block sd-encrypt lvm2 filesystems fsck)

boot
I'm using, for the first time, systemd boot

❯ cat /boot/loader/entries/arch.conf
title Arch Linux
linux /vmlinuz-linux
initrd /initramfs-linux.img
options	rd.luks.name=989XXX=cryptroot root=/dev/vg0/root rw

Last edited by tigerjack (2025-09-27 14:43:58)

Offline

#2 2025-09-27 09:19:52

frostschutz
Member
Registered: 2013-11-15
Posts: 1,573

Re: [SOLVED] ls on root hangs the system

Well, things look mostly normal in your outputs. Can you keep a terminal open with dmesg -w, is there anything when hangs start happening? You can also check your logs but when rootfs hangs, things are usually no longer logged either...

Offline

#3 2025-09-27 10:59:55

tigerjack
Member
Registered: 2017-08-20
Posts: 97

Re: [SOLVED] ls on root hangs the system

frostschutz wrote:

Well, things look mostly normal in your outputs. Can you keep a terminal open with dmesg -w, is there anything when hangs start happening? You can also check your logs but when rootfs hangs, things are usually no longer logged either...

At this moment, everything works normally. But I will try a few reboots soon to see what happens.

Is it possible that this is due to systemd? It is the first time I am using it (I usually used grub), and it's the only thing changed from my previous configuration.

Offline

#4 2025-09-27 13:21:44

tigerjack
Member
Registered: 2017-08-20
Posts: 97

Re: [SOLVED] ls on root hangs the system

frostschutz wrote:

Well, things look mostly normal in your outputs. Can you keep a terminal open with dmesg -w, is there anything when hangs start happening? You can also check your logs but when rootfs hangs, things are usually no longer logged either...

Ok, when I booted the system, everything worked normally. After I while, I started to observe the same strange behavior as before.
ls / hangs, and so is df. However, notice that sudo ls / works, and sudo ls -l / hangs.

There are a few messages that can be related in the journal (you can find all of them here, but I guess this may be problematic).

[   15.344424] FAT-fs (nvme0n1p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

I tried to check the boot partition, but I have no idea what to do here

❯ sudo fsck.vfat -v /dev/nvme0n1p1
fsck.fat 4.2 (2021-01-31)
Checking we can access the last sector of the filesystem
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
  65:01/00
1) Copy original to backup
2) Copy backup to original
3) No action
[123?q]?

However, I don't know if this is related to my issue.

Offline

#5 2025-09-27 14:30:49

frostschutz
Member
Registered: 2013-11-15
Posts: 1,573

Re: [SOLVED] ls on root hangs the system

try `strace df` (and/or ls), if it hangs on a specific syscall e.g. statfs("/boot" or somewhere else?

Quite mysterious since if it was some undetected filesystem corruption, it should happen from the start, not after a while.

Offline

#6 2025-09-27 14:39:47

tigerjack
Member
Registered: 2017-08-20
Posts: 97

Re: [SOLVED] ls on root hangs the system

frostschutz wrote:

try `strace df` (and/or ls), if it hangs on a specific syscall e.g. statfs("/boot" or somewhere else?

Quite mysterious since if it was some undetected filesystem corruption, it should happen from the start, not after a while.

Indeed, the problem seems to be related to keybase

statx(AT_FDCWD, "/keybase", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE

When I try to stop the systemd service, I get

systemd[1]: Failed unmounting /keybase.
░░ Subject: A stop job for unit keybase.mount has finished
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A stop job for unit keybase.mount has finished.
░░ 
░░ The job identifier is 3768 and the job result is failed.

And now I found a similar issue here github.com/orgs/community/discussions/77611
which pointed out to the following workaround

systemctl --user stop keybase-redirector.service
systemctl --user disable keybase-redirector.service

Offline

#7 2025-09-30 13:17:55

tigerjack
Member
Registered: 2017-08-20
Posts: 97

Re: [SOLVED] ls on root hangs the system

Sorry, wrong link. Here is the correct one.

github.com/orgs/community/discussions/7761

Offline

Pages: 1

Board footer

Powered by FluxBB