Arch Linux

PeerK

Member

Registered: 2020-06-24

Posts: 37

[SOLVED] Broken after update (Problem was: /tmp ran out of disk space)

Hi all,

I am using my Arch-installation for ca. 2 years on my laptop computer and update it all 3 to 10 days manually.

Last night I installed the latest update (pretty big, 1.7 Gigs) and think, I did not realize that something went wrong (was a little bit tired ..).

Now the system does not start any more, not even into LUKS password-prompt.

I started the current arch-iso from an usb-stick and mounted the LVM2-devices (root, /home, /boot, ..) on the LUKS-partition manually, but I have no idea, how to repair the broken installation (probably chroot into the broken system, to have a look with journalctl, what weht wrong last night ..).

Can anyone give me a helping hand?

Update:

* chrooted into the broken system
* looking into last log-file with journalctrl -1 (maybe pacman-loggings more useful?)

Thanks and kind regards,

Peer

Last edited by PeerK (2024-11-11 11:19:49)

PeerK

Member

Registered: 2020-06-24

Posts: 37

Re: [SOLVED] Broken after update (Problem was: /tmp ran out of disk space)

Ahh, I found something .. strange:

Theoretically, there should be enough space on my ssd:

# df -h
..
/dev/mapper/temp_crypt-root  196G   63G  124G  34% /mnt/lala5
/dev/mapper/temp_crypt-home   98G   14G   80G  15% /mnt/lala5/home
/dev/mapper/temp_crypt-rep   1.5T  692G  752G  48% /mnt/lala5/rep
/dev/nvme0n1p1             1022M   20M 1003M   2% /mnt/lala5/boot
..

but actually pacman throwed "out-of-disk-space" errors:

# less /var/log/pacman.log
..
[2024-11-11T00:37:49+0100] [ALPM] upgraded zanshin (24.08.2-1 -> 24.08.3-1)
[2024-11-11T00:37:50+0100] [ALPM] transaction completed
[2024-11-11T00:38:00+0100] [ALPM] running '20-systemd-sysusers.hook'...
[2024-11-11T00:38:00+0100] [ALPM] running '30-systemd-daemon-reload-system.hook'...
[2024-11-11T00:38:00+0100] [ALPM] running '30-systemd-daemon-reload-user.hook'...
[2024-11-11T00:38:00+0100] [ALPM] running '30-systemd-tmpfiles.hook'...
[2024-11-11T00:38:01+0100] [ALPM] running '30-systemd-udev-reload.hook'...
[2024-11-11T00:38:01+0100] [ALPM] running '30-systemd-update.hook'...
[2024-11-11T00:38:01+0100] [ALPM] running '30-update-mime-database.hook'...
[2024-11-11T00:38:02+0100] [ALPM] running '90-mkinitcpio-install.hook'...
[2024-11-11T00:38:02+0100] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2024-11-11T00:38:02+0100] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf'
[2024-11-11T00:38:02+0100] [ALPM-SCRIPTLET]   -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux.img
[2024-11-11T00:38:02+0100] [ALPM-SCRIPTLET] /usr/lib/initcpio/functions: Zeile 1160: echo: Schreibfehler: Auf dem Gerät ist kein Speicherplatz mehr verfügbar.
[2024-11-11T00:38:02+0100] [ALPM-SCRIPTLET] /usr/lib/initcpio/functions: Zeile 1181: printf: Schreibfehler: Auf dem Gerät ist kein Speicherplatz mehr verfügbar.
[2024-11-11T00:38:02+0100] [ALPM-SCRIPTLET] install: Fehler beim Schreiben von '/tmp/mkinitcpio.8LOAoB/root/usr/lib/os-release': Auf dem Gerät ist kein Speicherplatz mehr verfügbar
[2024-11-11T00:38:02+0100] [ALPM-SCRIPTLET] ==> Starting build: '6.11.6-arch1-1'
[2024-11-11T00:38:02+0100] [ALPM-SCRIPTLET]   -> Running build hook: [base]
[2024-11-11T00:38:02+0100] [ALPM-SCRIPTLET] cp: Fehler beim Schreiben von '/tmp/mkinitcpio.8LOAoB/root/bin/busybox': Auf dem Gerät ist kein Speicherplatz mehr verfügbar
[2024-11-11T00:38:02+0100] [ALPM-SCRIPTLET] cp: Fehler beim Schreiben von '/tmp/mkinitcpio.8LOAoB/root/usr/bin/kmod': Auf dem Gerät ist kein Speicherplatz mehr verfügbar
..

German: Schreibfehler: Auf dem Gerät ist kein Speicherplatz mehr verfügbar.
English: Write error: No space left on device.

Anyone an idea?
-> Uhh: now I see it: Boot disk is full no, was a misinterpretation. But only 20 MB of 1GB on /boot used?
-> Question: has the update proceess deleted all files on /dev/nvme0n1p1 and is 1 GB not enough for the boot device any more?

Last edited by PeerK (2024-11-11 08:54:09)

Awebb

Member

Registered: 2010-05-06

Posts: 6,602

Re: [SOLVED] Broken after update (Problem was: /tmp ran out of disk space)

/tmp/mkinitcpio... means there is no space in /tmp, which normally is a tmpfs, which is basically a RAM disk. I'm not sure if this is the case here, but it looks like you went OOM at some point during mkinitcpio.

You might be fine with arch-chrooting properly and mkinitcpio'ing manually, although any hook after  90-mkinitcpio... will not have been executed, even though the 90- pretty much means it's very late in the process.

Awebb

Member

Registered: 2010-05-06

Posts: 6,602

Re: [SOLVED] Broken after update (Problem was: /tmp ran out of disk space)

Addendum: This is what happens if you have no swap. Do you have an active swap method? Check free -m.

astralc

Member

Registered: 2022-09-17

Posts: 90

Re: [SOLVED] Broken after update (Problem was: /tmp ran out of disk space)

error is during mkinitcpio run, look like you didn't have space on /tmp (where the initcpio is built). maybe you just need to run mkinitcpio again?

PeerK

Member

Registered: 2020-06-24

Posts: 37

Re: [SOLVED] Broken after update (Problem was: /tmp ran out of disk space)

Ah, thank you @awebb and @astralc +1

The notebook has 16 GB of RAM and shoud have also 16 GB of swap-space. Due to usage, might have been not enough. - I would have loved Lenovo, if they could have offered the Ryzen8840-Yoga with 32 Gigs ;-)

I exited from chroot and
* mounted proc, sys, etc. and swap,
* chrooted again
* looked for memory ..

root@archiso /mnt # mount -t proc proc /mnt/lala5/proc 
root@archiso /mnt # mount -t sysfs sys /mnt/lala5/sys 
root@archiso /mnt # mount -o bind /dev /mnt/lala5/dev 
root@archiso /mnt # mount -t devpts /dev/pts /mnt/lala5/dev/pts 
root@archiso /mnt # mount -o bind /sys/firmware/efi/efivars /mnt/lala5/sys/firmware/efi/efivars 

root@archiso /mnt # swapon /dev/mapper/lala5crypt-swap 

root@archiso /mnt # arch-chroot ./lala5 /bin/bash

# free -m
               total        used        free      shared  buff/cache   available
Mem:           13727        1469       11913         823        1433       12257
Swap:          16383           0       16383

But mkinitcpio fails ..

[root@archiso /]# mkinitcpio
==> ERROR: '/lib/modules/6.11.5-arch1-1' is not a valid kernel module directory

# ls -la /lib/modules
total 380
drwxr-xr-x   3 root root   4096 Nov  4 08:25 .
drwxr-xr-x 216 root root 376832 Nov 11 00:37 ..
drwxr-xr-x   5 root root   4096 Nov  4 08:25 6.11.6-arch1-1

Seems somehow wrong with the kernel-versions ..

Last edited by PeerK (2024-11-11 10:20:00)

astralc

Member

Registered: 2022-09-17

Posts: 90

Re: [SOLVED] Broken after update (Problem was: /tmp ran out of disk space)

Run

mkinitcpio -P

instead.

Also, why you don't use arch-chroot command to chroot?

Last edited by astralc (2024-11-11 10:45:09)

PeerK

Member

Registered: 2020-06-24

Posts: 37

Re: [SOLVED] Broken after update (Problem was: /tmp ran out of disk space)

Ah, perfect!

Thank you all. my notebook is up an running again :-) and I have some small lessons learned, e.g. to have a look to /tmp when updating ;-)

PeerK

Member

Registered: 2020-06-24

Posts: 37

Re: [SOLVED] Broken after update (Problem was: /tmp ran out of disk space)

Run

mkinitcpio -P

instead.

+1 :-)

Also, why you don't use arch-chroot command to chroot?

I thought, that I used it (only omitted the "arch-") ;-)

Thanks and kind regards,

Last edited by PeerK (2024-11-11 11:29:09)