[SOLVED] kdump kernel hang/freeze

espritlibre · 2023-07-28 17:04:36

i want to set up kdump because i'm plagued by random kernel panics when turning on bluetooth, since i ever got this laptop... anyway...
i compiled a "linux-kdump kernel" and i'm booting from repo linux while doing this, i'm at the point where i want to start the kdump kernel with kexec in order to have the right command for the kdump.service file for automatic loading...

sudo kexec /boot/vmlinuz-linux-kdump --initrd=/boot/initramfs-linux-kdump.img --append="cryptdevice=UUID=4f1ca192-c9f6-4d00-8ce7-a9615f3b1635:cryptlvm cryptkey=rootfs:/root/cryptlvm.keyfile root=/dev/mapper/espritlibre_VolGroup-root single irqpoll nr_cpus=1 reset_devices"

results in the system getting rebootet to the point where i'm dropped into rescue mode after systemd did its thing and then the system is frozen. REISUB is still working but unfortunately the filesystem didn't get synced... pic

sudo kexec /boot/vmlinuz-linux-kdump --initrd=/boot/initramfs-linux-kdump.img --append="cryptdevice=UUID=4f1ca192-c9f6-4d00-8ce7-a9615f3b1635:cryptlvm cryptkey=rootfs:/root/cryptlvm.keyfile root=/dev/mapper/espritlibre_VolGroup-root ibt=off net.ifnames=0 lsm=landlock,yama,apparmor,integrity,bpf nvidia_drm.modeset=1 intel_iommu=on iommu=pt ipv6.disable=1 loglevel=3 audit=1 single irqpoll nr_cpus=1 reset_devices"

results in a similar situation... pic

i also tried to boot the system kernel without the lsm=... no joy...

where am i screwing this up?

/etc/mkinitcpio.conf

...
BINARIES=(... /usr/bin/makedumpfile)
...
FILES=(/etc/systemd/system/kdump-save.service)
...

cat /proc/cmdline

pti=on page_alloc.shuffle=1 BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/mapper/espritlibre_VolGroup-root rw cryptdevice=UUID=4f1ca192-c9f6-4d00-8ce7-a9615f3b1635:cryptlvm cryptkey=rootfs:/root/cryptlvm.keyfile ibt=off net.ifnames=0 lsm=landlock,yama,apparmor,integrity,bpf nvidia_drm.modeset=1 intel_iommu=on iommu=pt ipv6.disable=1 loglevel=3 audit=1 crashkernel=1024M

blkid

/dev/mapper/espritlibre_VolGroup-root: UUID="60d51440-8fd1-4979-a5b5-893601df59bd" BLOCK_SIZE="4096" TYPE="ext4"
/dev/nvme0n1p1: UUID="138E-D91B" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="8c0ad4ca-0ca5-47ab-a81c-f3b5644c95a4"
/dev/nvme0n1p2: UUID="4f1ca192-c9f6-4d00-8ce7-a9615f3b1635" TYPE="crypto_LUKS" PARTLABEL="Linux LUKS" PARTUUID="22292a7e-b349-48db-94e1-76e4db685864"
/dev/mapper/espritlibre_VolGroup-home: UUID="d51de7b7-07ed-4c47-a6b8-999250409d67" BLOCK_SIZE="4096" TYPE="ext4"
/dev/mapper/cryptlvm: UUID="BifA6o-UugM-Asc8-mrJE-BJEQ-Fu5X-dZuGwf" TYPE="LVM2_member"
/dev/mapper/espritlibre_VolGroup-files: UUID="0829ce3b-dfff-4489-920b-c7c9a35fb233" BLOCK_SIZE="4096" TYPE="ext4"

EDIT:
secure boot is enabled and vmlinuz-linux-kdump is signed

EDIT 2: adjusted the subject for better search results

Last edited by espritlibre (2023-07-29 22:41:24)

seth · 2023-07-28 19:44:35

The stock kernels should™ be kdump ready, so you didn't actually have to compile one.
I suspect the issue is the root device, it's already unlocked so you probably have to only pass the opened root partition, but I've never tried this.

espritlibre · 2023-07-28 20:03:57

thanks seth for being here. i just tried

kexec /boot/vmlinuz-linux-kdump --initrd=/boot/initramfs-linux-kdump.img --append="root=/dev/mapper/espritlibre_VolGroup-root ibt=off net.ifnames=0 lsm=landlock,yama,apparmor,integrity,bpf nvidia_drm.modeset=1 intel_iommu=on iommu=pt ipv6.disable=1 loglevel=3 audit=1 single irqpoll nr_cpus=1 reset_devices"

but it didn't work pic

EDIT:
i didn't know that i can use the same kernel for booting the system and for the dump, i think the wiki needs some overhaul.. btw my dump kernel is linux-kdump-6.4.5 but the booted kernel is linux-6.4.6 could this be an issue?

seth wrote:

The stock kernels should™ be kdump ready, so you didn't actually have to compile one.

true, i didn't have to change anything just pkgbase in PKGBUILD

Last edited by espritlibre (2023-07-28 20:11:12)

seth · 2023-07-28 20:11:00

So still needs to be unlocked.
What if you boot the main and kdump kernel w/o "lsm=landlock,yama,apparmor,integrity,bpf"?

Edit: the wiki says what needs to be in place for a kernel to be used and how to check it and the kernel version doesn't matter.
The uncompressed kernel might be useful for debug symbols, though.

Last edited by seth (2023-07-28 20:14:10)

espritlibre · 2023-07-28 20:12:03

i tried that already, didn't work

seth · 2023-07-28 20:18:15

Wait a second: simply ctrl+d doesn't get you ahead from the rescue.target, does it?

espritlibre · 2023-07-28 20:26:13

i haven't tried that because it was frozen in rescue mode BUT
i just successfuly booted the kdump kernel, i followed the man page of kexec

man kexec wrote:

...
To load a kernel, the syntax is as follows:
kexec -l kernel-image --append=command-line-options --ini‐trd=initrd-image
...
Passing the exact contents of /proc/cmdline into command-line-options is the safest way to ensure that correct values are passed to the rebooting kernel.

so i did:

sudo kexec /boot/vmlinuz-linux-kdump --append="BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/mapper/espritlibre_VolGroup-root rw cryptdevice=UUID=4f1ca192-c9f6-4d00-8ce7-a9615f3b1635:cryptlvm cryptkey=rootfs:/root/cryptlvm.keyfile ibt=off net.ifnames=0 lsm=landlock,yama,apparmor,integrity,bpf nvidia_drm.modeset=1 intel_iommu=on iommu=pt ipv6.disable=1 loglevel=3 audit=1 crashkernel=2048M" --initrd=/boot/initramfs-linux-kdump.img

and it booted the kdump kernel

EDIT:
so while the previous command booted directly into the gnome, there was still "single irqpoll maxcpus=1 reset_devices" missing. after adding them to the command:

sudo kexec /boot/vmlinuz-linux-kdump --append="BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/mapper/espritlibre_VolGroup-root rw cryptdevice=UUID=4f1ca192-c9f6-4d00-8ce7-a9615f3b1635:cryptlvm cryptkey=rootfs:/root/cryptlvm.keyfile ibt=off net.ifnames=0 lsm=landlock,yama,apparmor,integrity,bpf nvidia_drm.modeset=1 intel_iommu=on iommu=pt ipv6.disable=1 loglevel=3 audit=1 crashkernel=2048M single irqpoll maxcpus=1 reset_devices" --initrd=/boot/initramfs-linux-kdump.img

kernel boots into rescue mode and then the system freezes...

next gonna try to boot the dump kernel off of the booted distro linux kernel and see what happens...

EDIT 2: i made a mistake in my previous 2 commands mixing linux-kdump with linux, but they didn't cause any issue because one boot succeeded... correcting it and booting kexec with

sudo kexec /boot/vmlinuz-linux --append="BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/mapper/espritlibre_VolGroup-root rw cryptdevice=UUID=4f1ca192-c9f6-4d00-8ce7-a9615f3b1635:cryptlvm cryptkey=rootfs:/root/cryptlvm.keyfile ibt=off net.ifnames=0 lsm=landlock,yama,apparmor,integrity,bpf nvidia_drm.modeset=1 intel_iommu=on iommu=pt ipv6.disable=1 loglevel=3 audit=1 crashkernel=2048M single irqpoll nr_cpus=1 reset_devices" --initrd=/boot/initramfs-linux.img

freezes in rescue mode
i also tried this with tlp disabled and hyperthreading disabled in firmware
i'm suspecting one of the paramter

single irqpoll nr_cpus=1 reset_devices

causes the issues, because one boot succeeded without these... are they even needed?
i'm done with this for today, spent already too much time on this....

seth wrote:

Wait a second: simply ctrl+d doesn't get you ahead from the rescue.target, does it?

no, it didn't work

EDIT 3:
after some digging through the kdump documentation i found the "disable_cpu_apicid=" parameter. i'm no expert but i think this is needed for the dump kernel to disable the BSP to wake up multiple cpus, which otherwise would cause the hang because apic id 0 is in use by the first kernel? at the same i gave the "nr_cpus" more cpus and now the dump kernel loads without freezing the system.

so just for the record my kexec command looks like this now:

sudo kexec /boot/vmlinuz-linux --append="BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/mapper/espritlibre_VolGroup-root rw cryptdevice=UUID=4f1ca192-c9f6-4d00-8ce7-a9615f3b1635:cryptlvm cryptkey=rootfs:/root/cryptlvm.keyfile ibt=off net.ifnames=0 lsm=landlock,yama,apparmor,integrity,bpf nvidia_drm.modeset=1 intel_iommu=on iommu=pt ipv6.disable=1 loglevel=3 audit=1 crashkernel=2048M single irqpoll nr_cpus=4 disable_cpu_apicid=0 reset_devices" --initrd=/boot/initramfs-linux.img

and my /etc/systemd/system/kdump.service looks like this:

[Unit]
Description=Load dump capture kernel
After=local-fs.target

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/usr/bin/kexec -p /boot/vmlinuz-linux --append="BOOT_IMAGE=/boot/vmlinuz-linux root=/dev/mapper/espritlibre_VolGroup-root rw cryptdevice=UUID=4f1ca192-c9f6-4d00-8ce7-a9615f3b1635:cryptlvm cryptkey=rootfs:/root/cryptlvm.keyfile ibt=off net.ifnames=0 lsm=landlock,yama,apparmor,integrity,bpf nvidia_drm.modeset=1 intel_iommu=on iommu=pt ipv6.disable=1 loglevel=3 audit=1 crashkernel=2048M systemd.unit=kdump-save.service irqpoll nr_cpus=4 disable_cpu_apicid=0 reset_devices" --initrd=/boot/initramfs-linux.img
# convenience
ExecStartPost=/bin/sh -c 'mkdir -p /var/crash/ && /usr/bin/makedumpfile --dump-dmesg /proc/vmcore "/var/crash/crashdump-$$(date +%%F-%%T)".dmesg'
ExecStop=/usr/bin/kexec -p -u

[Install]
WantedBy=multi-user.target

btw: i ended up recompiling the latest 6.4.7 just to get the vmlinux file with debug symbols to satisfy crash's needs.

marking as solved

Last edited by espritlibre (2023-07-29 16:04:23)

Arch Linux

#1 2023-07-28 17:04:36

[SOLVED] kdump kernel hang/freeze

#2 2023-07-28 19:44:35

Re: [SOLVED] kdump kernel hang/freeze

#3 2023-07-28 20:03:57

Re: [SOLVED] kdump kernel hang/freeze

#4 2023-07-28 20:11:00

Re: [SOLVED] kdump kernel hang/freeze

#5 2023-07-28 20:12:03

Re: [SOLVED] kdump kernel hang/freeze

#6 2023-07-28 20:18:15

Re: [SOLVED] kdump kernel hang/freeze

#7 2023-07-28 20:26:13

Re: [SOLVED] kdump kernel hang/freeze

Board footer