Arch Linux

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

I have checked the kernels from https://bbs.archlinux.org/viewtopic.php … 0#p2052560
The "test kernels" did not boot, the "sanity check" kernels did.

I am using a UEFI setup with syslinux and booting into a LVM on LUKS drive.

$ cat /boot/EFI/syslinux/syslinux.cfg |grep "LABEL arch$" -A5
LABEL arch
    MENU LABEL Arch Linux
    LINUX ../../vmlinuz-linux
    APPEND cryptdevice=/dev/nvme0n1p5:encStorage root=/dev/encStorage/rootvol rw
    INITRD ../../amd-ucode.img,../../initramfs-linux.img

The system is a Lenovo ThinkPad X395. I have another system on a similar setup with the same behaviour where I will test the "test kernels" later today but I expect the same result. If that is not the case I will update this post.

I am happy to test anything which further helps dissecting this problem.

As others have this issue using GRUB: Is it connected to the bootloader? I don't see further progress being done/discussed, if there are other sources I can follow, I would be happy if anyone could share that resource with me.

loqs

Member

Registered: 2014-03-06

Posts: 18,132

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

git bisect bad
Bisecting: 1204 revisions left to test after this (roughly 10 steps)
[22922deae13fc8d3769790c2eb388e9afce9771d] Merge tag 'objtool-core-2022-05-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

https://drive.google.com/file/d/1w4a30X … sp=sharing linux-5.18.r1199.g22922deae13f-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/17Qykoc … sp=sharing linux-headers-5.18.r1199.g22922deae13f-1-x86_64.pkg.tar.zst

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

Thanks loqs, this kernel was bad as well.

loqs

Member

Registered: 2014-03-06

Posts: 18,132

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

git bisect bad
Bisecting: 605 revisions left to test after this (roughly 9 steps)
[03e1ccd45fa70904e43ddceda140854d22b7e871] Merge tag 'x86-irq-2022-05-23' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

https://drive.google.com/file/d/1tLpfqg … sp=sharing linux-5.18.r593.g03e1ccd45fa7-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1OUblgt … sp=sharing linux-headers-5.18.r593.g03e1ccd45fa7-1-x86_64.pkg.tar.zst

BwK

Member

Registered: 2019-09-25

Posts: 14

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

git bisect bad
Bisecting: 2538 revisions left to test after this (roughly 11 steps)
[3842007b1a33589d57f67eac479b132b77767514] Merge tag 'zonefs-5.19-rc1-fix' of git://git.kernel.org/pub/scm/linux/kernel/git/dlemoal/zonefs

https://drive.google.com/file/d/1PfVRMO … sp=sharing linux-5.18.r2404.g3842007b1a33-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1kPneHP … sp=sharing linux-headers-5.18.r2404.g3842007b1a33-1-x86_64.pkg.tar.zst
Also a sanity check,  does the kernel below work?
https://drive.google.com/file/d/1uHRoaz … sp=sharing linux-5.18-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1pdXEj3 … sp=sharing linux-headers-5.18-1-x86_64.pkg.tar.zst

Can confirm, r2404 does NOT boot, "sanity check" does boot.

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

Last one was good:

$ uname -r
5.18.0-1-00593-g03e1ccd45fa7

loqs

Member

Registered: 2014-03-06

Posts: 18,132

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

git bisect good
Bisecting: 311 revisions left to test after this (roughly 8 steps)
[d61306047533eb6f63a7bd51dfa7f868503bf0ba] Merge tag 'for-linus-5.19-rc1-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip

https://drive.google.com/file/d/1d6U2BK … sp=sharing linux-5.18.r887.gd61306047533-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1pIm9Ch … sp=sharing linux-headers-5.18.r887.gd61306047533-1-x86_64.pkg.tar.zst

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

That one was bad.

loqs

Member

Registered: 2014-03-06

Posts: 18,132

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

git bisect bad
Bisecting: 146 revisions left to test after this (roughly 7 steps)
[1de564b8c1a6f9f8bf3a106daa0be9f2cba7d045] Merge tag 'x86_build_for_v5.19_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

https://drive.google.com/file/d/1sXy-cX … sp=sharing linux-5.18.r740.g1de564b8c1a6-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1i4yvrU … sp=sharing linux-headers-5.18.r740.g1de564b8c1a6-1-x86_64.pkg.tar.zst

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

Bad again.

loqs

Member

Registered: 2014-03-06

Posts: 18,132

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

git bisect bad
Bisecting: 66 revisions left to test after this (roughly 6 steps)
[eb39e37d5cebdf0f63ee2a315fc23b035d81b4b0] Merge tag 'x86_sev_for_v5.19_rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

https://drive.google.com/file/d/1yn_mlg … sp=sharing linux-5.18.r673.geb39e37d5ceb-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1t_eNA7 … sp=sharing linux-headers-5.18.r673.geb39e37d5ceb-1-x86_64.pkg.tar.zst

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

Still bad
Feels like one could start betting on this

Last edited by Watnuss (2022-08-20 17:21:37)

loqs

Member

Registered: 2014-03-06

Posts: 18,132

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

git bisect bad
Bisecting: 39 revisions left to test after this (roughly 5 steps)
[ba37a1438aeb540cc48722d629f4b2e7e4398466] x86/sev: Add a sev= cmdline option

https://drive.google.com/file/d/1ziWqUW … sp=sharing linux-5.18rc1.r40.gba37a1438aeb-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1Aa0XGU … sp=sharing linux-headers-5.18rc1.r40.gba37a1438aeb-1-x86_64.pkg.tar.zst

Last edited by loqs (2022-08-20 18:30:44)

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

bad again

loqs

Member

Registered: 2014-03-06

Posts: 18,132

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

Does blacklisting the kvm module have any effect?
Edit:

git bisect bad
Bisecting: 19 revisions left to test after this (roughly 4 steps)
[9704c07bf9f7682a83aec4e66f2d9154dbd8577f] x86/kernel: Validate ROM memory before accessing when SEV-SNP is active

https://drive.google.com/file/d/1JauFaJ … sp=sharing linux-5.18rc1.r20.g9704c07bf9f7-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1vR167q … sp=sharing linux-headers-5.18rc1.r20.g9704c07bf9f7-1-x86_64.pkg.tar.zst

Last edited by loqs (2022-08-20 18:11:29)

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

I upgraded to linux 5.19.2.arch1-1 and added "module_blacklist=kvm" to the kernel command line:

$ grep "LABEL arch$" -A5 /boot/EFI/syslinux/syslinux.cfg
LABEL arch
    MENU LABEL Arch Linux
    LINUX ../../vmlinuz-linux
    APPEND cryptdevice=/dev/nvme0n1p5:encStorage root=/dev/encStorage/rootvol rw module_blacklist=kvm
    INITRD ../../amd-ucode.img,../../initramfs-linux.img

Blacklisting the kvm kernel module worked like that. Booting with this config ended with the boot loop.

lsmod |grep kvm
kvm_amd               167936  0
kvm                  1118208  1 kvm_amd
irqbypass              16384  1 kvm
ccp                   135168  1 kvm_amd

Will check the next kernels you edited.

Edit: The latest kernel was good:

$ uname -r
5.18.0-rc1-1-00020-g9704c07bf9f7

Edit2: Edited the post for correct configuration. Blacklisting the kvm module did not help on the new kernel. Still get a boot loop

Last edited by Watnuss (2022-08-20 18:39:27)

seth

Member

Registered: 2012-09-03

Posts: 59,882

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

"module_blacklist=kvm"

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

Thanks seth. I fixed my post before I saw yours. I don't want to leave your post looking "out of context".

loqs

Member

Registered: 2014-03-06

Posts: 18,132

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

git bisect good
Bisecting: 9 revisions left to test after this (roughly 3 steps)
[b66370db9a90b3fa4c4a1a732af3e7e38d6d4c7c] KVM: x86: Move lookup of indexed CPUID leafs to helper

https://drive.google.com/file/d/1pSDHH4 … sp=sharing linux-5.18rc1.r30.gb66370db9a90-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1He885C … sp=sharing linux-headers-5.18rc1.r30.gb66370db9a90-1-x86_64.pkg.tar.zst

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

$ uname -r
5.18.0-rc1-1-00030-gb66370db9a90

good commit

loqs

Member

Registered: 2014-03-06

Posts: 18,132

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

seth any guess in what is left https://github.com/torvalds/linux/compa … e7e4398466
Edit:

git bisect good
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[5f211f4fc49622473667e6983bb57beab755f6f6] x86/compressed: Use firmware-validated CPUID leaves for SEV-SNP guests

https://drive.google.com/file/d/189IGU5 … sp=sharing linux-5.18rc1.r35.g5f211f4fc496-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/12NCvbY … sp=sharing linux-headers-5.18rc1.r35.g5f211f4fc496-1-x86_64.pkg.tar.zst

Last edited by loqs (2022-08-20 19:13:28)

seth

Member

Registered: 2012-09-03

Posts: 59,882

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

You mean out of the remaining patches that all tap into the compressed image?
https://github.com/torvalds/linux/commi … 7f9bbc39f3
If the bootloader assumes the size of the struct, also the code has an explicit warning for bootloader authors above it…

Edit: yes it fixes the variable by reducing the padding - it's still the most sketchy thing I see on a glance

Last edited by seth (2022-08-20 19:19:31)

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

Watnuss

Member

Registered: 2013-03-04

Posts: 53

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

good commit as well:

$ uname -r
5.18.0-rc1-1-00035-g5f211f4fc496

seth

Member

Registered: 2012-09-03

Posts: 59,882

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

Because of #47, has anyone tried simply rebuilding syslinux?

---

How to upload text · How to boot w/o GUI · Disable Windows Fast-Start! · Fix your xinitrc

loqs

Member

Registered: 2014-03-06

Posts: 18,132

Re: Lock up on upgrade kernel 5.18.16-arch1-1 to 5.19.1-arch2-1

git bisect good
Bisecting: 2 revisions left to test after this (roughly 1 step)
[76f61e1e89b32f3e5d639f1b57413a919066da06] x86/compressed/64: Add identity mapping for Confidential Computing blob

https://drive.google.com/file/d/1RpZey9 … sp=sharing linux-5.18rc1.r37.g76f61e1e89b3-1-x86_64.pkg.tar.zst
https://drive.google.com/file/d/1jo-KM3 … sp=sharing linux-headers-5.18rc1.r37.g76f61e1e89b3-1-x86_64.pkg.tar.zst

syslinux did not build last year https://bugs.archlinux.org/task/70765 and it looks like it needs more work for glibc 2.36 https://lore.kernel.org/all/CA+chaQcSS4 … gmail.com/ as well as the kernel ABI changes.