Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

threefour · 2022-07-25 13:48:49

Exactly as title says. Did a package upgrade on my work machine this morning, all went routinely. Rebooted my system, and it got stuck on the "Starting arch..." or whatever message it shows just after GRUB. I arch-chrooted in, only downgraded the `linux` and `linux-headers` packages from 5.18.14 back to 5.18.12, and that fixed the problem, and it was bootable again. And here I am now, writing a post on it.

Anyone know what's going on?

ewaller · 2022-07-25 15:13:59

Please post the output of mount
And, what GPU are you using?

threefour · 2022-07-25 15:18:24

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=5984728k,nr_inodes=1496182,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
/dev/nvme0n1p2 on / type ext4 (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=12732)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,nr_inodes=1048576,inode64)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=1199488k,nr_inodes=299872,mode=700,uid=1000,gid=998,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=998)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=998)
net_cls on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls)
overlay on /var/lib/docker/overlay2/fa2281fb646964a9d324a8676f569839d08dd379354223a2bc63c9c701683e02/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/EITNO2XDMNYJDJUW75Z7XVPMUQ:/var/lib/docker/overlay2/l/FUDY6CIBW6ZL6DBE3MQMCU3WAP:/var/lib/docker/overlay2/l/XDL2H7L3R4RFRKIGFMIR5A736O:/var/lib/docker/overlay2/l/GA6G7YFMVIYY42X64X4OYXW7OF:/var/lib/docker/overlay2/l/VXFLZDMK5WHCWY2KDNKKTRLZMB:/var/lib/docker/overlay2/l/6ZSERQ6TMO34B63RC5A6QKFQBL:/var/lib/docker/overlay2/l/JOPIFUKBO7ECL4L3LEQMUFBZWV,upperdir=/var/lib/docker/overlay2/fa2281fb646964a9d324a8676f569839d08dd379354223a2bc63c9c701683e02/diff,workdir=/var/lib/docker/overlay2/fa2281fb646964a9d324a8676f569839d08dd379354223a2bc63c9c701683e02/work,index=off)
nsfs on /run/docker/netns/517e6575045f type nsfs (rw)

I am using a GT 710, with the "nvidia-470xx-..." packages from AUR, at their latest.

Last edited by threefour (2022-07-25 15:36:01)

loqs · 2022-07-25 15:27:20

Can you please post the full system journal for the failed boot?

ewaller · 2022-07-25 15:28:57

Be advised we use BBCode and not markup
https://bbs.archlinux.org/help.php#bbcode

So, unless I am missing it, I do not see where your boot volume is mounted on /boot. Is this an EFI system?

Normally, one mounts the boot partition on /boot. When the kernel and initramfs are updated they update the files in /boot. As /boot is supposed to be a mount point for the boot partition, the files are supposed to be written to said boot partition. If the boot partition is not mounted, the new kernel and initramfs are written to the DIRECTORY called /boot where they are pretty useless.

The next time you boot, the boot loader picks up the old kernel that is still on the boot partition, it starts, but then cannot find any of its kernel modules on / because they are not there anymore -- the modules for the new kernel are there.

Read this over. If this is what happened, we can talk you through cleaning this up and fixing it.

threefour · 2022-07-25 15:35:12

I've never used a forum with BBCode, so thanks for the heads up.

I don't see anything mounted in /boot either, however the directory does exist. It is an EFI system. For what it's worth, I've been using this system as-is for months, and updated the kernel many times. This is the first time it's done this to me.

Edit: I guess I should clarify, there are files in /boot, I'm just saying you didn't miss anything.

Last edited by threefour (2022-07-25 15:37:33)

threefour · 2022-07-25 15:42:28

I am mistaken. I guess at some point in the distant past I re-installed using legacy boot. The first partition appears to be "dos" type.

ewaller · 2022-07-25 15:49:17

My guess is that those files in /boot are initramfs-linux.img, initramfs-linux-fallback.img and vmlinuz-linux.
Next we need to mount your boot partition on /boot, You should then see several files in /boot inlcluding those listed above, an EFI directory and perhaps some files related to your boot loader or other operating systems.
At this point, update the kernel and reboot.
You should be able to boot to the new kernel at this point.

I'll pause here. If this all works I'll help you with the clean up and fixing this so it does not happen again.

As to previous kernel updates, don't know -- I can but speculate. Perhaps they were not major (or minor) updates but just a release change. Perhaps you changed your /etc/fstab.

Edit: Just saw your post -- So there may not be an EFI directory if this is legacy boot.

Last edited by ewaller (2022-07-25 15:50:54)

Morta · 2022-07-25 16:06:56

I have the same problem. See my logs or my thread in other section.

http://0x0.st/ojpk.txt

threefour · 2022-07-25 16:12:12

Next we need to mount your boot partition on /boot, You should then see several files in /boot inlcluding those listed above, an EFI directory and perhaps some files related to your boot loader or other operating systems.

Meaning simply do

doas mount /nvme0n1p1 /boot

in my particular case? Doing that causes the following existing items to be erased from the /boot directory for the duration of the mount:

initramfs-linux-fallback.img
initramfs-linux.img
intel-ucode.img
syslinux/
vmlinuz-linux

So my /boot directory now looks like:

EFI/
grub/
NvVars

Is that correct? I should run the upgrade like that? Also, yes, there is an EFI directory in there apparently. I'm slightly confused. Shouldn't my boot partition be formatted to GPT if it's EFI? Or is that not strictly required?

Last edited by threefour (2022-07-25 16:35:12)

ewaller · 2022-07-25 16:36:18

Well those files are not 'erased' they are masked. The boot DIRECTORY has those files in it (it shouldn't). When you mount a partition on /boot, making it a mount point, you now see the files from that partition under /boot -- you can no longer see the files in the DIRECTORY /boot.

It looks like you are booting using GRUB. But it looks like you tried to setup syslinux at some point.
So yes, go ahead and update the kernel with that mounted. Then reboot.
Then we can clean this up.

seth · 2022-07-25 16:53:55

loqs wrote:

Can you please post the full system journal for the failed boot?

And is this an nvidia system?

threefour · 2022-07-25 16:56:30

Gotcha. I know they're not "erased" per se, I just didn't know the correct term to use.

Okay, so I ran the upgrade and rebooted. I was met with:

xf86EnableIO: failed to enable I/O ports 0000-03ff (Operation not permitted)

As well as X.org telling me "no screens found", which makes it sound like this is a graphics driver issue.

threefour · 2022-07-25 17:14:54

And is this an nvidia system?

Yes, Nvidia GT 710 running the nvidia-470xx drivers from the AUR.

loqs · 2022-07-25 17:40:55

seth wrote:

loqs wrote:
Can you please post the full system journal for the failed boot?

Edit:
Also the full contents of Xorg.0.log.

Last edited by loqs (2022-07-25 18:01:53)

threefour · 2022-07-25 18:04:23

I'm not sure if the forum prefers a large file pasted inline, or having a link to a paste service. So I'll do the latter: https://paste.sr.ht/~seanld/28d03027dd7 … a700ec3c99

Do you need me to crash the system again with the new kernel before I send the Xorg log? I'd prefer not to, if not utterly necessary, as I have work to do.

Last edited by threefour (2022-07-25 18:08:36)

loqs · 2022-07-25 18:21:44

How do you produce the crash? By starting X? You could see if it is recorded in /var/log/Xorg.0.log.old

threefour · 2022-07-25 18:27:55

Yep, you're right. My terminal theme isn't fully optimized, so the old file blended in with the background and I didn't see it. Here you go: https://paste.sr.ht/~seanld/56013fbb670 … befb9d8ff7

seth · 2022-07-25 21:13:27

Fits https://bbs.archlinux.org/viewtopic.php?id=278312 but the outcome there is varying.
What if you straight up just blacklist i915?
https://wiki.archlinux.org/title/Kernel … and_line_2

The xorg log is from an 5.18.12 boot and ran fine for ~3h

threefour · 2022-07-25 21:20:20

I'll have to try that tomorrow and report back. This is on my work PC in the office, and I'm headed out for the day.

My situation seems to be slightly different, just taking a quick glance over the linked thread. Their screen completely blanks. Mine still has the "Starting arch..." message stuck on the screen until I force shutdown the system with the physical power button. Not sure if that's an important difference or not.

Last edited by threefour (2022-07-25 21:20:32)

seth · 2022-07-25 21:41:06

There's also https://bbs.archlinux.org/viewtopic.php?id=278351 - but that would allow you to (blindly) start X11 and there's also no i915 chip on that system.

And finally there's https://bbs.archlinux.org/viewtopic.php?id=278350
You should remove the "quiet" parameter to see where the boot actually stalls.

threefour · 2022-07-26 14:23:49

It looks like I already have i915 blacklisted.

And here's the journal from my failed boot with "quiet" removed: https://paste.sr.ht/~seanld/b7c7e1d319b … 548e8edc86

seth · 2022-07-26 14:28:48

It looks like I already have i915 blacklisted.

Nope.

Jul 26 08:06:28 inspirarch kernel: i915 0000:00:02.0: enabling device (0000 -> 0003)
Jul 26 08:06:28 inspirarch kernel: Console: switching to colour dummy device 80x25
Jul 26 08:06:28 inspirarch kernel: i915 0000:00:02.0: Direct firmware load for i915/gvt/vid_0x8086_did_0x9bc8_rid_0x03.golden_hw_state failed with error -2
Jul 26 08:06:28 inspirarch kernel: i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/kbl_dmc_ver1_04.bin (v1.4)
Jul 26 08:06:28 inspirarch kernel: [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
Jul 26 08:06:28 inspirarch kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
Jul 26 08:06:28 inspirarch kernel: i915 0000:00:02.0: MDEV: Registered

my failed boot with "quiet" removed

The plan was to check whether you could better see at which exact point (rather than "Starting arch...") the output sticks (that's not visible from the journal, the quiet parameter only has impact on the console output)

threefour · 2022-07-26 14:57:23

Weird. This is the output of:

modprobe --showconfig | grep i915

Also, sorry I misunderstood. When I rebooted with "quiet" removed, it hung immediately after "running early hook [udev]". Quick search yielded another Arch Forum post, where in their case, there was a bug in the kernel and Intel driver causing the failure to boot. So if I can figure out why it's still loading i915 despite me blacklisting it, and also running mkinitcpio, then I can see if that fixes it for me.

seth · 2022-07-26 15:24:25

If you've the module in the initramfs, modprobe.d on your HDD is too late.
Also only the /bin/true method will prevent the module from being explicitly loaded (though that should not be relevant)
=> Using the kernel commandline to blacklist the module should™ suffice (as it will hit the initramfs as well)

Arch Linux

#1 2022-07-25 13:48:49

Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#2 2022-07-25 15:13:59

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#3 2022-07-25 15:18:24

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#4 2022-07-25 15:27:20

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#5 2022-07-25 15:28:57

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#6 2022-07-25 15:35:12

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#7 2022-07-25 15:42:28

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#8 2022-07-25 15:49:17

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#9 2022-07-25 16:06:56

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#10 2022-07-25 16:12:12

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#11 2022-07-25 16:36:18

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#12 2022-07-25 16:53:55

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#13 2022-07-25 16:56:30

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#14 2022-07-25 17:14:54

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#15 2022-07-25 17:40:55

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#16 2022-07-25 18:04:23

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#17 2022-07-25 18:21:44

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#18 2022-07-25 18:27:55

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#19 2022-07-25 21:13:27

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#20 2022-07-25 21:20:20

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#21 2022-07-25 21:41:06

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#22 2022-07-26 14:23:49

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#23 2022-07-26 14:28:48

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#24 2022-07-26 14:57:23

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

#25 2022-07-26 15:24:25

Re: Cannot Boot Arch After Upgrading From Kernel 5.18.12 to 5.18.14

Board footer