Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

mokchira · 2022-12-02 01:42:33

I run pacman -Syu, reboot, and then get stuck in the boot screen, just after the grub menu when I select the kernel. I get some log messages printed and eventually it just halts. This has actually been going on for a while, starting with the 5.19 kernel back in August. Each time nothing seems to work keyboard wise and I have to push the power the button to shut it down. However, today when I gave it another go and it got stuck again I tried ssh'ing in from another machine and voila! I was able to get in. So possibly the whole time the machine was actually booting, but the display just freezes somewhere along the way. Its been about 20 minutes now since the restart, and the display still is just showing

:: running early hook [udev]
Starting systemd-udevd version 252.2-2-arch
:: running hook [udev]
:: Triggering uevents

from the boot console. So it seems to be a display issue?

The machine is an Alienware Area-51m laptop, running BIOS 1.22.2
CPU: i9-9900K
GPU: Nvidia GeForce RTX 2080
Everything is just stock hardware wise. Never opened it up or anything.

The CPU has an integrated GPU as well, but I've never tried to configure things so that it switches to it ever - I have like 30 minutes of battery life when its unplugged and I just accept that. Basically I'm saying I have never done anything fancy with hardware settings. I just run the regular community nvidia packages and up until 5.19 things I had no issues really.

The last nvidia package that worked, the one I'm reverting back to each time the upgrade fails, is 515.57. Ditto for nvidia-settings. The version I just upgraded to is 525.60, but I remember the last one 520 something also did not work.

The linux kernel I keep rolling back to is 5.18.9.arch1-1.

Fortunately since I can ssh in I should be able to provide journal output, pacman logs, whatever. I'm just not sure what would help diagnose the problem. Please let me know. I did run journalctl -b and checked for anything suspicious. I have a bit of red text but nothing I didn't also have before the upgrade. The only red text that shows up with nvidia is this bit

Dec 01 19:10:12 boswell kernel: nvidia-gpu 0000:01:00.3: i2c timeout error e0000000
Dec 01 19:10:12 boswell kernel: ucsi_ccg 6-0008: i2c_transfer failed -110
Dec 01 19:10:12 boswell kernel: ucsi_ccg 6-0008: ucsi_ccg_init failed - -110

but I googled that and most folk say its not a big deal so idk. I could also paste the whole boot log but its about 1700 lines. Not sure what the protocol is for that.

I'd really appreciate help solving this issue. I've sort of been limping along with this older kernel and packages but I know I can only avoid upgrading for so long before I need to just switch to another distro. I really don't want to though! So please help!

cfr · 2022-12-02 02:56:41

Post the journal from a relevant boot to a pastebin site (see link in my signature below). Also post pacman.log.

Do you get the same problems if you use the lts kernel (with the appropriate nvidia package)? Have you tried the nouveau drivers?

mokchira · 2022-12-02 03:35:34

Journal -> http://0x0.st/okKR.txt
Pacman log -> https://0x0.st/okPs.txt

When I try the linux-lts kernel it just hangs at

Loading initial ramdisk

and I can't ssh in. However, I did not know that there was a specific nvidia-lts package that had to be used with it. Thank you for mentioning that. Is it possible to install both that and the regular one? Or do I need to remove one to use the other?

And no I have not tried the nouveau drivers. I could see if those work.

mokchira · 2022-12-02 03:54:38

I went ahead and installed nvidia-lts, rebooted, tried booting into linux-lts, and still got hung on loading initial ramdisk. Also tried the same after removing the regular nvidia and nvidia-settings packages and got the same result.

cfr · 2022-12-02 04:09:39

I'm not 100% sure about the nvidia package, but I would be surprised if you couldn't have both installed, just as you can have the lts kernel alongside the regular one.

pacman -Qi <name-of-your-current-nvidia-package>

should include a 'Conflicts' line which tells you for certain. But the packages are different because they need to match the kernel version, so they shouldn't conflict unless there's something specific about the nvidia stuff. When you install the linux alongside linux-lts, you also install two sets of modules etc. and that's all perfectly fine, so packages which match kernels usually follow that pattern.

I don't know, but the following looked potentially interesting?

Dec 01 19:10:12 boswell kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module  525.60.11  Wed Nov 23 23:04:03 UTC 2022
Dec 01 19:10:12 boswell systemd-udevd[382]: nvidia: Process '/usr/bin/bash -c '/usr/bin/mknod -Z -m 666 /dev/nvidiactl c $(grep nvidia-frontend /proc/devices | cut -d \  -f 1) 255'' failed with exit code 1.
Dec 01 19:10:12 boswell systemd-udevd[382]: nvidia: Process '/usr/bin/bash -c 'for i in $(cat /proc/driver/nvidia/gpus/*/information | grep Minor | cut -d \  -f 4); do /usr/bin/mknod -Z -m 666 /dev/nvidia${i} c $(grep nvidia-frontend /proc/devices | cut -d \  -f 1) ${i}; done'' failed with exit code 1.

Do you get anything similar in the journal with the nvidia version which works? If so, it is probably insignificant (like the 'failures' systemd now reports if you aren't doing everything its way).

Dec 01 19:10:12 boswell kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  525.60.11  Wed Nov 23 22:49:17 UTC 2022
Dec 01 19:10:12 boswell kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.
Dec 01 19:10:12 boswell kernel: [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
Dec 01 19:10:12 boswell kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 1
Dec 01 19:10:12 boswell kernel: nvidia-gpu 0000:01:00.3: i2c timeout error e0000000
Dec 01 19:10:12 boswell kernel: ucsi_ccg 6-0008: i2c_transfer failed -110
Dec 01 19:10:12 boswell kernel: ucsi_ccg 6-0008: ucsi_ccg_init failed - -110
Dec 01 19:10:12 boswell kernel: ucsi_ccg: probe of 6-0008 failed with error -110

Again, I'd compare with a 'good' journal to decide if they're worth looking into. The next messages don't look catastrophic or anything.

Dec 01 19:10:12 boswell systemd[1]: Starting Load/Save Screen Backlight Brightness of backlight:acpi_video1...
Dec 01 19:10:12 boswell systemd[1]: Finished Load/Save Screen Backlight Brightness of backlight:acpi_video1.
Dec 01 19:10:12 boswell audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-backlight@backlight:acpi_video1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Dec 01 19:10:12 boswell systemd-modules-load[339]: Inserted module 'nvidia_uvm'
Dec 01 19:10:12 boswell kernel: nvidia-uvm: Loaded the UVM driver, major device number 511.

Can you boot to the multi-user.target? Have you tried switching to a tty when you boot normally with the problem versions?

Have you checked if there are any firmware updates for your machine (e.g. with fwupd, if your laptop is included, or otherwise, if not)?

seth · 2022-12-02 09:01:56

GPU: Nvidia GeForce RTX 2080

Dec 01 19:10:11 boswell kernel: i915 0000:00:02.0: [drm] failed to retrieve link info, disabling eDP

Do you run the thing attached to an extenal monitor?
Try to add "module_blacklist=i915" to the kernel commandline.

mokchira · 2022-12-02 16:36:24

Thank you both for taking a look.

Here is a journal from today's boot, since I'm back on the working version again -> http://0x0.st/okA6.txt

@cfr yes all those lines are in both journals it seems.
I'm sorry, I'm not sure what you mean by booting to the multi-user.target. All I know about the multi-user.target is I see it at the bottom of many of my systemd unit files (I've been meaning to learn more about systemd). How exactly would I try to boot to it?
I have tried switching terminals and have not been successful, but I can give that another go on my next attempt. (It is possible that on earlier kernels I had a different issue than the one I have now with 6.0.10. I think I remember on 5.19 getting more print statements out before the hang).
Not sure if this helps. but I can say that I don't use a desktop environment. I just boot to the default tty login prompt, and then after login I run startx to kick off xorg and my window manager.

I did actually take a look at what new firmware Dell has put out, and I upgraded my bios to the latest version last week. There are a few other firmware updates listed I could grab, but it was a sketchy experience because my machine also hung during the bios update and I had to let the battery die in order to reboot since my power button wasn't working. So I have held off. I have to do the firmware update from my windows partition - unfortunately my laptop itself is not listed by fwupd, not sure if some individuale controllers in my machine might be though?
And that reminds me to mention that I have a windows partition... if that is relevant. Grub doesn't "know" about it. I access it through the BIOS (holding F12 as the computer boots, before I get to grub). That exists on a separate physical drive, and have been able to access it fine throughout.
Anyway, I will give the other firmware updates a try. I imagine they will be less sketchy than the BIOS update.

@seth I do run with an external monitor attached. I have that line in my working boot log as well, although I noticed these lines follow it in the working version but not in the broken one:

Dec 02 10:38:48 boswell kernel: i915 0000:00:02.0: [drm] [ENCODER:94:DDI B/PHY B] is disabled/in DSI mode with an ungated DDI clock, gate it
Dec 02 10:38:48 boswell kernel: i915 0000:00:02.0: [drm] [ENCODER:111:DDI C/PHY C] is disabled/in DSI mode with an ungated DDI clock, gate it
Dec 02 10:38:48 boswell kernel: i915 0000:00:02.0: [drm] [ENCODER:115:DDI E/PHY E] is disabled/in DSI mode with an ungated DDI clock, gate it

I can try blacklisting as you suggest, and also just booting without the external monitor plugged in to see what happens.

Again, I appreciate the help very much.

seth · 2022-12-02 16:57:18

I'm sorry, I'm not sure what you mean by booting to the multi-user.target. All I know about the multi-user.target is I see it at the bottom of many of my systemd unit files (I've been meaning to learn more about systemd). How exactly would I try to boot to it?

2nd link below

Do you currently have the internal *and* external output enabled / in use?
Both end up "freezing"?
Either way, try to blacklist i915 (edit the kernel commandline at the bootloader, that's transient and gone w/ the next reboot)

mokchira · 2022-12-02 18:13:35

So good news! Adding that module_blacklist=i915 parm seems to have fixed it! I was able to boot after adding that. I've done 5 reboots with it. Four worked. One actually hung but I got farther along - I got Welcome to Archlinux and the last line before the hang was

[ OK ] Found device Samsung_SSD_860_PRO_1TB 1.

but I think this is a separate issue... I've had a few occasional boot hangs even with the 5.18 setup. If you have any suggestions there I'm all ears, otherwise I'll do some more investigation and possibly open a separate thread on it.

I did look at the available firmware updates but nothing was listed as Critical (the BIOS update I did earlier was). There was an Intel UHD graphics card update which I installed, but that unfortunately made no difference. The only thing that worked was the kernel parameter. Thank you so much for spotting that in the boot log.

Is the best thing to just keep using that kernel parameter, or now that we had an idea of the source of the problem are there better solutions you might suggest? Maybe Intel specific packages I might need? Is this a potential bug that should be reported somewhere? Curious about next steps in general.

Thanks for the tip about the multi-level.target thing. I had no idea about that. Will add it to my tool belt for debugging future issues.

Edit: yes the attached laptop monitor and exernal one should both be "enabled", though in practice its just the laptop one that ever actually displays anything on the boot terminal. The external one is just dark until I actually startx. So the laptop one is the one that was freezing. It froze even when I didn't have the external monitor plugged in.

Last edited by mokchira (2022-12-02 18:17:34)

cfr · 2022-12-02 18:22:12

You're booting to multi-user.target by default anyway, so my suggestion was irrelevant. multi-user is just console/virtual terminal without any GUI starting.

seth · 2022-12-02 18:31:43

You can alternatively make the i915 module pre-load the nvidia one, see eg. https://bbs.archlinux.org/viewtopic.php … 6#p2059606
The problem is w/ the nvidia module and recent kernels, afaiu it's addressed in nvidia-open (turing and newer only), but for the prop. nvidia driver you'll have to wait…

Possibly moving only the nvidia module into the initramfs might also work, https://wiki.archlinux.org/title/NVIDIA#Early_loading

mokchira · 2022-12-02 19:08:20

Ok, I'll look into those options.

Thank you both again for taking the time out of your days to help me solve this issue.

omniscient_potato · 2022-12-08 21:43:01

I'm currently facing the same issue.
Brand new hardware and arch installation with disk encryption.
After grub I can see the initramfs loading output, then it shows "Starting systemd-udevd version 252.2-3-arch" and "freezes".
However, if I type in my encryption password, it continues to my greeter, where I can put in my password but it never goes past that.
If I try to switch to another tty, it shows the "starting systemd..."-message again.

I tried blacklisting the i915 both in the kernel parameters and as a file in modprobe.d, both without sucess.

Any suggestions?

mokchira · 2022-12-08 22:00:31

Sounds like your issue is a little different from mine. Once my screen froze on the boot message is never got unfrozen, entering my password and logging in did not change that. Also nothing happened when I tried to change terminals. And, of course, changing that kernel parm did fix my issue. Is your hardware the same as mine?

You might want to make a new thread for your problem, and post you boot logs and such like i did.

seth · 2022-12-08 23:04:43

Please post an updated journal w/ i915 blacklisted.

omniscient_potato · 2022-12-09 10:26:02

Unfortunately, there is no journal being created, as long as I don't try to get pasty my disk encryption blindly.

I opened a new post here so that I don't takeover this one.

timminn · 2022-12-10 11:58:39

I've been compiling my own kernels recently,
and subsequently encountered the same problem.

After trying out so many times,
I'm pretty sure it occurs when patched from 5.18.12 to 5.18.13,
which seems to be quite a big set of patches.
ALL the kernel versions >= 5.18.13, including 6.0.x,
has the same problem.

Though I cannot find out which exact part is causing the problem.
Now I decided to stay at 5.18.12.

Hope this could help a little bit.

Last edited by timminn (2022-12-10 12:00:22)

j.mlr · 2022-12-11 12:33:47

I can confirm this happens to me with the Zen kernel (6.0.12), but not with the LTS kernel (5.15.82).

seth · 2022-12-11 14:15:21

System journal?

j.mlr · 2022-12-12 10:22:59

It seems that my issue might be different from what was reported in this thread.

When I boot with the Zen kernel (6.0.12), the system tells me to enter my passcode to decrypt my drive. Afterwards, it hangs at "Arch: clean ...", but the cursor continue to blink so it is not "frozen".

I tried to boot without GUI (systemctl set-default multi-user.target), but it can't get past "Arch: clean ..." either. I am not sure how to get the system journal in this case...

Blacklisting i915 did nothing.

I can boot with the LTS kernel (5.15.82) though.

Arch Linux

#1 2022-12-02 01:42:33

Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#2 2022-12-02 02:56:41

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#3 2022-12-02 03:35:34

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#4 2022-12-02 03:54:38

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#5 2022-12-02 04:09:39

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#6 2022-12-02 09:01:56

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#7 2022-12-02 16:36:24

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#8 2022-12-02 16:57:18

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#9 2022-12-02 18:13:35

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#10 2022-12-02 18:22:12

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#11 2022-12-02 18:31:43

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#12 2022-12-02 19:08:20

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#13 2022-12-08 21:43:01

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#14 2022-12-08 22:00:31

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#15 2022-12-08 23:04:43

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#16 2022-12-09 10:26:02

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#17 2022-12-10 11:58:39

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#18 2022-12-11 12:33:47

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#19 2022-12-11 14:15:21

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

#20 2022-12-12 10:22:59

Re: Boot screen freeze after upgrading from Linux 5.18.9 -> 6.0.10

Board footer