You are not logged in.

#1 2022-11-21 17:22:05

boredhero
Member
Registered: 2020-10-14
Posts: 23

[RESOLVED] After update last night, hung boot with failed services.

Hello. I did my daily system upgrade and turned my machine off last night. This morning, I am unable to boot

Starting systemd-udevd version 252.1-1-arch
/dev/md127p1: recovering journal
/dev/md127p1: clean, 1486933/29261824 files, 59423719/117019131 blocks
[FAILED] Failed to start Initialize hardware monitoring sensors.
[FAILED] Failed to start Wireless service.
[FAILED] Failed to start Wireless service.
[FAILED] Failed to start Wireless service.
[FAILED] Failed to start Wireless service.
[FAILED] Failed to start Refresh Pacman mirrorlist with Reflector.
[FAILED] Failed to start Docker Application Container Engine.

Things tried so far:
(1) Reinstall / make new config / initramfs for grub
(2) Uninstalling some services (I had tailscaled and some other random non-essential programs also failing to start and figured it might be one of their faults)
(3) Boot with debug kernel param. All I see is that it tries to start these services again over and over again after they fail the first time
(4) Reinstall linux and linux-headers

I'm stumped.

Last edited by boredhero (2022-11-21 19:24:34)

Offline

#2 2022-11-21 17:25:42

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,463

Re: [RESOLVED] After update last night, hung boot with failed services.

are you getting to a prompt? If so, what does `uname -r && pacman -Q linux` give you?

Offline

#3 2022-11-21 17:30:22

boredhero
Member
Registered: 2020-10-14
Posts: 23

Re: [RESOLVED] After update last night, hung boot with failed services.

Scimmia wrote:

are you getting to a prompt? If so, what does `uname -r && pacman -Q linux` give you?

Unfortunately not. It just hangs on those services that fail. Booted in debug mode, it just shows that it tries to start / audit those services over and over again but keeps failing.

I just tried uninstalling docker and iwd and now i'm left with just

[FAILED] Failed to start Initialize hardware monitoring sensors.
[FAILED] Failed to start Refresh Pacman mirrorlist with Reflector

I've been making all changes to the system with an installer and arch-chroot because I can't get to a terminal

Offline

#4 2022-11-21 17:31:58

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,463

Re: [RESOLVED] After update last night, hung boot with failed services.

Boot an install disk, mount everything, chroot in, and check the journal.

EDIT, before that, make sure you can't access TTY2.

Last edited by Scimmia (2022-11-21 17:32:54)

Offline

#5 2022-11-21 18:11:24

boredhero
Member
Registered: 2020-10-14
Posts: 23

Re: [RESOLVED] After update last night, hung boot with failed services.

Scimmia wrote:

Boot an install disk, mount everything, chroot in, and check the journal.

EDIT, before that, make sure you can't access TTY2.

Most interesting things I was able to find in journalctl for this boot were

systemd-modules-load[336]: Failed to find module 'crypto_user'
systemd-modules-load[336]: Failed to find module 'sg'
systemd-modules-load[336]: Failed to find module 'dm-multipath'
systemd-modules-load[336]: Failed to find module 'vmw_vmci'
systemd-modules-load[336]: Failed to find module 'vmmon'

EDIT: I did try to access TTY2 without success

Last edited by boredhero (2022-11-21 18:11:51)

Offline

#6 2022-11-21 18:13:26

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,463

Re: [RESOLVED] After update last night, hung boot with failed services.

That supports my theory; at the very beginning of the journal, it tells you what kernel it's booting. What is it?

If it matches what's installed, post the entire journal from a bad boot.

Edit: thinking about it more, I may be thinking of the xorg log instead of the journal.

Edit2: It's at the beginning of both, so check the journal for sure.

Last edited by Scimmia (2022-11-21 18:23:18)

Offline

#7 2022-11-21 18:25:29

boredhero
Member
Registered: 2020-10-14
Posts: 23

Re: [RESOLVED] After update last night, hung boot with failed services.

Scimmia wrote:

That supports my theory; at the very beginning of the journal, it tells you what kernel it's booting. What is it?

If it matches what's installed, post the entire journal from a bad boot.

Edit: thinking about it more, I may be thinking of the xorg log instead of the journal.

kernel: Linux version 6.0.8-arch1-1 (linux@archlinux) (gcc (GCC) 12.2.0, GNU ld (GNU Binutils) 2.39.0) #1 SMP PREEMPT_DYNAMIC

This is confusing because I have ran a system update and should presumably be running 6.0.9?

I tried reinstalling linux again and rebooting, only to get the same thing.

Offline

#8 2022-11-21 18:26:49

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,463

Re: [RESOLVED] After update last night, hung boot with failed services.

Yep, that's kind of what I was expecting. Do you have a separate /boot partition? This usually happens when you update without it mounted.

Offline

#9 2022-11-21 18:28:17

boredhero
Member
Registered: 2020-10-14
Posts: 23

Re: [RESOLVED] After update last night, hung boot with failed services.

Scimmia wrote:

Yep, that's kind of what I was expecting. Do you have a separate /boot partition? This usually happens when you update without it mounted.

Yeah it should be /dev/sdb1 in my rather confusing partition table. Although I guess I should check my fstab...

Offline

#10 2022-11-21 18:31:13

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,463

Re: [RESOLVED] After update last night, hung boot with failed services.

I have to take off, but you can check the kernel image with `file` to see what version it is. You either didn't have it mounted, or you did have it mounted, but your bootloader is loading the kernel from somewhere else. Good luck, hopefully you've got it fixed before I get back.

Offline

#11 2022-11-21 19:24:01

boredhero
Member
Registered: 2020-10-14
Posts: 23

Re: [RESOLVED] After update last night, hung boot with failed services.

Scimmia wrote:

I have to take off, but you can check the kernel image with `file` to see what version it is. You either didn't have it mounted, or you did have it mounted, but your bootloader is loading the kernel from somewhere else. Good luck, hopefully you've got it fixed before I get back.

Hey! Thanks so much. Realized that when I fixed my grub after the grub breakage that happened a while ago I accidentally mounted and reinstalled grub to the same partition as window's bootloader, but for some reason the system wasn't using that one somehow?

Anyways, editing my /etc/fstab to boot the right one, deleting the grub installation on the windows EFI partition, and reinstalling grub / reinstalling linux onto the correct partition which was now set to be mounted by fstab solved the issue. Thanks so much for helping! I never would've guessed that those module loading issues meant my mounting got borked!

Offline

Board footer

Powered by FluxBB