You are not logged in.

#1 2021-11-13 12:34:43

Strangiato
Member
Registered: 2020-01-10
Posts: 103

Unable to boot after yesterday's kernels (linux and linux-zen) updates

Hi

Both linux 5.14.16 (stable) and linux-zen 5.15 (testing) were installed on my system. Yesterday I updated my system by running 'pacman -Syu'
and today my system was unable to boot. I selected linux-zen entry from grub and then I got " kernel not found, press any key to continue" error message, or something similar. Then I rebooted and chose linux entry from grub and got the same result. Then I booted with a live usb, did arch-chroot to my install, updated grub by running

grub-mkconfig -o /boot/grub/grub.cfg

Rebooted, and Windows 7 was the only operating system listed in my grub.
Then I did arch-chroot again, reinstalled linux-zen and now my Arch is bootable again.
What has happened here? How to prevent this problem?

Here is pacman.log after problematic update:

[2021-11-12T19:17:45-0300] [PACMAN] Running 'pacman -Syu'
[2021-11-12T19:17:45-0300] [PACMAN] synchronizing package lists
[2021-11-12T19:17:57-0300] [PACMAN] starting full system upgrade
[2021-11-12T19:21:47-0300] [ALPM] running '60-mkinitcpio-remove.hook'...
[2021-11-12T19:21:47-0300] [ALPM] running '70-dkms-upgrade.hook'...
[2021-11-12T19:21:48-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m vhba-module -v 20211023 -k 5.15.0-zen1-1-zen
[2021-11-12T19:21:48-0300] [ALPM-SCRIPTLET] rmdir: failed to remove ' ': File or directory does not exist 
[2021-11-12T19:21:49-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m vhba-module -v 20211023 -k 5.14.16-arch1-1
[2021-11-12T19:21:49-0300] [ALPM] running '71-dkms-remove.hook'...
[2021-11-12T19:21:50-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m vboxhost -v 6.1.28_OSE -k 5.14.16-arch1-1
[2021-11-12T19:21:51-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m r8168 -v 8.049.02 -k 5.14.16-arch1-1
[2021-11-12T19:21:52-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m vboxhost -v 6.1.28_OSE -k 5.15.0-zen1-1-zen
[2021-11-12T19:21:53-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m r8168 -v 8.049.02 -k 5.15.0-zen1-1-zen
[2021-11-12T19:21:53-0300] [ALPM-SCRIPTLET] ==> depmod 5.14.16-arch1-1
[2021-11-12T22:26:06-0300] [PACKAGEKIT] synchronizing package lists

Last edited by Strangiato (2021-11-13 13:46:57)

Offline

#2 2021-11-13 12:47:23

icar
Member
From: Catalunya
Registered: 2020-07-31
Posts: 244

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

[2021-11-12T22:26:06-0300] [PACKAGEKIT] synchronizing package lists

Last time I tried PackageKit, it uninstalled my DE and many other packages for no apparent reason. Pretty sure this is its fault. Can you check its logs? I think they were in /var/log/Packagekit.log

Last edited by icar (2021-11-13 12:49:42)

Offline

#3 2021-11-13 12:48:15

Scimmia
Fellow
Registered: 2012-09-01
Posts: 8,747

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

If that's all that's in the log, it looks like pacman was killed before actually updating.

Online

#4 2021-11-13 12:49:14

Ammako
Member
Registered: 2021-07-16
Posts: 266

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

Strangiato wrote:

What has happened here? How to prevent this problem?

Well, for starters, you don't just blindly answer Yes to pacman -Syu. You're expected to check what it's doing first, if you spot anything wrong then you abort and figure out why it's trying to do what it's trying to do.

From the looks of your logs though, it never uninstalled any of your kernels. I'm guessing you misconfigured your system (fstab?) and grub is looking for kernel and initramfs in a different location than where the OS is placing them.

Offline

#5 2021-11-13 12:58:15

Strangiato
Member
Registered: 2020-01-10
Posts: 103

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

icar wrote:
[2021-11-12T22:26:06-0300] [PACKAGEKIT] synchronizing package lists

Last time I tried PackageKit, it uninstalled my DE and many other packages for no apparent reason. Pretty sure this is its fault. Can you check its logs? I think they were in /var/log/Packagekit.log

/var/log/pacman.PackageKit.log has thousands of lines saying

[date/time] [PACKAGEKIT] synchronizing package lists

and nothing else.

Ammako wrote:
Strangiato wrote:

What has happened here? How to prevent this problem?

Well, for starters, you don't just blindly answer Yes to pacman -Syu. You're expected to check what it's doing first, if you spot anything wrong then you abort and figure out why it's trying to do what it's trying to do.

From the looks of your logs though, it never uninstalled any of your kernels. I'm guessing you misconfigured your system (fstab?) and grub is looking for kernel and initramfs in a different location than where the OS is placing them.

You are right Ammako. Yesterday I edited my fstab to enable ACL permissions for ntfs partitions mounted with ntfs3 driver.
I was getting "denied permisison" error when trying to write files into folders created with another user account.

Scimmia wrote:

If that's all that's in the log, it looks like pacman was killed before actually updating.

it's the log of the last update before the problem. A few days ago pacman crashed on my system after kernel update.

$coredumpctl|grep pacman
Wed 2021-11-03 19:57:07 -03  1452    0    0 SIGSEGV missing  /usr/bin/pacman                               n/a

Last edited by Strangiato (2021-11-13 13:19:01)

Offline

#6 2021-11-13 13:09:07

Ammako
Member
Registered: 2021-07-16
Posts: 266

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

Strangiato wrote:

You are right Ammako. Yesterday I edited my fstab to enable ACL permissions for ntfs partitions mounted with ntfs3 driver.
I was getting "denied permisison" error when trying to write files into folders created with another user account.

Nah, forget what I said about that. 60-mkinitcpio-remove.hook was run, for some reason. That would be why your vmlinuz kernel files were deleted. Why? No clue. Your packages weren't uninstalled or upgraded.

Offline

#7 2021-11-13 16:24:21

seth
Member
Registered: 2012-09-03
Posts: 25,010

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

Ammako wrote:

60-mkinitcpio-remove.hook was run, for some reason.

Strangiato wrote:

Both linux 5.14.16 (stable) and linux-zen 5.15 (testing) were installed on my system. Yesterday I updated my system by running 'pacman -Syu'

To update the kernel, at least the vanilla one is at 5.15.2

Scimmia wrote:

If that's all that's in the log, it looks like pacman was killed before actually updating.

=> Upload the complete log to some pastebin service or google drive or wetransfer or whatever.

Offline

#8 2021-11-13 16:28:44

Ammako
Member
Registered: 2021-07-16
Posts: 266

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

seth wrote:

To update the kernel, at least the vanilla one is at 5.15.2.

Yes, but as far as the logs posted show, no package upgrade actually happened.

Last edited by Ammako (2021-11-13 16:29:24)

Offline

#9 2021-11-13 16:30:04

seth
Member
Registered: 2012-09-03
Posts: 25,010

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

Scimmia wrote:

If that's all that's in the log, it looks like pacman was killed before actually updating.

That's why we need to see the full log (and or get an explanation for its abrupt ending)

Offline

#10 2021-11-13 17:05:58

Strangiato
Member
Registered: 2020-01-10
Posts: 103

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

full log can be downloaded from the following link:

https://www.mediafire.com/file/p03ccwdj … n.log/file

The log is too long to share via pastebin services.

Offline

#11 2021-11-14 15:38:31

seth
Member
Registered: 2012-09-03
Posts: 25,010

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

So there's

[2021-11-12T19:17:45-0300] [PACMAN] Running 'pacman -Syu'
[2021-11-12T19:17:45-0300] [PACMAN] synchronizing package lists
[2021-11-12T19:17:57-0300] [PACMAN] starting full system upgrade
[2021-11-12T19:21:47-0300] [ALPM] running '60-mkinitcpio-remove.hook'...
[2021-11-12T19:21:47-0300] [ALPM] running '70-dkms-upgrade.hook'...
[2021-11-12T19:21:48-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m vhba-module -v 20211023 -k 5.15.0-zen1-1-zen
[2021-11-12T19:21:48-0300] [ALPM-SCRIPTLET] rmdir: falhou em remover '': Arquivo ou diretório inexistente
[2021-11-12T19:21:49-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m vhba-module -v 20211023 -k 5.14.16-arch1-1
[2021-11-12T19:21:49-0300] [ALPM] running '71-dkms-remove.hook'...
[2021-11-12T19:21:50-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m vboxhost -v 6.1.28_OSE -k 5.14.16-arch1-1
[2021-11-12T19:21:51-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m r8168 -v 8.049.02 -k 5.14.16-arch1-1
[2021-11-12T19:21:52-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m vboxhost -v 6.1.28_OSE -k 5.15.0-zen1-1-zen
[2021-11-12T19:21:53-0300] [ALPM-SCRIPTLET] ==> dkms remove --no-depmod -m r8168 -v 8.049.02 -k 5.15.0-zen1-1-zen
[2021-11-12T19:21:53-0300] [ALPM-SCRIPTLET] ==> depmod 5.14.16-arch1-1

[2021-11-12T22:26:06-0300] [PACKAGEKIT] synchronizing package lists

[2021-11-13T09:07:17-0300] [PACMAN] Running 'pacman -S linux-zen linux-zen-headers'
[2021-11-13T09:07:50-0300] [ALPM] running '60-mkinitcpio-remove.hook'...
[2021-11-13T09:07:50-0300] [ALPM] running '71-dkms-remove.hook'...
[2021-11-13T09:07:50-0300] [ALPM-SCRIPTLET] ==> Unable to remove module vboxhost/6.1.28_OSE for kernel 5.15.0-zen1-1-zen: Not found in dkms status output.
[2021-11-13T09:07:50-0300] [ALPM-SCRIPTLET] ==> Unable to remove module vhba-module/20211023 for kernel 5.15.0-zen1-1-zen: Not found in dkms status output.
[2021-11-13T09:07:50-0300] [ALPM-SCRIPTLET] ==> Unable to remove module r8168/8.049.02 for kernel 5.15.0-zen1-1-zen: Not found in dkms status output.
[2021-11-13T09:07:50-0300] [ALPM] transaction started

And I assume the -Syu on 11/12 was the compromising one.
It starts, cleans up syncs the package list and then does nothing and after ~3h PACKAGEKIT shows up (what seems an unrelated frequent injection) and then nothing happens until you ran 'pacman -S linux-zen linux-zen-headers' the next morning (11/13)

So: what happened that evening?
There's no explanation as to why the hooks were triggered, what was supposed to be updated at all and why apparently the update didn't happen.
The log shows a gap, but you'll have to fill in the blanks from your memory.

Offline

#12 2021-11-14 16:13:57

Strangiato
Member
Registered: 2020-01-10
Posts: 103

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

Thank you for your analisys, seth.
Really I don't remember doing anything related to pacman or packagekit in this internal

[2021-11-12T19:21:53-0300] [ALPM-SCRIPTLET] ==> depmod 5.14.16-arch1-1

[2021-11-12T22:26:06-0300] [PACKAGEKIT] synchronizing package lists

I can be wrong though. My memory is not very good.

Could

[2021-11-12T22:26:06-0300] [PACKAGEKIT] synchronizing package lists

be related to Discover app store (I use KDE Plasma) checking for updates in the background?

Offline

#13 2021-11-14 20:55:59

seth
Member
Registered: 2012-09-03
Posts: 25,010

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

Could [2021-11-12T22:26:06-0300] [PACKAGEKIT] synchronizing package lists be related to Discover app store (I use KDE Plasma) checking for updates in the background?

Probably. Not related to the problem, though.

How to prevent this problem?

Pay attention to the pacman outputs. If you're using some GUI or other wrapper around pacman - maybe don't.

Offline

#14 2021-11-14 23:00:59

Strangiato
Member
Registered: 2020-01-10
Posts: 103

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

I was doing multiple things at the same time during the update.
Is there any way to know if I accidentally closed my terminal emulator (Konsole) before pacman completing update process?

Offline

#15 2021-11-15 07:18:27

seth
Member
Registered: 2012-09-03
Posts: 25,010

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

KDE can/could log window closes, but it's rather not enabled by default - so "no".
But you can decouple terminals from windows: https://wiki.archlinux.org/title/Tmux

Offline

#16 2021-11-15 12:45:11

Strangiato
Member
Registered: 2020-01-10
Posts: 103

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

Thanks for suggesting Tmux.
Pacman.log should tell when Pacman is interrupted during a update, no?

Offline

#17 2021-11-15 12:51:40

seth
Member
Registered: 2012-09-03
Posts: 25,010

Re: Unable to boot after yesterday's kernels (linux and linux-zen) updates

If you sigkill'd the process, it's not gonna log anything for sure.
If you did a hard reboot, the log wasn't synced to disk.
Not sure whether it logs a sigterm.

And while I guess I've at some point in the past sigint'ed pacman, I don't see anything like that in the log (but I've also likely never done that *during* an update)

Offline

Board footer

Powered by FluxBB