System can no longer boot after upgrade [SOLVED]

stefano · 2020-05-10 22:35:18

After today's upgrade my server no longer boots. Grub is still there and loads, but it fails to find the kernel.

People in another, similar thread have suggested reinstalling grub and proceeding from there. I tried to boot into an emergency disk, chroot into the old system, and reinstall grub, but I no longer have a working pacman, it seems. Every combination of pacman commands I tried fails with no output. Specifying the -v flag has no effect either.

I checked /var/lib and I found a lock file, which seems to indicate the update somehow went awry, even though no error messages were produced at the time. Removing the lock file did not change pacman's behaviour either.

Last edited by stefano (2020-05-11 18:19:06)

loqs · 2020-05-10 22:44:55

What was in the upgrade? ( /var/log/pacman.log )
If pacman is broken you can use the pacman on the rescue media with --root and --cachedir instead of --sysroot e.g.

pacman -Qkk --root=/mnt #check for corruption in packages
pacman -Syu --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg

stefano · 2020-05-10 23:19:44

The upgrade contained linux, linux-lts and grub, in addition to other packages that do not seem related to the current problem (I don't have remote access to the system at the moment, so I can't post the log unless I type it in).

I tried the two pacman commands from the rescue drive, the problem was not solved. In particular:

pacman -Qkk --root=/mnt #check for corruption in packages

apparently found some corruption and fixed it

pacman -Syu --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg

ran successfully but found no package to upgrade. Chrooting into /mnt after these two commands exhibited no new behavior. Pacman fails silently. Rebooting fails as well

I rebooted into the emergency disk and tried to reinstall grub from it as indicated, but pacman failed complaining of "invalid value Path" for most of the hooks in /usr/share/libalpm/hooks/

jasonwryan · 2020-05-10 23:22:44

stefano wrote:

In particular:
pacman -Qkk --root=/mnt #check for corruption in packages
apparently found some corruption and fixed it

That command just reports issues, it does nothing about them. It certainly won't "fix" anything...

stefano · 2020-05-10 23:28:02

jasonwryan wrote:

stefano wrote:
In particular:
pacman -Qkk --root=/mnt #check for corruption in packages
apparently found some corruption and fixed it
That command just reports issues, it does nothing about them. It certainly won't "fix" anything...

I stand corrected---I misinterpreted the output.
It had quite a few reports of "Modification time mismatch" and "Size mismatch"). I have no idea what those mean.

Last edited by stefano (2020-05-10 23:28:21)

stefano · 2020-05-10 23:30:11

BTW, is there a way to ssh into the emergency disk (the standard Archlinux one)?

jasonwryan · 2020-05-10 23:33:09

https://wiki.archlinux.org/index.php/In … x_from_SSH

stefano · 2020-05-10 23:37:07

jasonwryan wrote:

https://wiki.archlinux.org/index.php/In … x_from_SSH

Thanks, I had forgotten about the required password

stefano · 2020-05-10 23:54:56

So, now that I have ssh access I can post the logs. Here is pacman's, with today's update:

[2020-05-10T08:56:15-0500] [PACMAN] Running 'pacman -Syu'
[2020-05-10T08:56:15-0500] [PACMAN] synchronizing package lists
[2020-05-10T08:56:23-0500] [PACMAN] starting full system upgrade
[2020-05-10T09:00:02-0500] [ALPM] running '60-mkinitcpio-remove.hook'...
[2020-05-10T09:00:02-0500] [ALPM] running '70-dkms-remove.hook'...
[2020-05-10T09:00:03-0500] [ALPM-SCRIPTLET] ==> dkms remove zfs/0.8.3 -k 5.4.36-1-lts
[2020-05-10T09:01:40-0500] [ALPM] transaction started
[2020-05-10T09:01:46-0500] [ALPM] upgraded filesystem (2019.10-2 -> 2020.05.03-1)
[2020-05-10T09:03:06-0500] [ALPM] upgraded attr (2.4.48-2 -> 2.4.48-3)
[2020-05-10T09:03:10-0500] [ALPM] upgraded acl (2.2.53-2 -> 2.2.53-3)
[2020-05-10T09:03:13-0500] [ALPM] upgraded argon2 (20190702-2 -> 20190702-3)
[2020-05-10T09:03:13-0500] [ALPM] upgraded libp11-kit (0.23.20-4 -> 0.23.20-5)
[2020-05-10T09:03:16-0500] [ALPM] upgraded p11-kit (0.23.20-4 -> 0.23.20-5)
[2020-05-10T09:03:16-0500] [ALPM] upgraded ca-certificates-mozilla (3.51.1-1 -> 3.52-1)
[2020-05-10T09:03:16-0500] [ALPM] upgraded device-mapper (2.02.187-1 -> 2.02.187-2)
[2020-05-10T09:03:16-0500] [ALPM] upgraded json-c (0.14-1 -> 0.14-4)
[2020-05-10T09:03:16-0500] [ALPM] upgraded cryptsetup (2.3.2-1 -> 2.3.2-2)
[2020-05-10T09:03:16-0500] [ALPM] upgraded e2fsprogs (1.45.6-1 -> 1.45.6-2)
[2020-05-10T09:03:24-0500] [ALPM] upgraded geoip-database (20180403-1 -> 20200507-2)
[2020-05-10T09:03:24-0500] [ALPM] upgraded nettle (3.5.1-2 -> 3.6-1)
[2020-05-10T09:03:24-0500] [ALPM] upgraded gnutls (3.6.13-1 -> 3.6.13-2)
[2020-05-10T09:03:24-0500] [ALPM] upgraded gpgme (1.13.1-3 -> 1.13.1-5)
[2020-05-10T09:03:24-0500] [ALPM] upgraded grub (2:2.04-5 -> 2:2.04-6)
[2020-05-10T09:03:24-0500] [ALPM] upgraded libtool (2.4.6+42+gb88cebd5-11 -> 2.4.6+42+gb88cebd5-12)
[2020-05-10T09:03:31-0500] [ALPM] upgraded imagemagick (7.0.10.10-1 -> 7.0.10.11-1)
[2020-05-10T09:03:31-0500] [ALPM] upgraded keyutils (1.6.1-3 -> 1.6.1-4)
[2020-05-10T09:03:31-0500] [ALPM] upgraded libmicrohttpd (0.9.70-1 -> 0.9.70-3)
[2020-05-10T09:03:31-0500] [ALPM] upgraded libsm (1.2.3-1 -> 1.2.3-2)
[2020-05-10T09:03:49-0500] [ALPM] upgraded linux (5.6.8.arch1-1 -> 5.6.11.arch1-1)
[2020-05-10T09:05:39-0500] [ALPM] upgraded linux-headers (5.6.8.arch1-1 -> 5.6.11.arch1-1)
[2020-05-10T09:12:01-0500] [ALPM] upgraded linux-lts (5.4.36-1 -> 5.4.39-1)
[2020-05-10T09:16:46-0500] [ALPM] upgraded linux-lts-headers (5.4.36-1 -> 5.4.39-1)
[2020-05-10T09:16:46-0500] [ALPM] upgraded lmdb (0.9.24-1 -> 0.9.25-1)
[2020-05-10T09:16:46-0500] [ALPM] upgraded lvm2 (2.02.187-1 -> 2.02.187-2)
[2020-05-10T09:16:46-0500] [ALPM] upgraded netctl (1.22-1 -> 1.23-1)
[2020-05-10T09:16:46-0500] [ALPM] upgraded nss (3.51.1-1 -> 3.52-1)
[2020-05-10T09:16:50-0500] [ALPM] upgraded pacman (5.2.1-4 -> 5.2.1-5)
[2020-05-10T09:18:04-0500] [ALPM] upgraded perl-alien-build (2.21-1 -> 2.22-1)
[2020-05-10T09:18:17-0500] [ALPM] upgraded perl-xml-libxml (2.0204-1 -> 2.0205-1)
[2020-05-10T09:18:34-0500] [ALPM] upgraded poppler (0.87.0-1 -> 0.88.0-1)
[2020-05-10T09:19:35-0500] [ALPM] upgraded poppler-data (0.4.9-1 -> 0.4.9-2)
[2020-05-10T09:19:35-0500] [ALPM] upgraded pygobject-devel (3.36.0-2 -> 3.36.1-1)
[2020-05-10T09:19:45-0500] [ALPM] upgraded python-gobject (3.36.0-2 -> 3.36.1-1)
[2020-05-10T09:19:49-0500] [ALPM] upgraded zile (2.4.14-1 -> 2.4.14-2)
[2020-05-10T09:20:03-0500] [ALPM] transaction completed
[2020-05-10T09:20:04-0500] [ALPM] running '20-systemd-sysusers.hook'...
[2020-05-10T09:20:33-0500] [ALPM] running '30-systemd-daemon-reload.hook'...
[2020-05-10T09:20:37-0500] [ALPM] running '30-systemd-sysctl.hook'...
[2020-05-10T09:20:37-0500] [ALPM] running '30-systemd-tmpfiles.hook'...
[2020-05-10T09:20:37-0500] [ALPM] running '30-systemd-udev-reload.hook'...
[2020-05-10T09:20:37-0500] [ALPM] running '30-systemd-update.hook'...
[2020-05-10T09:20:37-0500] [ALPM] running '60-depmod.hook'...
^@^@^@^@^@^@^@^@^@^@^@

followed by the update I tried from the emergency disk a short while ago:

[2020-05-10 22:57] [PACMAN] Running 'pacman -Syu --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg'
[2020-05-10 22:57] [PACMAN] synchronizing package lists
[2020-05-10 23:00] [PACMAN] Running 'pacman -Syu --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg'
[2020-05-10 23:00] [PACMAN] synchronizing package lists
[2020-05-10 23:00] [PACMAN] starting full system upgrade
[2020-05-10 23:12] [PACMAN] Running 'pacman -Sv --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg grub'
[2020-05-10 23:28] [PACMAN] Running 'pacman -Sv --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg grub'

Here is the result of my failed attempt to reinstall grub:

root@archiso ~ # pacman -Sv grub --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg
Root      : /mnt/
Conf File : /etc/pacman.conf
DB Path   : /mnt/var/lib/pacman/
Cache Dirs: /mnt/var/cache/pacman/pkg/  
Hook Dirs : /mnt/usr/share/libalpm/hooks/  /etc/pacman.d/hooks/  
Lock File : /mnt/var/lib/pacman/db.lck
Log File  : /mnt/var/log/pacman.log
GPG Dir   : /etc/pacman.d/gnupg/
Targets   : grub
warning: grub-2:2.04-6 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (1) grub-2:2.04-6

Total Installed Size:  32.82 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                                                          [###################################################] 100%
(1/1) checking package integrity                                                        [###################################################] 100%
(1/1) loading package files                                                             [###################################################] 100%
(1/1) checking for file conflicts                                                       [###################################################] 100%
error: hook /mnt/usr/share/libalpm/hooks/fontconfig.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-tmpfiles.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/texinfo-remove.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-daemon-reload.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-hwdb.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-sysctl.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-update.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/70-dkms-remove.hook line 4: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-catalog.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-udev-reload.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/texinfo-install.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/glib-compile-schemas.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/20-systemd-sysusers.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/detect-old-perl-modules.hook line 4: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-binfmt.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/gio-querymodules.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/70-dkms-install.hook line 4: invalid value Path
error: failed to commit transaction (failed to run transaction hooks)
Errors occurred, no packages were upgraded.

and finally, here are the errors that pacman's query command found (the whole output is too long to post. Pastebin here):

s, 0 altered files
libgpg-error: 94 total files, 0 altered files
error: error while reading file /mnt/var/lib/pacman/local/nss-3.52-1/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/pacman-5.2.1-5/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/perl-alien-build-2.22-1/mtree: Unrecognized archive format
perl-error: 18 total files, 0 altered files
perl-netaddr-ip: 84 total files, 0 altered fierror: error while reading file /mnt/var/lib/pacman/local/perl-xml-libxml-2.0205-1/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/poppler-0.88.0-1/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/poppler-data-0.4.9-2/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/pygobject-devel-3.36.1-1/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/python-gobject-3.36.1-1/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/zile-2.4.14-2/mtree: Unrecognized archive format
root@archiso

loqs · 2020-05-11 00:04:17

error: error while reading file /mnt/var/lib/pacman/local/nss-3.52-1/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/pacman-5.2.1-5/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/perl-alien-build-2.22-1/mtree: Unrecognized archive format
perl-alien-build: no mtree file
error while reading file /mnt/var/lib/pacman/local/perl-xml-libxml-2.0205-1/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/poppler-0.88.0-1/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/poppler-data-0.4.9-2/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/pygobject-devel-3.36.1-1/mtree: Unrecognized archive format
error: error while reading file /mnt/var/lib/pacman/local/python-gobject-3.36.1-1/mtree: Unrecognized archive format
perl-xml-libxml: no mtree file
poppler: no mtree file
poppler-data: no mtree file
pygobject-devel: no mtree file
python-gobject: no mtree file
zile: no mtree file

All of the above definitely need to be reinstalled.

zfs-utils
zfs-dkms
netctl
go

The above probably need to be reinstalled.
Edit:
For the first issue:

pacman --root=/mnt --dbonly -S nss pacman perl-alien-build perl-xml-libxml poppler poppler-data pygobject-devel python-gobject zile #fix up the pacman mtree files
pacman --root=/mnt --cachedir=/mnt/var/cache/pacman/pkg -S nss pacman perl-alien-build perl-xml-libxml poppler poppler-data pygobject-devel python-gobject zile #reinstall the packages contents

Last edited by loqs (2020-05-11 00:11:39)

stefano · 2020-05-11 03:25:41

Unfortunately I keep getting the same "invalid value Path" errors on the hooks and no upgrade happens.
It certainly would be great to know what that error is

[root@archiso ~ # pacman --root=/mnt --dbonly -S nss pacman perl-alien-build perl-xml-libxml poppler poppler-data pygobject-devel python-gobject zile
warning: nss-3.52-1 is up to date -- reinstalling
warning: pacman-5.2.1-5 is up to date -- reinstalling
warning: perl-alien-build-2.22-1 is up to date -- reinstalling
warning: perl-xml-libxml-2.0205-1 is up to date -- reinstalling
warning: poppler-0.88.0-1 is up to date -- reinstalling
warning: poppler-data-0.4.9-2 is up to date -- reinstalling
warning: pygobject-devel-3.36.1-1 is up to date -- reinstalling
warning: python-gobject-3.36.1-1 is up to date -- reinstalling
warning: zile-2.4.14-2 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Packages (9) nss-3.52-1  pacman-5.2.1-5  perl-alien-build-2.22-1  perl-xml-libxml-2.0205-1  poppler-0.88.0-1  poppler-data-0.4.9-2
             pygobject-devel-3.36.1-1  python-gobject-3.36.1-1  zile-2.4.14-2

Total Download Size:    6.45 MiB
Total Installed Size:  29.10 MiB

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 nss-3.52-1-x86_64                                            1489.2 KiB  1160K/s 00:01 [###################################################] 100%
 pacman-5.2.1-5-x86_64                                         836.7 KiB  11.7M/s 00:00 [###################################################] 100%
 perl-alien-build-2.22-1-any                                   324.0 KiB  13.8M/s 00:00 [###################################################] 100%
 perl-xml-libxml-2.0205-1-x86_64                               280.7 KiB  13.7M/s 00:00 [###################################################] 100%
 poppler-0.88.0-1-x86_64                                      1346.8 KiB  11.2M/s 00:00 [###################################################] 100%
 poppler-data-0.4.9-2-any                                     1933.5 KiB  9.44M/s 00:00 [###################################################] 100%
 pygobject-devel-3.36.1-1-x86_64                                 9.6 KiB  0.00B/s 00:00 [###################################################] 100%
 python-gobject-3.36.1-1-x86_64                                248.8 KiB  24.3M/s 00:00 [###################################################] 100%
 zile-2.4.14-2-x86_64                                          131.4 KiB  42.8M/s 00:00 [###################################################] 100%
(9/9) checking keys in keyring                                                          [###################################################] 100%
(9/9) checking package integrity                                                        [###################################################] 100%
(9/9) loading package files                                                             [###################################################] 100%
error: hook /mnt/usr/share/libalpm/hooks/fontconfig.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-tmpfiles.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/texinfo-remove.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-daemon-reload.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-hwdb.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-sysctl.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-update.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/70-dkms-remove.hook line 4: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-catalog.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-udev-reload.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/texinfo-install.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/glib-compile-schemas.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/20-systemd-sysusers.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/detect-old-perl-modules.hook line 4: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/30-systemd-binfmt.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/gio-querymodules.hook line 2: invalid value Path
error: hook /mnt/usr/share/libalpm/hooks/70-dkms-install.hook line 4: invalid value Path
error: failed to commit transaction (failed to run transaction hooks)
Errors occurred, no packages were upgraded.

jasonwryan · 2020-05-11 03:27:30

Your pacman version is too old...

stefano · 2020-05-11 04:42:54

jasonwryan wrote:

Your pacman version is too old...

Yep, I figured that one out from another thread.

So I created a new emergency disk with a brand new archiso version and retried everything.

This time I could successfully run pacman and reinstall the corrupted packages, no more value errors in the hook files.

Unfortunately that did not help. System still can't boot---still having exactly the same problem I started from.

jasonwryan · 2020-05-11 04:51:19

Post the journal and dmesg from a failed boot.

eschwartz · 2020-05-11 04:55:01

Also please post the output of `pacman-conf`.

stefano · 2020-05-11 18:18:46

So issues kept piling up, but there was light at the end of the tunnel

1. I was no longer able to boot into the emergency drive (latest Archiso install), being greeted with a blank screen at the end of the boot process. I eventually got out of that problem by setting nomodeset and nouveau.nomodeset=0 in the kernel boot parameter. I wonder why I needed to do that, though, if the boot process from exactly the same media was fine only a few minutes earlier?

2. Then I was no longer able to ssh remotely---I would always get the error message "PTY allocation request failed on channel 0" on the client. After a few more searches, I managed to ssh in passing the "/bin/zsh -i" argument to ssh. I have no idea why I needed to do that, but it worked, albeit with somewhat limited functionality

3. Finally, I checked the /boot directory and found out that in fact there were no kernels or initiramfs's (much to my surprise). So grub was actually correct in its complaints..

4. Arch-chrooted into the mounted system, reinstalled linux and linux-lts and rebooted

5. System now boots, finally. However, there is still an issue with zfs-dkms that I am working on right now (building the modules fails for the lts kernel)

So I'm marking the issue as solved and would like to tjanks everyone for their help. i really would like to know how did it happen that my kernels vanished into thin air, though...

Arch Linux

#1 2020-05-10 22:35:18

System can no longer boot after upgrade [SOLVED]

#2 2020-05-10 22:44:55

Re: System can no longer boot after upgrade [SOLVED]

#3 2020-05-10 23:19:44

Re: System can no longer boot after upgrade [SOLVED]

#4 2020-05-10 23:22:44

Re: System can no longer boot after upgrade [SOLVED]

#5 2020-05-10 23:28:02

Re: System can no longer boot after upgrade [SOLVED]

#6 2020-05-10 23:30:11

Re: System can no longer boot after upgrade [SOLVED]

#7 2020-05-10 23:33:09

Re: System can no longer boot after upgrade [SOLVED]

#8 2020-05-10 23:37:07

Re: System can no longer boot after upgrade [SOLVED]

#9 2020-05-10 23:54:56

Re: System can no longer boot after upgrade [SOLVED]

#10 2020-05-11 00:04:17

Re: System can no longer boot after upgrade [SOLVED]

#11 2020-05-11 03:25:41

Re: System can no longer boot after upgrade [SOLVED]

#12 2020-05-11 03:27:30

Re: System can no longer boot after upgrade [SOLVED]

#13 2020-05-11 04:42:54

Re: System can no longer boot after upgrade [SOLVED]

#14 2020-05-11 04:51:19

Re: System can no longer boot after upgrade [SOLVED]

#15 2020-05-11 04:55:01

Re: System can no longer boot after upgrade [SOLVED]

#16 2020-05-11 18:18:46

Re: System can no longer boot after upgrade [SOLVED]

Board footer