Arch Linux

bjcubsfan

Member

From: Oklahoma City, OK, USA

Registered: 2012-05-04

Posts: 32

Website

Manual kernel reinstall required after upgrades

For the last several of my approximately monthly Arch Linux upgrades on my 2 Dell PowerEdge R430 servers, I have had to manually reinstall the Linux kernel to get the machine to boot after the upgrade. Just after the upgrade, when the machines do not come up, I see this message on the screen which seems to be from something Dell put on the machine:

UEFI0002: The system was reset due to a timeout horn the watchdog timer.
Check the System Event Log (SEL) or crash dumps from Operating System to
identify the source that triggered the watchdog timer reset.
Update the firmware or driver for the identified device.

Auailable Actions:
F1 to Continue and Retry Boot Order
F2 for System Setup (BIOS)
F10 for LifeCycle Controller
- Enable/Configure iDRAC
- Update or Backup/Restore Server Firmware
- Help Install an Operating System
- F11 for Boot Manager

If I hit `F1` and try to boot, it gives this error from the Arch boot process:

:: running early hook [udev]
kmod: error while loading shared libraries: liblzma.so.5: cannot open shared obj
ect file: No such file or directory
systemd-tmpfiles: error while loading shared libraries: liblzma.so.5: cannot ope
n shared object file: No such file or directory
/usr/lib/systemd/systemd-udevd: error while loading shared libraries: liblzma.so
.5: cannot open shared object file: No such file or directory
:: running hook [udev]
:: Triggering uevents...
udevadm: error while loading shared libraries: liblzma.so.5: cannot open shared
object file: No such file or directory
udevadm: error while loading shared libraries: liblzma.so.5: cannot open shared
object file: No such file or directory
udevadm: error while loading shared libraries: liblzma.so.5: cannot open shared
object file: No such file or directory
Waiting 10 seconds for device /dev/disk/by-uuid/43e3cff2-80e-4887-afdd-897m12d
6dc4 ...
ERROR: device 'UUID=43e3cff2-8e7e-4007-afaa-897acl2d6dc41 not found. Skipping fs
ck.
:: mounting 'OUID=43e3cff2-80e-4887-afaa-897ac12d6dc41 on real root
mount: /new_root: can't find UUID=43e3cff2-80e-4887-afaa-897ac12d6dc4.
You are now being dropped into an emergency shell.
sh: can't access tty; job control turned off
[rootfs ]# _

To get the machine running again, I execute steps like this after booting off of a rescue USB drive:

fdisk -l  # Find the boot drive
mkdir /mnt/root/
mount /dev/sdb2 /mnt/root  # Mount the boot drive
arch-chroot /mnt/root
cd /var/cache/pacman/pkg
pacman -U linux-4.14.5-1-x86_64.pkg.tar.xz linux-firmware-20171206.fdee922-1-any.pkg.tar.xz linux-api-headers-4.12.7-1-any.pkg.tar.xz  # Reinstall packages

Does anyone have ideas of what might be going on here? I would like to get back to being able to upgrade these servers without having to make a physical appearance afterwards.

Awebb

Member

Registered: 2010-05-06

Posts: 6,286

Re: Manual kernel reinstall required after upgrades

Compare your fstab (or systemd generators, whatever you use) with the entries in your bootloader. It tries to mount something that isn't there. Weird missing libraries during the boot process could be old libraries expected from the old kernel image. I suspect your /boot isn't mounted properly when you do the updates.

bjcubsfan

Member

From: Oklahoma City, OK, USA

Registered: 2012-05-04

Posts: 32

Website

Re: Manual kernel reinstall required after upgrades

I don't have "/boot" on a separate partition, it's just on "/". Heres' the snippet from "/boot/syslinux/syslinux.cfg":

LABEL arch
    MENU LABEL Arch Linux
    LINUX ../vmlinuz-linux
    INITRD ../intel-ucode.img,../initramfs-linux.img
    APPEND root=UUID=43e3cff2-8e7e-4887-afaa-897ac12d6dc4

Which matches the fstab:

# Static information about the filesystems.
# See fstab(5) for details.
#
# <file system> <dir> <type> <options> <dump> <pass>
# /dev/sdb2
UUID=43e3cff2-8e7e-4887-afaa-897ac12d6dc4	/         	ext4      	rw,relatime,data=ordered	0 1

# /dev/sdb1
/dev/mapper/swap                         	none      	swap      	defaults  	0 0

# /dev/sda1
UUID=d2605736-5fb8-4dbb-b0dd-c67a41f54ccf       /array          ext4            rw,relatime,data=ordered        0 2

Those look OK to me. It also doesn't line up that a kernel reinstall would fix an issue with the boot (not that anything does to my mind).

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 19,772

Re: Manual kernel reinstall required after upgrades

You have a UEFI system.  The EFI partition must be VFAT. You do not have a separate boot partition.  You do not appear to have a separate EFI partition mounted at /boot/efi.
But, / is ext4. 

Um?

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

bjcubsfan

Member

From: Oklahoma City, OK, USA

Registered: 2012-05-04

Posts: 32

Website

Re: Manual kernel reinstall required after upgrades

I will look into how to configure UEFI, it sounds like I might be doing something wrong there.

I thought I disabled UEFI in the BIOS so that I wouldn't have to deal with it, but maybe I didn't or maybe it's not possible.

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 19,772

Re: Manual kernel reinstall required after upgrades

So, you think it is configured as MBR?  Could be that you are in compatibility mode.  I based by assertion on

UEFI0002: The system was reset due to a timeout horn the watchdog timer.

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

jasonwryan

Anarchist

From: .nz

Registered: 2009-05-09

Posts: 30,424

Website

Re: Manual kernel reinstall required after upgrades

I thought I disabled UEFI in the BIOS...

Read this: https://www.happyassassin.net/2014/01/2 … work-then/

---

Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

bjcubsfan

Member

From: Oklahoma City, OK, USA

Registered: 2012-05-04

Posts: 32

Website

Re: Manual kernel reinstall required after upgrades

OK, I had my terminology wrong. I should have said that I configured the machine through the firmware UI to use BIOS compatibility mode for booting.

I do seem to have combined BIOS+GPT which that article recommends against. Perhaps I'll try switching to UEFI-native booting.

21:20:01 bpotter@resiliency syslinux sudo parted /dev/sdb
GNU Parted 3.2
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Model: DELL PERC H730 Mini (scsi)
Disk /dev/sdb: 199GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  1075MB  1074MB
 2      1075MB  199GB   198GB   ext4               legacy_boot

loqs

Member

Registered: 2014-03-06

Posts: 17,323

Re: Manual kernel reinstall required after upgrades

UEFI0002: The system was reset due to a timeout horn the watchdog timer.

could be coming from the firmware itself and the system is booting arch using CSM

eschwartz

Fellow

Registered: 2014-08-08

Posts: 4,097

Re: Manual kernel reinstall required after upgrades

You have a UEFI system.  The EFI partition must be VFAT. You do not have a separate boot partition.  You do not appear to have a separate EFI partition mounted at /boot/efi.
But, / is ext4. 

Um?

Added: OP uses syslinux, which unlike refind and grub do not support UEFI filesystem drivers for ext4 which allow booting without a separate boot partition.

---

Managing AUR repos The Right Way -- aurpublish (now a standalone tool)