Unable to update kernel - Rackspace

goldfinch · 2017-11-20 18:58:10

I have a rackspace Arch instance. It is kept up to date. It was recently discovered the kernel updates were not actually being used (for quite some time, disturbingly). I have the latest kernel installed, however it's not getting placed into the grub.cfg. Efforts to correct it have unsuccessfully result in a 2 hour re-image from a recent backup. If the grub.cfg is updated to reflect the new kernel, the system becomes unavailable. By unavailable I mean all communication to the rackspace instance is permanently destroyed. Even the rackspace emergency console can not see the server, bypassing the normal network ssh access. Support from rackspace confirms it also becomes inaccessible to them.

Similar issue I found using search:
https://bbs.archlinux.org/viewtopic.php?id=181223
https://bbs.archlinux.org/viewtopic.php?id=228383
https://bbs.archlinux.org/viewtopic.php?id=230084

(Yes, ignoring fail2ban is intentional. The kernel problem was discovered via iptables missing the LOG target and it's important fail2ban continues to work.)

[root@rack ~]# pacman -Syyu --ignore fail2ban
:: Synchronizing package databases...
 core                                                                                                                 124.4 KiB  1207K/s 00:00 [#######################################################################################] 100%
 extra                                                                                                               1660.2 KiB  11.8M/s 00:00 [#######################################################################################] 100%
 community                                                                                                              4.1 MiB  29.5M/s 00:00 [#######################################################################################] 100%
 multilib                                                                                                             172.6 KiB  5.11M/s 00:00 [#######################################################################################] 100%
 archlinuxfr                                                                                                           13.8 KiB   109K/s 00:00 [#######################################################################################] 100%
:: Starting full system upgrade...
warning: fail2ban: ignoring package upgrade (0.9.7-1 => 0.10.1-2)
 there is nothing to do

See the kernel mismatch:

[root@rack ~]# uname -r
3.14.1
[root@rack ~]# pacman -Qi linux
Name            : linux
Version         : 4.13.12-1
Description     : The Linux kernel and modules
Architecture    : x86_64
URL             : https://www.kernel.org/
Licenses        : GPL2
Groups          : base
Provides        : None
Depends On      : coreutils  linux-firmware  kmod  mkinitcpio>=0.7
Optional Deps   : crda: to set the correct wireless channels of your country
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 92.72 MiB
Packager        : Tobias Powalowski <tpowa@archlinux.org>
Build Date      : Wed Nov 8 10:55:50 2017
Install Date    : Sat Nov 18 17:19:40 2017
Install Reason  : Explicitly installed
Install Script  : Yes
Validated By    : Signature

The file system:

[root@rack ~]# cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/xvda1  /               ext4    errors=remount-ro,noatime,barrier=0 0       1
#/dev/xvdc1 none            swap    sw              0       0

[root@rack ~]# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=61937,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=100156k,mode=755)
/dev/xvda1 on / type ext4 (rw,noatime,nobarrier,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
mqueue on /dev/mqueue type mqueue (rw,relatime)
xenfs on /proc/xen type xenfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=50076k,mode=700,uid=1001,gid=100)

[root@rack ~]# lsblk -f
NAME    FSTYPE LABEL UUID                                 MOUNTPOINT
xvda
`-xvda1 ext4         Hidden_MY_UUID1 /
xvdc
`-xvdc1 swap        Hidden_MY_UUID2

The files in question:

[root@rack ~]# ls -ltr /boot/
total 45764
-rw-r--r-- 1 root root  4242287 Jan 31  2014 growpart-initramfs
-rw-r--r-- 1 root root  4078304 Apr 26  2014 vmlinuz-linux-3.14.1
-rw-r--r-- 1 root root  2160885 Apr 26  2014 initramfs-linux-3.14.1.img
-rw-r--r-- 1 root root  5502736 Nov  8 10:55 vmlinuz-linux
-rw-r--r-- 1 root root  6365989 Nov 18 17:19 initramfs-linux.img
-rw-r--r-- 1 root root 24494276 Nov 18 17:20 initramfs-linux-fallback.img
drwxr-xr-x 6 root root     4096 Nov 19 04:04 grub

[root@rack ~]# ls -ltr /boot/grub/
total 48
-rw------- 1 root root  6072 Apr 26  2014 grub.cfg.example
drwxr-xr-x 2 root root  4096 Apr 26  2014 locale
drwxr-xr-x 3 root root  4096 Apr 26  2014 themes
drwxr-xr-x 2 root root  4096 Apr 26  2014 fonts
-rw-r--r-- 1 root root  1024 Apr 26  2014 grubenv
drwxr-xr-x 2 root root 12288 Apr 26  2014 i386-pc
-rw------- 1 root root   322 Nov 19 04:04 grub.cfg

The grub.cfg that boots to the old kernel (because it was automatically generated to specify to use that old kernel):

[root@rack ~]# cat /boot/grub/grub.cfg
timeout=15

menuentry 'Arch Linux 3.14.1' {
root=hd0,1
linux /boot/vmlinuz-linux-3.14.1 root=/dev/xvda1 init=/usr/lib/systemd/systemd
initrd /boot/growpart-initramfs
}

jasonwryan · 2017-11-20 19:37:41

So just write your own grub stanza.

And Arch doesn't name it's kernels like that, so it is something Rackspace is doing (ie., is unsupported).

Moving to NC...

goldfinch · 2017-11-20 20:11:44

jasonwryan wrote:

So just write your own grub stanza.
And Arch doesn't name it's kernels like that, so it is something Rackspace is doing (ie., is unsupported).

Moving to NC...

Thx Jason. I agree, change the grub stanza manually to point to the correct kernel. However that causes the system to be completely inaccessible upon reboot. Here's my changes that don't work:

[root@rack ~]# cat /boot/grub/grub.cfg
timeout=15

menuentry 'Arch Linux' {
root=hd0,1
linux /boot/vmlinuz-linux root=/dev/xvda1 init=/usr/lib/systemd/systemd
initrd /boot/growpart-initramfs
}

To prove these files are what is expected:

[root@rack ~]# file /boot/vmlinuz-linux-3.14.1
/boot/vmlinuz-linux-3.14.1: Linux kernel x86 boot executable bzImage, version 3.14.1 (root@livecd) #1 SMP Sat Apr 26 09:03:36 UTC 2014, RO-rootFS, swap_dev 0x3, Normal VGA

[root@rack ~]# file /boot/vmlinuz-linux
/boot/vmlinuz-linux: Linux kernel x86 boot executable bzImage, version 4.13.12-1-ARCH (builduser@tobias) #1 SMP PREEMPT Wed Nov 8 11:54:06 CET 2017, RO-rootFS, swap_dev 0x5, Normal VGA

[root@rack ~]# file /boot/growpart-initramfs
/boot/growpart-initramfs: gzip compressed data, last modified: Fri Jan 31 18:19:29 2014, max compression, from Unix

jasonwryan · 2017-11-20 20:18:00

So, their initrd is expecting their kernel?

goldfinch · 2017-11-20 21:28:52

jasonwryan wrote:

So, their initrd is expecting their kernel?

That might be the case. I can't determine if there is a way to prove their initrd is expecting their kernel version. Any idea how to query an existing initramfs?

Following that idea, I looked into the growpart-initramfs for any information. I suspect it is part of the cloud server creation process. I poorly understand the root fs is 'grown' prior to every boot process?

The cloud-utils package might have something to do with how rackspace uses growpart-initramfs?
https://aur.archlinux.org/packages/cloud-utils

[root@rack~]$ yaourt -Ss cloud-utils
aur/cloud-utils 0.29-1 (0) (0.00)
    A useful set of utilities for interacting with a cloud
aur/cloud-utils-bzr 0.27.r269-1 (0) (0.00)
    A useful set of utilities for interacting with a cloud

Or is mkinitcpio-growrootfs the hint I'm missing?
https://aur.archlinux.org/packages/mkin … rowrootfs/

[root@rack ~]$ yaourt -Ss mkinitcpio-growrootfs
aur/mkinitcpio-growrootfs 0.1-4 (0) (0.00)
    mkinitcpio hook to autoscale the rootfs to the size of the partition

jasonwryan · 2017-11-20 21:31:11

You can compare them: https://wiki.archlinux.org/index.php/Mk … _the_image

goldfinch · 2017-11-20 22:27:13

Yes they are different. The initramfs-linux-3.14.1.img and initramfs-linux.img look like the files I expect to see. Although the initrd my grub uses, is different (growpart-initramfs). Do I need to regenerate growpart-initramfs against the newer kernel?

[root@rack initramfs-extracted]# lsinitcpio -a /boot/initramfs-linux-3.14.1.img
==> Image: /boot/initramfs-linux-3.14.1.img
==> Created with mkinitcpio 17
==> Kernel: 3.14.1
==> Size: 2.06 MiB
==> Compressed with: gzip
  -> Uncompressed size: 4.77 MiB (.431 ratio)
  -> Estimated extraction time: 0.061s

==> Included binaries:
blkid                   busybox                 fsck                    fsck.ext4               kmod                    mount                   switch_root             systemd-tmpfiles        udevadm

==> Early hook run order:
  udev

==> Hook run order:
  udev

==> Cleanup hook run order:
  udev

[root@rack initramfs-extracted]# lsinitcpio -a /boot/initramfs-linux.img
==> Image: /boot/initramfs-linux.img
==> Created with mkinitcpio 24
==> Kernel: 4.13.12-1-ARCH
==> Size: 6.07 MiB
==> Compressed with: gzip
  -> Uncompressed size: 15.1 MiB (.402 ratio)
  -> Estimated extraction time: 0.285s

==> Included modules:
crc16           crc32c-intel    crc32c_generic  ext4            fscrypto        hid             jbd2            mbcache         scsi_mod        sd_mod          usb-common      usbcore         usbhid          xen-blkfront

==> Included binaries:
blkid                   busybox                 fsck                    fsck.ext4               kmod                    mount                   switch_root             systemd-tmpfiles        udevadm

==> Early hook run order:
  udev

==> Hook run order:
  udev

==> Cleanup hook run order:
  udev

[root@rack initramfs-extracted]# lsinitcpio -a /boot/growpart-initramfs
find: '/tmp/lsinitcpio.X0XgNN/usr/bin': No such file or directory
/usr/bin/lsinitcpio: line 183: /tmp/lsinitcpio.X0XgNN/VERSION: No such file or directory
/usr/bin/lsinitcpio: line 186: /tmp/lsinitcpio.X0XgNN/config: No such file or directory
==> Image: /boot/growpart-initramfs
==> Kernel: *
==> Size: 4.05 MiB
==> Compressed with: gzip
  -> Uncompressed size: 8.81 MiB (.459 ratio)
  -> Estimated extraction time: 0.112s

==> Included binaries:

[root@rack initramfs-extracted]# lsinitcpio /boot/growpart-initramfs
.
init
lib64
lib64/libblkid.so.1
lib64/libext2fs.so.2
lib64/ld-linux-x86-64.so.2
lib64/libpthread.so.0
lib64/libe2p.so.2
lib64/libc.so.6
lib64/libcom_err.so.2
lib64/libuuid.so.1
conf
conf/arch.conf
conf/conf.d
conf/initramfs.conf
scripts
scripts/local-premount
scripts/local-premount/resume
scripts/local-premount/ORDER
scripts/init-top
scripts/init-top/all_generic_ide
scripts/init-top/cloud-initramfs-dyn-netconf
scripts/init-top/blacklist
scripts/init-top/keymap
scripts/init-top/ORDER
scripts/init-top/udev
scripts/local-bottom
scripts/local-bottom/ORDER
scripts/local
scripts/functions
scripts/nfs
scripts/init-bottom
scripts/init-bottom/cloud-initramfs-dyn-netconf
scripts/init-bottom/ORDER
scripts/init-bottom/udev
run
bin
bin/awk
bin/run-init
bin/resume
bin/cat
bin/mount
bin/sh
bin/ln
bin/nuke
bin/nfsmount
bin/false
bin/mkfifo
bin/sync
bin/pivot_root
bin/resize2fs
bin/mknod
bin/fstype
bin/poweroff
bin/e2fsck
bin/dd
bin/kmod
bin/rm
bin/mv
bin/ls
bin/insmod
bin/cpio
bin/halt
bin/umount
bin/ipconfig
bin/mkdir
bin/uname
bin/chroot
bin/minips
bin/sed
bin/sleep
bin/reboot
bin/readlink
bin/dmesg
bin/busybox
bin/gunzip
bin/gzip
bin/true
bin/kill
bin/losetup
sbin
sbin/modprobe
sbin/udevadm
sbin/growpart
sbin/rmmod
sbin/sfdisk
sbin/blkid
sbin/dmsetup
sbin/udevd
lib
lib/x86_64-linux-gnu
lib/x86_64-linux-gnu/libblkid.so.1
lib/x86_64-linux-gnu/libext2fs.so.2
lib/x86_64-linux-gnu/librt.so.1
lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
lib/x86_64-linux-gnu/libm.so.6
lib/x86_64-linux-gnu/libpthread.so.0
lib/x86_64-linux-gnu/libdl.so.2
lib/x86_64-linux-gnu/libe2p.so.2
lib/x86_64-linux-gnu/libc.so.6
lib/x86_64-linux-gnu/libcom_err.so.2
lib/x86_64-linux-gnu/libkmod.so.2
lib/x86_64-linux-gnu/libudev.so.0
lib/x86_64-linux-gnu/libuuid.so.1
lib/x86_64-linux-gnu/libselinux.so.1
lib/modules
lib/klibc-2xtYrByCrj5OEwaInv4tMSjej98.so
lib/udev
lib/udev/ata_id
lib/udev/scsi_id
lib/udev/rules.d
lib/udev/rules.d/91-permissions.rules
lib/udev/rules.d/80-drivers.rules
lib/udev/rules.d/50-udev-default.rules
lib/udev/rules.d/55-dm.rules
lib/udev/rules.d/60-persistent-storage.rules
lib/udev/rules.d/60-persistent-storage-dm.rules
lib/udev/hotplug.functions
lib/udev/edd_id
lib/udev/firmware.agent
lib/modprobe.d
lib/modprobe.d/aliases.conf
etc
etc/ld.so.conf.d
etc/ld.so.conf.d/libc.conf
etc/ld.so.conf.d/x86_64-linux-gnu.conf
etc/ld.so.conf
etc/ld.so.cache
etc/udev
etc/udev/udev.conf
etc/modprobe.d
etc/modprobe.d/fbdev-blacklist.conf

jasonwryan · 2017-11-20 22:30:53

Presumably. But I have no clue about growpart-initramfs and this seems more like a ticket with rackspace than it does an Arch issue.

Arch Linux

#1 2017-11-20 18:58:10

Unable to update kernel - Rackspace

#2 2017-11-20 19:37:41

Re: Unable to update kernel - Rackspace

#3 2017-11-20 20:11:44

Re: Unable to update kernel - Rackspace

#4 2017-11-20 20:18:00

Re: Unable to update kernel - Rackspace

#5 2017-11-20 21:28:52

Re: Unable to update kernel - Rackspace

#6 2017-11-20 21:31:11

Re: Unable to update kernel - Rackspace

#7 2017-11-20 22:27:13

Re: Unable to update kernel - Rackspace

#8 2017-11-20 22:30:53

Re: Unable to update kernel - Rackspace

Board footer