KVM grub 2 ERROR device UUID not found

MountainX · 2018-03-16 23:44:05

I'm installing Arch on a KVM VPS. I did this a couple days ago at one VPS provider and I was able to get Arch up and running without much problem. However, I'm trying out another VPS provider today and I ran into a problem I haven't been able to solve.

I'm installing Arch by mounting an ISO image as a CD Rom, booting into the Arch ISO and doing a manual install. Everything seems to go fine, but when I unmount the "CD Rom" and reboot, the error is:

ERROR: device 'UUID=fd92f3e7-f092-48c7-9f16-d94ba3de51fb' not found. Skipping fsck.
mount: /new_root: can't find UUID=fd92f3e7-f092-48c7-9f16-d94ba3de51fb.
You are now being dropped into an emergency shell.
sh: can't access tty: job control turned off
[rootfs ]#

I'm using GTP and a bios boot partition as shown in these partitioning steps:

gdisk /dev/sda

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            4095   1024.0 KiB  EF02  BIOS boot partition
   2            4096        25169919   12.0 GiB    8300  Linux filesystem
   3        25169920        41943006   8.0 GiB     8200  Linux swap

Here are my steps leading up to pacstrap:

mkfs.btrfsk /dev/sda2
mkswap /dev/sda3
swapon /dev/sda3

mount /dev/sda2 /mnt
pacstrap -i /mnt <my list of packages includes grub>
genfstab -U -p /mnt >> /mnt/etc/fstab

Here' is the generated fstab:

# /dev/sda2
UUID=fd92f3e7-f092-48c7-9f16-d94ba3de51fb       /               btrfs           rw,relatime,space_cache,subvolid=5,subvol=/     0 0

# /dev/sda3
UUID=b1388b5c-0bd1-444b-9b2e-d1b39537326d       none            swap            defaults        0 0

I typically change the mount options to these:

UUID=fd92f3e7-f092-48c7-9f16-d94ba3de51fb       /               btrfs           rw,noatime,nodiratime,compress=lzo,space_cache,defaults,subvol=/        0 0

(BTW, both fstab versions give the same error shown above on rebooting.)

Next I do `arch-chroot /mnt` and then the normal installation steps... I'm keeping these details brief so I can show the steps related to grub, which is where I assume the problem is.

To /etc/mkinitcpio.conf, I add this binary:

BINARIES=(/usr/bin/btrfs)

Then I run the next two standard grub commands (output shown):

grub-install /dev/sda

	Installing for i386-pc platform.
	Installation finished. No error reported.

# grub-mkconfig -o /boot/grub/grub.cfg

	Generating grub configuration file ...
 	Found linux image: /boot/vmlinuz-linux-lts
	Found initrd image: /boot/initramfs-linux-lts.img
	Found fallback initrd image(s) in /boot: initramfs-linux-lts-fallback.img
	Found linux image: /boot/vmlinuz-linux
	Found initrd image: /boot/initramfs-linux.img
	Found fallback initrd image(s) in /boot: initramfs-linux-fallback.img
	done

after getting the network ready, etc., I reboot and remove the CD Rom. But the system fails to boot and gives the error shown above.

Here are all the config files and output that I think might be relevant:

root@archiso ~ # cat /mnt/boot/grub/grub.cfg

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod btrfs
set root='hd0,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  fd92f3e7-f092-48c7-9f16-d94ba3de51fb
else
  search --no-floppy --fs-uuid --set=root fd92f3e7-f092-48c7-9f16-d94ba3de51fb
fi
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_input console
terminal_output gfxterm
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-fd92f3e7-f092-48c7-9f16-d94ba3de51fb' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod btrfs
        set root='hd0,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  fd92f3e7-f092-48c7-9f16-d94ba3de51fb
        else
          search --no-floppy --fs-uuid --set=root fd92f3e7-f092-48c7-9f16-d94ba3de51fb
        fi
        echo    'Loading Linux linux-lts ...'
        linux   /boot/vmlinuz-linux-lts root=UUID=fd92f3e7-f092-48c7-9f16-d94ba3de51fb rw  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initramfs-linux-lts.img
}
submenu 'Advanced options for Arch Linux' $menuentry_id_option 'gnulinux-advanced-fd92f3e7-f092-48c7-9f16-d94ba3de51fb' {
        menuentry 'Arch Linux, with Linux linux-lts' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts-advanced-fd92f3e7-f092-48c7-9f16-d94ba3de51fb' {
                load_video
                set gfxpayload=keep
                insmod gzio
                insmod part_gpt
                insmod btrfs
                set root='hd0,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  fd92f3e7-f092-48c7-9f16-d94ba3de51fb
                else
                  search --no-floppy --fs-uuid --set=root fd92f3e7-f092-48c7-9f16-d94ba3de51fb
                fi
                echo    'Loading Linux linux-lts ...'
                linux   /boot/vmlinuz-linux-lts root=UUID=fd92f3e7-f092-48c7-9f16-d94ba3de51fb rw  quiet
                echo    'Loading initial ramdisk ...'
                initrd  /boot/initramfs-linux-lts.img
        }
        menuentry 'Arch Linux, with Linux linux-lts (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-lts-fallback-fd92f3e7-f092-48c7-9f16-d94ba3de51fb' {
                load_video
                set gfxpayload=keep
                insmod gzio
                insmod part_gpt
                insmod btrfs
                set root='hd0,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  fd92f3e7-f092-48c7-9f16-d94ba3de51fb
                else
                  search --no-floppy --fs-uuid --set=root fd92f3e7-f092-48c7-9f16-d94ba3de51fb
                fi
                echo    'Loading Linux linux-lts ...'
                linux   /boot/vmlinuz-linux-lts root=UUID=fd92f3e7-f092-48c7-9f16-d94ba3de51fb rw  quiet
                echo    'Loading initial ramdisk ...'
                initrd  /boot/initramfs-linux-lts-fallback.img
        }
        menuentry 'Arch Linux, with Linux linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-advanced-fd92f3e7-f092-48c7-9f16-d94ba3de51fb' {
                load_video
                set gfxpayload=keep
                insmod gzio
                insmod part_gpt
                insmod btrfs
                set root='hd0,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  fd92f3e7-f092-48c7-9f16-d94ba3de51fb
                else
                  search --no-floppy --fs-uuid --set=root fd92f3e7-f092-48c7-9f16-d94ba3de51fb
                fi
                echo    'Loading Linux linux ...'
                linux   /boot/vmlinuz-linux root=UUID=fd92f3e7-f092-48c7-9f16-d94ba3de51fb rw  quiet
                echo    'Loading initial ramdisk ...'
                initrd  /boot/initramfs-linux.img
        }
        menuentry 'Arch Linux, with Linux linux (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-fallback-fd92f3e7-f092-48c7-9f16-d94ba3de51fb' {
                load_video
                set gfxpayload=keep
                insmod gzio
                insmod part_gpt
                insmod btrfs
                set root='hd0,gpt2'
                if [ x$feature_platform_search_hint = xy ]; then
                  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  fd92f3e7-f092-48c7-9f16-d94ba3de51fb
                else
                  search --no-floppy --fs-uuid --set=root fd92f3e7-f092-48c7-9f16-d94ba3de51fb
                fi
                echo    'Loading Linux linux ...'
                linux   /boot/vmlinuz-linux root=UUID=fd92f3e7-f092-48c7-9f16-d94ba3de51fb rw  quiet
                echo    'Loading initial ramdisk ...'
                initrd  /boot/initramfs-linux-fallback.img
        }
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

[root@archiso /]# cat /etc/default/grub

# GRUB boot loader configuration

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable booting from LUKS encrypted devices
#GRUB_ENABLE_CRYPTODISK=y

# Uncomment to enable Hidden Menu, and optionally hide the timeout count
#GRUB_HIDDEN_TIMEOUT=5
#GRUB_HIDDEN_TIMEOUT_QUIET=true

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
#GRUB_COLOR_NORMAL="light-blue/black"
#GRUB_COLOR_HIGHLIGHT="light-cyan/blue"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/path/to/wallpaper"
#GRUB_THEME="/path/to/gfxtheme"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

# Uncomment to make GRUB remember the last selection. This requires to
# set 'GRUB_DEFAULT=saved' above.
#GRUB_SAVEDEFAULT="true"

[root@archiso /]# blkid

/dev/sr0: UUID="2017-04-01-11-29-57-00" LABEL="ARCH_201704" TYPE="iso9660" PTUUID="48d46635" PTTYPE="dos"
/dev/sda1: PARTLABEL="BIOS boot partition" PARTUUID="0102316e-d6ba-4896-b184-7d93bbd8fd1b"
/dev/sda2: LABEL="top_level" UUID="fd92f3e7-f092-48c7-9f16-d94ba3de51fb" UUID_SUB="9c7e449b-26f5-43d0-88a0-f7d01479cf82" TYPE="btrfs" PARTLABEL="Linux filesystem" PARTUUID="2758217d-bf0b-4036-aea0-26b68ae733f5"
/dev/sda3: UUID="b1388b5c-0bd1-444b-9b2e-d1b39537326d" TYPE="swap" PARTLABEL="Linux swap" PARTUUID="c48910f1-2483-4228-a0f4-952b2e9f101a"
/dev/loop0: TYPE="squashfs"

root@archiso ~ # lsblk -f

NAME   FSTYPE   LABEL       UUID                                 MOUNTPOINT
loop0  squashfs                                                  /run/archiso/sfs/airootfs
sda
├─sda1
├─sda2 btrfs    top_level   fd92f3e7-f092-48c7-9f16-d94ba3de51fb /mnt
└─sda3 swap                 b1388b5c-0bd1-444b-9b2e-d1b39537326d [SWAP]
sr0    iso9660  ARCH_201704 2017-04-01-11-29-57-00               /run/archiso/bootmnt

Last edited by MountainX (2018-03-16 23:46:00)

Lone_Wolf · 2018-03-17 13:24:35

The setup you describe appears to be for a bios boot on drives using gpt .

You did verify whether this vps provider uses bios or uefi boot for virtual machines ?
(from install iso look for /sys/firmware/efi/efivars )

MountainX · 2018-03-17 20:04:36

Lone_Wolf wrote:

The setup you describe appears to be for a bios boot on drives using gpt .
You did verify whether this vps provider uses bios or uefi boot for virtual machines ?
(from install iso look for /sys/firmware/efi/efivars )

This VPS uses SeaBIOS. BTW, I think all or almost all KVM VPS's use SeaBIOS.

My setup is indeed bios boot on GPT using grub.

Last edited by MountainX (2018-03-17 20:06:09)

Lone_Wolf · 2018-03-18 12:41:45

The error looks like it comes from systemd emergency mode in the initramfs , not from grub itself.

what is in your /etc/mkinitcpio.conf ?

You have many options in your grub boot menu including several that use the lts kernel.
please remove quiet from the kernel boot lines.
Do all choices fail ?

What happens if you boot from the iso and choose "boot into existing system" ?

MountainX · 2018-03-18 17:33:02

Lone_Wolf wrote:

what is in your /etc/mkinitcpio.conf ?

The only change I made was to add the btrfs binary.

BINARIES=(/usr/bin/btrfs)

Lone_Wolf wrote:

Do all choices fail ?
What happens if you boot from the iso and choose "boot into existing system" ?

Don't have that info. I had done an identical installation that worked on another VPS (also KVM) and the only difference I could find with this one was that the ISO image for the installation was fairly old:

> iso9660 ARCH_201704 2017-04-01-11-29-57-00

I compared every config file (including mkinitcpio.conf, grub, and all others) and there were no differences in any of them (other than IP addresses, device names, etc.) So I concluded that the issue must have been related to using an old ISO image, in particular something around grub and/or btrfs tools. Do you think that is a valid conclusion? If so, any thoughts on how it would have played a role?

My first test of this theory was to repeat the same installation steps exactly, but to use ext4 instead of btrfs. That succeeded. No problems.

Next, I was able to have the VPS provide update the Arch ISO image to a current one. At this point, my next test (if I have time to do it) would be to test the exact same installation steps (using the same configurations / settings) with the newer ISO image and btrfs. However, I've actually done that already at another KVM-based VPS provider and everything went as expected. The VPS instance is up and running with Arch and btrfs. That's why I now think the whole issue is caused by the older ISO image.

It would be very helpful to have a theory to explain this rather than just a simple "it worked" or "it didn't work". Also, in a case where I could not have gotten the VPS provider to update the ISO image, is there a simple work-around (while using the same ISO-based installation method, not the rsync or some other installation method)? Maybe it is as simple as manually loading new btrfs kernel modules after running pacstrap (but before arch-chroot)... is that even possible?

Lone_Wolf · 2018-03-19 11:43:54

I was hoping these providers were using kvm as a type 1 hypervisor, but your description suggests otherwise.

Are you using an installation iso from the provider or a generic one ?
Which kernel version do they support with archlinux ?

MountainX · 2018-03-19 22:16:47

Lone_Wolf wrote:

I was hoping these providers were using kvm as a type 1 hypervisor, but your description suggests otherwise.
Are you using an installation iso from the provider or a generic one ?
Which kernel version do they support with archlinux ?

I think the distinction that is most relevant is the one between KVM, XEN and OpenVZ, as those are the only three common choices on low end VPS servers like what I'm using. OpenVZ does not let the guest OS run its own kernel, while KVM and XEN do. As far as I know, the distinction between a type 1 hypervisior and type 2 is not the key issue in regard to my situation. With KVM, I can run any kernel version available in the OS I'm installing.

So in regard to your questions:

1. The installation ISO is the same "generic" one downloaded from archlinux.org. The only issue is that (at the time of my question) it was a year old. They have since downloaded and made available a new Arch ISO image.

2. KVM allows using any kernel version available in Arch.

My problem, as I currently understand it (which may be wrong), resulted from the kernel on the installation ISO being a year old. I suppose one could replicate this and test my theory on any hardware. I don't think it had anything to do with KVM except that KVM required a specific installation of grub. So if one used a year old ISO, installed a grub2 bootloader (type 386-pc), used btrfs, and did the other steps I outlined in my OP, I think one would run into the same issue. That's just my guess, of course.

Reference in regard to whether KVM is a type 1 or type 2 hypervisor:

Type 1 and Type 2 Hypervisors Explained -- Virtualization Review https://virtualizationreview.com/blogs/ … ained.aspx

Some links for fruther reading are provided there, but the bottom line is that KVM is not a clear case because it could be categorized as either type 1 or type 2 (some of both). And, the conclusion of that article is "The relevance of Type 1 and Type 2 distinction is academic." (In my opinion, it is also not relevant to this issue.)

Last edited by MountainX (2018-03-19 22:34:34)

Arch Linux

#1 2018-03-16 23:44:05

KVM grub 2 ERROR device UUID not found

#2 2018-03-17 13:24:35

Re: KVM grub 2 ERROR device UUID not found

#3 2018-03-17 20:04:36

Re: KVM grub 2 ERROR device UUID not found

#4 2018-03-18 12:41:45

Re: KVM grub 2 ERROR device UUID not found

#5 2018-03-18 17:33:02

Re: KVM grub 2 ERROR device UUID not found

#6 2018-03-19 11:43:54

Re: KVM grub 2 ERROR device UUID not found

#7 2018-03-19 22:16:47

Re: KVM grub 2 ERROR device UUID not found

Board footer