Messed up GRUB and I can't recover

zombieagain · 2020-08-26 01:09:37

Hello,

after years of relative stability, after upgrades, in the past couple of months I occasionally received error message about failing to mounting /boot. In the past I got through it by reinstalling grub, but this time I messed up something and I can't recover the situation, basically, everything I try sends me to grub Minimal bash...

My setup is relatively simple (I removed all my storage disks from fstab while I debug this):
/dev/sda1 is the boot partition. It was initially setup to mount on /boot, but in my last attempts I modified fstab to mount on /boot/efi
/dev/sda2 is the root partition.
Motherboard's brand is MSI, single operating system.

My last attempt was:
start from a live disk
mount partitions
arch-chroot
eliminate old UEFI entries with efibootmgr
run this commands:
grub-install --target=x86_64-efi --bootloader-id=GRUB --efi-directory=/boot/efi (I also attempted a second time adding "--removable" as it helped with a previous MSI motherboard I had)
grub-mkconfig -o /boot/grub/grub.cfg
exit arch-chroot
restart
result: GRUB minimal bash

of course I'm not skilled enough to be able to start the system from the GRUB minimal bash.

output of ls when in grub bash:

(hd0) (hd0,msdos1) (hd1) (hd2) (hd2,gpt2) (hd2,gpt1)

Please be patient with me, I've used Arch for many years and I usually get through little issues myself, but this time I'm stumped.

What is the right way to "reset" the boot loader?

fistrosan · 2020-08-26 06:25:42

I am myself a noob, but maybe I can help after all.

What you describe should have worked, I think ... maybe you forgot to create the images with mkinitcpio before running the grub commands ? (https://wiki.archlinux.org/index.php/In … #Initramfs) Another question: you mention that you have changed from /boot to /boot/efi. Do you mean that you did not have an UEFI boot originally and now you are trying to make it so ? More questions: in your description you did not say that you unmounted your partitions after exiting chroot. I hope you did.

Anyhow, you can boot from your grub bash, but you need to do a few things:

1) find your root partition with "ls (hd0,msdos1)/", "ls (hd2/gpt1)/" etc
2) assume it is in (hd0,msdos1) then mount it with "set root=(hd0,msdos1)"
3) then the kernel "linux /boot/vmlinuz-linux root=/dev/sda2" -> this is where your root parition was yes ? sda2 ? also, you can autocomplete in case you have different kernels (i.e. lts)
4) "initrd /boot/initramfs-linux.img" (you can also select lts or anything other image) -> if the image file does not appear that would explain why your method did not work. In this case repeat what you did but rrunning the mkinitcpio command.
5) "boot"

And the system should boot. The thing is: once you boot you should recreate grub with the exact commands you used, so I see no reason as to why they should work correctly now ... I am afraid therefore that what I am offering is only half a fix, as I think you will have to do it everytime until you actually repair your grub ...

Head_on_a_Stick · 2020-08-26 09:50:15

zombieagain wrote:

I occasionally received error message about failing to mounting /boot

That's a bit vague, can you remember the exact message?

zombieagain wrote:

everything I try sends me to grub Minimal bash

Do you mean the 'grub>' prompt? If so then that indicates that either the configuration file cannot be found or that the configuration file does not contain any 'menuentry' stanzas. Have you checked /boot/grub/grub.cfg? It's probably best to post the full content here.

If you see the 'grub rescue>' prompt then that means that GRUB cannot find any modules (which are located under /boot/grub/).

zombieagain wrote:

My setup is relatively simple (I removed all my storage disks from fstab while I debug this):
/dev/sda1 is the boot partition. It was initially setup to mount on /boot, but in my last attempts I modified fstab to mount on /boot/efi
/dev/sda2 is the root partition.

So is this a UEFI system? Is /dev/sda1 an EFI system partition? The full output of 'parted --list' would be clearer and less confusing, the command can be run from a graphical live environment for ease of pasting (or see https://wiki.archlinux.org/index.php/Li … in_clients).

zombieagain wrote:

My last attempt was:
start from a live disk
mount partitions
arch-chroot
eliminate old UEFI entries with efibootmgr

Again, this is a bit vague — please post the *exact* commands that you used.

We should also see the full output of

efibootmgr -v

^ That command can also be run from any live environment.

zombieagain wrote:

output of ls when in grub bash:

The output of the set command would be more useful, that will show where GRUB is looking for the configuration file.

zombieagain · 2020-08-26 10:17:21

Hello,
thanks for all the help. Following the suggestions, I was able to boot the system from the grub minimal bash. Then from the graphical environment I ran these 3 commands:

 sudo mkinitcpio -p linux
sudo grub-install --target=x86_64-efi --bootloader-id=GRUB --efi-directory=/boot/efi
sudo grub-mkconfig -o /boot/grub/grub.cfg

rebooted and everything seems to be ok. I rebooted a few times, just in case ;-)

The error the started it all is this one:

https://imgur.com/a/SZNvHpI

sorry about the ugly phone photo, but at that point keyboard and mouse became unresponsive and I didn't know any other way to capture that.

For reference:
output of parted --list:

 Model: ATA Samsung SSD 840 (scsi)
Disk /dev/sda: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End    Size   File system  Name  Flags
 1      1049kB  538MB  537MB  fat32              boot, esp
 2      538MB   128GB  127GB  ext4

output of efibootmgr -v:

BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0000,0008,0006,0007,0002,0005
Boot0000* GRUB  HD(1,GPT,3640a733-6416-4331-9420-adc642352286,0x800,0x100000)/File(\EFI\GRUB\GRUBX64.EFI)
Boot0002* CD/DVD Drive  BBS(CDROM,,0x0)/VenHw(5ce8128b-2cec-40f0-8372-80640e3dc858,0300)..GO..NO..........T.S.S.T.c.o.r.p. .D.V.D.W.B.D. .S.H.-.B.1.2.3.L...................\.,.@.r.d.=.X..........A...........................>..Gd-.;.A..MQ..L.8.R.9.4.G.6.B.E.0.A.2.3.8.0. . . . . . ........BO
Boot0005* USB KEY       BBS(HD,,0x0)/VenHw(5ce8128b-2cec-40f0-8372-80640e3dc858,0b00)..GO..NO........u.H.P. .v.1.1.5.w. .0...0.0...................\.,.@.r.d.=.X..........A.......................2..Gd-.;.A..MQ..L.0.0.0.0.0.0.0.0.0.0.0.5.B.E........BO
Boot0006* UEFI: HP v115w 0.00, Partition 1      PciRoot(0x0)/Pci(0x14,0x0)/USB(12,0)/HD(1,MBR,0x2c1c9742,0x800,0xf0a048)..BO
Boot0007* Hard Drive    BBS(HD,,0x0)/VenHw(5ce8128b-2cec-40f0-8372-80640e3dc858,0200)..GO..NO..........S.a.m.s.u.n.g. .S.S.D. .8.4.0. .P.R.O. .S.e.r.i.e.s...................\.,.@.r.d.=.X..........A...........................>..Gd-.;.A..MQ..L.1.S.P.2.E.N.C.A.0.C.0.4.9.5. .Z. . . . ........BO
Boot0008* UEFI OS       HD(1,GPT,3640a733-6416-4331-9420-adc642352286,0x800,0x100000)/File(\EFI\BOOT\BOOTX64.EFI)..BO

full content of grub.cfg:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod ext2
set root='hd0,gpt2'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-ieee1275='ieee1275//disk@0,gpt2' --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  8ce9c4d1-acf5-4b32-ad52-66011a0c174d
else
  search --no-floppy --fs-uuid --set=root 8ce9c4d1-acf5-4b32-ad52-66011a0c174d
fi
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_input console
terminal_output gfxterm
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-8ce9c4d1-acf5-4b32-ad52-66011a0c174d' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod ext2
	set root='hd0,gpt2'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint-ieee1275='ieee1275//disk@0,gpt2' --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  8ce9c4d1-acf5-4b32-ad52-66011a0c174d
	else
	  search --no-floppy --fs-uuid --set=root 8ce9c4d1-acf5-4b32-ad52-66011a0c174d
	fi
	echo	'Loading Linux linux ...'
	linux	/boot/vmlinuz-linux root=UUID=8ce9c4d1-acf5-4b32-ad52-66011a0c174d rw  loglevel=3 quiet
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initramfs-linux.img
}
submenu 'Advanced options for Arch Linux' $menuentry_id_option 'gnulinux-advanced-8ce9c4d1-acf5-4b32-ad52-66011a0c174d' {
	menuentry 'Arch Linux, with Linux linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-advanced-8ce9c4d1-acf5-4b32-ad52-66011a0c174d' {
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		set root='hd0,gpt2'
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root --hint-ieee1275='ieee1275//disk@0,gpt2' --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  8ce9c4d1-acf5-4b32-ad52-66011a0c174d
		else
		  search --no-floppy --fs-uuid --set=root 8ce9c4d1-acf5-4b32-ad52-66011a0c174d
		fi
		echo	'Loading Linux linux ...'
		linux	/boot/vmlinuz-linux root=UUID=8ce9c4d1-acf5-4b32-ad52-66011a0c174d rw  loglevel=3 quiet
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initramfs-linux.img
	}
	menuentry 'Arch Linux, with Linux linux (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-fallback-8ce9c4d1-acf5-4b32-ad52-66011a0c174d' {
		load_video
		set gfxpayload=keep
		insmod gzio
		insmod part_gpt
		insmod ext2
		set root='hd0,gpt2'
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root --hint-ieee1275='ieee1275//disk@0,gpt2' --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  8ce9c4d1-acf5-4b32-ad52-66011a0c174d
		else
		  search --no-floppy --fs-uuid --set=root 8ce9c4d1-acf5-4b32-ad52-66011a0c174d
		fi
		echo	'Loading Linux linux ...'
		linux	/boot/vmlinuz-linux root=UUID=8ce9c4d1-acf5-4b32-ad52-66011a0c174d rw  loglevel=3 quiet
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initramfs-linux-fallback.img
	}
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

thanks to the suggestions I'm up and running, but I feel I don't exactly understand what I did, so I worry that the system is brittle and will break again at some point, so if you see something not right, please let me know.

Thanks!

fistrosan · 2020-08-26 18:43:02

Glad it helped, although I am not sure I know why. It seems to me that all you did new was to recreate the images using mkinitcpio, but those should have been there from the beginning otherwise you would not have been able to boot from the grub bash prompt. Maybe you misplaced them when you changed from /boot to /boot/EFI in your fstab ? Do you use UUID in your fstab ?

Head_on_a_Stick · 2020-08-27 17:46:05

zombieagain wrote:

I feel I don't exactly understand what I did

Looks like the booted kernel did not match the installed modules, perhaps because /boot wasn't mounted during a kernel upgrade.

I can't explain why your attempts to fix the problem failed without knowing the exact commands that you entered.

zombieagain · 2020-08-28 05:57:21

Looks like the booted kernel did not match the installed modules, perhaps because /boot wasn't mounted during a kernel upgrade.
I can't explain why your attempts to fix the problem failed without knowing the exact commands that you entered.

Thanks! If the problem comes back I'll be more careful in posting here with all the info, before messing up even more.

Arch Linux

#1 2020-08-26 01:09:37

Messed up GRUB and I can't recover

#2 2020-08-26 06:25:42

Re: Messed up GRUB and I can't recover

#3 2020-08-26 09:50:15

Re: Messed up GRUB and I can't recover

#4 2020-08-26 10:17:21

Re: Messed up GRUB and I can't recover

#5 2020-08-26 18:43:02

Re: Messed up GRUB and I can't recover

#6 2020-08-27 17:46:05

Re: Messed up GRUB and I can't recover

#7 2020-08-28 05:57:21

Re: Messed up GRUB and I can't recover

Board footer