Unable to boot and mount root partition [SOLVED]

Hickin · 2018-03-21 21:32:13

I've had a problem with my Archlinux server for a while now and I am having a hard time trying to figure out what to do.
It started I did a upgrade of kernel, which version I am uncertain of but quite new since it was a couple of mounts ago it all happend.

When booting GRUB does not detect any booting partition and it ends up in emergency mode.
Booting up a Archlinux live USB I can see that fdisk reports my disk sdb has a corrupt GPL table.
But it shows all my partition at sdb:

/dev/sdb1 Root
/dev/sdb2 BIOS BOOT
/dev/sdb3 Home
/dev/sdb4 Swap

Trying to mount the disk I get:

mount: special device /dev/sdb1 does not exist

What would be the next step?
I will try to get the outputs from fdisk and blkid by saving to a file on the live USB.

Last edited by Hickin (2018-04-02 18:57:39)

blackout · 2018-03-22 09:16:06

yeah provide blkid and grub.cfg
and try
lvscan
maybe there is something with your partitioning
# blockdev --rereadpt

Hickin · 2018-03-29 14:09:54

Ok, after much time I've got SSH working and could login from another computer so I could paste the result from the texts:

root@archiso ~ # lvscan
  ACTIVE            '/dev/VolGroup/lvolhome' [<7.28 TiB] inherit

and

root@archiso ~ # blkid
/dev/sda: UUID="HKJ14R-AiOF-yuxs-wQ1V-XJ0c-PwMv-YWwQZu" TYPE="LVM2_member"
/dev/sdc: UUID="tDxbLr-bfIg-5jj6-XcWx-MbFS-d4oD-mBelcN" TYPE="LVM2_member"
/dev/sdd1: LABEL="ARCH_201803" UUID="66CD-C8AA" TYPE="vfat" PARTUUID="00a2a341-01"
/dev/loop0: TYPE="squashfs"
/dev/mapper/VolGroup-lvolhome: UUID="01f05586-3ee1-4551-9d94-98f90c7b7f28" TYPE="ext4"
/dev/sdb: PTUUID="d909924c-e7a2-4f2e-ada7-ebc2b940b00e" PTTYPE="gpt"

and

1 root@archiso ~ # fdisk -l                                                                                                              :(
Disk /dev/sda: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


The primary GPT table is corrupt, but the backup appears OK, so that will be used.
Disk /dev/sdb: 139.8 GiB, 150039945216 bytes, 293046768 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: D909924C-E7A2-4F2E-ADA7-EBC2B940B00E

Device        Start       End   Sectors  Size Type
/dev/sdb1      2048  62916607  62914560   30G Linux root (x86-64)
/dev/sdb2        34      2047      2014 1007K BIOS boot
/dev/sdb3  62916608  83888127  20971520   10G Linux swap
/dev/sdb4  83888128 293046734 209158607 99.8G Linux home

Partition table entries are not in disk order.


Disk /dev/sdc: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdd: 14.9 GiB, 16013942784 bytes, 31277232 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00a2a341

Device     Boot Start      End  Sectors  Size Id Type
/dev/sdd1  *     2048 31277231 31275184 14.9G  c W95 FAT32 (LBA)


Disk /dev/loop0: 442.1 MiB, 463548416 bytes, 905368 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/VolGroup-lvolhome: 7.3 TiB, 8001566015488 bytes, 15628058624 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

and last

root@archiso ~ # cat /boot/grub/grub.cfg
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
if [ -s $prefix/grubenv ]; then
  load_env
fi
set default="0"

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_msdos
insmod ext2
set root='hd0,msdos5'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos5 --hint-efi=hd0,msdos5 --hint-baremetal=ahci0,msdos5  ad4103fa-d940-47ca-8506-301d8071d467
else
  search --no-floppy --fs-uuid --set=root ad4103fa-d940-47ca-8506-301d8071d467
fi
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_input console
terminal_output gfxterm
set timeout=5
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux, with Linux core repo kernel' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-core repo kernel-true-ad4103fa-d940-47ca-8506-301d8071d467' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos5'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos5 --hint-efi=hd0,msdos5 --hint-baremetal=ahci0,msdos5  ad4103fa-d940-47ca-8506-301d8071d467
        else
          search --no-floppy --fs-uuid --set=root ad4103fa-d940-47ca-8506-301d8071d467
        fi
        echo    'Loading Linux core repo kernel ...'
        linux   /boot/vmlinuz-linux root=UUID=ad4103fa-d940-47ca-8506-301d8071d467 rw  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initramfs-linux.img
}
menuentry 'Arch Linux, with Linux core repo kernel (Fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-core repo kernel-fallback-ad4103fa-d940-47ca-8506-301d8071d467' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos5'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos5 --hint-efi=hd0,msdos5 --hint-baremetal=ahci0,msdos5  ad4103fa-d940-47ca-8506-301d8071d467
        else
          search --no-floppy --fs-uuid --set=root ad4103fa-d940-47ca-8506-301d8071d467
        fi
        echo    'Loading Linux core repo kernel ...'
        linux   /boot/vmlinuz-linux root=UUID=ad4103fa-d940-47ca-8506-301d8071d467 rw  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initramfs-linux-fallback.img
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_memtest86+ ###
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

I was unable to run this command

130 root@archiso ~ # blockdev --rereadpt /dev/sdb                                                                                        :(

mxfm · 2018-03-31 05:33:28

I don't know GRUB intrinsics but this line from fdisk output:

The primary GPT table is corrupt, but the backup appears OK, so that will be used.

suggests that you need to open relevant disk in partition editor and save changes (assuming fdisk/gptfdisk wil read backup GPT). GRUB may fail on erroneous primary GPT.

Last edited by mxfm (2018-03-31 05:34:01)

Hickin · 2018-03-31 07:55:44

I agree that the GPT table is hopefully the problem and did occur when I upgraded the kernel, leading to being unable to start up the server.
However, like I said in the first post, I am unable to mount the disk and therefor unable to make any adjustments on the system at all.
Is there any way to reach the system files without mounting it?

mxfm · 2018-03-31 08:05:32

Technically you need to mount filesystems/partitions before using them. If you suspect file system corruption, you can use relevant filesystem tools to repair filesystem/partition, then mount it, and try to reinstall arch linux kernel package and to run mkinitcpio to create relevant initrd. This is minimum to boot the system. In your case thee is visible primary GPT header corruption, so good start is to use fdisk/gptfdisk tool to fix this issue. It is very unlikely that primary GPT header was corrupted by system update.

sadboi389 · 2018-03-31 08:30:34

IIRC, gdisk should be used instead of fdisk when having a GPT partition table type.

In your live system, execute `gdisk /dev/sdb`, hit `p` and verify the partitions are correct, then hit `w` to write the partition table.
That should fix the table.

In case it doesn't: please execute `gdisk /dev/sdb` and then the `v` command. Post its output.

I advise to perform `fsck` as soon as possible.

Last edited by sadboi389 (2018-03-31 08:32:41)

Hickin · 2018-04-02 19:00:08

Thanks sadboi389, that did it!
After running the commands

gdisk /dev/sdb

and hitting 'w', the partition table was rewritten and my system booted up again, many thanks!
Just a question more, you say I should run 'fsck', will have to look into it.
Unable to run it when the system was mounted and nothing happend when running the live cd.

Arch Linux

#1 2018-03-21 21:32:13

Unable to boot and mount root partition [SOLVED]

#2 2018-03-22 09:16:06

Re: Unable to boot and mount root partition [SOLVED]

#3 2018-03-29 14:09:54

Re: Unable to boot and mount root partition [SOLVED]

#4 2018-03-31 05:33:28

Re: Unable to boot and mount root partition [SOLVED]

#5 2018-03-31 07:55:44

Re: Unable to boot and mount root partition [SOLVED]

#6 2018-03-31 08:05:32

Re: Unable to boot and mount root partition [SOLVED]

#7 2018-03-31 08:30:34

Re: Unable to boot and mount root partition [SOLVED]

#8 2018-04-02 19:00:08

Re: Unable to boot and mount root partition [SOLVED]

Board footer