No boot, placed into emergency shell after removing fallback image

justwerks · 2023-04-28 12:14:06

Hello,

I have been using Arch Linux regularly since the beginning of the year, updating just about daily.

Yesterday, I encountered the same issue as this post here: https://bbs.archlinux.org/viewtopic.php?id=285378 , where during an update, the /boot partition was too full, canceling the operation as a result.

After doing a brief search, I saw that 100 MB is apparently very small for a /boot partition. I attempted to resize the /boot FAT partition from 100 MB to 200 MB through downsizing the /root partition on the same drive by 100 mb using a partition manager. That did not work, as I was unable to up size even after unmounting the /boot partition. I followed the other user's direction, deleting the fallback image, and was able to successfully update, but I failed to read that I had to reinstall the kernel. I shutdown that night thinking everything was fine.

This morning, after booting, I was granted with the following:

mount: /new_root: wrong fs type, bad option, bad superblock on /dev/nvme1n1p5, missing codepage or helper program, or other error
dmesg(1) may have more information after failed system mount call.
You are now being dropped into an emergency shell.
sh: can't access tty; job control turned off
[rootfs ]# _

Where should I go from here to restore my system?

V1del · 2023-04-28 13:03:41

Generally just from this message, you should ensure you are not using potentially changing device names (like /dev/nvme1n1p5) wherever they are referenced (root line in your bootloader and fstab): https://wiki.archlinux.org/title/Persis … ice_naming

The easiest would be to get yourself a live disk, chroot into your system with all partitions correctly mounted and reinstalling the linux package/regenerating the initramfs and fixing your fstab and bootloader cmdline to use persistent names

Last edited by V1del (2023-04-28 13:15:54)

justwerks · 2023-04-28 15:32:56

V1del wrote:

Generally just from this message, you should ensure you are not using potentially changing device names (like /dev/nvme1n1p5) wherever they are referenced (root line in your bootloader and fstab): https://wiki.archlinux.org/title/Persis … ice_naming
The easiest would be to get yourself a live disk, chroot into your system with all partitions correctly mounted and reinstalling the linux package/regenerating the initramfs and fixing your fstab and bootloader cmdline to use persistent names

/dev/nvme1n1p5 is the correct device for my root partition. I don't have a label attached to it. From my understanding, I think that the loader just doesn't like the /root being shrunk all of a sudden. Upon inputting the command:

mount /dev/nvme1n1p5 /mnt

I get:

[ 1298.791854] BTRFS error (device nvme1n1p5): device total_bytes should be at most 249857835008 but found 249962692608
[ 1298. 793504] BTRFS error (device nvme1n1p5): failed to read chunk tree: -22
[ 1298.795204] BTRFS error (device nvme1n1p5): open_ctree failed
mount: /mnt: wrong fs type, bad option, bad superblock on /dev/nvme1n1p5, missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call

I think if I add back the 100 MB I took from the root partition, it should function normally. I can then reinstall the linux package, perhaps from the live disk. Am I able to go about this while in the live ISO?

Last edited by justwerks (2023-04-28 15:38:26)

V1del · 2023-04-28 15:41:54

Probably yes, but that means you took the partition space incorrectly. When shrinking a partition you need to generally shrink the filesystem before shrinking the partition and the reverse when doing a growing operation. Which "partition manager" did you use exactly? You might also be able to try a https://wiki.archlinux.org/title/Btrfs#Resizing on the root partition to make the FS match the (smaller) partition size again. But making the partition correctly larger is probably the less error prone and safer operation (... though there's no telling you might have already broken something by reducing the partition size without shrinking the filesystem first)

As for the device naming, no that name is not guaranteed to always be that or be correct (the fact it's nvme1 means there's likely a nvme0 and the two can randomly swap places depending on multithreaded initialisation order), you don't need a LABEL, use the UUID or the PARTUUID instead in that case

Last edited by V1del (2023-04-28 15:44:24)

justwerks · 2023-04-28 16:07:33

V1del wrote:

Probably yes, but that means you took the partition space incorrectly. When shrinking a partition you need to generally shrink the filesystem before shrinking the partition and the reverse when doing a growing operation. Which "partition manager" did you use exactly? You might also be able to try a resize2fs on the root partition to make the FS match the (smaller) partition size again.
As for the device naming, no that name is not guaranteed to always be that or be correct (the fact it's nvme1 means there's likely a nvme0 and the two can randomly swap places depending on multithreaded initialisation order), you don't need a LABEL, use the UUID or the PARTUUID instead in that case

Regarding device naming, I can confirm that nvme1n1p5 is the correct partition for my root directory, nvme0 is an entirely different device, and is formatted in ext4, as opposed to my root, which is in BTRFS. No random swaps have been made.

I used KDE Partition Manager. I put 100 MB in 'Free Size After': https://docs.kde.org/stable5/en/partiti … ition.html , hit OK, performed the operation using 'Apply', then realized I couldn't add the 100 MB I freed onto the /boot partition (nvme1n1p1), and left it there.

I tried using the command

btrfs filesystem resize max /dev/nvme1n1p5

but it resulted in:

ERROR: not a directory: /dev/nvme1n1p5
ERROR: resize works on mounted filesystems and accepts only directories as argument. Passing file contains a btrfs image
would resize the underlying filesystem instead of the image

So, strange, if I can't mount it, because the system thinks its too small then I can't resize it? I'm probably misunderstanding something here. I am dual-booting my install, so there's always the option of swapping to the other OS (Windows) to (hopefully) resize the partition as an option.

justwerks · 2023-04-28 22:45:16

justwerks wrote:

V1del wrote:
Probably yes, but that means you took the partition space incorrectly. When shrinking a partition you need to generally shrink the filesystem before shrinking the partition and the reverse when doing a growing operation. Which "partition manager" did you use exactly? You might also be able to try a resize2fs on the root partition to make the FS match the (smaller) partition size again.
As for the device naming, no that name is not guaranteed to always be that or be correct (the fact it's nvme1 means there's likely a nvme0 and the two can randomly swap places depending on multithreaded initialisation order), you don't need a LABEL, use the UUID or the PARTUUID instead in that case
Regarding device naming, I can confirm that nvme1n1p5 is the correct partition for my root directory, nvme0 is an entirely different device, and is formatted in ext4, as opposed to my root, which is in BTRFS. No random swaps have been made.
I used KDE Partition Manager. I put 100 MB in 'Free Size After': https://docs.kde.org/stable5/en/partiti … ition.html , hit OK, performed the operation using 'Apply', then realized I couldn't add the 100 MB I freed onto the /boot partition (nvme1n1p1), and left it there.
I tried using the command
btrfs filesystem resize max /dev/nvme1n1p5
but it resulted in:
ERROR: not a directory: /dev/nvme1n1p5
ERROR: resize works on mounted filesystems and accepts only directories as argument. Passing file contains a btrfs image
would resize the underlying filesystem instead of the image
So, strange, if I can't mount it, because the system thinks its too small then I can't resize it? I'm probably misunderstanding something here. I am dual-booting my install, so there's always the option of swapping to the other OS (Windows) to (hopefully) resize the partition as an option.

I booted up GParted, attempting to grow the partition, but the same error is occurring. I feel like I'm just about SOL here.

>Grow /dev/nvme1n1p5 from 232.70 GiB to 232.80 GiB
>calibrate /dev/nvme1n1p5
path: /dev/nvme1n1p5 (partition)
start: 487477248
end 975480831
size: 488003584 (232.70 GiB)
>check file system on /dev/nvme1n1p5 for errors and (if possible) fix them
>btrfs check '/dev/nvme1n1p5'
Opening filesystem to check
Checking filesystem on /dev/nvme1n1p5
UUID: 9ca48590-b377-4987-a6ba-f73ff0d489fd
found 108117024768 bytes used, error(s) found
total csum bytes: 104652864
total tree bytes: 768376832
total fs tree bytes: 594919424
total extent bytes: 41353216
btree space waste bytes: 140431378
file data blocks allocated: 1955129470976
referenced 109931466752

[1/7] checking root items
[2/7] checking extent
ERROR: block device size is smaller than total_bytes in device item, has 249857835008 expect >= 249962692608
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)

seth · 2023-04-29 06:38:52

The FS is still bigger than the partition.
If you're lucky and haven't written around too much (notably on the boot partition), restoring the original partition layout w/ plain (c)fdisk will hopefully align FS and partition and the FS is still there and intact.

DO NOT TRY TO "FIX" THE FS ON THE SHORT PARTITION!

Shrinking partitions, notably to the right (ie. cutting away the head of the partition) is a delicate process and even if you know what you're doing, you want to have backups.
If you don't know what you're doing, you're very lucky to not trash your system.

justwerks · 2023-04-29 15:56:28

seth wrote:

The FS is still bigger than the partition.
If you're lucky and haven't written around too much (notably on the boot partition), restoring the original partition layout w/ plain (c)fdisk will hopefully align FS and partition and the FS is still there and intact.
DO NOT TRY TO "FIX" THE FS ON THE SHORT PARTITION!
Shrinking partitions, notably to the right (ie. cutting away the head of the partition) is a delicate process and even if you know what you're doing, you want to have backups.
If you don't know what you're doing, you're very lucky to not trash your system.

So restoring the original partition layout would be done w/ the command

sfdisk /dev/(device/partition where /boot is held) < sda.dump

? If so, wish me luck.

seth · 2023-04-29 15:59:29

If you have access to an intact dump of the original partition table, NO!
Not "device/partition where /boot is held" but the disk, ie. eg. sda, NOT sda1!

Arch Linux

#1 2023-04-28 12:14:06

No boot, placed into emergency shell after removing fallback image

#2 2023-04-28 13:03:41

Re: No boot, placed into emergency shell after removing fallback image

#3 2023-04-28 15:32:56

Re: No boot, placed into emergency shell after removing fallback image

#4 2023-04-28 15:41:54

Re: No boot, placed into emergency shell after removing fallback image

#5 2023-04-28 16:07:33

Re: No boot, placed into emergency shell after removing fallback image

#6 2023-04-28 22:45:16

Re: No boot, placed into emergency shell after removing fallback image

#7 2023-04-29 06:38:52

Re: No boot, placed into emergency shell after removing fallback image

#8 2023-04-29 15:56:28

Re: No boot, placed into emergency shell after removing fallback image

#9 2023-04-29 15:59:29

Re: No boot, placed into emergency shell after removing fallback image

Board footer