[SOLVED] Grub.cfg: Unable to identify typo (syntax error)

SeagullFish · 2023-08-27 17:12:53

Hello.

I have both Linux Mint and Arch Linux installed on separate partitions. My computer boots from UEFI, and the storage device has a GPT-based partition table that looks like this:

nvme0n1     UUID                                 Mount point (Mint)  Mount point (Arch)  Comment
├─nvme0n1p1 xxxx-xxxx                            /boot/efi           /boot/efi           EFI system partition
├─nvme0n1p2                                      N/A                 N/A                 Microsoft reserved. Leftovers from a long gone MS Windows installation. (I have never dared to remove this, in case of bricking/breaking anything.)
├─nvme0n1p3 10f7xxxx-xxxx-xxxx-xxxx-xxxxxxxxae73 [SWAP]              [SWAP]
├─nvme0n1p4 d4c3xxxx-xxxx-xxxx-xxxx-xxxxxxxxc1e4 /                   N/A                 Linux Mint root partition, including a /boot directory.
├─nvme0n1p8 7758xxxx-xxxx-xxxx-xxxx-xxxxxxxx8ffa N/A                 /                   Arch Linux root partition. A /boot directory is NOT included on this partition.
└─nvme0n1p9 7115xxxx-xxxx-xxxx-xxxx-xxxxxxxx9eb0 N/A                 /boot               Separate boot-partition for Arch Linux.

(Please note: Some partitions that are irrelevant to the issue has been omitted from the list.)

From the answer to this post at stackexchange.com, I learn that:
If I edit the configuration file for the GRUB instance provided with Linux Mint, it should be able to read the configuration file for the GRUB instance provided by Arch Linux, thereby making a boot entry for Arch Linux, and display the boot menu of Arch Linux when that boot entry is selected.

Very well! So this is what I did:

Booted into Linux Mint
Edited /etc/grub.d/40_custom

Adapted the code from stackexchange.com, and put it into the 40_custom file, making the following contents:

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.

## Added manually:
#Arch linux:
menuentry 'Arch Linux' {
    search --no-floppy --fs-uuid --set=root 7115xxxx-xxxx-xxxx-xxxx-xxxxxxxx9eb0
    set prefix=$(root)'/grub'
    configfile $prefix/grub.cfg
}

Executed the grub-mkconfig command, to regenerate /boot/grub/grub.cfg. This process failed, giving me the following output:

foo@bar:~$ sudo grub-mkconfig -o /boot/grub/grub.cfg
[sudo] password for foo:      
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/50_linuxmint.cfg'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.2.0-26-generic
Found initrd image: /boot/initrd.img-6.2.0-26-generic
Found linux image: /boot/vmlinuz-5.15.0-79-generic
Found initrd image: /boot/initrd.img-5.15.0-79-generic
Found linux image: /boot/vmlinuz-5.15.0-76-generic
Found initrd image: /boot/initrd.img-5.15.0-76-generic
Warning: os-prober will be executed to detect other bootable partitions.
Its output will be used to detect bootable binaries on them and create new boot entries.
Found Windows Boot Manager on /dev/nvme0n1p1@/EFI/Microsoft/Boot/bootmgfw.efi
Found Arch Linux on /dev/nvme0n1p8
Adding boot menu entry for UEFI Firmware Settings ...
error: $.
error: syntax error.
error: Incorrect command.
error: syntax error.
Syntax error at line 288
Syntax errors are detected in generated GRUB config file.
Ensure that there are no errors in /etc/default/grub
and /etc/grub.d/* files or please file a bug report with
/boot/grub/grub.cfg.new file attached.

So I opened the file /boot/grub/grub.cfg.new to review its contents. Here is a small extract from its contents:

Line Contents
285  #Arch linux:
286  menuentry 'Arch Linux' {
287      search --no-floppy --fs-uuid --set=root 7115xxxx-xxxx-xxxx-xxxx-xxxxxxxx9eb0
288      set prefix=$(root)'/grub'
289      configfile $prefix/grub.cfg
290  }

My question:

What am I doing wrong with the syntax at line 288? I reproduced that line exactly as it was written in the post at stackexchange.com.

Last edited by SeagullFish (2023-09-25 20:46:28)

frostschutz · 2023-08-27 17:26:10

try ($root) instead of $(root)?

SeagullFish · 2023-08-27 18:31:49

frostschutz wrote:

try ($root) instead of $(root)?

Thank you! Apparently, your suggestion solved the issue with the syntax error, and a new grub.cfg file was created successfully.
However, when rebooting and testing the new configuration file, an error occurs if I select the new boot entry for Arch Linux:

error: no such device: 7115xxxx-xxxx-xxxx-xxxx-xxxxxxxx9eb0
Press any key to continue ...

After pressing any key, GRUB returns to the boot menu of Linux Mint. And subsequently, if I now select the boot entry for Linux Mint, another error occurs:

error: could not find the file «/grub/grubenv».
Press any key to continue ...

After pressing any key, Linux Mint successfully boots without further errors.
However, if I reboot the computer, and then select the boot entry for Linux Mint, without selecting the boot entry for Arch Linux in advance, then no error occurs and Linux Mint boots successfully.

Does anyone have further suggestions? Or is it unlikely that what I am trying to do is going to work at all?

(For the record: I did not actually put in all those x'es into the config file. These are just substitutes for the real ciphers and letters of the UUID's of my partitions, to avoid exposing my entire UUID's on this forum.)

frostschutz · 2023-08-27 18:44:27

Make sure the UUID is set correctly? Also that Grub has loaded the modules necessary to identify UUIDs for that filesystem.

The menuentry (search --set=root command) changes the root variable which is used in a lot of places. Once set incorrectly, you get additional errors.

SeagullFish · 2023-09-19 21:40:09

frostschutz wrote:

Make sure the UUID is set correctly? Also that Grub has loaded the modules necessary to identify UUIDs for that filesystem.
The menuentry (search --set=root command) changes the root variable which is used in a lot of places. Once set incorrectly, you get additional errors.

Hello again. A few weeks have gone by now. I have been troubleshooting this matter occasionally, when I have had some spare time. I have been studying the GNU GRUB Manual 2.06, and learned more about how to configure GRUB .

As described in the introduction, both Mint and Arch each provide their own instance of GRUB. The UEFI is currently set to load the Mint instance automatically at power-up. However, by entering the UEFI system settings at power-up, I can do a one-time override of this setting, and manually load the Arch instance instead. This is a clumsy and cumbersome way of selecting which OS to boot, but at least it makes me able to load both instances of GRUB (individually) and compare their behavior.

As you may already know, GRUB offers the option to hit C to enter a shell prompt when the GRUB boot menu is displayed. I have done this within both GRUB instances, and executed some carefully selected commands. These are the results I got:

From the Mint instance of GRUB:

GNU GRUB version 2.06
You may edit lines in BASH style. The first word in each line can be auto-completed to a command name with TAB. The second word can be TAB-ed to complete device or file names. Exit at any time with ESC.

grub> ls
(proc) (memdisk) (hd0) (hd0,gpt9), (hd0,gpt8), (hd0,gpt4), (hd0,gpt3), (hd0,gpt2), (hd0,gpt1)
grub> probe –fs-uuid hd0,gpt1
xxxx-xxxx
grub> probe –fs-uuid hd0,gpt2
error: unknown filesystem.
grub> probe –fs-uuid hd0,gpt3
error: unknown filesystem.
grub> probe –fs-uuid hd0,gpt4
d4c3xxxx-xxxx-xxxx-xxxx-xxxxxxxxc1e4
grub> probe –fs-uuid hd0,gpt8
7758xxxx-xxxx-xxxx-xxxx-xxxxxxxx8ffa
grub> probe –fs-uuid hd0,gpt9
error: unknown filesystem.
grub> lsmod

(Some lines scroll past the screen too fast for me to catch them, but here is the end of the list.)

gcry_sa512	        2			crypto
part_gpt	        1
part_msdos              1
part_apple	        1
ntfs		        1			fshelp
minicmd	                1
memdisk	                1
lssal		        1
lsefilsystab	        1
lsefi		        1
ls		        1			normal,datetime,extcmd
linux		        2			mmap,video,boot,relocator
reloactor	        3			mmap
loopback	        1			extcmd
loadenv	                1			disk,extcmd
disk		        2
keystatus	        1			bitmap,bufio
iso9660	                1			fshelp
hfsplus		        1			fshelp
help		        1			normal,extcmd
halt		        1			ascpi
acpi		        2			mmap,extcmd
mmap		        8
gfxterm_background	1	                bitmap_scale,video_colors,video,bitmap,gfxterm,extcmd
gfxmenu	                1		        trig,bitmap_scale,video_colors,video,bitmap,normal,font,gfxterm
gfxterm	                4			video,font
video_colors	        3
bitmap_scale	        3			bitmap
bitmap		        8
trig		        2
font		        6			video,bufio
fat		        2			fshelp
ext2		        2			fshelp
fshelp		        10
efiwsetup	        1
echo		        1			extcmd
configfile	        1			normal
normal		        9			terminal,gettext,bufio,net,boot,crypto,datetime,extcmd
gettext		        11
terminal	        10
chain		        1			efinet,net,boot
efinet		        2			net
net		        13			bufio,boot,datetime,priority_queue
priority_queue		14
datetime	        24
bufio		        31
cat		        1			extcmd
extcmd		        32
btrfs		        1			zstd,raid6rec,izopio,gzio
gzio		        6			gcry_crc
gcry_crc	        7			crypto
lzopio		        3			crypto
crypto		        67
raid6rec	        2			diskfilter
diskfilter	        7
zstd		        2
boot		        27
all_video	        2			video_cirrus,video_bochs,efi_uga,efi_gop
efi_gop	                3			video,video_fb
efi_uga	                3			video,video_fb
video_bochs	        3			video,video_fb
video_cirrus	        3			video,video_fb
vide_fb	                13
video		        27

In the mint instance, I also made a “stupid” additional test, to verify that I don’t get the same error message both when probing an unreadable partition and when probing a partition that does not exist:

grub> probe –fs-uuid hd0,gpt10
error: could not find disk “hd0,10”

From the Arch instance of GRUB:

GNU GRUB version 2:2.12rci1-1
Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. Anywhere else TAB lists possible device or file completions. To enable less(1)-like paging, “set pager=1”. Exit at any time with ESC.

grub> ls
(hd0) (hd0,gpt9), (hd0,gpt8), (hd0,gpt4), (hd0,gpt3), (hd0,gpt2), (hd0,gpt1)
grub> probe –fs-uuid hd0,gpt1
xxxx-xxxx
grub> probe –fs-uuid hd0,gpt2
error: unknown filesystem.
grub> probe –fs-uuid hd0,gpt3
error: unknown filesystem.
grub> probe –fs-uuid hd0,gpt4
d4c3xxxx-xxxx-xxxx-xxxx-xxxxxxxxc1e4
grub> probe –fs-uuid hd0,gpt8
7758xxxx-xxxx-xxxx-xxxx-xxxxxxxx8ffa
grub> probe –fs-uuid hd0,gpt9
7115xxxx-xxxx-xxxx-xxxx-xxxxxxxx9eb0
grub> lsmod
Name			Refernce quantity	Dependencies
minicmd		        1
efifwsetup		1
bli			1
gfxterm		        1			video,font
all_video		1			video_cirrus,video_bochs,efi_uga,efi_gop
video_cirrus		2			video_fb,video
video_bochs		2			video_fb,video
efi_uga		        2			video_fb,video
efi_gop		        2			video_fb,video
video_fb		12
font			3			vide_bufio
video			18
loadenv		        1
disk			2
test			1
part_msdos		1
normal			1			terminal,net,gettext,extcmd,datetime,crypto,bufio,boot
gzio			0			gcry_crc
gcry_crc		1			crypto
terminal		2
net			2			priority_queue,datetime,bufio,boot
priority_queueu	        3
gettext			3
extcmd			4
datetime		5
crypto			4
bufio			9
boot			5
part_gpt		2
ext2			3			fshelp
fshelp			4

I realize that the Mint instance, for unknown reasons, is unable to read the partition (hd0,gpt9). This is obviously the partition named /dev/nvme0n1p9 within both Mint and Arch, namely where the Arch instance of GRUB is stored. I can’t figure out what’s the matter.
After booting Mint, I am fully able to mount the partition /dev/nvme0n1p9. I can both read from and write to it without issues.

I don’t get this.

My questions:

What am I overlooking here?
What’s wrong with this partition?
All 3 partitions (hd0,gpt4), (hd0,gpt8) and (hd0,gpt9) seems to be formatted with the same ext4 file system. Thus, if GRUB is able to read one of them, then shouldn’t GRUB be able to read all of them?
Both GRUB instances are the same piece of software. So, if one of them can read a partition, then shouldn’t both of them be able read it?

Any help will be appreciated.

frostschutz · 2023-09-20 06:10:52

I'm not sure.

You could check the filesystems with `tune2fs -l` and see if their properties are different. There may be some new filesystem feature enabled which an older version of Grub does not yet support?

For a small boot partition it's completely fine to use ext2 w/o any journal, instead of ext4. It may improve compatibility.

seth · 2023-09-20 07:06:02

Tried

set root='hd0,gpt9'

?

SeagullFish · 2023-09-23 22:06:46

frostschutz wrote:

I'm not sure.
You could check the filesystems with `tune2fs -l` and see if their properties are different. There may be some new filesystem feature enabled which an older version of Grub does not yet support?
For a small boot partition it's completely fine to use ext2 w/o any journal, instead of ext4. It may improve compatibility.

It seems like you may have just hit the nail on the head here! By executing `tune2fs -l /dev/nvme0n1p#`, I get:

For /dev/nvme0n1p4:

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum

For /dev/nvme0n1p8:

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum

For /dev/nvme0n1p9:

Filesystem features:      has_journal ext_attr resize_inode dir_index FEATURE_C12 filetype extent flex_bg metadata_csum_seed sparse_super large_file huge_file dir_nlink extra_isize metadata_csum

From several posts at other forums and web sites, I learn that certain “new” ext4 filesystem features (introduced within the last 5-6 years) could make compatibility issues with GRUB2. The following features have been specifically mentioned:

inline_data
metadata_csum_seed
ea_inode
large_dir
casefold

I collected this information from here:

A blog post at badrpc.net, dated 2023-07-28
Bug #1844012 at bugs.launchpad.net, reported 2019-09-14
Bug #56897 at savannah.gnu.org, submitted 2019-09-15
A post at unix.stackexchange.com, posted 2023-03-11

In my case, the filesystem feature `metadata_csum_seed` has been added to the troubled partition (dev/nvme0n1p9). I suspect this may be the cause of the problem.

The subsequent question that arises is:

How can I remove that feature from the partition? According to paragraph `-O` at the man page for tune2fs(8), the following filesystem features can be set or cleared using tune2fs:

`dir_index`
`filetype`
`flex_bg`
`has_journal`
`large_file`
`resize_inode`
`sparse_super`
`uninit_bg`

The feature `metadata_csum_seed` is not mentioned. Will I make a mess if I try to remove it anyway?

Last edited by SeagullFish (2023-09-23 22:10:13)

SeagullFish · 2023-09-23 22:08:37

seth wrote:

Tried

set root='hd0,gpt9'

?

This is the result:

grub> set prefix=(hd0,gpt9)/grub
grub> set root=(hd0,gpt9)
grub> linux /vmlinuz-linux root=UUID=7758xxxx-xxxx-xxxx-xxxx-xxxxxxxx8ffa rw loglevel=3 quiet
error: unknown file system.
grub> initrd /intel-ucode.img /initramfs-linux.img
error: you need to load the kernel first.

seth · 2023-09-24 06:29:11

God knows from what version of tune2fs that manpage was.
https://man.archlinux.org/man/core/e2fs … _csum_seed

SeagullFish · 2023-09-25 20:45:34

seth wrote:

God knows from what version of tune2fs that manpage was.
https://man.archlinux.org/man/core/e2fs … _csum_seed

Thank you! The manpage provided by Arch was a lot more helpful. I executed this:

sudo tune2fs -O ^metadata_csum_seed /dev/nvme0n1p9
sudo grub-mkconfig -o /boot/grub/grub.cfg

After reboot, the Mint instance of GRUB is now able to read the troubled partition. But then a new issue arises. The chainload-attempt fails, resulting in the computer rebooting after the following command:

configfile $prefix/grub.cfg

However, what actually does work, is to put the following code into /etc/grub.d/40_custom:

menuentry 'Arch Linux (Hard load)' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod ext2
	search --no-floppy --fs-uuid --set=root 7115xxxx-xxxx-xxxx-xxxx-xxxxxxxx9eb0
	echo	'Loading Arch Linux ...'
	linux /vmlinuz-linux root=UUID=7758xxxx-xxxx-xxxx-xxxx-xxxxxxxx8ffa rw loglevel=3 quiet
	echo	'Loading initial ramdisk ...'
	initrd /intel-ucode.img /initramfs-linux.img
}

After executing the following command and rebooting, Arch boots perfectly fine from the Mint instance of GRUB:

sudo grub-mkconfig -o /boot/grub/grub.cfg

So I guess that the new chainloading issue could be related to the different instances of GRUB being not the exact same version of GRUB. Hence, this could be a matter of compatibility issues, and is not related to the initial question in this thread.

So I assume that I should change the status of this thread to [SOLVED] now. Thank you for your suggestions and help.

P.S. I am a little puzzled that the different manpages-sites around the web differs so significantly, although they describe the usage of the same application/function. This makes me wonder: Which manpages-sites can I really trust are being kept updated? I thought that a distro-independent manpages-site would be the best choice, as the information would be distro-independent. (Hence, the information would be applicable to all linux distros.) Obviously, I was wrong...

seth · 2023-09-25 21:05:56

man.archlinux.org is relevant to archlinux, you cannot expect features described there to be necessarily available on eg. debian stable.
The difference in the online manpages is basically their date of origin and the most relevant/trustworthy ones are the ones you installed with the tool in question on the distro where you're using it.

Arch Linux

#1 2023-08-27 17:12:53