Arch Linux

Cheeto

Member

Registered: 2022-09-11

Posts: 64

[SOLVED] btrfs snapshots are read-only despite using grub-btrfs

EDIT: apparently `grub-btrfs` doesn't work with the linux-zen kernel. Switching to the regular linux kernel appears to resolve the issue.

---

The issue happens on 100% of bare metal installs and in VMs on 2 machines. I use archinstall to set up base configuration. I followed the archwiki here https://wiki.archlinux.org/title/snappe … _snapshots and here https://wiki.archlinux.org/title/snappe … ount_point and likely more pages that I don't recall.

The exact instructions I made for myself from reading that and trial and error are these:

- install the packages

sudo pacman -S snapper inotify-tools grub-btrfs --noconfirm

- add `grub-btrfs-overlayfs` to the end of the HOOKS array in `/etc/mkinitcpio.conf`, for example:

HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block filesystems fsck grub-btrfs-overlayfs)

- regenerate your initramfs:

sudo mkinitcpio -P

- unmount /.snapshots and delete it

sudo umount /.snapshots
sudo rm -rf /.snapshots

- now you can create the config

sudo snapper -c root create-config /

- delete the the erroneous subvol that `create-config` in the previous step made

sudo btrfs subvolume delete /.snapshots

- recreate /.snapshots

sudo mkdir /.snapshots

- use `sudo mount -a` to mount `@.snapshots` to `/.snapshots` using the existing `/etc/fstab` entry
    - if there isn't one (there should be if you used archinstall) then follow the next two steps:

1. mount `@.snapshots` to `/.snapshots`

sudo mount -o subvol=@.snapshots /dev/nvme1n1p2 /.snapshots

2. make the mount permanent by ensuring it's in `/etc/fstab`

# /dev/nvme1n1p2
UUID=1760eb1c-5daf-4eb4-bc45-09f0dfd6740f       /.snapshots     btrfs           rw,relatime,ssd,discard=async,space_cache=v2,subvolid=260,subvol=/@.snapshots   0 0

- give `/.snapshots` 750 permissions

sudo chmod 750 /.snapshots

- in the snapper config (it's at `/etc/snapper/configs/root`) allow wheel

ALLOW_GROUPS="wheel"

- change default for `/` from 5 to 256

sudo btrfs subvol set-default 256 /

- enable the "systemd unit" (daemon?) for `grub-btrfs`

sudo systemctl enable --now grub-btrfsd

# then check its status for no errors
sudo systemctl status grub-btrfsd

- update grub config

sudo grub-mkconfig -o /boot/grub/grub.cfg

- enable the "systemd unit" (daemon?) for snapper automatic snapshot management

sudo systemctl enable --now snapper-timeline.timer
sudo systemctl enable --now snapper-cleanup.timer

- create a snapshot

sudo snapper -c root create --description "enter a description here"

At this point, snapshots work and show up in grub as expected. I make a rw copy of one which will show in grub and boot normally, using:

sudo btrfs subvolume snapshot /.snapshots/95/snapshot /mnt/snapshot_rw

But the DM never loads if I boot into a regular snapshot. The first arch wiki link I mentioned is supposed to fix that. At that point I can switch to a TTY and run

journalctl -u gdm

The output will be:

-- Boot 6c82216e5b6848c7ad73d8ae6b4d5958 --
Apr 08 12:12:23 archlinux systemd[1]: Starting GNOME Display Manager...
Apr 08 12:12:23 archlinux systemd[1]: Started GNOME Display Manager.
Apr 08 12:12:23 archlinux gdm[509]: Gdm: GdmLaunchEnvironment: Failed to set owner of /var/lib/gdm: Read-only file system
Apr 08 12:12:23 archlinux gdm[509]: GLib-GObject: g_object_set: assertion 'G_IS_OBJECT (object)' failed

OTHER CONTEXT:

When logged in normally:

~$sudo btrfs subvol list /
ID 256 gen 5850 top level 5 path @
ID 257 gen 5850 top level 5 path @home
ID 258 gen 5850 top level 5 path @log
ID 259 gen 5658 top level 5 path @pkg
ID 260 gen 5767 top level 5 path @.snapshots
ID 261 gen 115 top level 256 path var/lib/portables
ID 262 gen 116 top level 256 path var/lib/machines

When at the TTY of a read-only snapshot that was supposed to be using overlayfs:

~$ sudo mkdir /tmp/upper
~$ sudo mount -t overlay -o lowerdir=/,upperdir=/tmp/upper,workdir=/tmp/upper/overlay_workdir none /
mount: /: special device does not exist.
dmesg(1) may have more information after failed mount system call.

These both produce no output. That's because the directories are empty, because the readme for grub-btrfs says you don't need to put files in there if you installed via pacman.

~$ ls /etc/initcpio/install

~$ ls /etc/initcpio/hooks

The contents of `/usr/lib/initcpio/hooks/grub-btrfs-overlayfs`:

~$ cat /usr/lib/initcpio/hooks/grub-btrfs-overlayfs 
#!/usr/bin/ash

run_latehook() {
	local root_mnt="/new_root"
	local current_dev=$(resolve_device "$root"); # resolve devices for blkid
	if [[ $(blkid "${current_dev}" -s TYPE -o value) = "btrfs" ]] && [[ $(btrfs property get ${root_mnt} ro) != "ro=false" ]]; then # run only on a read only snapshot
		local lower_dir=$(mktemp -d -p /)
		local ram_dir=$(mktemp -d -p /)
		mount --move ${root_mnt} ${lower_dir} # move new_root to lower_dir
		mount -t tmpfs cowspace ${ram_dir}  #meuh!!! space, you can't test !
		mkdir -p ${ram_dir}/upper
		mkdir -p ${ram_dir}/work
		mount -t overlay -o lowerdir=${lower_dir},upperdir=${ram_dir}/upper,workdir=${ram_dir}/work rootfs ${root_mnt}
	fi
}

Last edited by Cheeto (2023-04-11 03:10:17)

BluishHumility

Member

Registered: 2023-03-31

Posts: 26

Re: [SOLVED] btrfs snapshots are read-only despite using grub-btrfs

EDIT: apparently `grub-btrfs` doesn't work with the linux-zen kernel.

I can confirm this is not accurate.

Are you sure you are not trying to restore a snapshot from before the btrfs-overlay hook was built into the initramfs?

It looks like your other issue posted here is now resolved. It is customary to edit the title to mark it [SOLVED], see here:

Please mark solved threads as [SOLVED]
Marking threads as [SOLVED], after a solution has been found will help others to find an expedient solution for their own issues. Please be considerate and mark those threads when an answer is found.
Simply edit the original post by clicking on the 'edit' button at the lower right corner of it and prepend [SOLVED] to the title.

Cheeto

Member

Registered: 2022-09-11

Posts: 64

Re: [SOLVED] btrfs snapshots are read-only despite using grub-btrfs

EDIT: apparently `grub-btrfs` doesn't work with the linux-zen kernel.

I can confirm this is not accurate.

Are you sure you are not trying to restore a snapshot from before the btrfs-overlay hook was built into the initramfs?

It looks like your other issue posted here is now resolved. It is customary to edit the title to mark it [SOLVED], see here:

Please mark solved threads as [SOLVED]
Marking threads as [SOLVED], after a solution has been found will help others to find an expedient solution for their own issues. Please be considerate and mark those threads when an answer is found.
Simply edit the original post by clicking on the 'edit' button at the lower right corner of it and prepend [SOLVED] to the title.

Thanks I updated it.

Yes, per my steps the snapshot is created at the very end. I've tried countless combinations of steps to try to get to the bottom of this. I've tried on bare metal on multiple machines, vm, external SSD, bios, uefi, linux, linux-zen, omitting `snap-pac`, with and without setting the default for `/` from 5 to 256, and other stuff I can't remember.

The reason I posted my steps is because at this point, the only thing I can assume is that I'm doing something EVERY SINGLE OTHER PERSON isn't doing, and whatever that thing is happens to be the one thing that breaks this process. It's also possible I'm just dumb (this is posted in the newbie corner after all) and am overlooking a step or entering a wrong command or doing a step out of order or forgetting a step entirely. That's why I posted the exact process and steps I'm using. And I know you guys are big on this so yes, I reference the arch wiki constantly.

Last edited by Cheeto (2023-04-10 06:06:51)

BluishHumility

Member

Registered: 2023-03-31

Posts: 26

Re: [SOLVED] btrfs snapshots are read-only despite using grub-btrfs

I took a look today and found this issue opened on the grub-btrfs GitHub page, which appears to be yours. In that thread, Antynea has pointed out the HOOKS line is misleading.

HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block filesystems fsck grub-btrfs-overlayfs)

Nothing wrong with that hooks line, but it seems that was a copy/paste from the ArchWiki (instead of a copy/paste from the actual /etc/mkinitcpio.conf).

Deeper in the thread, Antynea was able to glean that the incompatible systemd hook was in use, despite not being disclosed in the HOOKS line.

Be aware that the systemd hook is incompatible with the grub-btrfs-overlayfs hook.

So I switched systemd to udev, and it regenerated stuff and now new snapshots work as expected.

To be honest, I was a bit stumped on this one so I am glad to see it is resolved. When you have a chance, don't forget to mark the title so people are aware you are all set.

Cheeto

Member

Registered: 2022-09-11

Posts: 64

Re: [SOLVED] btrfs snapshots are read-only despite using grub-btrfs

Thanks again for your your help and for trying!

On the topic of that GitHub issue, I just want to clarify something small but significant to me.

The OP had initially showed explicitly, unambiguously, that the hooks section was copy/pasted from the arch wiki, by way of me stating exactly that with a link to the wiki section, then showing the copy pasted code unchanged from the wiki including the wiki's descriptive steps of each line. Importantly, the wiki hooks section states, "for example:" to indicate that the line is visually what you should be looking for, as opposed to suggesting that the line IS anyone's particular line. All of this was in a distinct section, separate from the rest of the "these are the steps I used" section.

At the end, when antynea pointed out that he thought I had initially used udev, it took some investigation on my part to realize why. I don't think he claimed the OP was misleading, but I can see how someone as busy as him glancing through it would have easily overlooked the context.

I also pointed out that the morning of the day he made that realization, I had updated the OP for brevity (because I was linking it elsewhere after days of failing to find a solution, and I was interested in making it as concise as possible for new readers—the distinct section was quite large and could bore busy people, was my mindset). So while anyone looking at it now could be much more easily mislead (though it still says "for example:"), just understand it was not that way through 99% of the conversation.

Anyways, I do appreciate everyone who took time to help with this. I learned that I'll need to be even more explicit when I show steps like that—a lesson I can appreciate as I'm critical of others' guides as well. My feedback to the dev would be to consider stating literally anywhere in the readme that the udev hook is required, as there are now two big issues where the resolution was realizing that fact.

None of this should be taken as a swipe at the dev, who does great work apart from the documentation, and who generously took time out of his day to offer some support.

---

One final note for the newbs who find this thread: in my research, it seems for most standard setups switching `systemd` for `udev` in that hooks line should pose no issues and doesn't have any impact on your systems use of systemd outside of the initial boot phase.

Last edited by Cheeto (2023-04-11 05:35:14)