Using an overlay to create an indestructible system

bedtime · 2023-10-07 21:04:38

Yesterday, I messed up my system 3 times whilst installing an OS onto USB... I want something that will not be persistent after reboot. I'd like to have an option at boot to boot into a completely overlay'd system. I've got something working so far, but it's quite hackish. I'd like to figure out a better way to do it.

Here's what I have:

I installed this package in aur:
https://github.com/felixsinger/mkinitcp … ree/master

Installing the program automatically puts an 'overlayfs' hook in your mkinitcpio.conf and updates every single kernel with that hook, so I just re-added back the hooks for the default kernel and left the 'fallback' kernel to be the overlay'd one:

echo 'ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux"
ALL_microcode=(/boot/*-ucode.img)

PRESETS=("default")

default_image="/boot/initramfs-linux.img"
fallback_options="-S autodetect"' > /etc/mkinitcpio.d/linux.preset

# reset hooks
sed -i 's/HOOKS=.*/HOOKS=\(systemd autodetect modconf keyboard sd-vconsole block filesystems resume\)/' /etc/mkinitcpio.conf

mkinitcpio -p linux

This leaves the fallback kernel to be the one still running the overlayfs hook. This worked just fine (i.e., changes to the root partition will not be persistent.) The issue now is that the /efi partition is still writable. So I booted into the default (persistent) kernel and edited /etc/fstab to give the /efi partition the 'ro' parameter. Then I rebooted back into default (persistent) kernel again. I took a btrfs snapshot and named it 'readonly' and had it added to grub with grub-btrfs (which allows the user to run snapshots from grub). Then when I run the 'readonly' snapshot under the fallback kernel, I get an overlay'd root partition + readonly /efi; so, basically, a nearly indestructible system.

The issue is that this seems like a huge run-around to get setup. And as time goes on, that 'readonly' snapshot will become older and potentially corrupted?

If anyone has ideas on how to make this less hackish, that would be great.

seth · 2023-10-08 06:44:42

Yesterday, I messed up my system 3 times whilst installing an OS onto USB...

Then when I run the 'readonly' snapshot under the fallback kernel, I get an overlay'd root partition + readonly /efi; so, basically, a nearly indestructible system.

https://youtu.be/7FfU-k9L_KQ?t=67
I see your "indestructible" and raise my "dd"

You're essentially looking for some live distro like knoppix (or a custom archiso) that you put onto a non-writable medium (nb. that the latch on an SD card is merely a friendly request to not write it, the device driver can perfectly ignore that; there're usb keys w/ actual write protection and of course coasters) and yes: a system that you cannot update by design will inevitably get outdated.

Is the end-goal here to have a non-compromisable, transient system or to prevent yourself from accidentally shredding your OS - and private data! (which regardless poses a different problem because you *have* to write that) - with dd?

bedtime · 2023-10-08 17:35:27

seth wrote:

Yesterday, I messed up my system 3 times whilst installing an OS onto USB...
Then when I run the 'readonly' snapshot under the fallback kernel, I get an overlay'd root partition + readonly /efi; so, basically, a nearly indestructible system.
https://youtu.be/7FfU-k9L_KQ?t=67
I see your "indestructible" and raise my "dd"

Lol, that's pretty much how I feel.

You're essentially looking for some live distro like knoppix (or a custom archiso) that you put onto a non-writable medium (nb. that the latch on an SD card is merely a friendly request to not write it, the device driver can perfectly ignore that; there're usb keys w/ actual write protection and of course coasters) ...

Yep, and I found that out yesterday when yet again I messed my entire system up by purposely deleting system files in overlay mode and rebooting.

Is the end-goal here to have a non-compromisable, transient system or to prevent yourself from accidentally shredding your OS - and private data! (which regardless poses a different problem because you *have* to write that) - with dd?

I've made a script that installs arch to a USB or computer. This arch OS can be easily made into a desktop system or used as a rescue disk. So there are two things I'd like:

1. To have a mode that is mutable and allows changes as if it was a regular user system.

and

2. To have a mode that is *indestructible*. This mode would allow me to mess around with my computer in any way I want without causing damage/changes.

The thing that switches the mode would be grub, I hope. I could just tell grub to enter regular or immutable mode.

One of the ways I've thought of doing this was to mount / as ro in the kernal params but then have a few necessary directories in tmpfs mode, so the system could write and be happy: /var/{log,cache,tmp} and some home dirs ~/{.local,.config,.cache}. I don't know how I could switch this on/off at grub though. Maybe something that can unmount the tmpfs files when the computer boots into rw mode? But that is quite hackish.

seth · 2023-10-08 18:58:08

If you're at all exposing device nodes to physically writable devices, "indescrutible" is off the table. Period.

As long as you're booting the system in overlay mode, you're not persistenly updating it - that's the point.
But if you boot it "flat" and update it, the next overlay boot will pick up that. So the above doesn't matter.

https://wiki.archlinux.org/title/profile-sync-daemon

https://www.freedesktop.org/software/sy … .volatile=
https://www.freedesktop.org/software/sy … tml#fstab=
https://www.freedesktop.org/software/sy … mount.usr=

bedtime · 2023-10-09 19:08:35

seth wrote:

https://wiki.archlinux.org/title/profile-sync-daemon
https://www.freedesktop.org/software/sy … .volatile=
https://www.freedesktop.org/software/sy … tml#fstab=
https://www.freedesktop.org/software/sy … mount.usr=

Thanks for this info. I tried to get the systemd.volatile kern param working. When I do:

... ro systemd.volatile=yes ...

I just get a black screen with no tty. I have no access to logs as the system is ro.

...

So far, I managed to add readonly 'ro' kernel param options. Later in the boot, a systemd daemon picks up that the system is running in readonly mode, so it mounts ~.{cache,config,local,mozilla} as tmpfs and extracts a working copy of those directories and files back in place, as if they were always there. You end up logging into a functional window manager and firefox that was just how you left it.

Here is the code:

Make required directories writable so the system won't complain:

/etc/fstab:

...
tmpfs    /var/cache  tmpfs   rw,nodev,nosuid,mode=1755,size=2G   0 0
tmpfs    /var/log    tmpfs   rw,nodev,nosuid,mode=1775,size=2G   0 0
tmpfs    /var/run    tmpfs   rw,nodev,nosuid,mode=1777,size=2G   0 0

Edit grub config and add readonly option with 'ro' kernel params, then regenerate:

# cp /etc/grub.d/10_linux /etc/grub.d/10_linux-readonly
# sed -i 's/\"\$title\"/\"\$title \(readonly\)\"/g; s/ rw / ro /g' /etc/grub.d/10_linux-readonly
# grub-mkconfig -o /boot/grub/grub.cfg

Make a file that will detect if the root device is readonly and mount required dirs as tmpfs. Then replace the user's config files so the user will be setup properly:

/usr/local/bin/mount-user-tmpfs.sh:

#!/bin/bash

user=user

if [[ $(mount | grep " on / " | grep "ro") ]] || [[ "$1" = "-a" ]]; then

   tmp_dir=/home/$user/.cache

   mount -o uid=$user -t tmpfs tmpfs $tmp_dir

   cd /home/$user
   sudo -u $user cp -r -p .{local,config,mozilla} $tmp_dir/

   mount -o uid=$user -t tmpfs tmpfs /home/$user/.config
   mount -o uid=$user -t tmpfs tmpfs /home/$user/.local
   mount -o uid=$user -t tmpfs tmpfs /home/$user/.mozilla

   sudo -u $user cp -r $tmp_dir/.{config,local,mozilla} /home/$user/

fi

Make a service to run the above file before we log in.

/etc/systemd/system/readonly.service:

[Unit]
Description=Run script with systemd right before login prompt
After=rc-local.service
Before=getty.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/mount-user-tmpfs.sh

[Install]
WantedBy=multi-user.target

Enable the service:

# systemctl enable readonly.service

Stop systemd from remounting as 'rw':

# systemctl mask systemd-remount-fs.service

When you boot into grub, you'll be greeted with choices that are something like:

linux
linux-fallback
linux (readonly)
linux-fallback (readonly)
...

Could you, seth, or anyone else tell me what they think of this solution?

Last edited by bedtime (2023-10-09 20:50:06)

seth · 2023-10-09 19:36:48

I just get a black screen with no tty. I have no access to logs as the system is ro.

No, because

manpage wrote:

/etc/ and /var/ (as well as all other resources shipped in the root file system) are reset at boot and lost on shutdown

Why did you not go for "systemd.volatile=overlay"?

bedtime · 2023-10-09 20:24:43

seth wrote:

I just get a black screen with no tty. I have no access to logs as the system is ro.
No, because
manpage wrote:
/etc/ and /var/ (as well as all other resources shipped in the root file system) are reset at boot and lost on shutdown
Why did you not go for "systemd.volatile=overlay"?

I tried that too, and the system just goes black as well. Can't go to any other tty or alt-ctl-del.

Here is my /etc/fstab if that helps:

UUID=1bbccece-789a-4a05-b114-abff3ba8b7c0       /               btrfs           rw,noatime,compress-force=zstd:1,ssd,discard=async,space_cache=v2,subvol=/@       0 0

# /dev/sda1 LABEL=EFI
UUID=7C78-8FB5          /efi            vfat ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro     0 2

Note that / is mounted as 'rw'. Not sure if that affects it.

Arch Linux

#1 2023-10-07 21:04:38

Using an overlay to create an indestructible system

#2 2023-10-08 06:44:42

Re: Using an overlay to create an indestructible system

#3 2023-10-08 17:35:27

Re: Using an overlay to create an indestructible system

#4 2023-10-08 18:58:08

Re: Using an overlay to create an indestructible system

#5 2023-10-09 19:08:35

Re: Using an overlay to create an indestructible system

#6 2023-10-09 19:36:48

Re: Using an overlay to create an indestructible system

#7 2023-10-09 20:24:43

Re: Using an overlay to create an indestructible system

Board footer