Does this exist: RAM filesystem with manual snapshot to persistent?

fdmillion · 2017-09-19 02:51:11

Apologies if this is the wrong topic, please feel free to move as appropriate if necessary.

This is more of an inquiry to find out either if something I'm interested in exists, or if I can emulate it using other tools.

The scenario is that I have a few different systems where I am booting Arch from flash media, most notably ArchARM on a Raspberry Pi, but also some x86 machines booting from USB drives.

We all know about the issues of limited write endurance on flash media. The media used in consumer grade and commodity flash drives is pretty low-grade, and has pretty bad write endurance in my experience. It's possible to "wear out" a drive in only a matter of months if you're running an OS off of it that does regular logging and such. Some of my use cases do involve a significant amount of regular logging, which means constant writes to the USB device or SD card and, consequently, short lifespan.

So my idea is this. I'd like to have a RAM drive, say mounted on /var. But, periodically, I'd like to be able to snapshot the current contents of that RAM filesystem off to disk. Then upon boot, the last snapshot stored on persistent storage is restored. This would give me the ability to control exactly when and how often disk writes occur. I could make the decision whether I want to incur more writes or lose more data on a power loss. Better case is I would make the root filesystem a union FS and then put the upper layer on such a filesystem.

Why not just mount a tmpfs and tar it occasionally? 1. because open files can cause issues, as in if a file is locked exclusive such that tar can't read it, or if it's open for writing and hasn't been flushed so an incomplete or even corrupt file results. And 2. because that would preclude the idea of a union mounted filesystem as I stated, because you'd end up tarring the entire system, not just the RAM portion.

In my "perfect" world, there'd be a tmpfs with a persistent file option, and a userspace command to flush. Like "mount -t ptmpfs ptmpfs /var -o size=512M,persistent_filename=/root/var.ptmpfs" and "ptmpfs --flush /var" or something. Since I don't think that exists as is, is there an alternative method?

circleface · 2017-09-19 03:23:01

I'm not sure about most of your post, since I'm not too advanced myself, but the 'sync' command will flush any buffers to disk, so that might help with open files and such.

Slithery · 2017-09-19 08:58:40

Have you seen Graysky's anything-sync-daemon...

frostschutz · 2017-09-19 09:07:02

You could probably do something with snapshots/overlays backed by a zram / ramdisk / tmpfs loop device. Snapshots can be synced to disk, or not. Although LVM does snapshots, I'm not sure how well it will behave if you add temporary PV...

The alternative would be using the device mapper directly. One such example (used in data recovery) is here: https://raid.wiki.kernel.org/index.php/ … erlay_file

Actually making it sync is slightly different from this. Also you'd have to set this up in your initramfs so it will mount the device mapper and not the backing device.

Device Mapper snapshot and snapshot merge is described here: https://www.kernel.org/doc/Documentatio … apshot.txt

----

Of course, this is a block oriented approach. It would even write files that have been deleted in the meantime, unless these layers also support TRIM / discard (loop devices do, if the backing filesystem supports sparse files - not sure about the rest).

Another option would be to simply rsync the files but it would transfer whole files even if only a portion of files actually changed.

Last edited by frostschutz (2017-09-19 09:11:46)

ugjka · 2017-09-19 10:31:29

https://github.com/bluerider/liveroot

Arch Linux

#1 2017-09-19 02:51:11

Does this exist: RAM filesystem with manual snapshot to persistent?

#2 2017-09-19 03:23:01

Re: Does this exist: RAM filesystem with manual snapshot to persistent?

#3 2017-09-19 08:58:40

Re: Does this exist: RAM filesystem with manual snapshot to persistent?

#4 2017-09-19 09:07:02

Re: Does this exist: RAM filesystem with manual snapshot to persistent?

#5 2017-09-19 10:31:29

Re: Does this exist: RAM filesystem with manual snapshot to persistent?

Board footer