Mounting the Root File System into RAM

AGT · 2014-05-17 00:40:03

Hi,

I had been wondering, recently, how can one copy the entire root hierarchy, or wanted parts of it, into RAM, mount it at startup, and use it as the root itself. At shutdown, the modified files and directories would be synchronized back to the non-volatile storage. This synchronization could also be performed manually, before shutting down.

I have now succeeded, at least it seems, in performing such a task. There are still some issues.

For anyone interested, I will be describing how I have done it, and I will provide the files that I have worked with.

A custom kernel hook is used to (overall):

Mount the non-volatile root in a mountpoint in the initramfs. I used /root_source
Mount the volatile ramdisk in a mountpoint in the initramfs. I used /root_ram
Copy the non-volatile content into the ramdisk.
Remount by binding each of these two mountpoints in the new root, so that we can have access to both volumes in the new ramdisk root itself once the root is changed, to synchronize back any modified RAM content to the non-volatile storage medium: /rootfs/rootfs_{source,ram}
A mount handler is set (mount_handler) to a custom function, which mounts, by binding, the new ramdisk root into a root that will be switched to by the kernel.

To integrate this hook into a initramfs, a preset is needed.
I added this hook (named "ram") as the last one in mkinitcpio.conf. -- Adding it before some other hooks did not seem to work; and even now, it sometimes does not detect the physical disk.
The kernel needs to be passed some custom arguments; at a minimum, these are required: ram=1
When shutting down, the ramdisk contents is synchronized back with the source root, by the means of a bash script. This script can be run manually to save one's work before/without shutting down. For this (shutdown) event, I made a custom systemd service file.

I chose to use unison to synchronize between the volatile and the non-volatile mediums. When synchronizing, nothing in the directory structure should be modified, because unison will not synchronize those changes in the end; it will complain, and exit with an error, although it will still synchronize the rest. Thus, I recommend that if you synch manually (by running /root/Documents/rootfs/unmount-root-fs.sh, for example), do not execute any other command before synchronization has completed, because ~/.bash_history, for example, would be updated, and unison would not update this file.

Some prerequisites exist (by default):
Packages: unison(, cp), find, cpio, rsync and, of course, any any other packages which you can mount your root file system (type) with. I have included these: mount.{,cifs,fuse,ntfs,ntfs-3g,lowntfs-3g,nfs,nfs4}, so you may need to install ntfs-3g the nfs-related packages (nfs-utils?), or remove the unwanted "mount.+" entires from /etc/initcpio/install/ram.
Referencing paths:
The variables:
source=
temporary=
...should have the same value in all of these files:
"/etc/initcpio/hooks/ram"
"/root/Documents/rootfs/unmount-root-fs.sh"
"/root/.rsync/exclude.txt" -- Should correspond.
This is needed to sync the RAM disk back to the hard disk.
I think that it is required to have the old root and the new root mountpoints directly residing at the root / of the initramfs, from what I have noticed. For example, "/new_root" and "/old_root".

Here are all the accepted and used parameters:
Parameter Allowed Values Default Value Considered Values Description
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
root Default (UUID=+,/dev/disk/by-*/*) None Any string The source root
rootfstype Default of "-t <types>" of "mount" "auto" Any string The FS type of the source root.
rootflags Default of "-o <options>" of "mount" None Any string Options when mounting the source root.
ram Any string None "1" If this hook sould be run.
ramfstype Default of "-t <types>" of "mount" "auto" Any string The FS type of the RAM disk.
ramflags Default of "-o <options>" of "mount" "size=50%" Any string Options when mounting the RAM disk.
ramcleanup Any string None "0" If any left-overs should be cleaned.
ramcleanup_source Any string None "1" If the source root should be unmounted.
ram_transfer_tool cp,find,cpio,rsync,unison unison cp,find,cpio,rsync What tool to use to transfer the root into RAM.
ram_unison_fastcheck true,false,default,yes,no,auto "default" true,false,default,yes,no,auto Argument to unison's "fastcheck" parameter. Relevant if ram_transfer_tool=unison.
ramdisk_cache_use 0,1 None 0 If unison should use any available cache. Relevant if ram_transfer_tool=unison.
ramdisk_cache_update 0,1 None 0 If unison should copy the cache to the RAM disk. Relevant if ram_transfer_tool=unison.

This is the basic setup.
Optionally:
I disabled /tmp as a tmpfs mountpoint: "systemctl mask tmp.mount" which executes "ln -s '/dev/null' '/etc/systemd/system/tmp.mount' ". I have included "/etc/systemd/system/tmp.mount" amongst the files.
I unmount /dev/shm at each startup, using ExecStart from "/etc/systemd/system/ram.service".

Here are the updated (version 3) files, archived: Root_RAM_FS.tar (I did not find a way to attach files -- does Arch forums allow attachments?)
I decided to separate the functionalities "mounting from various sources", and "mounting the root into RAM". Currently, I am working only on mounting the root into RAM. This is why the names of some files changed.

Of course, use what you need from the provided files.

Here are the values for the time spend copying during startup for each transfer tool. The size of the entire root FS was 1.2 GB:
find+cpio: 2:10s (2:12s on slower hardware)
unison: 3:10s - 4:00s
cp: 4 minutes (31 minutes on slower hardware)
rsync: 4:40s (55 minutes on slower hardware)

Beware that the find/cpio option is currently broken; it is available to be selected, but it will not work when being used.

These are the remaining issues:
find+cpio option does not create any destination files.
(On some older hardware) When booting up, the source disk is not always detected.
When booting up, the custom initramfs is not detected, after it has been updated from the RAM disk. I think this represents an issue with synchronizing back to the source root.
Inconveniences:
Unison needs to perform an update detection at each startup.
initramfs' ash does not parse wild characters to use "cp".

That's about what I can think of for now.

I will gladly try to answer any questions.

I don't consider myself a UNIX expert, so I would like to know your suggestions for improvement, especially from who consider themselves so.

Last edited by AGT (2014-05-20 23:21:45)

AGT · 2014-05-17 00:47:47

Does anyone know how can unison be told to skip traversing the roots (unison terminology) to check for any changes? I haven't found any such option in its manual.
Assume that the source root (physical) is completely changed, and that the destination root (ramdisk) is completely dated, to just proceed actually copying.
This can speed up the startup.

AGT · 2014-05-17 18:39:35

Is there a way to make systemd always wait for a service to exit before shutting down? It says that there is "no limit" for the service that is stopping when shutting down, but I want to be sure.
I think that that may be the reason that after updating the initramfs, it is corrupted/not found after some reboots.
The service that I am mentioning is the one provided in the archive, which synchronizes the RAM root back to disk. I think that when shutting down, it does not have time to complete synchronizing.

Last edited by AGT (2014-05-17 18:44:03)

AGT · 2014-05-17 20:37:49

I have found that "-fastcheck true" was simply needed to be set in unison's command line arguments to allow faster initial update detection. But, this did not significantly increase the initial update detection in my case, and it does not skip the initial update detection, so I am trying to use cp. But I am having some issues: it seems that in the initramfs, the * wild character is not interpreted, and it looks for a literal file/folder.
Is there any way to use this wild character in the initramfs or to use "cp" or a similar tool to copy the contents of a folder into another folder, and not to copy the source folder itself into the destination folder?

Last edited by AGT (2014-05-17 22:51:43)

Rexilion · 2014-05-19 13:39:56

I wanted to suggest you used rsync, but the fact that it takes 20x times longer than unison surprises me. Rsync is a better fit for your case (unidirectional back-ups) compared to unison (bi-directional back-ups).

I took at quick look at your tarball:

- maybe try running rsync without the --inplace argument
- position your custom initcpio scripts in /etc/initcpio/hooks and /etc/initcpio/install . They do not belong in /usr/lib/initcpio

AGT · 2014-05-19 14:03:33

Thank you for the suggestions.
I will try not using "inplace" with rsync. Nevertheless, unison can also backup unidirectionally, according to its manual, by using the -force <source root> parameter.
I was not aware that custom hooks have another location.

AGT · 2014-05-19 15:30:55

Removing --inplace did not speed up rsync; it still took 55 minutes.
Unison takes about 3 minutes and 30 seconds to synchronize.

Rexilion · 2014-05-19 16:46:48

Ah, it seems that unison uses an archive maintains a list of files along with their dates and sizes. It then compares this with the in ram contents (in this specific case).

alphaniner · 2014-05-19 16:54:33

Unless I'm really missing something here, I think there's a problem with your code. There is no way it should take 55 mins to handle a few GiB, even if you're comparing each pair of files.

Rexilion · 2014-05-19 16:59:13

alphaniner wrote:

Unless I'm really missing something here, I think there's a problem with your code. There is no way it should take 55 mins to handle a few GiB, even if you're comparing each pair of files.

Yeah, maybe it's an idea to let it run under the invocation of 'time' and set stdout/stderr to /dev/console?

AGT · 2014-05-19 17:36:08

Rexilion wrote:

alphaniner wrote:
Unless I'm really missing something here, I think there's a problem with your code. There is no way it should take 55 mins to handle a few GiB, even if you're comparing each pair of files.
Yeah, maybe it's an idea to let it run under the invocation of 'time' and set stdout/stderr to /dev/console?

Well, my first test, with --inplace, started at 6:27 and finished at 7:22; the second test, without --inplace, started at 10:19 and finished at 11:14.
I checked the time with my phone, though, if you meant that I was checking against the PC time.
The transfer was being displayed on the screen, file-by-file.

Unison takes about 3 minutes and 10 seconds overall. It takes about 5 seconds for boot, usr and var (the current instance of my test installation) each.
The transfer itself takes (maybe) about 15-20 seconds (the other paths besides boot, usr and var are copied almost instantly).
Most of the time that unison spends is at detecting the updates, as mentioned amongst its FAQs.
I was thinking of providing pre-made cache files, but I'm not sure how to do it, as they are not simple text files, and they contain the list of files that need to be transfered/updated, which may change at each startup, so user-created paths in the root / might be bypassed if unison bases its synchronization on invalid cache.

Last edited by AGT (2014-05-19 22:18:02)

alphaniner · 2014-05-19 18:52:23

I didn't mean your timing code, I meant your use of rsync. For example, on my running system, rsync'ing my ~2G /usr (on single 7200RPM HDD) to /tmp (tmpfs) took about 4 minutes. But I may be completely misunderstanding what's going on here.

AGT · 2014-05-19 19:54:25

Your example run of rsync is what is happening. I'm copying from a hard disk drive to a tmpfs. The only difference may be that my drive is 5400 RPM and it is conneted via USB 2(.1?) to a USB 3.0 port.
So, how should rsync be used? What is the command that you have used? This is the command that I use:
rsync --archive --inplace --hard-links --acls --xattrs --devices --specials --one-file-system --8-bit-output --human-readable --progress --exclude-from="/root/.rsync/exclude.txt" "${source}/" "${temporary}"
Where $source is "/root_source" and $temporary is "/root_ram".
/root_source is the HDD and /root_ram is the tmpfs (or ramfs).

Rexilion · 2014-05-19 20:10:38

Could you time how long it takes for the cp and cpio/find options?

I think that unison is fast because of the fact that it archives.

alphaniner · 2014-05-19 20:21:44

Initially I ran rsync -a /usr /tmp . The time is pretty much the same if I run your command (minus the --exclude) though.

AGT · 2014-05-19 21:30:38

"cp" took 31 minutes: from 4:23 to 4:54. Here is how fast cp was copying files: root_ram_cp.mp4. Is it slow compared to "standard"?
The "find/cpio" option does not work: it seems to successfully copy everything (in 2 minutes and 12 secods), but when trying to boot into the system, there are no files present: "Root device mounted successfully, but /sbin/init does not exist." "ls" does not display anything either. The same command works correctly in user space: am I missing some files (executables, libraries) in the cpio image?
Also, these last 2 attempts took 3 reboots each to detect the HDD. This should be corrected.
I think that the bottleneck could be my hardware, but I don't have any other machine to test on.
Unison's FAQ mentions that the number of files is more significant for performance than the their size, which I would say is my case. "-fastcheck true" performs a pseudo-verification for each file, instead of a complete, more robust, one.
I doubt that unison is faster because it creates archives, because, as mentioned previously, it reacreates those archives each time, because it loses them, as the FS is in RAM; my reasoning is reinforced by the fact that the unison command is executed in the initramfs, and there is no $HOME set. At least, that is what I think. I'll delete any remaining /root/.unison/{ar,fp}* files from the disk, just to be sure.

Last edited by AGT (2014-05-20 12:34:24)

AGT · 2014-05-20 01:09:38

Unison remains at the same speed after deleting its cache on the hard disk, and no cache files have been recreated since, after rebooting and using unison as the transfer tool.

Rexilion · 2014-05-20 04:22:58

AGT wrote:

Unison remains at the same speed after deleting its cache on the hard disk, and no cache files have been recreated since, after rebooting and using unison as the transfer tool.

That is quite amazing. I'm certainly going to have a look at that piece of software.

As for your observations, I'm out of bright idea's.

As for the HDD detection issue, maybe try to poll or mount by UUID? If you use a pure systemd initrd, you could use a hook that starts after udev.

AGT · 2014-05-20 12:33:05

I am using the disk's UUID (/dev/disk/by-uuid) to mount it. I always have.
My custom hook runs last: HOOKS="base udev modconf net block mdadm_udev lvm2 resume filesystems pcmcia usr fsck btrfs shutdown keyboard ram". That's because I had problems when it wasn't.
I will try booting in a virtual machine.

Last edited by AGT (2014-05-20 12:36:13)

Rexilion · 2014-05-20 12:35:57

Looks like a race condition. You can use the poll_device function from init_functions to wait for a device. I'm not sure if that works for UUID though. It's located at /usr/lib/initcpio/init_functions.

AGT · 2014-05-20 12:53:18

How about synchronizing back to disk? That's a rather important issue, I would say.

If I mkinitcpio in the RAM disk and if I let systemd service sync the image at shutdown automatically, there are issues after rebooting:
The cpio image might not be detected at all, even thought it is there.
If the cpio image is detected, the root= paramater from the kernel is not detected in the hook.
If the root= parameter is detected in the hook, it does not find the disk.
If I mkinitcpio in the RAM but if I run the the sync script manually, there are no problems. This is why I think that the problem could be related to systemd.
The systemd service, during shuutting down, says "no limit" when waiting for the service to finish, though. I added TimeoutStopSec=0 to disable timing out, just in case, but that did not resolve it.

Last edited by AGT (2014-05-20 12:56:34)

AGT · 2014-05-20 14:47:03

I have booted into a VM hosted by a faster machine, and the times have sped up, overall:
find, cpio: 2:10s
unison: 4 minutes
cp: 4 minutes
rsync: 4 minutes, 40 s
cp and rsync scrolled faster.

It seems that systemd is not the problem for the undetected initramfs image:
Even after I used cp to copy over the image from the RAM to the disk (after making it in the RAM disk), it was still not detected at the next reboot. Is there anything else that mkinitcip does than just generating the image file? I think this may mean that syncing back is somehow an issue; and this could break other thinks.

Also the disk has not been undetected at any boot up in the VM, so far. It seems that my hardware is the problem.

Last edited by AGT (2014-05-20 15:07:52)

Rexilion · 2014-05-20 15:31:21

I have just started to deploy unison here. It's rediculously fast becaue it's using an archive in ~/.unison . The VM probably has better caching.

What do you exactly imply by 'undetected initramfs image'? Is grub unable to find it? Does the kernel panic?

The fact that your disk is properly detected in the VM but not on real hardware still does not exclude the possibility of a race condition.

EDIT: I just took another look at your configuration files. The only reason why unison is so fast is because at boot, you pre-create it's archive. That is why it has an advantage over other methods.

Last edited by Rexilion (2014-05-20 15:48:23)

AGT · 2014-05-20 16:36:04

How did you use/test unison? In my case, unison, of course, is used in the cpio image, where there are no cache files, because unison has not been run yet in the initcpio image, before it had a chance to be used during boot time, to generate them; and during start up is when it is used; when it creates the archives. ...a circular dependency. Yet, files changed by the user would still need to be traversed to detect changes. So, I think that even providing pre-made cache files would not guarantee that they would be valid at start up, for all configurations of installation. -- I think, though, that these cache files could be copied/saved from the initcpio image to the root (disk and RAM), after they have been created, and used next time by copying them in the initcpio image during each start up. I think $HOME would need to be set.
Unison was not using any cache previously anyway. I was aware of that, but I wanted to prove it by deleting any cache files remaining.
Unison, actually, was slower (4 minutes) the first time it ran in the VM, compared to the physical hardware (3:10s). I have not measured the time for its subsequent runs, but It seemed that it was faster after the first run. The VM was hosted on a newer machine than what I have used so far: the VM host has an i3-3227U at 1.9 GHz CPU with 2 cores/4 threads and 8 GB of RAM (4 GB ware dedicated to the VM); my hardware has a Pentium B940 at 2 GHz CPU with 2 cores/2 threads and 4 GB of RAM.
I could see that, in the VM, rsync and cp were copying faster than on my hardware; they were scrolling quicker.

Grub, initially complains that there is no image, and shows a "Press any key to continue" message; if you continue, the kernel panics.

I'll try using "poll_device()". What arguments does it need? More than just the device; also the number of seconds to wait?

Last edited by AGT (2014-05-20 16:49:35)

AGT · 2014-05-20 16:37:22

Rexilion wrote:

EDIT: I just took another look at your configuration files. The only reason why unison is so fast is because at boot, you pre-create it's archive. That is why it has an advantage over other methods.

Where in my configuration are you refering to? As I mentioned, I don't think that unison uses any cache, in this case.

Last edited by AGT (2014-05-20 16:40:38)

Arch Linux

#1 2014-05-17 00:40:03

Mounting the Root File System into RAM

#2 2014-05-17 00:47:47

Re: Mounting the Root File System into RAM

#3 2014-05-17 18:39:35

Re: Mounting the Root File System into RAM

#4 2014-05-17 20:37:49

Re: Mounting the Root File System into RAM

#5 2014-05-19 13:39:56

Re: Mounting the Root File System into RAM

#6 2014-05-19 14:03:33

Re: Mounting the Root File System into RAM

#7 2014-05-19 15:30:55

Re: Mounting the Root File System into RAM

#8 2014-05-19 16:46:48

Re: Mounting the Root File System into RAM

#9 2014-05-19 16:54:33

Re: Mounting the Root File System into RAM

#10 2014-05-19 16:59:13

Re: Mounting the Root File System into RAM

#11 2014-05-19 17:36:08

Re: Mounting the Root File System into RAM

#12 2014-05-19 18:52:23

Re: Mounting the Root File System into RAM

#13 2014-05-19 19:54:25

Re: Mounting the Root File System into RAM

#14 2014-05-19 20:10:38

Re: Mounting the Root File System into RAM

#15 2014-05-19 20:21:44

Re: Mounting the Root File System into RAM

#16 2014-05-19 21:30:38

Re: Mounting the Root File System into RAM

#17 2014-05-20 01:09:38

Re: Mounting the Root File System into RAM

#18 2014-05-20 04:22:58

Re: Mounting the Root File System into RAM

#19 2014-05-20 12:33:05

Re: Mounting the Root File System into RAM

#20 2014-05-20 12:35:57

Re: Mounting the Root File System into RAM

#21 2014-05-20 12:53:18

Re: Mounting the Root File System into RAM

#22 2014-05-20 14:47:03

Re: Mounting the Root File System into RAM

#23 2014-05-20 15:31:21

Re: Mounting the Root File System into RAM

#24 2014-05-20 16:36:04

Re: Mounting the Root File System into RAM

#25 2014-05-20 16:37:22

Re: Mounting the Root File System into RAM

Board footer