You are not logged in.
ArchTaz Live Boot: A method to boot and run an Arch system entirely in RAM using tmpfs.
As the name suggests it was inspired by (copied entirely from) SliTaz http://www.slitaz.org/ and boots in exacly the same way.
It needs a custom init script in the root directory and for some reason that I can't work out a hook for sysinit_premount in rc.sysinit to remount / rw.
The /usr/bin/archfs script is just a chunk of Slitaz's Tazusb utility with some minor changes (and I haven't properly tested the lzma or no compression).
The /init script is modified from the usual initramfs /usr/lib/initcpio/init with hints from the SliTaz /init script (and some personal additions for my system).
Copy init to the root directory, run archfs (eg. # archfs writefs gzip) and move the resulting archfs.gz to your boot partition/directory and boot with archfs.gz as your initrd.
I run a fairly light system (Openbox, Firefox, Gnome-Mplayer, SpaceFM, no dev packages) and gzip compression gives me an archfs.gz just over 230MB that boots in just over 30 seconds using 138MB RAM on startup, and most applications start instantly (2GHz Core2Duo, 4G RAM). Lzma compression generally gives a smaller archfs.gz but takes longer to boot. I have a persistent home partition, I install packages as I need them and then they're gone on next reboot if I don't do a new archfs.gz to keep them. I delete man pages, unneeded locales, etc. Obviously if you're running a hefty install on a limited box you may run into problems.
Any tips, cleanups, corrections welcome. It's been a mostly copy and paste and mash stuff together process so it isn't neat. And I think I may need some serious help when systemd becomes default.
/usr/bin/archfs:
#!/bin/sh
#
# Archfs - entirely copied from part of the Tazusb utility supplied with
# SLiTaz GNU Linux - with some cosmetic changes.
#
#
### Tazusb - SliTaz LiveUSB
###
### Tazusb is an utility to generate, configure and manipulate SliTaz LiveUSB
### bootable media and/or USB /home partition, such as flash keys, SD card or
### USB harddisk.
###
### Authors : Christophe Lincoln (Pankso) <pankso@slitaz.org>
### Andrew Miller (Spode) <spode@spodesabode.com>
#
COMMAND=$1
case $COMMAND in
writefs)
# Writefs to rootfs.gz
if [ -z $2 ]; then
COMPRESSION=none
else
COMPRESSION=$2
fi
# Start info
echo ""
echo -e "\033[1mWrite filesystem\033[0m
===============================================================================
The command writefs will write all the current filesystem into a suitable cpio
archive (rootfs.gz) usable on a bootable LiveUSB media.
Archive compression: $COMPRESSION"
echo ""
# Create list of files
find /bin /etc /init /sbin /var /lib /lib64 /root /usr >/tmp/list
for dir in /boot /dev /home /proc /sys /tmp /mnt /media /run
do
echo $dir >>/tmp/list
done
# Generate initramfs with specified compression
if [ "$COMPRESSION" = "lzma" ]; then
echo -n "Creating archfs.gz with lzma compression... "
cat /tmp/list | cpio -o -H newc | lzma > /archfs.gz
elif [ "$COMPRESSION" = "gzip" ]; then
echo -n "Creating archfs.gz with gzip compression... "
cat /tmp/list | cpio -o -H newc | gzip -9 > /archfs.gz
else
echo -n "Creating archfs.gz without compression... "
cat /tmp/list | cpio -o -H newc > /archfs.gz
fi
# Get initramfs size
size=`du -sh /archfs.gz | cut -f 1`
echo "==============================================================================="
echo "Root filesystem size: $size"
echo ""
echo -en "----\nENTER to continue..."; read i
;;
esac
exit 0
/init:
#!/bin/bash
PATH=/bin:/usr/bin:/usr/sbin:/sbin
udevd_running=0
if [ -x /usr/bin/systemd-timestamp ]; then
RD_TIMESTAMP=$(systemd-timestamp)
fi
. /usr/lib/initcpio/init_functions
mount -t proc proc /proc -o nosuid,noexec,nodev
mount -t sysfs sys /sys -o nosuid,noexec,nodev
mount -t devtmpfs dev /dev -o mode=0755,nosuid
mount -t tmpfs run /run -o nosuid,nodev,mode=0755
mkdir -m755 /run/initramfs
# parse the kernel command line
parse_cmdline
for d in ${disablehooks//,/ }; do
[ -e "/hooks/$d" ] && chmod 644 "/hooks/$d"
done
#. /config
run_hookfunctions 'run_earlyhook' 'early hook' $EARLYHOOKS
[ -n "${earlymodules//[[:space:]]}" ] && modprobe -qab ${earlymodules//,/ }
[ -n "${MODULES//[[:space:]]}" ] && modprobe -qab $MODULES
# If rootdelay is empty or not a non-negative integer, set it to 10
if [ -z "${rootdelay}" ] || ! [ "${rootdelay}" -ge 0 ]; then
rootdelay=10
fi
run_hookfunctions 'run_hook' 'hook' $HOOKS
# honor the old behavior of break=y as a synonym for break=premount
if [ "${break}" = "y" ] || [ "${break}" = "premount" ]; then
echo ":: Pre-mount break requested, type 'exit' to resume operation"
launch_interactive_shell
fi
rootdev=$(resolve_device "$root") && root=$rootdev
unset rootdev
#fsck_root
######################################################
# Make /new_root
mkdir /new_root
# Mount root at /new_root
#${mount_handler:-default_mount_handler} /new_root
echo -e "Switching / to tmpfs..."
mount -t tmpfs tmpfs /new_root
run_hookfunctions 'run_latehook' 'late hook' $LATEHOOKS
run_hookfunctions 'run_cleanuphook' 'cleanup hook' $CLEANUPHOOKS
# Stop udevd if is running
if [ "${udevd_running}" -eq 1 ]; then
udevadm control --exit
udevadm info --cleanup-db
fi
# Copy root
echo -e "Copying root..."
for i in $(ls -a /); do
case "$i" in
.|..) ;;
mnt) mkdir /new_root/mnt;;
proc) mkdir /new_root/proc;;
sys) mkdir /new_root/sys;;
dev) mkdir /new_root/dev;;
run) mkdir /new_root/run;;
new_root) ;;
*) cp -a /$i /new_root
esac
done
# Create mountpoints
mkdir /new_root/mnt/sda1
mkdir /new_root/mnt/sda2
mkdir /new_root/mnt/sda3
mkdir /new_root/mnt/sda4
init=${init:-/sbin/init}
echo -e "Switching root..."
exec env -i \
"TERM=$TERM" \
"RD_TIMESTAMP=$RD_TIMESTAMP" \
/sbin/switch_root /new_root $init "$@"
######################################################
## Mount root at /new_root
#${mount_handler:-default_mount_handler} /new_root
#run_hookfunctions 'run_latehook' 'late hook' $LATEHOOKS
#run_hookfunctions 'run_cleanuphook' 'cleanup hook' $CLEANUPHOOKS
init=${init:-/sbin/init}
if [ "$(stat -c %D /)" = "$(stat -c %D /new_root)" ]; then
# Nothing got mounted on /new_root. This is the end, we don't know what to do anymore
# We fall back into a shell, but the shell has now PID 1
# This way, manual recovery is still possible.
err "Failed to mount the real root device."
echo "Bailing out, you are on your own. Good luck."
echo
launch_interactive_shell --exec
elif [ ! -x "/new_root${init}" ]; then
# Successfully mounted /new_root, but ${init} is missing
# The same logic as above applies
err "Root device mounted successfully, but ${init} does not exist."
echo "Bailing out, you are on your own. Good luck."
echo
launch_interactive_shell --exec
fi
if [ "${break}" = "postmount" ]; then
echo ":: Post-mount break requested, type 'exit' to resume operation"
launch_interactive_shell
fi
exec env -i \
"TERM=$TERM" \
"RD_TIMESTAMP=$RD_TIMESTAMP" \
/sbin/switch_root /new_root $init "$@"
# vim: set ft=sh ts=4 sw=4 et:
/etc/rc.d/functions.d/root_rw:
root_rw() {
mount -o remount,rw /
}
add_hook sysinit_premount root_rw
archfs:
http://pastebin.com/RNTDWhFc
init:
http://pastebin.com/n4vcqG62
root_rw:
http://pastebin.com/i1LV61SV
Last edited by 0ddba11 (2012-12-30 10:12:35)
Offline
Forgive me for being a complete noob, but how much RAM would be required to not run into big problems doing this? My Arch install is relatively lightweight but maybe a bit full of apps I install and never use
And would it be possible to edit your scripts slightly to only include certain applications/directories, to cut the RAM requirement down? I was thinking of leaving out my /home because that is jam-packed full of music
Again, completely oblivious to the clever side of this, all I know is RAM=fast, hence my possibly idiot qustions
Last edited by Reded (2012-10-15 21:53:16)
"Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying "End-of-the-World Switch. PLEASE DO NOT TOUCH", the paint wouldn't even have time to dry."
Offline
Not sure about what the archfs.gz size to RAM needed ratio would be. Slitaz requires 192MB of RAM to run but is very lightweight (from the ground up so not even all kernel modules are included by default and the ones that are are gzipped). Experiment and see what you can get away with. That's one of the reasons I like running like this - create an archfs.gz of your install then take away/add what you like if it all goes horribly wrong just go back to the previous archfs.gz (or normal install) that worked. I started with a base install and worked up which keeps bloat to a minimum.
All the clever stuff in the scripts is from the Slitaz team I just adapted bits through trial and error. The line that starts "for dir in..." under the "Create list of files " section is basically a list of the directories that are not included in the archfs.gz (but the actual empty directories are needed in the filesystem). So that inlcudes /home and /mnt. I mount my home directory in fstab. I symlink /var/lib/pacman/sync and /var/cache/pacman to directories in my home directory so that they don't take up space in the archfs.gz but I can still use the cache and be up to date with pacman no matter how old the archfs.gz. You could potentially mount or symlink any directories you don't want in the archfs.gz (creating an archfs.gz preserves symlinks but doesn't follow them).
The "find.." line is the line of directories whose full tree and files are archived in the archfs.gz (so you wouldn't need /lib64 in there if you don't have an x86_64 system). The archfs script also leaves out directories I don't have any need for (/srv and /opt and possibly others) so they don't appear in the / directory at all. It's obviously adviseable to keep system directories all inside the archfs.gz though.
According to df -h my install takes up 600MB altogether when uncompressed into RAM. Mounting tmpfs automatically uses half the available RAM unless you tell it otherwise. But this doesn't directly impact the amount of RAM available to the OS as far as I know. The only time I've run into problems is upgrading everything and running out of space in the root directory (default half your RAM). So if you've only got 1G RAM then you'll probably run into problems if your install is already 900MB for example.
I'll shut up now, but I hope I helped a bit. The Slitaz website has got some pretty good doumentation about how to start from scratch.
Offline
I'm having a problem getting /usr/bin/archfs to run.
I ran chmod +x on it, but when I run it it does nothing.
Any Ideas?
Wait, how am I supposed to run the script? What arguments do I give it?
I got it!
archfs writefs gzip
Last edited by xens1x (2012-12-30 03:24:46)
Offline
Good point, I've edited the original post. Thanks
Offline
It might be an idea to set up a git for this project. Am interested in what the advantages are over using archiso? Please feel free to mail me if you need any help.
Mr Green
Offline
Advantages (for me at least) are that you can execute one command to create the initramfs from your running system. So all the set up can be done and tested as working before you create it (but should work from chroot into a another install). I run it after I've installed/changed anything I want to keep or updated everything. It boots everything into RAM which (probably) makes everything faster. You can have several different iniramfs's for different tasks or I've also done the same but used scripts that install/remove packages on startup based on runlevel set in Grub. Which only takes an extra couple of seconds (if the packages are in your package cache). I also have script outside the initramfs (tied to a hook in init scripts) that works as an rc.local that you alter without running archfs every time. I need to work out how to do the same with systemd when I switch but it seems like it would be just a case of creating a few custom services.
Disadvantages: lots I would think. The whole iniramfs is loaded into memory at boot so adds alot to the boot time. And as I said before if you're running anything heavy or you have limited resources you'll run into trouble I think. I do alot of farting about deleting locale files and removing packages I don't need (bloody perl and bloody python!) to keep the weight down. Scripts and custom package builds would probably sort most of this but that's extra work.
I'm considering changing the boot process to unpack the rootfs archive into tmpfs using hooks in the normal initramfs which might speed the boot up but might only happen after I switch to systemd.
A git repo seems like extra work on top of that (when I always should be doing other things anyway) but I'll investigate.
Offline
It would not be too difficult to replace rc.local as it just basically a script. A simple service file could run it job done. man pages take up a lot of room
Mr Green
Offline
Oh yes any man directory was symlinked to /tmp long ago, so they don't even get installed.
Right, new boot process:
Using hooks in normal initramfs-linux.img to unpack archfs.gz into tmpfs and boot it.
Create an initcpio preset file /etc/mkinitcpio.d/archfs.preset:
# mkinitcpio preset file for the 'archfs' package
ALL_config="/etc/mkinitcpio.conf"
ALL_kver="/boot/vmlinuz-linux"
PRESETS=('archfs')
archfs_config="/etc/mkinitcpio.conf"
archfs_image="/boot/archfs-linux.lzo"
archfs_options="-S fsck -A tmpfs -z lzop"
Or append to the standard 'linux.preset' file so the initramfs is created automatically when you upgrade your kernel. Or alter /etc/mkinitcpio.conf to add options.
Create the file /lib/initcpio/hooks/tmpfs:
#!/usr/bin/ash
run_latehook() {
# Mount tmpfs root
mount -t tmpfs tmpfs /mnt
# Unpack rootfs
msg "Unpacking rootfs..."
cd /mnt
# parse the kernel command line
parse_cmdline
for archfs in ${archfs//,/ }; do
if [ "$archfs" = "xz" ]; then
xz -d < /new_root/boot/archfs.xz | cpio -imd --no-absolute-filenames > /dev/null 2>&1
elif [ "$archfs" = "gzip" ]; then
gzip -dc /new_root/boot/archfs.gz | cpio -imd --no-absolute-filenames > /dev/null 2>&1
elif [ "$archfs" = "lzo" ]; then
lzop -d < /new_root/boot/archfs.lzo | cpio -imd --no-absolute-filenames > /dev/null 2>&1
elif [ "$archfs" = "img" ]; then
cpio -imd --no-absolute-filenames /new_root/boot/archfs.img > /dev/null 2>&1
fi
done
# Switch tmpfs to new root
umount /new_root
mount -M /mnt /new_root
}
# vim: set ft=sh ts=4 sw=4 et:
And the file /lib/initcpio/install/tmpfs:
#!/bin/bash
build() {
add_dir /mnt
add_module ext3
add_binary cpio
add_binary xz
add_binary gzip
add_binary lzop
add_binary fsck.ext4
add_symlink /usr/bin/fsck.ext2 fsck.ext4
add_symlink /usr/bin/fsck.ext3 fsck.ext4
add_runscript
}
help() {
cat <<HELPEOF
This hook creates the tmpfs root, mounts the device that contains the
rootfs and extracts it into the tmpfs root.
HELPEOF
}
# vim: set ft=sh ts=4 sw=4 et:
This loads modules and fsck binaries for the filesystem of my /boot partition into the initramfs. It also adds all the relevant compression binaries (some which can also be compiled into busybox or left out if not needed). I'm working on automatically reading the filesystem from your /boot partition based on the "root=" kernel command line option and adding the appropriate modules.
Create a new archfs.gz (or new extensions) with new cleaned up /usr/bin/archfs:
http://pastebin.com/xiqgyYyL
Now run with:
# archfs lzo
(or whichever crompression you choose). It also takes care of any mounting directories you have in /mnt. And "writefs" is no longer needed, that's all it does anyway.
Then simply add the "root=(your boot partition)" and "archfs=lzo" (or whichever compression you chose) command line options to your grub.cfg.
Run:
# mkinitcpio -p archfs
That'll create an initramfs: archfs-linux.lzo that you need to move to your boot partition with your kernel and archfs.gz. Then point grub to that as your initrd.
This will boot and the tmpfs hook will do it's work just after init has mounted your boot partition and then init will switch roots and run /sbin/init as it would on any normal install.
Mix and match as needed. You can remove the /init file from the root of your filesystem or leave it there and all but an xz compressed archfs.gz will boot the old way aswell (the kernel can't handle the xz version). It should all be fairly upgrade proof, barring any major changes to mkinitcpio.
Boot for me with lzo compression is at least twice as fast as old method. Partly due to lzo compression which is super fast (especially creating the archfs.lzo). It makes for a bigger file but it's still only 350MB (pretty much half my uncompressed root filesystem).
Offline
See what you mean about the Git repo Mr Green, would have made that post alot easier. If I'd known git was the lazy option...
Offline
/etc/mkinitcpio.d/archfs.preset
http://pastebin.com/iaX3708V
/lib/initcpio/hooks/tmpfs
http://pastebin.com/D7b58Qgd
/lib/initcpio/install/tmpfs
http://pastebin.com/W7k1j7Dn
/usr/bin/archfs
http://pastebin.com/xiqgyYyL
Offline
And it works fine with a pure systemd install!
Right now to try it with FreeBSD.
Offline
Is there any way to do this while encrypting and password protecting the zip file? Feels like it's a little bit more straightforward than encrypting the hard drive.
Offline
Perhaps, I've never tried but maybe something could be added to the zip/unzip lines in the respective scripts. I personally encrypt partition 2 (and use the encrypted root hook in the initramfs to decrypt it before anything else is done, doesn't matter that it's not my root) and have that mounted as /home and any sensitive files on there symlinked to the filesystem if neccessary since it's only really a few text files at most (wifi passwords, etc.). The kernel and archfs.lzo and my boot.rc (basically rc.local) are on partition 1 unencrypted since most of it is a vanilla install with a few settings changed which I don't consider sensitive.
Offline
Yeah, encrypting a partition that stores sensitive data is probably smarter than encrypting the zip. That way you can have an encrypted home folder only and store mountains of data in it safely if needed.
You point to /etc/rc.d/functions.d/root_rw
This path (/etc/rc.d/) does not exist in arch, what is that file supposed to do and are you sure that's where it should go?
I'm also not sure about how /etc/fstab should be configured.
I want to have /dev/sda1 mounted as /mnt/Linux/ (In there I will store big folders, like /home and /opt and /var/cache/pacman/pkg which will then in turn be symlinked to the system that has been loaded to RAM), but /dev/sda1 is coincidentally where my system is currently installed (i.e. I don't plan to keep the installation around on my SSD after it's been zipped to /boot)
Do I need to configure fstab to mount the root filesystem as ramfs/tmpfs before I run archfs?
Last edited by rabcor (2014-07-12 19:11:44)
Offline
Nope all the mounting of the root filesystem is handled by the boot/unpacking process in the initramfs so shouldn't need any reference in fstab. The 'encrypt' hook in mkinitcpio will take care of mounting your encrypted volume and then you'll probably need a line in fstab for the unencrypted filesystem. I have zfs on mine so it takes care of everything so there is only a line to mount sda1 (ext4) on /boot in my fstab (not actually neccessary but useful). The wiki should cover any of those aspects as they shouldn't be any different to a normal system once the archfs.gz is unpacked.
The /etc/rc.d/functions.d/root_rw line is no longer in the current scripts (see post #11) which use hooks in the initramfs rather than an ugly (on my part) /init script hack.
If you follow the instructions in post #9 and copy the resulting archfs-linux.lzo and archfs.lzo (or whichever compression you chose) to your boot directory you can set up another grub.cfg entry to boot from that and play with it as much as you want without affecting your current install. When you're sure everything works you can move stuff around/delete as much as you want. Boot from any partition, USB stick, network boot probably, whatever with just a few changes to grub.cfg. That's the beauty so long as you've got enough RAM.
Offline
A while back I also experimented with a 'path=' boot code and ram filesystem options so I'm using a hook called memfs hook that does the same job as the old tmpfs hook with a few extra options.
Then my 'linux' line in grub.cfg includes:
cryptdevice=[your home part] root=[your boot part] archfs=lzo path=boot memfs=tmpfs
I didn't find much use for other ram filesystems (other than lots of crashing) but haven't experimented in a while, but the path= option helps when needing to go back to old an archfs.gz (saved in /boot/old whenever I update the system)
Offline
Hi,
So, did I understand this right? First you install the system to a disk and then you apply the changes you posted, in order to run it from RAM?
I ask, because I am interested in running a small server from RAM, which shall get booted from a USB stick (thus I have all SATA free for data), and I want to prepare it the right way for setup.
Offline
Out of curiosity - what's point in doing that? I own 16GB DDR3 ram modules but i also use SSD. I dont think that i would gain anythnig from doing that when i have Arch on SSD.
Offline
amix, yes you can do that or I have done it by installing into a chroot using arch-chroot so you can create it very precisely from the ground up.
firekage, I don't know if you'd see any speed difference compared with an SSD, although the boot for ArchTaz will always be slower even compared to an normal hard drive. Personally I like running my system like this, it's my 'normal' and the advantages outweigh the disadvantages for me. Small install size (less than 500MB for my install), fast launching applications, any changes easily reversed with a line change in grub, potential increased security (changes only written to file system when I uprgade/change anything).
Offline
amix, yes you can do that or I have done it by installing into a chroot using arch-chroot so you can create it very precisely from the ground up.
Thanks a lot!
Offline
@0ddba11
Thanks for this thread and the effort you put into this. It helped me set up a system to my liking. Just one question:
starting version 229
... [sda] No Caching mode page found
... [sda] Assuming drive cache: write through
During boot into ram, after these messages, I need to hit enter, since otherwise the system wouldn't continue loading. Do you have any ideas as to why, or how to fix this?
Btw, I followed/applied the scripts from post #11
There are no errors in dmesg. Ther was one error: "tmpfs: Bad mount option data" and the systemd service failing was "systemd-remount-fs.service". It was apparently related to the fstab still trying to mount the disk, but I fixed that for now by commenting out the entry in the fstab and running mkinitcpio -p archfs.
Do you or does anyone have any ideas why pressing enter matters on ramboot?
It's not happening when booting to the hard drive.
e:
dmesg says boot stops (waits for some sort of confirmation) after the "EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)" entry, and before the "random: nonblocking pool is initialized" entry. Sometimes after the latter.
Hi,
So, did I understand this right? First you install the system to a disk and then you apply the changes you posted, in order to run it from RAM?
I ask, because I am interested in running a small server from RAM, which shall get booted from a USB stick (thus I have all SATA free for data), and I want to prepare it the right way for setup.
You don't need ArchTaz/Copy to Ram in order to do that. Just install arch on an USB stick to keep the system separated from your data-drives. It also makes it easy to pull an image from your system / have a backup system (another stick) at hand.
Out of curiosity - what's point in doing that? I own 16GB DDR3 ram modules but i also use SSD. I dont think that i would gain anythnig from doing that when i have Arch on SSD.
I feel there are a couple of reasons:
to speed up a system (if disk access was the bottleneck)
to not occupy the boot-drive post boot.
For instance, I run arch from an usb-drive and load into ram -> I can remove the usb-stick as soon as the system finished booting. This allows to use the stick on a different computer / to free up that usb-port / do crazy things to the installation -> it will be reset upon next boot / other
e:
I just found out about liveroot, which has an option to boot into ram (oroot=live):
https://bbs.archlinux.org/viewtopic.php?id=193461
https://aur.archlinux.org/packages/liveroot/
Since I already bumped this thread, I thought it's best to at least mention it here. Didn't encounter the boot-issue on liveroot.
Last edited by gmk (2016-04-03 02:19:17)
Offline