Running Arch Entirely in RAM!!??

lilsirecho · 2009-02-02 22:40:42

I re-tried the faunos forums and was pleased to access them.

According to Raymano in a discussion with Gradgrind the Faunos kernel is patched with an fbsplash and no arch logo. However, it produces an arch logo(older logo) if one boots in root.

He further stated that many packages are faunos format, such as ndiswrapper.faunos and these would have to be replaced if one desired pacman to respond to -Syu.

These remarks are minimal but do suggest that in addition to packages the kernel can be upgraded as long as the mkinitcpio is also upgraded.

The basic system boots to guest and the user must do su root to get command control in terminal.

I would expect that an upgrade is probable with some changes to faunos and then pacman can exercise control.

The pacget program lists the installed packages and those which are compatible with faunos installed kernel with the ID indicating the faunos special package.

The fact that the arch logo appears when boot in root suggests that pacman might already be in control????? Not likely with the flash data fixed in kernel26faunos.

I figure it should be possible but what do I know?

Heller_Barde · 2009-02-04 09:18:55

lilsirecho wrote:

The fact that the arch logo appears when boot in root suggests that pacman might already be in control????? Not likely with the flash data fixed in kernel26faunos.

i can't comment on the rest of your post due to lack of knowledge but pacman is only a package manager, it simply, only and exclusively manages packages, so any logos or similar graphical stuff is probably not due to pacman "being in control".

good luck with your project
cheers Barde

lilsirecho · 2009-02-04 15:40:11

The reference I made to pacman being in control was a bit joking I admit but it follows from the log-in to archlinux Desktop which occurs in Faunos when log-in to root is performed.

The archlinux older style log-in display is presented and operations are performed as root.

Perhaps in the near future the system will be regenerated.

schuay · 2009-02-05 12:56:36

apaige wrote:

I just tried an alternative: booting from a RAID 1 array made of a writable storage device (HDD, flash drive, whatever) and a ramdisk.

I'll second that, could you please expand a bit on your explanation? Sounds *very* interesting

Heller_Barde · 2009-02-05 14:14:55

schuay wrote:

I'll second that, could you please expand a bit on your explanation? Sounds *very* interesting

one for me too, please... actually, what i don't get is this:
don't you have to format the ramdisk again every boot with linux raid partition type and stuff?

apaige · 2009-02-05 16:23:50

Heller_Barde wrote:

don't you have to format the ramdisk again every boot with linux raid partition type and stuff?

No, you just add /dev/rd/0 to the array directly; the md driver will automatically "rebuild" the array (i.e. populate the ramdisk) in the background with idle bandwidth (i.e. whenever the permanent storage device isn't being accessed).

Here is one way of doing it:

* First you choose the permanent storage device you want to use; I chose an eSATA flash drive, but a USB flash drive or a HDD would work just as well.

* Create a partition on the device, with type "fd" (linux raid autodetect). Be careful when setting its size, since it will have to be backed by a ramdisk of the exact same size. I made mine exactly 2,147,483,648 bytes (2 GiB, 2,097,152 KiB). Let's say the partition is /dev/sdc1.

* Optionally create a /boot partition for grub/lilo. Grub at least should be able to boot a RAID 1 partition directly, but I haven't bothered to try. Since I'm not using the entire drive for the RAID 1 partition anyway, I figured I might as well set up the other partition (for data) as the /boot partition.

* Make sure /dev/rd/0 is available. I set the ramdisk parameters in my kernel command line (/boot/grub/menu.lst): brd.rd_nr=1 brd.rd_size=2097152. I also added raid=autodetect in there. All three parameters can be hardcoded when compiling the kernel, with make menuconfig.

* Create the array:

mdadm -C [b]/dev/md0[/b] -n 2 -l 1 /dev/rd/0 -W [b]/dev/sdc1[/b]

Note: the choice of the /dev/md[0-9] device number is permanent; if you are already running a RAID array and have to use /dev/md1 for instance, you'll have to specify the same device (/dev/md1) as the root of the filesystem when booting.
If you want to delay writing to /dev/sdc1, add both the --write-behind and -b internal parameters:

mdadm -C /dev/md0 -n 2 -l 1 [b]-b internal[/b] /dev/rd/0 -W [b]--write-behind[/b] /dev/sdc1

I'm still not sure how much of an impact it has on general responsiveness; you should test it yourself (YMMV). See the md(4) and mdadm(8) man pages for more information.

* If the command is successful, the md driver will start populating the ramdisk with the (albeit empty) contents of /dev/sdc1. You can watch its progression by running cat /proc/mdstat, or:

watch -n 1 'cat /proc/mdstat'

* Create a filesystem on your RAID device. I chose XFS:

mkfs.xfs -L RAID1 /dev/md0

* Your hybrid RAID1 device is created, you can now mount its filesystem, like with any other device. Copy on it the contents of what will be your root filesystem, while making sure you have modified the appropriate files like /etc/fstab (with /dev/md0 for /), etc… Personally, I previously booted an Arch installation USB drive and made a tarball of my entire / partition; then, after I created the RAID1 array, I simply unpacked the tarball onto the RAID1 filesystem:

tar -C /mnt/md0 -x -p -f backup.tar

* In /etc/rc.local of the RAID filesystem, add this line: mdadm /dev/md0 -a /dev/rd/0

* Make sure /dev/sdc is bootable (set up grub on it). Use either /dev/sdc1 or /dev/sdc2 as the boot partition (depending on your earlier choice), not /dev/md0 (I'm not sure that would work).

* In /boot/grub/menu.lst of /dev/sdc1 or /dev/sdc2, set root=/dev/md0 in your kernel command line, as well as the other three parameters I mentioned above (raid=autodetect and brd.rd_nr=1 brd.rd_size=2097152). I also added rootwait just to be sure the kernel would wait for the RAID1 device to be detected before it attempts to mount it; it's particularly important if /dev/sdc is a flash drive. If that doesn't work (kernel panic before it could find the root device), use rootdelay=20.

That's pretty much it. Reboot. The kernel will detect /dev/sdc1 as being part of a RAID 1 array and mount it as root, even though it can't find its second member (the ramdisk). That's alright. When /etc/rc.local is finally executed (at the end), the ramdisk gets added to the RAID array, and md populates it; meanwhile, you can do anything you want on the filesystem (read/write). The "rebuilding" process is pretty fast (as fast as the permanent storage device allows) because it is done on a block device level, as opposed to a filesystem level: transfer blocks are fairly large (128KiB by default, it seems). The faster the device at large sequential reads, the faster the rebuilding. That's why a HDD is perfectly acceptable in such a setup.

I haven't yet tried starting X and my other apps immediately (I've waited for the rebuilding process to complete, which takes a few seconds here); but once the ramdisk is populated, everything loads instantly. It would be pretty hard to go back to a normal setup…

schuay · 2009-02-06 07:53:15

Thanks! Going to give that a try as soon as I get home.

lilsirecho · 2009-02-06 19:27:40

I have three CF cards in a raid0 array which provides 133MB/sec read speed .

Changing to raid1 would reduce the speed to 42MB/sec.

Could raid0 be utilized in your ram OS instead?

apaige · 2009-02-06 19:49:29

lilsirecho: I guess you could set up a RAID 0+1 array.

lilsirecho · 2009-02-06 20:13:32

That seems like a winner, assuming the +1 is a partition on the raid0 devices.

Perhaps this would provide a speedier setup to ram although not a big difference but it would allow my present layout to adopt the ideas in this thread.

schuay · 2009-02-07 07:31:14

I had a go at this yesterday.. Made a new minimal arch install on a 1.5 gig partition, set up the raid (loaded modules brd + raid1), made a new initramfs with raid enabled, rebooted.. and got an error "cannot mount EXT4 filesystem, please update userspace programs".

Strange.

The Raid works in one session but after a reboot everything is gone, i need to recreate the FS and reinstall. Have I missed a step somewhere? Do I need to reenable the raid after boot? Maybe it's because the ramdisk + partition weren't exactly the same size?

As you can tell I'm new to software RAID .. Sorry if this is getting too offtopic.

Last edited by schuay (2009-02-07 07:32:08)

banshee28 · 2009-02-07 21:25:46

apaige, how much ram do you have in your system? I only have 2Gb on this PC for now and want to know if that will work. I can always upgrade but was not sure how much ram this method takes and whats left over for everything else?

apaige · 2009-02-08 11:15:01

schuay: I don't know about ext4. I use XFS, and a self-compiled kernel with XFS and RAID compiled in.

banshee28: it takes as much ram as the size of your ramdisk, which you set in your kernel command line. Of course it needs to be the same size as your permanent storage device partition, and both need to be large enough to store what you want to put on them. I have 8GiB of RAM total, with a 2GiB ramdisk (and 2GiB partition) for my RAID 1 array; with my current Arch Linux installation, I have 350 MiB of free space left on that. Your Arch Linux install could be larger or smaller depending on how much stuff you install.
Note that Arch isn't the lightest distribution out there, since every package comes with its /usr/include files, as well as locales, both of which take much space. Alternatively, you could move /usr/include (and /usr/share/locale, etc…) on another partition that's not part of the RAID 1 array, and mount them later on.

Last edited by apaige (2009-02-08 11:17:39)

lilsirecho · 2009-02-09 00:12:20

apaige;

You posted that you use an esata flash drive.

Is that flash drive UDMA capable(45MB/sec?)

apaige · 2009-02-09 06:57:54

lilsirecho: UDMA/100 (I usually get about 80MB/s for reads, 30MB/s for writes). It's an OCZ Throttle eSATA drive.

saten · 2009-02-09 12:44:30

hi everyone!
this is a very addictive topic
i want to try out apaige's solution, but first i'd like to know more.

let's assume i'm making a clean install. how can i get /dev/rd/0 to be more than arch's default 16MB ? i've tried giving kernel options at boot time with no luck. currently i'm cross compiling latest stock kernel for i686 with a different size, but i wonder if there's a easier way. (EDITED BELOW)

if i choose to have /boot on a separate partition, do i need to set it type FD or may i only set it ext2 flagged bootable?

what if you exceed the disk space with ramfs rather than tmpfs? would it be possible to have some good results mounting /usr/bin and some other stuff as tmpfs and then go raid 1 on them? just wondering

saten

EDIT: maybe it's enough to use modprobe brd with options yes! you can just use modprobe brd rd_nr=1 rd_size=size.. and proceed with normal installation media.

Last edited by saten (2009-02-09 16:35:29)

lilsirecho · 2009-02-09 16:29:21

apaige;

Very nice esata device.

Wondering if a pci-e to esata card with two independent interface ports will respond to a mdadm raid setup.

What is the identifier for serial ATA drives; sdx?

saten · 2009-02-09 18:52:41

i can't get it boot after installing without problems. get kernel panic even using rootdelay. what can it be?
i'm installing on a eee pc 701 (the 4 GB ssd disk one).

lilsirecho · 2009-02-13 03:40:03

AFAIK ramdisc is limited to one/half of physical memory.

Tmpfs can use all of memory and then use swap.

Just a reminder....

Ranguvar · 2009-02-13 15:52:53

lilsirecho wrote:

AFAIK ramdisc is limited to one/half of physical memory.
Tmpfs can use all of memory and then use swap.
Just a reminder....

That's if you're using /dev/shm, which is itself a limited-to-half-memory tmpfs. And you can change the limit by modifying your fstab.

apaige · 2009-02-14 12:40:43

Nothing is limited to one half of the physical memory. It's just that a tmpfs mount (no matter where it's mounted) has a default limit equal to half the RAM. Like Ranguvar said, you can change that limit with the size= mount parameter.

The number of ramdisks (brd driver) and their size is whatever you say they are, either directly in the kernel's configuration, or as driver/module parameters (brd.rd_nr and brd.rd_size). Also, in case that wasn't clear, you can't make a RAID array with tmpfs/ramfs mounts, since those are blockless (virtual) filesystems; the md driver needs block devices, hence the use of ramdisks (brd driver).

saten: what does it say exactly? Once you understand the boot process, it won't be difficult to fix your problem.

lilsirecho · 2009-02-14 15:40:35

I am aware of the tmpfs arrangement and have used remount to adjust its size in rc.local.

I mention the subject to alert all users since it is a portion of the system being discussed.

I often introduce comments which expand the scope and may seem wrong or out of place but does encourage others to provide better helpful insight into the subject.

I am not always the last word!

Thanks for the responses!

I may be too old for this...83years+.

Odysseus · 2009-02-15 21:59:26

I tried this with a USB Flash drive. I had some trouble getting the array to assemble on boot: http://bbs.archlinux.org/viewtopic.php?id=65407

Even with the --write-behind related options, performance still seems limited at times.
Installing gnome was taking forever until I just removed the usb drive itself from the array. It was kind of cool to know that I was running entirely in ram.

And before I shutdown, I just had to hot-add the USB drive back in and let it synchronize.

Last edited by Odysseus (2009-02-15 22:00:13)

saten · 2009-02-16 12:20:41

apaige wrote:

saten: what does it say exactly? Once you understand the boot process, it won't be difficult to fix your problem.

sorry i don't remember the exact error. i managed to get something working by mounting only /usr /except /usr/share in raid 0 with ram.
i modified /etc/mdadm.conf accordingly
what i don't understand now is why my ram seems to be unused. when i launch something i see the led of the ssd on, so i believe that reads are actually done on the ssd. i've used your command to build /dev/md0.

apaige · 2009-02-16 17:55:37

saten: paste the ouput of sudo mdadm --detail /dev/md0

Arch Linux

#26 2009-02-02 22:40:42

Re: Running Arch Entirely in RAM!!??

#27 2009-02-04 09:18:55

Re: Running Arch Entirely in RAM!!??

#28 2009-02-04 15:40:11

Re: Running Arch Entirely in RAM!!??

#29 2009-02-05 12:56:36

Re: Running Arch Entirely in RAM!!??

#30 2009-02-05 14:14:55

Re: Running Arch Entirely in RAM!!??

#31 2009-02-05 16:23:50

Re: Running Arch Entirely in RAM!!??

#32 2009-02-06 07:53:15

Re: Running Arch Entirely in RAM!!??

#33 2009-02-06 19:27:40

Re: Running Arch Entirely in RAM!!??

#34 2009-02-06 19:49:29

Re: Running Arch Entirely in RAM!!??

#35 2009-02-06 20:13:32

Re: Running Arch Entirely in RAM!!??

#36 2009-02-07 07:31:14

Re: Running Arch Entirely in RAM!!??

#37 2009-02-07 21:25:46

Re: Running Arch Entirely in RAM!!??

#38 2009-02-08 11:15:01

Re: Running Arch Entirely in RAM!!??

#39 2009-02-09 00:12:20

Re: Running Arch Entirely in RAM!!??

#40 2009-02-09 06:57:54

Re: Running Arch Entirely in RAM!!??

#41 2009-02-09 12:44:30

Re: Running Arch Entirely in RAM!!??

#42 2009-02-09 16:29:21

Re: Running Arch Entirely in RAM!!??

#43 2009-02-09 18:52:41

Re: Running Arch Entirely in RAM!!??

#44 2009-02-13 03:40:03

Re: Running Arch Entirely in RAM!!??

#45 2009-02-13 15:52:53

Re: Running Arch Entirely in RAM!!??

#46 2009-02-14 12:40:43

Re: Running Arch Entirely in RAM!!??

#47 2009-02-14 15:40:35

Re: Running Arch Entirely in RAM!!??

#48 2009-02-15 21:59:26

Re: Running Arch Entirely in RAM!!??

#49 2009-02-16 12:20:41

Re: Running Arch Entirely in RAM!!??

#50 2009-02-16 17:55:37

Re: Running Arch Entirely in RAM!!??

Board footer