how do I use unionfs to merge two squashfs files

bwh1969 · 2009-02-05 23:02:54

I have a live system and would like to "merge" my mods.sqf with the system.sqf... then i would replace the system.sqf with the result.

I know how to mount the squashfs files, but want to mount the system one first, then the mods, then squashfs the resulting merged filesystem. I know I need unionfs but can't figure out quite how to do this part (the merge of directories and overlaying of the mods to the system.

Commands would be great, but I will settle for a good resource. I don't need to add this to fstab, as I would probably do this from a different running system:-)

gradgrind · 2009-02-10 13:21:51

I only just saw this, maybe it's too late now ...

Firstly we need to know how the mods.sqf was created. If it was created as an overlay using aufs or unionfs, it will probably contain 'whiteouts' (to cover up files in the layer below it). In that case you must use the same utility (i.e. aufs or unionfs - they are not exchangeable) to perform the merge. Information concerning use of aufs is available on aufs.sourceforge.net. I'm not sure where unionfs is documented, I think in the kernel patch. If you have the patience you can try to figure out how they are mounted in larch - it is in the larch-live package, in the hooks folder (larch2 is there in two versions, one for unionfs, one for aufs, and larch3 does the mounting). If you need to use aufs, beware that it seems to be broken at the moment, though someone suggested recompiling it would fix it.

I haven't tried this, but in principle it should be possible to rebuild a running larch system using 'larchify /'. There would need to be a lot of RAM, or else one could mount a partition to /.larch (or maybe symlink something there).

bwh1969 · 2009-02-11 02:44:18

Hmm, not sure which,,, would I be able to tell by attempting to mount the file? if I tried to mount the file in a regular distro using

mount -t squashfs -o loop /path/mods.sqf /path/mountingdirectory

if it mountts, it is squashfs? In the /.iivesys/mods and /.livesys/overlay there are .wh files as you mention. The latter overlay is some sort of tar file that is created with the mods.sqf called overlay.ovl Unetbootin creates two overlays... and the latter is just sort of a bad idea because it is a tar file (if you are not familiar with Unetbootin, I can't fill you in too much, as I am not completely versed in how it works. When shutdown occurs, it asks to save session, merge overlay, or no save. Just saving the session just saves the overlay.ovl and does not modify the mods.sqf. The overlay is cumbersome and is just a tar file, and if it is too big, it takes forever to boot (I assume it is a tar file as on boot, a bunch of text goes by for every file that is put back in place, and each line starts with "tar"

At any rate, these two files are merged with the original system.sqf from the original iso file (from Chakra Live or whatever distro) that Unetbootin placed on the "install" device. I would like to merge them one unified file, rename it as system.sqf so that my session saves using the "merge" don't take 15 minutes.

FaunOS, another live Arch distro (but is no longer being worked on) gave the user a bit of a break by allowing one to save just the home directroy changes, as it was a separate squashfs ooverlay.

I am somewhat curoius about 'larchify /' but it sounds like i would just be taking my nice compact install that runs mostly from ramdisk and be turning it into a regular install, which in some applications could be very appealing.

I wish i were not such a novice when it comes to things like this:-) I know enough to want to do it and that somehow I can.

gradgrind · 2009-02-11 13:11:03

Are you saying that it is a chakra iso you started with?

I haven't studied unetbootin - could you say why you are using it?

Anyway, if it is only the boot time that is troubling you, you should be able to overcome this by simply choosing the 'merge overlay' option at shutdown. The tar.lzo overlay will be merged into mods.sqf - at least that's the way it works in unadulterated larch, I don't know what effect unetbootin would have or how your system deviates from standard larch.

As to how to tell if you are using aufs or unionfs - I guess if the aufs package is installed on your system you can probably assume you are using it. Also, at boot towards the end of the initcpio stuff a line is normally output in which aufs or unionfs is mentioned (so long as your boot messages are displayed) - or of course you could do lsmod and see which of aufs and unionfs appears. If it should turn out to be unionfs, the 'merge overlay' feature won't work.

bwh1969 · 2009-02-11 14:32:07

I am using Unetbootin/Chakra on two of my PC's because the user is loaded into ramdisk and it is very fast. The "install" is also very small compared to an actual install... and this works like a dream on my ASUS eee-pc (I have a 701 with a 4 GB SSD). I also have an ASUS-eee-box and it runs very fast from there as well. There is no HD activity unless I load a program. Once loaded, it just flies along. I know there is a blog somewhere about running arch out of ram, but I don't have that much ram. 1.5 GB here.

I know Chakra beta two has or will come out with its own USB image with overlay saving, so I suppose my need for Unetbootin is mute, but I have things fairly customized and it would be nice to just work with what I have.

I think Unetbootin just makes a live cd iso into a bootable usb or hard disk, so whatever features the live distro has, then so will the result of the resulting bootable usb/hard disk. I think in that sense, it is a function of Arch then, because an Ubuntu disk set up Unetbootin, it does not ask to save the session at shutdown. I guess I was misunderstanding that Unetbootin set up the system this way, when in reality, it is because it is Arch?!

Based on what you said, and what I see as far as the output below, it must be aufs... but I was assuming because mods is called "mods.sqf," the sqf extention meant that it was squashfs. From my experience, this is the chosen extention for that particular file system.

[brian@chakra ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
aufs                  907M  191M  716M  22% /
tmpfs                 907M  191M  716M  22% /.livesys
/dev/sdd1             5.4G  2.9G  2.5G  54% /.livesys/medium
none                  756M     0  756M   0% /dev/shm
/dev/sdd2             2.1G  371M  1.8G  18% /media/STORAGE
/dev/sdc1             466G  144G  322G  31% /media/My Book
[brian@chakra ~]$

My mods.sqf is close to 300MB right now (I upgraded kde 4.1 to 4.2) so saving the overlay with "1" takes forever and a day, so that is why I was inqiring if somehow I could merge everything into a unified overlay file (make a new system.sqf), so that my mods.sqf would again be small. The simple save to overlay.ovl has an annoying issue for me as well... for some reason, even if my time is correct in BIOS and in the running system, it displays an error on every file loaded into ramdisk from overlay.ovl So, if I decide to update/sync pacman, and should I choose to save with the "simple" method, the resulting next boot takes 2-3 minutes of annoying text going by. Something about timestamp set too far in the future. It doesn't affect the system once it is up, but I think it takes forever to load because it displays that error for, like I said, every file.

You seem to be quite an expert, or at least to me... so is what I am wanting to do within the realms of a mere mortals and worth an effort, or is it next to impossible (for someone like me to figure out)? I just want mods.sqf and system.sqf to become my new system.sqf.

I would go back to FaunOS, because of the separate home directory overlay... it took like 30 seconds to save and because it was an .sqf overlay, it didn't get "loaded" into ramdisk, sucking up my available home directory space. Unfortunately, it is a "dead "project for now, ingenious as it is. Actually, more like comatose. It is alive, but isn't being worked on.

Last edited by bwh1969 (2009-02-11 21:15:28)

gradgrind · 2009-02-12 20:07:59

bwh1969 wrote:

Based on what you said, and what I see as far as the output below, it must be aufs... but I was assuming because mods is called "mods.sqf," the sqf extention meant that it was squashfs.

It is both, sort of: mods.sqf is a squashfs file system which is mounted as an aufs layer within the live system.

bwh1969 wrote:

You seem to be quite an expert, or at least to me... so is what I am wanting to do within the realms of a mere mortals and worth an effort, or is it next to impossible (for someone like me to figure out)? I just want mods.sqf and system.sqf to become my new system.sqf.

Well, I only seem to be an expert because I wrote the larch scripts used by faunos and chakra, so one would hope I know at least a little about them ...

Anyway what you want should certainly be possible. A complete rebuild using (say) 'larchify -u /' should work - but I don't know what changes the chakra people have made, and you will probably have a problem with insufficient memory, which it should be possible to overcome by mounting something at /.larch or making this a symlink to a large enough place. But this method hasn't been tested ...

The other way, which I think really should work is to mount system.sqf and mods.sqf as aufs layers (see the aufs man page) and then to resquash the unioned result as the new system.sqf. You will also need an empty mods.sqf. Something like this (but be careful - if you have a mods.sqf_ file this is the newest version and you must use this instead of mods.sqf):

mkdir system
mkdir mods
mksquashfs mods mods.sqf_new
mkdir union
mount -o loop -t squashfs system.sqf system
mount -o loop -t squashfs mods.sqf mods
mount -t aufs -o br:mods=rr+wh:system=rr aufs union
mksquashfs union system.sqf_new
umount union
umount mods
umount system

No Guarantees! But it might be worth a try. If you do it, save your old files somewhere, so that if it goes wrong, you can restore it to what you had.

I hope all this will be a bit easier in the next version of larch, and a separate home directory is probably a good idea, but I'm afraid I don't know when the next version will be ready.

Last edited by gradgrind (2009-02-12 20:11:21)

bwh1969 · 2009-02-12 22:14:30

I am on vacation but wish I brought my one machine to try this! I"ll attempt this next week sometime and post back my results. I

I am assuming this would not be done in the running system and that I can modify your code and process slightly as in

mksquashfs mods mods.sqf_new

could be

mksquashfs mods /path2another_disk/mods.sqf

and

mksquashfs union system.sqf_new

could be

mksquashfs union /path2another-disk/system.sqf

Then I could boot with something else, move the old files(mods and system) from the boot device to some location, and the new ones from another_disk back to the boot device (it is an SD card)

Is this a correct assumption?

What about overlay.ovl?

I also noticed that in FaunOS, the etc.sqf is an overlay also but there was no way to select that to be saved separately. I just created a line in rc.local to call a script from /home/user/bin .

I liked how one could save just home by modifying /larch/overlay and removing the 1 from the line that saved the system overlay so it read

overlay=""
home="1"

It would be more user friendly if implemented if it were part of the shutodown options

Like

Save session "enter" (this can be the lzop overlay.ovl)
Save changes to home only ="1" (have this be home.sqf)
Merge overlays="2"
No session saving="3"
Choice=?

Last edited by bwh1969 (2009-02-12 22:26:48)

gradgrind · 2009-02-14 12:18:54

bwh1969 wrote:

I am on vacation but wish I brought my one machine to try this! I"ll attempt this next week sometime and post back my results. I
I am assuming this would not be done in the running system and that I can modify your code and process slightly as in
mksquashfs mods mods.sqf_new
could be
mksquashfs mods /path2another_disk/mods.sqf
and
mksquashfs union system.sqf_new
could be
mksquashfs union /path2another-disk/system.sqf
Then I could boot with something else, move the old files(mods and system) from the boot device to some location, and the new ones from another_disk back to the boot device (it is an SD card)
Is this a correct assumption?

Yes, that sounds about right.

bwh1969 wrote:

What about overlay.ovl?

Leave it as it is, you are just merging the layers underneath it.

bwh1969 wrote:

I also noticed that in FaunOS, the etc.sqf is an overlay also but there was no way to select that to be saved separately. I just created a line in rc.local to call a script from /home/user/bin .
I liked how one could save just home by modifying /larch/overlay and removing the 1 from the line that saved the system overlay so it read
overlay=""
home="1"
It would be more user friendly if implemented if it were part of the shutodown options
Like
Save session "enter" (this can be the lzop overlay.ovl)
Save changes to home only ="1" (have this be home.sqf)
Merge overlays="2"
No session saving="3"
Choice=?

faunos was based on larch-4 which handled the overlays in a rather different way. It is possible to make /home be saved separately with larch-5, but you would need to script it yourself. I am now considering how to handle this in an easier way for the next version of larch.

bwh1969 · 2009-02-17 23:05:27

It works:-)

Of course it does:-)

So now I have a really small mods.sqf to save.

Excellent!

Arch Linux

#1 2009-02-05 23:02:54

how do I use unionfs to merge two squashfs files

#2 2009-02-10 13:21:51

Re: how do I use unionfs to merge two squashfs files

#3 2009-02-11 02:44:18

Re: how do I use unionfs to merge two squashfs files

#4 2009-02-11 13:11:03

Re: how do I use unionfs to merge two squashfs files

#5 2009-02-11 14:32:07

Re: how do I use unionfs to merge two squashfs files

#6 2009-02-12 20:07:59

Re: how do I use unionfs to merge two squashfs files

#7 2009-02-12 22:14:30

Re: how do I use unionfs to merge two squashfs files

#8 2009-02-14 12:18:54

Re: how do I use unionfs to merge two squashfs files

#9 2009-02-17 23:05:27

Re: how do I use unionfs to merge two squashfs files

Board footer