larch-4 testing, can also 'livify' existing Arch installation

gradgrind · 2007-10-03 07:53:37

I've uploaded a testing repository for larch-4 to:
ftp://archie.dotsrc.org/projects/archie/larch/testing

It has some added features which need testing - especially with USB, which is difficult for me to test because my computers are rather old and don't support USB boot.

It should now be possible to adjust the session-saving behaviour without modifying the larch scripts - additional scripts (hooks / event-handlers, call them what you will ...) can be placed in a 'scripts' folder in the root directory of the boot device (using the profile mechanism) to adjust loading and saving of overlay files. Initial description (sorry, it's still rather sketchy) in the session-saving doc page.

The home directory is now a separate sqf (not overlayed).

Added options to mklarch to build a CD/USB-stick from an existing Arch installation. This can be done, for example, from a live CD (using the -x option) or even from the system itself while it is running (option -X). This could be rather dangerous - don't do it on your main installation. I've tried it out - without obvious damage - on a test installation, and it looked quite promising, but be careful.

Added option -r to mklarch to ensure regeneration of base and system sqfs when 'reusing' an installation (with options -b, -x, -X).

Changed the place where the live CD is built (default /home/larchroot) and the structure within this folder. The installation is now made directly to this location and the building is done in subfolder .larch, where the final iso also lands.

To use it, just copy the larch-setup.sh script to an empty folder and run it (as before). Then you have the new mklarch set up and ready to go.

Feedback welcome.

wantilles · 2007-10-10 19:31:35

This has been happening for several days now:

error: failed to synchronize larch-testing: File unavailable (e.g., file not found, no access)

gradgrind · 2007-10-11 05:00:58

wantilles wrote:

This has been happening for several days now:
error: failed to synchronize larch-testing: File unavailable (e.g., file not found, no access)

larch-testing? There is no such thing. The new larch scripts are in a separate repository, address as above. Fetch the larch-setup.sh script from there using a browser or wget, and place it in a new folder. Make it executable and run it, or else run:

sh larch-setup.sh

This will download and unpack the stuff you need to run mklarch.

wantilles · 2007-10-11 10:24:30

gradgrind wrote:

larch-testing? There is no such thing.

Of course there is.

I can put a custom repo in my pacman.conf with any name I like.

i.e.

[larch-testing]
Server = ftp://archie.dotsrc.org/projects/archie/larch/testing

Now if the repository is structured correctly, the above will work.

gradgrind wrote:

The new larch scripts are in a separate repository, address as above. Fetch the larch-setup.sh script from there using a browser or wget, and place it in a new folder. Make it executable and run it, or else run:
sh larch-setup.sh
This will download and unpack the stuff you need to run mklarch.

I do not like circumventing my package manager.

retsaw · 2007-10-11 11:06:26

wantilles wrote:

I can put a custom repo in my pacman.conf with any name I like.
i.e.
[larch-testing]
Server = ftp://archie.dotsrc.org/projects/archie/larch/testing
Now if the repository is structured correctly, the above will work.

Where did you get that idea? If you want it to work you have to give it the correct name, which unless gradgrind decides to change it is "larch" not "larch-testing". I did initially make this mistake myself, but realised that pacman looks for a db.tar.gz file of the name you have given the repo in your pacman.conf.

gradgrind · 2007-10-11 16:04:12

wantilles wrote:

I do not like circumventing my package manager.

'Circumventing' sounds too negative to me. It's just an alternative. I personally find it much more convenient to have larch unpacked in a single folder within /home somewhere, rather than having to fiddle around with pacman.conf and reinstall packages every time I alter something.

wantilles · 2007-10-18 11:46:30

retsaw wrote:

I did initially make this mistake myself, but realised that pacman looks for a db.tar.gz file of the name you have given the repo in your pacman.conf.

I did not know that.

Thank you.

wantilles · 2007-10-18 11:50:13

About the "livifying" an existing installation feature:

Is the larch-4 script smart enough to compare the default files of the profile being used with the equivalent of the existing installation and only include in the overlays those that are different?

Or does it "blindly" copy the whole installation into a huge overlay file?

gradgrind · 2007-10-18 17:45:36

wantilles wrote:

About the "livifying" an existing installation feature:
Is the larch-4 script smart enough to compare the default files of the profile being used with the equivalent of the existing installation and only include in the overlays those that are different?
Or does it "blindly" copy the whole installation into a huge overlay file?

I'm not sure exactly what you're asking here. The initial overlay is normally pretty small, just those files in the profile plus a few extras (unless you make an enormous overlay.xpk - by default there isn't any). The default profile for livifying an existing installation is really very small. Nearly all the files in the installation will end up in system.sqf and home.sqf.

spookykid · 2007-10-19 13:43:34

Hi gradgrind, this tool is really what I'm looking for. I have 3 partitions: /dev/sda1 -> swap; /dev/sda2 -> reiserfs (partition where my system is installed); /dev/sda3 -> reiserfs (partition where I'm running mklarch script). I want to try larch but I have a few questions.

1) To create a LiveCD from my current arch setup I only need to run

./mklarch -X

?
2) Is it possible to add gensplash to larch?
3) For what I've read about larch I can have a system installed with 4 GB and that will fit on a 700 MB CD?
4) Larch will only go for my /dev/sda2 partition (where i have my system) or will it add all available partitions to the live image?

Thank you for your answer!

gradgrind · 2007-10-19 16:03:11

spookykid wrote:

Hi gradgrind, this tool is really what I'm looking for. I have 3 partitions: /dev/sda1 -> swap; /dev/sda2 -> reiserfs (partition where my system is installed); /dev/sda3 -> reiserfs (partition where I'm running mklarch script). I want to try larch but I have a few questions.
1) To create a LiveCD from my current arch setup I only need to run
./mklarch -X
?

If you're running the system you want to save, yes. But from your description it sounds like you're doing something else. Do I understand correctly that you have some other system unrelated to sda2 on sda3 and you want to run mklarch from sda3? And what you want to livify is sda2? Then you need to do something like:

# mount -o dev,exec /dev/sda2 /mnt/sda2
# ./mklarch -x /mnt/sda2

Note the small 'x'.

spookykid wrote:

2) Is it possible to add gensplash to larch?

Sorry, I have no idea about gensplash. If it's supported by Arch then it should work with larch too, but it's not something in which I have any interest.

spookykid wrote:

3) For what I've read about larch I can have a system installed with 4 GB and that will fit on a 700 MB CD?

I hope I didn't write that. You might need that much free space to build a 700MB CD, but a more accurate estimate of what fits would be about 2GB. If you want more you'll need correspondingly more free space and a DVD or a large USB-stick.

spookykid wrote:

4) Larch will only go for my /dev/sda2 partition (where i have my system) or will it add all available partitions to the live image?

It just packs up what it finds at the mount point, so if you've got other stuff mounted within the system (I guess in your case you probably haven't) don't be surprised if it tries to pack that up too. I think (I hope!) it ignores stuff mounted within /mnt and /media, but I'd have to look at the code again to be sure. For people who have /home or /var or whatever on separate partitions it is necessary to mount these before running mklarch.

If you use 'mklarch -X' (big 'X') to build a live CD from the currently running system, all the mounts should be ok anyway, but I think that in general building from a separate system, as you seem to want to do might be slightly safer.

In any case, please back up anything important before running mklarch - just in case! It runs as root and could do all sorts of nasty things. I don't think it will, but don't blame me if it feeds your grandmother to the ravenous bugblatter beast of Traal

spookykid · 2007-10-19 16:54:19

Thank you for your reply gradgrind! My intentions are to make my custom Arch LiveCD instead of installing everything again and again, i install my system, configure and save it on a LiveCD, then I just install from LiveCD. My sda1 partition is my swap partition, sda2 is my root partition and sda3 is just a backup partition i use for storing settings and files. All partitions are mounted on fstab and acessible via Nautilus, but I just want to copy sda2 to a LiveCD. Will

./mklarch -X /dev/sda2

do the trick?
Ohh and about gensplash I think raymano with FaunOS has managed to get it working, I'm not sure about this. I'm about to download and test it now. Many thanks gradgrind.

gradgrind · 2007-10-19 18:04:38

Ahh, so you are running from /dev/sda2. Then

./mklarch -X

without the /dev/sda2. sda3 will be ignored if it is mounted in /mnt somewhere. If you want to install from the live CD it's probably a good idea to install larchin, which should make installation quite easy.

And good luck with gensplash ...

GCN · 2007-10-22 12:30:43

Very useful and helpful set of scripts! It is working wonderfully here for making a bootable Arch on a USB stick. Or I must say it *was* working. Since Arch's kernel release bumped to 2.6.23 it is not working anymore.

It is complaining about aufs:

WARNING:
  You seem to have installed a package containing modules
which aren't compatible with your kernel.
Please check that this won't cause problems.
Maybe you need the corresponding package
for your kernel?
/lib/modules/2.6.22-ARCH/kernel/fs/aufs/aufs.ko is owned by aufs 2.6.22cvs-2
   Package: aufs

I'm sure you'll fix that minor issue very quickly . Thanks for you excellent work gradgrind.

gradgrind · 2007-10-22 15:06:42

Should now work with 2.6.23. I've updated the repository on the archie ftp server.

Flying Saxman · 2007-10-22 15:24:29

just a question: anyone tested yet, if it works with unionfs again?

GCN · 2007-10-22 16:06:29

gradgrind wrote:

Should now work with 2.6.23. I've updated the repository on the archie ftp server.

I will try tomorrow as I need this only for a project at work and I don't have any USB Stick with me currently.

Thanks.

wantilles · 2007-10-30 11:16:09

gradgrind wrote:

Should now work with 2.6.23. I've updated the repository on the archie ftp server.

Any progress on amd64 on the following restriction?

Can 64-bit live Arch CDs be built now that we have the new 2.6.23 kernel?

gradgrind wrote:

Unfortunately there is one catch. I am recommending using aufs at the moment instead of unionfs, because unionfs does not work with kernel 2.6.22. aufs does not yet support 64-bit processors.
Practically that means, I guess, that only 32-bit live systems can be built at the moment. As to whether that could be done from your 64-bit environment, I'm afraid I have no idea - I don't have a 64-bit processor and so I can't try anything out. If you haven't got space for or don't want a 32-bit system on your disk, you could try from a (32-bit) live CD - just make sure that the larch build gets done somewhere where there's enough space.

I know you are not responsible and it is not your fault.

gradgrind · 2007-10-30 16:38:07

Hi wantilles,

if you are really interested in a 64-bit larch, I can only suggest that you give it a try. It would mean rebuilding the packages, but that's not difficult. Apparently it has been reported that aufs does/might work on 64-bit - you'd have to test it. Otherwise, maybe unionfs works again ...

Anyway, let me know if you want to have a go - I could try to walk you through the package building. The 'source' code is all in ftp://archie.dotsrc.org/projects/archie … 025.tar.gz

wantilles · 2007-10-31 10:38:51

Okay.

Now if you could please give me some links to the PKGBUILDs of the packages (you said they are architecture-independent) and to the source tarballs themselves (or any other necessary files) so that I could build them on x86_64, that would be nice.

Thank you.

gradgrind · 2007-10-31 13:21:58

wantilles wrote:

Okay.
Now if you could please give me some links to the PKGBUILDs of the packages (you said they are architecture-independent) and to the source tarballs themselves (or any other necessary files) so that I could build them on x86_64, that would be nice.
Thank you.

1) Download ftp://archie.dotsrc.org/projects/archie … ild.tar.gz and unpack it. That's PKGBUILDs for the larch packages.

2) Download ftp://archie.dotsrc.org/projects/archie … 025.tar.gz and unpack this within the newbuild directory you got when you did step 1.

3) Run 'makepkg -d' in directories archin, initcpio_larch, initcpio_larch_aufs, larch, larchin, pacin.

4) Move all the built packages to an empty directory, your larch repository.

5) Go to trunk/abs/klibc-larch and run makepkg, move the resulting package to your repository.

6) With aufs you can either try to modify my PKGBUILD at trunk/abs/aufs, or the official one in /var/abs/extra/system (assuming you have run abs recently) - you don't need aufs-utils. If this works, move the resulting package to your repository. If it doesn't work, try asking on the aufs mailing list.

7) Go to trunk/BuildScripts/. Edit repodb so that it points at your repository, then run it (you might need to be root for this).

8) Edit archin-setup.sh and larch-setup.sh to point at your repository, then copy them into it.

If all went well, you should now have your own 64-bit larch repository. Good luck!

wantilles · 2007-11-01 17:13:01

gradgrind wrote:

Go to trunk/abs/klibc-larch and run makepkg, move the resulting package to your repository.

There is a problem in the abs/klibc-larch PKGBUILD.

There is a patch missing:

==> ERROR: multiple_raid_assembly_fix.patch was not found in the build directory and is not a proper URL.
==> Aborting...

gradgrind wrote:

With aufs you can either try to modify my PKGBUILD at trunk/abs/aufs, or the official one in /var/abs/extra/system (assuming you have run abs recently) - you don't need aufs-utils. If this works, move the resulting package to your repository. If it doesn't work, try asking on the aufs mailing list.

The default in the official x86_64 abs seems okay.

Why and how should I modify it?

You mean just the 'arch' field?

It is already there in the x86_64 official abs.

gradgrind · 2007-11-01 21:21:08

wantilles wrote:

gradgrind wrote:
Go to trunk/abs/klibc-larch and run makepkg, move the resulting package to your repository.
There is a problem in the abs/klibc-larch PKGBUILD.
There is a patch missing:
==> ERROR: multiple_raid_assembly_fix.patch was not found in the build directory and is not a proper URL.
==> Aborting...

Sorry about that. You can either copy the patch over from klibc in abs (core/base) or strip out all the patch stuff from the PKGBUILD. I also noticed another problem - you'll need to change _kver to 2.6.23-ARCH (the klibc in abs also hasn't been updated yet).

wantilles wrote:

gradgrind wrote:
With aufs you can either try to modify my PKGBUILD at trunk/abs/aufs, or the official one in /var/abs/extra/system (assuming you have run abs recently) - you don't need aufs-utils. If this works, move the resulting package to your repository. If it doesn't work, try asking on the aufs mailing list.
The default in the official x86_64 abs seems okay.
Why and how should I modify it?
You mean just the 'arch' field?
It is already there in the x86_64 official abs.

Ah, great - aufs is already in extra for x86_64, too. So you don't need to build it. Forget step 6.

wantilles · 2007-11-02 00:00:30

Okay, here is the first feedback on x86_64:

It does not work.

It builds the live system (although I am not sure whether it builds it correctly and fully, see further on why).

But then it fails to build the ISO image.

When I answer "yes" to build the ISO image, after the kernel (with the hooks) is made, it fails shortly afterwards:

  /usr/lib/libgcc_s.so.1
     .. copied
  /lib/libc.so.6
cp: cannot stat `/mnt/temp/larch/build/lib/ld-linux.so.2': No such file or directory
ls: cannot access /lib/ld-linux.so.2: No such file or directory
// creating compressed image of linux system
//  ... base.sqf
Parallel mksquashfs: Using 1 processor
Creating little endian 3.0 filesystem on /.larch/tmp/base.sqf, block size 65536.

And then it gets into an endless loop, with full user CPU usage on all CPU cores at 100%, producing output like:

[                                                                ]   1/
[=                                                               ]   2/
[=                                                               ]   3/
[==                                                              ]   4/
[==                                                              ]   5/
[===                                                             ]   6/
[===                                                             ]   7/
[====                                                            ]   8/
[====                                                            ]   9/
[=====                                                           ]  10/

until it reaches 100% and then the cursors into the terminal blinks and jumps at random points.

The only option is to press Ctrl+C.

And then I get:

-1   0%ERROR: squash failed --- mksquashfs / /.larch/tmp/system.sqf -ef /.larch/basefiles -e dev home mnt media proc sys tmp .larch

If I press Ctrl+C before it reaches 100%, at a random point, I may get something like this:

-1  32%ERROR: squash failed --- mksquashfs /.larch/tmp/base /.larch/tmp/base.sqf

I tried it four times, so that there is no possibility or error.

The same exact profile builds normally on i686.

gradgrind · 2007-11-02 06:57:47

Your problem with ld-linux.so.2 is quite simple, in x86_64 it is called something else. Please edit (preferably in your sources and then rebuild the larch package with a new version number) (blah/blah/larch/)share/larch/buildlive, line 517:
Was:
move /lib/ld-linux.so.2
New:
move /lib/$( cd ${INSTLDIR}/lib; ls ld-linux*.so.2 )

The next problem, with the funny terminal output, is less obvious. The heavy CPU usage is not a problem, that is expected - it has to do a lot of work to compress the data. I would suggest first playing around with mksquashfs outside of larch to see if you get the same problem. You could also try in a different terminal program.

Arch Linux

#1 2007-10-03 07:53:37

larch-4 testing, can also 'livify' existing Arch installation

#2 2007-10-10 19:31:35

Re: larch-4 testing, can also 'livify' existing Arch installation

#3 2007-10-11 05:00:58

Re: larch-4 testing, can also 'livify' existing Arch installation

#4 2007-10-11 10:24:30

Re: larch-4 testing, can also 'livify' existing Arch installation

#5 2007-10-11 11:06:26

Re: larch-4 testing, can also 'livify' existing Arch installation

#6 2007-10-11 16:04:12

Re: larch-4 testing, can also 'livify' existing Arch installation

#7 2007-10-18 11:46:30

Re: larch-4 testing, can also 'livify' existing Arch installation

#8 2007-10-18 11:50:13

Re: larch-4 testing, can also 'livify' existing Arch installation

#9 2007-10-18 17:45:36

Re: larch-4 testing, can also 'livify' existing Arch installation

#10 2007-10-19 13:43:34

Re: larch-4 testing, can also 'livify' existing Arch installation

#11 2007-10-19 16:03:11

Re: larch-4 testing, can also 'livify' existing Arch installation

#12 2007-10-19 16:54:19

Re: larch-4 testing, can also 'livify' existing Arch installation

#13 2007-10-19 18:04:38

Re: larch-4 testing, can also 'livify' existing Arch installation

#14 2007-10-22 12:30:43

Re: larch-4 testing, can also 'livify' existing Arch installation

#15 2007-10-22 15:06:42

Re: larch-4 testing, can also 'livify' existing Arch installation

#16 2007-10-22 15:24:29

Re: larch-4 testing, can also 'livify' existing Arch installation

#17 2007-10-22 16:06:29

Re: larch-4 testing, can also 'livify' existing Arch installation

#18 2007-10-30 11:16:09

Re: larch-4 testing, can also 'livify' existing Arch installation

#19 2007-10-30 16:38:07

Re: larch-4 testing, can also 'livify' existing Arch installation

#20 2007-10-31 10:38:51

Re: larch-4 testing, can also 'livify' existing Arch installation

#21 2007-10-31 13:21:58

Re: larch-4 testing, can also 'livify' existing Arch installation

#22 2007-11-01 17:13:01

Re: larch-4 testing, can also 'livify' existing Arch installation

#23 2007-11-01 21:21:08

Re: larch-4 testing, can also 'livify' existing Arch installation

#24 2007-11-02 00:00:30

Re: larch-4 testing, can also 'livify' existing Arch installation

#25 2007-11-02 06:57:47

Re: larch-4 testing, can also 'livify' existing Arch installation

Board footer