live Arch CD construction kit (larch)

raymano · 2007-06-14 21:27:39

Hi gradgrind,

Wanted to thank you again for larch. It's been working great in FaunOS.

I'm having some problems with latest pacman upgrade. Here's what I did step-by-step:

- While running from the USB key, I upgraded pacman using: pacman -Sy pacman to latest version in arch repos, version 3.0.4-4.
- I saved my session at boot time.
- Rebooted and did a pacman -Syu

And here's what I get:

sudo pacman -Syu | more
:: Synchronizing package databases...
 current is up to date
 extra is up to date
 community is up to date
:: Starting full system upgrade...
error: could not open file /var/lib/pacman/current//acpi-0.09-1/desc: No such fi
le or directory
error: could not open file /var/lib/pacman/current//acpid-1.0.4-2/desc: No such
file or directory
error: could not open file /var/lib/pacman/current//acroread-7.0.9-1/desc: No su
ch file or directory
.
.
.

Notice the double slashes after the repo (current//acpi....).

Any ideas on what's causing this? Is it an aufs issue?

P.S. I also tried updating pacman and then updating the rest of the packages in the same session and same thing happens.

gradgrind · 2007-06-15 09:25:10

Hi raymano,
Thanks for the report - I have managed to find a couple of problem areas. But first a question: How did you get
'pacman -Sy pacman' to work? I thought we had established that for some reason it doesn't work in aufs.

1) There does seem to be a problem with aufs. After the first 'pacman -Sy' (where there are no pacman db files present) it can take quite a long time for the contents of /var/lib/pacman to actually appear there - they are immediately in the write-branch of the union but not yet visible in the unified /var. It seems to happen quite repeatedly here, so I assume you should also notice this.

2) Secondly, and probably most relevant to your immediate problem, you have helped me find a bug in the merge_overlay script, where not all the contents of completely rewritten (i.e. whited-out) directories are copied to the overlay. I'll try to get a fix done, but unfortunately this is one of my least favourite areas of the code! If you're interested it's about line 105 for the normal files - that should not be very difficult, but I might have to consider directories as well.

raymano · 2007-06-15 13:40:11

gradgrind wrote:

Hi raymano,
Thanks for the report - I have managed to find a couple of problem areas. But first a question: How did you get
'pacman -Sy pacman' to work? I thought we had established that for some reason it doesn't work in aufs.
1) There does seem to be a problem with aufs. After the first 'pacman -Sy' (where there are no pacman db files present) it can take quite a long time for the contents of /var/lib/pacman to actually appear there - they are immediately in the write-branch of the union but not yet visible in the unified /var. It seems to happen quite repeatedly here, so I assume you should also notice this.

Yes. You are right. pacman -Sy pacman doesn't really work correctly either and I forgot to mention it. I found out that it works if you run pacman-optimize first which recreates the pacman file tree. But this doesn't help the subsequent pacman -Syu.

The confusing thing is why does this happen only when updating packages? Installing new packages without updating seems to work just fine.

Also, off topic, I have created a forum for FaunOS at http://forum.faunos.com and slowly building a community.

Our goal with FaunOS is not creating a fork from Arch. But rather a different way of distributing it and making it more accessible to more people. We will always use packages from the Arch repos and if someday the need for some new packages rises, we'll release them to AUR and have only these additional packages in an additional repo named "faunos". If and when these packages are moved to the arch community repo by the Arch devs, we'll remove them from the faunos repo.

We would love to have you join us at the FaunOS forums. I think your input and thoughts, and our interaction will be very beneficial to larch, Arch, and FaunOS.

Thanks,
Raymano

gradgrind · 2007-06-16 14:21:50

I've just uploaded larch-3.9, with fixes to merge_overlay. I hope session-saving will work better now. The mistakes causing some files to be missing in rebuilt overlays should be fixed. Anyone using session-saving should definitely update.
I've also updated the aufs package. This seems to have cured the lag problem, so anyone using aufs should also update.

raymano · 2007-06-16 18:07:55

Thanks gradgrind. I'll give it a shot.

raymano · 2007-06-17 18:45:50

Gradgrind,

It seems like there is still something wrong with aufs lag time. Here's the problem:

- I made a live cd using your latest larch scripts.
- I booted into the live cd and did a "pacman -Sl" and pacman lists all packages in the repos, about 3925 packages.
- I did a "pacman -Syu" and all my packages were up to date. Immediately after that I do a "pacman -Sl" again and nothing is listed anymore.
- I wait about 1 minute and do a "pacman -Sl" and the 3925 packages are listed again.

These issues with aufs are starting to make me nervous. Do you recommend moving back to unionfs?

Thanks,
Raymano

gradgrind · 2007-06-17 19:23:25

raymano wrote:

Gradgrind,
It seems like there is still something wrong with aufs lag time. Here's the problem:
- I made a live cd using your latest larch scripts.
- I booted into the live cd and did a "pacman -Sl" and pacman lists all packages in the repos, about 3925 packages.
- I did a "pacman -Syu" and all my packages were up to date. Immediately after that I do a "pacman -Sl" again and nothing is listed anymore.
- I wait about 1 minute and do a "pacman -Sl" and the 3925 packages are listed again.
These issues with aufs are starting to make me nervous. Do you recommend moving back to unionfs?
Thanks,
Raymano

It's annoying, isn't it. Maybe for the moment unionfs might be the better choice, worth a try anyway. It might also be worth getting in touch with the aufs developer - perhaps he can help.

raymano · 2007-06-18 04:50:33

I switched back to unionsfs but now the session save takes 2 minutes even for the smallest changes like saving 1 text file.

Since pacman is under /var and most of the problems seem to mostly effect pacman would it make any difference if /var was separated into its own sqf file.

gradgrind · 2007-06-18 06:56:18

raymano wrote:

I switched back to unionsfs but now the session save takes 2 minutes even for the smallest changes like saving 1 text file.

Painful! I think if I wanted to regularly save sessions I would consider using a somewhat different approach - especially for usb-stick. Maybe without squashfs, maybe a separate partition for /home, maybe, maybe ... (I don't really know what, I haven't thought about it that much). For rare changes like updating software a longer delay is less of a problem.

raymano wrote:

Since pacman is under /var and most of the problems seem to mostly effect pacman would it make any difference if /var was separated into its own sqf file.

I don't really know. Squashfs seems pretty quick and scalable on reading but does take a while to build. If you can adjust things so that you only have to rebuild a smaller subset of the overlay, it might improve session-saving time, at the expense of some complication.

I'll pm you the report I'm sending to the aufs mailing list.

raymano · 2007-06-18 13:13:02

It seems that the pacman issues are note specific to aufs but the happen with unionfs as well.

Here's what happens with unionfs when trying to update pacman:

Targets: pacman-3.0.5-1

Total Package Size:   0.80 MB

Proceed with installation? [Y/n] y
checking package integrity... done.
cleaning up... done.
(1/1) checking for file conflicts                   [#################################################################################################] 100%
error: cannot remove file '/usr/bin/pacman': Text file busy
error: could not commit transaction
error: failed to commit transaction (transaction aborted)

Here are the mounts:

rootfs / rootfs rw 0 0
tmpfs / tmpfs rw 0 0
/dev/sdb2 /.livesys/livecd ext2 ro 0 0
/dev/loop0 /.livesys/base squashfs ro 0 0
/dev/loop1 /.livesys/etc squashfs ro 0 0
/dev/loop0 /bin squashfs ro 0 0
/dev/loop0 /lib squashfs ro 0 0
/dev/loop0 /sbin squashfs ro 0 0
/dev/loop0 /usr squashfs ro 0 0
/dev/loop1 /etc squashfs ro 0 0
none /proc proc rw 0 0
/dev/loop2 /.livesys/system squashfs ro 0 0
/dev/loop3 /.livesys/overlay squashfs ro 0 0
unionfs /bin unionfs rw,dirs=/.livesys/.bin_w=rw:/.livesys/base/bin=ro:/.livesys/system/bin=ro 0 0
unionfs /boot unionfs rw,dirs=/.livesys/.boot_w=rw:/.livesys/system/boot=ro 0 0
unionfs /etc unionfs rw,dirs=/.livesys/.etc_w=rw:/.livesys/etc/etc=ro:/.livesys/system/etc=ro 0 0
unionfs /home unionfs rw,dirs=/.livesys/.home_w=rw:/.livesys/overlay/home=ro:/.livesys/system/home=ro 0 0
unionfs /lib unionfs rw,dirs=/.livesys/.lib_w=rw:/.livesys/overlay/lib=ro:/.livesys/base/lib=ro:/.livesys/system/lib=ro 0 0
unionfs /lost+found unionfs rw,dirs=/.livesys/.lost+found_w=rw:/.livesys/system/lost+found=ro 0 0
unionfs /opt unionfs rw,dirs=/.livesys/.opt_w=rw:/.livesys/system/opt=ro 0 0
unionfs /root unionfs rw,dirs=/.livesys/.root_w=rw:/.livesys/overlay/root=ro:/.livesys/system/root=ro 0 0
unionfs /sbin unionfs rw,dirs=/.livesys/.sbin_w=rw:/.livesys/base/sbin=ro:/.livesys/system/sbin=ro 0 0
unionfs /srv unionfs rw,dirs=/.livesys/.srv_w=rw:/.livesys/system/srv=ro 0 0
unionfs /usr unionfs rw,dirs=/.livesys/.usr_w=rw:/.livesys/overlay/usr=ro:/.livesys/base/usr=ro:/.livesys/system/usr=ro 0 0
unionfs /var unionfs rw,dirs=/.livesys/.var_w=rw:/.livesys/overlay/var=ro:/.livesys/system/var=ro 0 0
unionfs /larch unionfs rw,dirs=/.livesys/.larch_w=rw:/.livesys/overlay/larch=ro 0 0
none /sys sysfs rw 0 0
none /dev ramfs rw 0 0
none /proc/bus/usb usbfs rw 0 0
none /dev/pts devpts rw 0 0
none /dev/shm tmpfs rw 0 0

lilsirecho · 2007-06-18 16:49:43

`

gradgrind · 2007-06-19 07:05:06

raymano wrote:

It seems that the pacman issues are note specific to aufs but the happen with unionfs as well.

Here's what happens with unionfs when trying to update pacman:

Targets: pacman-3.0.5-1

Total Package Size:   0.80 MB

Proceed with installation? [Y/n] y
checking package integrity... done.
cleaning up... done.
(1/1) checking for file conflicts                   [#################################################################################################] 100%
error: cannot remove file '/usr/bin/pacman': Text file busy
error: could not commit transaction
error: failed to commit transaction (transaction aborted)

Yes, I'd already tested that - as I said before, the only way seems to be to rename/copy /usr/bin/pacman to /usr/bin/pacman2 and run pacman2 -S pacman. But I think that is a different problem to the delay one.

raymano · 2007-06-19 19:39:54

gradgrind wrote:

Yes, I'd already tested that - as I said before, the only way seems to be to rename/copy /usr/bin/pacman to /usr/bin/pacman2 and run pacman2 -S pacman. But I think that is a different problem to the delay one.

Thanks. I thought maybe your latest changes had fixed it.

I think I'm going to stick with aufs for now. My work around is running pacman-optimize right after doing a pacman -Sy and before doing a pacman -Su. It seems to work.

Last edited by raymano (2007-06-19 19:40:54)

gradgrind · 2007-06-19 20:03:55

raymano wrote:

gradgrind wrote:
Yes, I'd already tested that - as I said before, the only way seems to be to rename/copy /usr/bin/pacman to /usr/bin/pacman2 and run pacman2 -S pacman. But I think that is a different problem to the delay one.
Thanks. I thought maybe your latest changes had fixed it.
I think I'm going to stick with aufs for now. My work around is running pacman-optimize right after doing a pacman -Sy and before doing a pacman -Su. It seems to work.

In my experiments (with the newer aufs) the delay only occurred after pacman -Syu. After pacman -Sy the changes were immediately visible, so (apart from updating pacman itself) using pacman -Sy followed by pacman -Su should work 'as expected'.

raymano · 2007-06-19 22:19:32

You are right. I just tested your way and it works. No need for pacman-optimize in the middle.

Thanks!

raymano · 2007-06-23 15:20:04

Hi gradgrind,

Have you ever thought of implementing a way to save sessions at runtime? Do you think it would be possible?

gradgrind · 2007-06-23 15:42:14

raymano wrote:

Hi gradgrind,
Have you ever thought of implementing a way to save sessions at runtime? Do you think it would be possible?

The possibilities are of course endless ...

There are some things which might be a bit tricky or undesirable (things left lying around by running programs, for example), but if one is careful, or restricts oneself in the scope of what is to be saved, there might be some room for manoeuvre. What did you have in mind?

raymano · 2007-06-23 16:01:53

What I have in mind is providing this capability in FaunOS. Basically an icon on the KDE desktop that would save the session from within KDE.

gradgrind · 2007-06-23 19:43:02

Is there any particular reason for wanting this?

I guess it might work, but you'd need to be a bit careful about what you try to save. Taking the merge_overlay script just as it is might not be a good idea - it might try to save things like lock-files and temporary files that would have a bad effect on a restarted session. I would suggest having a look at the aufs write branches while the system is running to see what sort of things are lurking in there, and also having a look through the merge_overlay script to see what it tries to do.

Flying Saxman · 2007-06-23 20:31:12

Is thera a possibility to automatically create an user-account (with password and group-memberships) and a root-pw during the installation (not during boot-up of larch!), so that you simply boot the stick the first time and everything runs "out-of-the-box"?

gradgrind · 2007-06-23 20:54:07

Flying Saxman wrote:

Is thera a possibility to automatically create an user-account (with password and group-memberships) and a root-pw during the installation (not during boot-up of larch!), so that you simply boot the stick the first time and everything runs "out-of-the-box"?

Err, yes, there are various possibilities, but I haven't put any of them explicitly 'into-the-box'. That was probably as clear as mud, but it's past my bedtime. The easiest is, in my opinion - and especially if you are running from usb-stick - to create your system, then run it and set up your accounts and passwords, then do a session-save before rebooting. That's of course not the first time you boot into larch, but I think it's really the easiest of the possibilities.

Flying Saxman · 2007-06-24 10:19:41

I ask, because I would like to integrate this user and his (probably very big!) home-directory into the overlay. You understand what I'm trying to say?

gradgrind · 2007-06-24 11:11:17

Flying Saxman wrote:

I ask, because I would like to integrate this user and his (probably very big!) home-directory into the overlay. You understand what I'm trying to say?

Well the only possible problem I see is the size of the overlay data. If you save it just once, that's fine, but if you want to use session-saves thereafter there are two difficulties:

1) The way the session-saving works at the moment is that all the old overlays are retained (it was originally developed for CD, where this was unavoidable), so each time a new one is created, you are using up more space on the device. This could be solved quite easily by changing the merge_overlay script to delete the old overlay.

2) Building a large squashfs file takes a long time, so you are looking at a lot of waiting. Getting around this is not so easy as the optimum solution depends on the pattern of data usage. For example maybe some of the data could be stored unsquashed, or in separate squashfs files, so that not everything needs updating with every save.

Flying Saxman · 2007-06-26 20:12:53

just to explain my situation: I am still building larch on USB for my uncle. I do some experiments with packages etc. and so I do have to reinstall larch very often (I mean big packages...). And I also want to install larch on USB-sticks for other interested people knowing me (and there are some). So I wanted to configure the whole thing once with user and a home-directory for this user so I can simply make an "mklarch -p <profile> -i -u" (or how it worked) and after this the stick is ready to run for this user without any post-configuration. Understand what I mean? (I'm a little bit tired and my english seems not to be the best today...)

gradgrind · 2007-06-27 06:08:24

Do you mean that the username should be different in each case, or would it always be, say, 'user'?
And to what extent will the user accounts be customized/configured? Will that always be the same?

In any case I think using a saved session will be the easiest way to go. You can unpack the new overlay and take out the bits you need for your new profile. Just be careful with file and directory permissions/owners.

Arch Linux

#176 2007-06-14 21:27:39

Re: live Arch CD construction kit (larch)

#177 2007-06-15 09:25:10

Re: live Arch CD construction kit (larch)

#178 2007-06-15 13:40:11

Re: live Arch CD construction kit (larch)

#179 2007-06-16 14:21:50

Re: live Arch CD construction kit (larch)

#180 2007-06-16 18:07:55

Re: live Arch CD construction kit (larch)

#181 2007-06-17 18:45:50

Re: live Arch CD construction kit (larch)

#182 2007-06-17 19:23:25

Re: live Arch CD construction kit (larch)

#183 2007-06-18 04:50:33

Re: live Arch CD construction kit (larch)

#184 2007-06-18 06:56:18

Re: live Arch CD construction kit (larch)

#185 2007-06-18 13:13:02

Re: live Arch CD construction kit (larch)

#186 2007-06-18 16:49:43

Re: live Arch CD construction kit (larch)

#187 2007-06-19 07:05:06

Re: live Arch CD construction kit (larch)

#188 2007-06-19 19:39:54

Re: live Arch CD construction kit (larch)

#189 2007-06-19 20:03:55

Re: live Arch CD construction kit (larch)

#190 2007-06-19 22:19:32

Re: live Arch CD construction kit (larch)

#191 2007-06-23 15:20:04

Re: live Arch CD construction kit (larch)

#192 2007-06-23 15:42:14

Re: live Arch CD construction kit (larch)

#193 2007-06-23 16:01:53

Re: live Arch CD construction kit (larch)

#194 2007-06-23 19:43:02

Re: live Arch CD construction kit (larch)

#195 2007-06-23 20:31:12

Re: live Arch CD construction kit (larch)

#196 2007-06-23 20:54:07

Re: live Arch CD construction kit (larch)

#197 2007-06-24 10:19:41

Re: live Arch CD construction kit (larch)

#198 2007-06-24 11:11:17

Re: live Arch CD construction kit (larch)

#199 2007-06-26 20:12:53

Re: live Arch CD construction kit (larch)

#200 2007-06-27 06:08:24

Re: live Arch CD construction kit (larch)

Board footer