Archie – down and bleeding

dtw · 2006-03-08 11:08:13

As some of you may or may not know Ziggy is the head dev on Archie but alas real life has finally caught up with him. Added to this my internet access has dried up and Archie is suffering as a consequence.

My strengths lie in packaging, administration, organization and scripting but Ziggy was the brains behind the whole ram disk and busybox thing.

As you will know there have been some big changes in kernel land recently and this has presented some new opportunities but also caused some problems with the way that Archie currently builds. The changes to udev have also bought some positives and negatives.

We now have an opportunity to bring some Archie functionality bang up to date. We no longer need lshwd to auto-load modules as it seems that udev can now do it all for us. We now also have pre-built scripts for building ram disks, which surely we can take advantage of.

Up until this point Archie kernels have been built with initrd and bootsplash in mind but ideally we'd like to move over to fbsplash, which means initramfs. Iphitus has been supporting these developments with changes to mkinitramfs, which we can now call from Archie's build scripts.

So, what's the problem? I don't know how busybox works and may forays into working initramfs and it have not been successful. If we can get that working we should be able to completely switch to the –archck patchset and will no longer need to use a custom build. That means that all the kernel modules we maintain for Archie will now be useful to all –archck users and we can also benefit from modules built in the official repos.

So I basically need volunteers to help us out for the short term and maybe into the longer term. You need to be familiar with busybox, bash scripting and kernel building. It would also be good if you understood how gensplash/fbsplash is currently implemented as we need to increase support for that in the busybox "thing". A decent box for compiling would also be a bonus as my 1Ghz lappy ain't up to frequent kernel rebuilds.

It would also be useful if you had experience of managing source code with CVS, administrating Arch Linux repositories and working with FTP.

Dusty · 2006-03-08 16:38:17

Ah man, I'd love to help.... but I don't have the time. I love Archie. Another thing that needs volunteering is some Archie documentation, especially for building custom CDs. As usual, I only have time to make suggestions and not make contributions (F#$#% NEWBIE!!! ;-)). But I did want to offer my support to the project. I can't even offer future support because I have no idea when I will have free time and if I do I want to focus it on magnum....

ok, enough of my life ramblings.... people out there who feel they owe me a favour can repay it by working on Archie... wooooooooo! Get to it! :-D

I want to see Archie officially integrated into Arch... maybe an Arch-dev would be interested (points at phrakture and iphitus).

Hope Archie will recognize my SATA hard drives after the udev upgrade.... :-)

Dusty

dtw · 2006-03-08 17:39:24

Aye, it should be able to do so much more. If I had proper internet access I think I could fix it myself but with once weekly internet cafe trips I just don't have the research time or download access. It's a bugger! Especially as the whole AEGIS project depends upon it!

I'd be surprised if Iphitus or Phrakture can take it up - I'd have thought they were full on busy already.

Dusty · 2006-03-08 18:25:01

Phrakture's never full on busy. He just does more and more and more and more and more and more and more.... He's like an inside out black hole -- Mr. Accomplishment. He's very wise, you see....

Dusty

tomk · 2006-03-08 22:45:49

I'd be only too happy to help out with Archie, but of the three requirements you mentioned, kernel-building would be all I could offer. I do that a lot - archck6 is building as I post, just over there
<===
Bash-scripting - complete novice, and busybox, well, I just know it exists. But if there's anything I can do, let me know.

rayjgu3 · 2006-03-09 02:42:09

i lack all the qualites you ask for ive built a kernel or 2 thats about it
i know less than next to nothing bout busybox , initramfs ,& bash scripting
but id be willing to learn . i do have a good internet connection W/ some time to spare
to do some of the work , i just need a some hand holding or a violent shove in the right direction to learn about these things

Dusty · 2006-03-09 03:01:27

Hey dudes, you only learn stuff you don't know how to do if you try.... :-) I've learned there's a HUGE difference between "I can do that" and "I know how to do that". I always say the former....

Dusty

tomk · 2006-03-09 08:27:00

Wise words, Dusty, as always. However, I'm not going to jump in and say I can do that, because I can't - not right now. I will continue to work on bash stuff, and I'll have a look at busybox, but I don't believe there's any point in giving anyone the wrong impression of my current abilities. Hopefully, someone with the right skillset will join up in the meantime.

tpowa · 2006-03-09 08:43:23

busybox is only a binary that provides different commands, like insmod etc.

gradgrind · 2006-03-09 09:26:34

Hey, people, I've done some of this already! It still needs tweaking and testing and ... but on this very list I recently announced my larch CD kit, which might be a basis for what you are looking for. It uses archck, initramfs, latest udev, DOCUMENTATION, ...
I might even have a little time to help out with this or that.
Please have a look at what I've already done, it might save someone time and effort: www.fsphost.com/gradgrind

Oh, and by the way, I've just posted a fix to lshwd (bug 4055) so that it works with Xorg7. See the posting for a repaired lshwd.c, if you can't wait for someone to redo the package.

iphitus · 2006-03-09 09:48:05

gradgrind wrote:

Hey, people, I've done some of this already! It still needs tweaking and testing and ... but on this very list I recently announced my larch CD kit, which might be a basis for what you are looking for. It uses archck, initramfs, latest udev, DOCUMENTATION, ...
I might even have a little time to help out with this or that.
Please have a look at what I've already done, it might save someone time and effort: www.fsphost.com/gradgrind
Oh, and by the way, I've just posted a fix to lshwd (bug 4055) so that it works with Xorg7. See the posting for a repaired lshwd.c, if you can't wait for someone to redo the package.

You should collaborate with dibble, combine larch and archie-scripts, to generate one set of liveCD scripts. contact, dibble, I'm sure he'd love to.

iphitus

giddygiddyBA · 2006-03-09 14:42:04

I'd be willing to help out.

dtw · 2006-03-09 15:18:14

gradgrind: great that you have a lot of this stuff working and it would be awesome if you could help us through this tricky patch - you seem to have done all the hard bits.

If we too can switch straight to a vanilla -archck kernel then larch will also be able to take advantage of the kernel modules that we maintain for Archie.

I will say that the architecture and infrastructure of Archie is way too established for any radical changes so support with in the current framework is greatly appreciated.

Were you aware of Archie before you started on larch? Seems we could have used your help from the start and made much faster progress together!

However, looking at your larch pages you are way ahead of us in documentation. A brief glance suggests to me that Archie is a bit more simple to use than larch but we do have a lot to document.

I'll check out your larch.tar.gz in the meantime you can view the archie scripts in CVS here. You can change the tag to TESTING to see my most recent efforts

gradgrind · 2006-03-09 17:19:12

I would be glad to help out, but without having a clear idea of what the aim is, what the structure of the project is (what is special/different, how the bits fit together, etc.) I find it a bit like walking in a cloud. That is why documentation is so important to me. I have been working on larch since about November but didn't want to start shouting until I could - hopefully clearly - explain to people what it is and how it's done, and of course the udev, initramfs and Xorg changes meant a lot of annoying rewriting.

I especially don't like having to read shell scripts to find out what the point of something is. I think I should first communicate the point and then describe how it's done and then even a shell script can be interesting reading matter (well, if you're that way inclined). If you look at my installer scripts you'll see that they are unfortunately not described in detail - sorry for that, they are still only a rough sketch, though they seem to work - but I've tried to describe what they do and at least they are written in python (well, they run using the python interpreter ...).

I tried Archie quite a long time ago, but in itself it was no real match for kanotix as a live CD, and I didn't manage to get a custom build going. I tried again last week, but I am really only interested in custom builds and of course it's not quite as simple as just running 'mkliveiso'. As larch fulfils all my present needs, I didn't feel like scraping through the script to find out how to make it work - especially as it was already clear that certain elements are not at all up-to-date. My initial aim was something to ease installation of an up-to-date system, with rescue capabilities.

What also annoyed me was the need to modify my system in order to build Archie - larch should build on any Linux without modification, apart from adding pacman.static (and maybe rebuilding mksquashfs, which is dynamically linked to glibc). I tested this bold statement by trying it out on Ubuntu and it worked without a hitch, just like on Arch - except that I first copied over the package cache to save having to download all the packages.

Anyway, after not finding what I was looking for in Archie I turned to the Linux-Live scripts, whose purpose and way of functioning was in my opinion a bit better described, and started working on these.

As to ease of use, it's difficult to judge your own creations. It also depends what you want to do with the system. And as the flexibility of a system increases, as it gains features and capabilities there is an inherent tendency to decreasing ease of use.

Finally, to the practical side (sigh of relief). I think what would be really helpful to anyone who wants to work on Archie (or any project really) would be a clear statement of the aims, what is there, what should be there, what should not be there (a features and non-features list?), a brief sketch of the architecture and components, tasks to do, etc. What would be especially helpful to me personally would be a comparison with larch (if you have the patience to read through my texts, try it out, etc.) so that I can get an idea of what the differences are (or should be). Maybe this is something that can be worked out through dialogue - feel free to contact me via e-mail if that helps. As you can see, I'll happily write documentation, code, debug, so long as I know what should come out at the end. Also the Archie web page could do with some more concrete and poignant content.

I'll stop now (another sigh of relief).

Sigi · 2006-03-10 08:44:11

how about adding larch to the AUR?

dtw · 2006-03-10 09:41:12

gradgrind: I think ziggy always prefered to develop Archie without any heirs and graces. It's far from perfect, he knows it, I know. Ziggy is in fact largely reluctant to make any public changes to Archie because it is so developmental.

It's quite clear that larch does something entirely different from Archie, with an entirely different purpose. Archie is about building Arch from Arch, using Arch tools and methods.

As far as the bootsplash vs fbsplash pkg conflicts - that's one major thing we want to address and my current biggest complaint.

I'm going to take a look at your scripts to see how you have tackled our major showstoppers and I'd like to contact you if I need any clarification.

Ta

gradgrind · 2006-03-10 10:13:48

dibble:

It's quite clear that larch does something entirely different from Archie, with an entirely different purpose. Archie is about building Arch from Arch, using Arch tools and methods.

I am not sure how this differs from larch, please explain so that even I can understand this! Perhaps my main problem with Archie is understanding exactly what it is trying to do/be ...

Sigi:

1) It is not really a package of a single piece of software, it is more of a howto with scripts to do it all ... I am not sure how well this ties in with the idea of an Arch package. Where should I install it all to, for example? I just use a sub-directory in my home directory.

2) It would/should contain a few binaries of components that are a bit tricky to compile (using uclibc). This is not allowed in AUR.

iphitus · 2006-03-10 10:23:29

gradgrind wrote:

dibble:
It's quite clear that larch does something entirely different from Archie, with an entirely different purpose. Archie is about building Arch from Arch, using Arch tools and methods.
I am not sure how this differs from larch, please explain so that even I can understand this! Perhaps my main problem with Archie is understanding exactly what it is trying to do/be ...

Nobody said it had to be different from larch. Maybe larch needs to be different from Archie . Archie *was* as easy as you claim larch to be now, now it is in need of some love and care as it has fallen behind in terms of the state of Arch and the trends.

Sigi:
1) It is not really a package of a single piece of software, it is more of a howto with scripts to do it all ... I am not sure how well this ties in with the idea of an Arch package. Where should I install it all to, for example? I just use a sub-directory in my home directory.

well... use the unix file structure.
scripts to bin. any libs to lib, and miscellaneous files to share, and configs to etc.

2) It would/should contain a few binaries of components that are a bit tricky to compile (using uclibc). This is not allowed in AUR.

?
There aint a rule that tricky to compile stuff isnt allowed in the AUR. Conversely, it's encouraged that you put tricky to compile stuff in as it makes it easiers for others.

iphitus

tomk · 2006-03-10 10:52:10

Actually, there IS a rule prohibiting binaries in the AUR - I believe that's what gradgrind is talking about.

Sigi · 2006-03-10 11:46:24

tomk: but there is IMHO no binary needed, or is it really impossible to write a PKGBUILD that does handle the uclibc-stuff properly? :?

Why not putting it into /usr/share/larch? Just an idea..

Cheers Sigi

dtw · 2006-03-10 13:18:26

gradgrind wrote:

dibble:
It's quite clear that larch does something entirely different from Archie, with an entirely different purpose. Archie is about building Arch from Arch, using Arch tools and methods.
I am not sure how this differs from larch

As you say larch can be built from outside Arch - maybe I should have said "from WITHIN Arch" rather than "from Arch" - which suggests constituent parts. Archie is entirely focused on building from within an Arch system and so the adding repos etc, which you seem to see as a weakness, is actually part of the "fun" (tm). The build method is not supposed to be ultra-portable, it's not supposed to "catch-all" and it's not even REALLY at a release standard. As iph says, it needs a LOT of work, which is why I am asking for help.

Dusty · 2006-03-10 16:42:07

regardless of all other issues, Archie is the fastest live CD I've seen, which is a really good reason to a) not let it die and b) borrow ideas from it.

Dusty

gradgrind · 2006-03-11 07:41:01

I'm afraid I still don't really understand where the conflict/difference lies. I would be glad to help out as far as I can. But I must gently reiterate that I can only write software or documentation if I have something like a specification.

One or two of you seem to have at least a vague idea of what Archie should and shouldn't be. Would it be possible to compile this into a feature list, or whatever, so that potential contributors have a clear idea of what the aims are? Then one could maybe sort out a task list and look for volunteers to take on the various jobs.

Dusty:
I don't know what aspect of Archie you find fast, but I must warn that my initial experiences of the new udev running from CD are not so convincing as regards speed - it adds quite a delay to start up. I put it in larch because I wanted larch to be 100% up-to-date standard Arch. lshwd and hwdetect were quicker (I don't know how they compare for correctness of module detection, though).

Dusty · 2006-03-11 15:56:25

I have no idea what Ziggy did for speed, he was always balancing speed and space -- compressed filesystems and whatnot. I'm just saying that was my primary reason for using it over Knoppix. I've looked at your site, my first impulse on reading the documentation was "this guy should write for arch". ;-) I don't have time to really get into it, but I think it would be great that if larch (any relation to the monty python flying circus episode "how to identify trees from quite a distance away"?) becomes the next semi-official Arch live CD system that it either also be optimized for speed or that such optimizations be available -- since it appears to be a very flexible system indeed.

Dusty

dtw · 2006-03-20 14:30:02

Ok, I have done some basic research on the Archie problem.

Essential the whole issue revolves around the initramfs. We would like to switch to initramfs, from initrd, for several reasons:

1) fbsplash over bootsplash
2) Can use vanilla –archck from [extra] and associated modules

Now, the basic stumbling block of my efforts to switch to an initramfs was that I merely replaced the Archie initrd (created by mkliveiso) with the initramfs created by mkinitramfs for a special kernel I built. This was totally flawed as the Archie initrd appears to have some special settings, especially dir structure, and seems to create device nodes in another way.

Also the original Archie initrd was VERY simple and only included support (as far as I could see for) for very few options – mkinitrd and mkiniramfs provide many more options and therefore increased compatibility. This is something we should take advantage of AND, for those who build really customized Archie disks, we really can reduce the size for unwanted modules.

As mkintramfs does so much of the hard work for us I think it would be ideal if we could use that to create the basic initramfs and then write some code within mkliveliso to customize it. Iphitus has already included a few patches to mkinitramfs that allow you to pass it a custom config and also to output to a custom location.

The code that creates the Archie initrd is very simple and can be found in mkliveiso. The linuxrc could be created, I think, by merging the Archie specific parts into the linuxrc file created by mkinitramfs. The custom liblinuxrc file can remain as it is and be sourced by linuxrc. These files do need updating first tho for gensplash compatibility.

That's my basic idea but, for me personally, the differences between the current Archie initrd and the initramfs created by mkinitramfs are a huge mystery. One generates nodes from linuxrc during startup, another creates them statically (one way is faster?). One has loads of symlinking to the busybox binary, the other doesn't. I'm sure there are people out there that can clarify this situation very easily and I'd appreciate it!

Ziggy always had issues with speed and maybe the original initrd was created with speed in mind but personally, with the dynamic changes and developments currently taking place with kernel I think it's a case of keep it as simple as possible and sacrifice that little bit of speed for the time being.

Arch Linux

#1 2006-03-08 11:08:13

Archie – down and bleeding

#2 2006-03-08 16:38:17

Re: Archie – down and bleeding

#3 2006-03-08 17:39:24

Re: Archie – down and bleeding

#4 2006-03-08 18:25:01

Re: Archie – down and bleeding

#5 2006-03-08 22:45:49

Re: Archie – down and bleeding

#6 2006-03-09 02:42:09

Re: Archie – down and bleeding

#7 2006-03-09 03:01:27

Re: Archie – down and bleeding

#8 2006-03-09 08:27:00

Re: Archie – down and bleeding

#9 2006-03-09 08:43:23

Re: Archie – down and bleeding

#10 2006-03-09 09:26:34

Re: Archie – down and bleeding

#11 2006-03-09 09:48:05

Re: Archie – down and bleeding

#12 2006-03-09 14:42:04

Re: Archie – down and bleeding

#13 2006-03-09 15:18:14

Re: Archie – down and bleeding

#14 2006-03-09 17:19:12

Re: Archie – down and bleeding

#15 2006-03-10 08:44:11

Re: Archie – down and bleeding

#16 2006-03-10 09:41:12

Re: Archie – down and bleeding

#17 2006-03-10 10:13:48

Re: Archie – down and bleeding

#18 2006-03-10 10:23:29

Re: Archie – down and bleeding

#19 2006-03-10 10:52:10

Re: Archie – down and bleeding

#20 2006-03-10 11:46:24

Re: Archie – down and bleeding

#21 2006-03-10 13:18:26

Re: Archie – down and bleeding

#22 2006-03-10 16:42:07

Re: Archie – down and bleeding

#23 2006-03-11 07:41:01

Re: Archie – down and bleeding

#24 2006-03-11 15:56:25

Re: Archie – down and bleeding

#25 2006-03-20 14:30:02

Re: Archie – down and bleeding

Board footer