[Thought] Distributed Package Building

esquiso · 2008-05-21 23:31:54

Well, this is my first post @ BBS, so, hello! In the first place, i'd like to say that my english is very rudimentar, but I guess it's better than your Portuguese :D
This subject may be a little awkward, but it's a good way do describe my idea (at this point, it's nothing more than that).

I've seen a lot of complaints because some packages are outdated for a long time. And I see the usual excuses: "the devs have a life/wife/kids/problems", "Flag it as outdated and the dev will update it ASAP", "That package is orphaned, so you'll have to wait for someone else to update it". And yes, these are very reasonable and acceptable excuses. And yes, we also have AUR, and we can even compile it! But hey! Arch uses binary, and if it's possible to avoid compilation, I'd be very happy. This is not Gentoo :D

And my idea is a way to end with that (I hope).
I guess you're all familiarized with the concept of Distributed Computing. Projects like Folding@Home, SETI@Home, Rosetta@Home, etc. And my idea is to implement a system similar to those above.
Let me explain: the devs will do the pkgbuild. They'll upload it to a central server, and those who want to contribute and have the right conditions to build will download it their PC/Mac and makepkg'it. Then, after build it, they will upload it again to the server and that's it.
Yes, I know it's not just "and that's it". Implementing this will be slow and a little bit painfull. And for those who build packages, will have to follow strict rules (same versions of gcc, libtool, m4, etc). But, in the end, all the community will benefit!

Yes, this idea is a little bit (oh well, a lot!) green and a little bit confusing. But I think it's clear enough for you to understand it.

- esquiso

Last edited by esquiso (2008-05-21 23:35:30)

finferflu · 2008-05-21 23:38:18

Hello and welcome on the forums

For what I can see, the packages being outdated are a very minimal portion.
So the problem can be solved, for example, by voluntary Archers who upload their binaries somewhere. That has been done quite a few times for Wine.
I would consider your solution if the number of outdated packages was massive. Things seem to go quite smoothly, mainly, so I wouldn't like to touch the balance. However I'm not a developer, so that's an opinion of an outsider.

esquiso · 2008-05-21 23:46:40

finferflu wrote:

Hello and welcome on the forums
For what I can see, the packages being outdated are a very minimal portion.
So the problem can be solved, for example, by voluntary Archers who upload their binaries somewhere. That has been done quite a few times for Wine.
I would consider your solution if the number of outdated packages was massive. Things seem to go quite smoothly, mainly, so I wouldn't like to touch the balance. However I'm not a developer, so that's an opinion of an outsider.

You may be right, this is an ideia where outdated packages are numerous.
But i cannot agree when you say "very minimal portion". ArchLinux has 11 xorg-related packages outdated (Core and Testing). Sonata is outdated (and has a nasty bug in 1.5). The nvidia-96 and -71 are also outdated. 6 -mm's related packages are outdates (incluiding kernel26-mm). Pidgin is also outdated. And that's what I can remember just in a short time.

Last edited by esquiso (2008-05-21 23:46:58)

finferflu · 2008-05-21 23:52:16

I was mainly referring to iphitus' statistics for my figures.

neotuli · 2008-05-22 00:12:19

Look up pacbuild. It's a mostly working but unused implementation of what you're talking about.

bender02 · 2008-05-22 00:12:35

esquiso wrote:

You may be right, this is an ideia where outdated packages are numerous.
But i cannot agree when you say "very minimal portion". ArchLinux has 11 xorg-related packages outdated (Core and Testing). Sonata is outdated (and has a nasty bug in 1.5). The nvidia-96 and -71 are also outdated. 6 -mm's related packages are outdates (incluiding kernel26-mm). Pidgin is also outdated. And that's what I can remember just in a short time.

There's approx 100 xorg related packages. Sonata's been out since May 3 (2 weeks ago). New pidgin's out since May 17 (4 days ago). I think you're asking for too much. I guess if you want even more fresh, there's ABS for you.

LTSmash · 2008-05-22 03:01:48

bender02 wrote:

esquiso wrote:
You may be right, this is an ideia where outdated packages are numerous.
But i cannot agree when you say "very minimal portion". ArchLinux has 11 xorg-related packages outdated (Core and Testing). Sonata is outdated (and has a nasty bug in 1.5). The nvidia-96 and -71 are also outdated. 6 -mm's related packages are outdates (incluiding kernel26-mm). Pidgin is also outdated. And that's what I can remember just in a short time.
There's approx 100 xorg related packages. Sonata's been out since May 3 (2 weeks ago). New pidgin's out since May 17 (4 days ago). I think you're asking for too much. I guess if you want even more fresh, there's ABS for you.

What he doesn't want is to compile.

Let me explain: the devs will do the pkgbuild. They'll upload it to a central server, and those who want to contribute and have the right conditions to build will download it their PC/Mac and makepkg'it. Then, after build it, they will upload it again to the server and that's it.

The problem with that is not the devs providing a PKGBUILD since those already exist in ABS, but the real problem is that some evil users could use this to distribute malware...

DaNiMoTh · 2008-05-27 13:18:42

I was thinking about a distribuited compilation using distcc, like one person that send a compilation and some people that receive the source file and returns the object file: with the ADSL ( 4Mbit/256k ) that we have in Italy, an AthlonXp 1600+ lose 2 minuts compiling compiz ( 8 minuts ).

Using pacbuild instead, you need to comprehend the idea behind which is one PKGBUILD to one person.

But from the test that I've done, it doesn't work (probably for bad configurations).

Actually the best thing to do is setting an open cluster up.. like 4-5 Pentium at 2GHz connected in a LAN and to internet with a cable connection... but it is a dream

shining · 2008-05-27 14:23:48

Is build time really the main problem?

Misfit138 · 2008-05-27 14:42:47

shining wrote:

Is build time really the main problem?

I'm glad you asked this question, shining.
I am not sure I can understand why there are threads that pop up from time to time complaining about "no binary available for version x.x". With ABS and makepkg, it takes a minimal amount of time to simply build, package and install- especially when compared to some of the elaborate ideas that have been suggested and the logistics and infrastructure they would require. Arch is still the most bleeding edge distro out there overall. Is a package here and there really a dealbreaker?
Don't get me wrong, distributed computing projects pique my interest (I run f@h for 2 teams) but the obvious and pragmatic solution is simply to use ABS.
I have seen posts here by users who complain over a period of days about out-of-date packages. You cannot tell me that their machines couldn't compile and package an updated version in less time.
esquiso, the distributed computing concept is really neat. I am just not sure it is practical or necessary- in my opinion.
Am I missing the point?

Arch Linux

#1 2008-05-21 23:31:54

[Thought] Distributed Package Building

#2 2008-05-21 23:38:18

Re: [Thought] Distributed Package Building

#3 2008-05-21 23:46:40

Re: [Thought] Distributed Package Building

#4 2008-05-21 23:52:16

Re: [Thought] Distributed Package Building

#5 2008-05-22 00:12:19

Re: [Thought] Distributed Package Building

#6 2008-05-22 00:12:35

Re: [Thought] Distributed Package Building

#7 2008-05-22 03:01:48

Re: [Thought] Distributed Package Building

#8 2008-05-27 13:18:42

Re: [Thought] Distributed Package Building

#9 2008-05-27 14:23:48

Re: [Thought] Distributed Package Building

#10 2008-05-27 14:42:47

Re: [Thought] Distributed Package Building

Board footer