You are not logged in.

#1 2013-03-07 21:57:04

Bluerider
Member
Registered: 2012-07-22
Posts: 97

Building a Compiling Farm

Would anyone be interested in building an automated compiling farm for Arch Linux? I can see several benefits, including let Arch users test out their pkgbuilds on different architectures (by piggybacking off of qemu). Long package compilations could also be offloaded to these servers. In addition, we could have a special git repo, where pkgbuilds that clone git would be built nightly. These packages could be distributed by syncing with a special git repo.

Offline

#2 2013-03-08 08:37:13

theblazehen
Member
Registered: 2012-12-05
Posts: 32

Re: Building a Compiling Farm

There is a git repo for the aur. I have attempted to create a script which will git clone and compile but it always fails. Will paste it when i'm at home. Running arch install in a chroot over ssh. Fails with some kind of pipe error

Offline

#3 2013-03-08 08:41:17

Awebb
Member
Registered: 2010-05-06
Posts: 6,275

Re: Building a Compiling Farm

This is vague. Do you think about collecting some money and rent a bunch of servers of different architecture?

Offline

#4 2013-03-08 09:36:17

Bluerider
Member
Registered: 2012-07-22
Posts: 97

Re: Building a Compiling Farm

Sort of. I have some experience building computational clusters (did it back in university as part of my research). I had plans to build a powerful cluster out of AMD Bulldozer cores (but I graduated before I could put that into play). It wouldn't really have to be renting servers, I could just build the servers. We'd need a place to keep the stuff, a good internet connection, and a steady power supply.

We could emulate other architectures (like ARM) using qemu.

I just feel the entire Arch Linux community could benefit from having a compiling farm of sorts for testing out code. In addition, it would help alleviate compiling stress from developer machines, and automate package building for certain packages. Think of it like an automated binary updater (based on git repos) for Arch Linux, which by necessity requires a good testing environment.

We just need some seed money (I figure roughly $400 per node (Vishera CPU + SSD RAID0)), and then try to back up the server based on advertising revenues.

Offline

#5 2013-03-08 10:06:36

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: Building a Compiling Farm

I think we'd need something to manage it if this idea has any legs.  Fedora has koji for example.  I read an article somewhere about an opensuse project that can compile ARCH packages... Anyway.

Last edited by graysky (2013-03-08 10:07:14)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#6 2013-03-08 13:18:30

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: Building a Compiling Farm

Maybe look at pkgbuild.com / brynhild that's used by arch devs and TUs ?

There chroots + devtools are used to build stuff.

Last edited by Lone_Wolf (2013-03-08 13:19:15)


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#7 2013-03-08 17:35:40

Bluerider
Member
Registered: 2012-07-22
Posts: 97

Re: Building a Compiling Farm

Pkgbuild.com is only open to trusted users and developers. I'd like a compiling farm that's open to all Arch users. I don't trust systems in which too much power is concentrated in a selected few.

Offline

#8 2013-03-08 17:56:42

bohoomil
Member
Registered: 2010-09-04
Posts: 2,376
Website

Re: Building a Compiling Farm

FWIW, I find the openSUSE Build Service quite satisfactory and worth checking out...


:: Registered Linux User No. 223384

:: github
:: infinality-bundle+fonts: good looking fonts made easy

Offline

#9 2013-03-08 19:41:40

Bluerider
Member
Registered: 2012-07-22
Posts: 97

Re: Building a Compiling Farm

Although an in-house solution would be best, I agree we should look into the openSUSE Build service. It satisfies many of the requirements of the system I proposed. I imagine this as a method of sort of reversing cloud-computing (in some respects). Instead of sending data to remote servers and having these servers run binaries (which we assume is up to date), we send updated binaries to Arch User computers (so they have a handle on their own data). Arch has been doing this for a while but on a per developer basis (we need them to build and upload their packages).

Offline

#10 2013-03-08 21:17:04

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: Building a Compiling Farm

bohoomil wrote:

FWIW, I find the openSUSE Build Service quite satisfactory and worth checking out...

I'm too lazy and busy right now to answer this myself: what the rules surrounding the usage of what you linked?  In other words, can anyone create an account, upload their PKGBUILD and get a compiled bin somewhere?


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#11 2013-03-08 22:29:24

bohoomil
Member
Registered: 2010-09-04
Posts: 2,376
Website

Re: Building a Compiling Farm

graysky wrote:

In other words, can anyone create an account, upload their PKGBUILD and get a compiled bin somewhere?

Yes, exactly. I haven't had enough spare time to scrutinize the thing thoroughly (I'm interested how far it can be pushed as far as distro specific customizations are concerned), so I have barely built a few packages and created the repo (here and here), but I believe that having spent some time learning how to do this properly, one can better use the potential of OBS. There is a scripting interface that can help a lot. There are also packages in the AUR that provide a cli interface to the system (1, 2, 3) which makes the whole thing nicely integrated with a client's PC. Working with Arch package format is available out of the box, without any prior struggle with config options. Briefly, it simply works, offers a lot and all that wealth comes absolutely for free.


:: Registered Linux User No. 223384

:: github
:: infinality-bundle+fonts: good looking fonts made easy

Offline

#12 2013-03-09 23:42:27

UnhappymealQQ
Member
Registered: 2013-02-07
Posts: 5

Re: Building a Compiling Farm

If you really need a TON of servers why not look at amazon EC2?  You could easily buy as much computing time as you want, and I think their most insane system type has Nvidia GPU's for cuda even.  Would solve your hardware problem, and probably be cheaper too cause you dont have to worry about power+cooling.

Last edited by UnhappymealQQ (2013-03-09 23:43:26)

Offline

Board footer

Powered by FluxBB