Building a Compiling Farm

Bluerider · 2013-03-07 21:57:04

Would anyone be interested in building an automated compiling farm for Arch Linux? I can see several benefits, including let Arch users test out their pkgbuilds on different architectures (by piggybacking off of qemu). Long package compilations could also be offloaded to these servers. In addition, we could have a special git repo, where pkgbuilds that clone git would be built nightly. These packages could be distributed by syncing with a special git repo.

theblazehen · 2013-03-08 08:37:13

There is a git repo for the aur. I have attempted to create a script which will git clone and compile but it always fails. Will paste it when i'm at home. Running arch install in a chroot over ssh. Fails with some kind of pipe error

Awebb · 2013-03-08 08:41:17

This is vague. Do you think about collecting some money and rent a bunch of servers of different architecture?

Bluerider · 2013-03-08 09:36:17

Sort of. I have some experience building computational clusters (did it back in university as part of my research). I had plans to build a powerful cluster out of AMD Bulldozer cores (but I graduated before I could put that into play). It wouldn't really have to be renting servers, I could just build the servers. We'd need a place to keep the stuff, a good internet connection, and a steady power supply.

We could emulate other architectures (like ARM) using qemu.

I just feel the entire Arch Linux community could benefit from having a compiling farm of sorts for testing out code. In addition, it would help alleviate compiling stress from developer machines, and automate package building for certain packages. Think of it like an automated binary updater (based on git repos) for Arch Linux, which by necessity requires a good testing environment.

We just need some seed money (I figure roughly $400 per node (Vishera CPU + SSD RAID0)), and then try to back up the server based on advertising revenues.

graysky · 2013-03-08 10:06:36

I think we'd need something to manage it if this idea has any legs. Fedora has koji for example. I read an article somewhere about an opensuse project that can compile ARCH packages... Anyway.

Last edited by graysky (2013-03-08 10:07:14)

Lone_Wolf · 2013-03-08 13:18:30

Maybe look at pkgbuild.com / brynhild that's used by arch devs and TUs ?

There chroots + devtools are used to build stuff.

Last edited by Lone_Wolf (2013-03-08 13:19:15)

Bluerider · 2013-03-08 17:35:40

Pkgbuild.com is only open to trusted users and developers. I'd like a compiling farm that's open to all Arch users. I don't trust systems in which too much power is concentrated in a selected few.

bohoomil · 2013-03-08 17:56:42

FWIW, I find the openSUSE Build Service quite satisfactory and worth checking out...

Bluerider · 2013-03-08 19:41:40

Although an in-house solution would be best, I agree we should look into the openSUSE Build service. It satisfies many of the requirements of the system I proposed. I imagine this as a method of sort of reversing cloud-computing (in some respects). Instead of sending data to remote servers and having these servers run binaries (which we assume is up to date), we send updated binaries to Arch User computers (so they have a handle on their own data). Arch has been doing this for a while but on a per developer basis (we need them to build and upload their packages).

graysky · 2013-03-08 21:17:04

bohoomil wrote:

FWIW, I find the openSUSE Build Service quite satisfactory and worth checking out...

I'm too lazy and busy right now to answer this myself: what the rules surrounding the usage of what you linked? In other words, can anyone create an account, upload their PKGBUILD and get a compiled bin somewhere?

bohoomil · 2013-03-08 22:29:24

graysky wrote:

In other words, can anyone create an account, upload their PKGBUILD and get a compiled bin somewhere?

Yes, exactly. I haven't had enough spare time to scrutinize the thing thoroughly (I'm interested how far it can be pushed as far as distro specific customizations are concerned), so I have barely built a few packages and created the repo (here and here), but I believe that having spent some time learning how to do this properly, one can better use the potential of OBS. There is a scripting interface that can help a lot. There are also packages in the AUR that provide a cli interface to the system (1, 2, 3) which makes the whole thing nicely integrated with a client's PC. Working with Arch package format is available out of the box, without any prior struggle with config options. Briefly, it simply works, offers a lot and all that wealth comes absolutely for free.

UnhappymealQQ · 2013-03-09 23:42:27

If you really need a TON of servers why not look at amazon EC2? You could easily buy as much computing time as you want, and I think their most insane system type has Nvidia GPU's for cuda even. Would solve your hardware problem, and probably be cheaper too cause you dont have to worry about power+cooling.

Last edited by UnhappymealQQ (2013-03-09 23:43:26)

Arch Linux

#1 2013-03-07 21:57:04

Building a Compiling Farm

#2 2013-03-08 08:37:13

Re: Building a Compiling Farm

#3 2013-03-08 08:41:17

Re: Building a Compiling Farm

#4 2013-03-08 09:36:17

Re: Building a Compiling Farm

#5 2013-03-08 10:06:36

Re: Building a Compiling Farm

#6 2013-03-08 13:18:30

Re: Building a Compiling Farm

#7 2013-03-08 17:35:40

Re: Building a Compiling Farm

#8 2013-03-08 17:56:42

Re: Building a Compiling Farm

#9 2013-03-08 19:41:40

Re: Building a Compiling Farm

#10 2013-03-08 21:17:04

Re: Building a Compiling Farm

#11 2013-03-08 22:29:24

Re: Building a Compiling Farm

#12 2013-03-09 23:42:27

Re: Building a Compiling Farm

Board footer