You are not logged in.

#1 2016-03-20 01:55:01

jantman
Member
Registered: 2013-09-14
Posts: 7

Automatic building of packages - does it exist? interest?

Greetings,

I've been using Arch for a few years now, and I've now got four machines running it, as well as a number of friends and co-workers. I originally used yaourt, but when I had some issues with it no longer working as root, I followed the advice of a number of seasoned Arch users and people on IRC, and switched from using an AUR helper to building all of the packages I want manually, and maintaining a personal Arch repository on Amazon S3 for all of my machines, and my friends.

The one thing that bothers me is the process of manually checking for updates and then pulling them down, rebuilding all of the packages and their dependencies, and uploading them to my personal repository. Right now, I'm updating all of my machines about once a week, and spend a good chunk of time Friday nights checking AUR for updates, building any updated packages, and re-publishing to my repo.

I know that, aside from just using an AUR helper, there are a number of tools that can help with different parts of this, but there doesn't seem to be anything I could find that does all of it.

(1) For the people who are AUR package maintainers, do any of you have tooling that you can just give a list of Git repositories (i.e. the https://aur.archlinux.org/foo.git repos for your packages) and automatically build all of them, or all of them that have been updated since some event (i.e. being published to a personal repository)?

(2) If not... I'm an automation engineer by profession, so this is sort of my thing - I've spent a good chunk of the last decade automating infrastructure and software build/release processes. Would there be any interest in tooling for this?

I've started the skeleton of a project for this (https://github.com/jantman/archautorepo), with the general plan of:

  1. As input, take a file with a list of Git repository URLs. I'd have some method of marking them as trusted or not, where trusted repos will be built automatically and untrusted repos will need manual review of the PKGBUILD and other files before building... likely via a separate command-line tool.

  2. Parse metadata from all of the PKGBUILDS, filter out ones which haven't been updated since the last build, and formulate a build order list taking the dependencies into account (either using aurchain from aurutils, or by building a directed graph of them)

  3. Build each of the packages in order, installing dependencies as needed. I know that one of the various chroot tools is the preferred way of doing this, but my plan was to do it in a Docker container to maintain complete isolation from the host system - this would still build with the host's kernel version, but would allow building against the latest dependencies without changing the host, and would also provide much greater protection against possible malicious code.

  4. Once the build is done, copy the built packages to a specific location on disk, optionally run repo-add for new packages, and optionally run a command to sync it somewhere else (i.e. rsync, S3, etc.).

My overall plan is that I would have a nightly cronjob which updates all packges in a base (i.e. equivalent to a clean chroot) Docker container to their latest versions, and then executes the builds, building each package in a container that has only the base build packages plus the required dependencies. Some sort of report would be output or sent on completion.

Does such a thing exist yet? If not, is there any interest in such a thing?

Thanks,
Jason

PS - It would almost certainly be implemented in Python, as not only is that by far my strongest language, but there's also some existing code I could leverage.

PPS - I'd be very interested in finding out how packages are built for the official repos, both whether there's any tooling/code that I could use, and what the setup and installed packages are like on the machines that do the building...

Last edited by jantman (2016-03-20 01:56:24)

Offline

#2 2016-03-20 02:24:05

Awebb
Member
Registered: 2010-05-06
Posts: 6,271

Re: Automatic building of packages - does it exist? interest?

I have a couple of lines in bash, that run a git pull (or clone, if they are not there yet) on all the packages reported by "cower -u" (because I'm too lazy to implement a version check on all foreign packages against AurJson with libalpm, and cower is much faster than a pacman -Qqem), copies the folders over to /tmp/aurbuild, where I manually check those PKGBUILD's for signs of clusterfuck (you should always check AUR packages, never install them blindly) and then run a bash for loop over the folders with makepkg -si. I do not automatically catch changes in dependencies, because if I allow AUR packages to pull each other aboard without my explicit wish, then I probably don't know what's on my machine after a week.

Regarding AUR package maintenance automation, I hope nobody does that.

Offline

#3 2016-03-20 02:47:01

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Automatic building of packages - does it exist? interest?

I see potential overlap with some of my own tools so I'd like to mention some of them here. Package names are in parenthesis when different from the tool.

  • Command-line tool to check trust: bb-query_trust (bauerbill). Run bauerbill -S --aur some_aur_pkg to generate the download and build scripts to see how it's used.

  • AUR metadata parsing: AUR.RPC module in python3-aur

  • Dependency graph parsing and build ordering: see the determine_dependency_graph function in Bauerbill.py (bauerbill)

  • Building in order with dependency resolution: check how bauerbill does it. If it's not general/modular enough I can try to rework it.

  • Build isolation: patch archbuild from the devtools package to use a custom pacman.conf (and optionally a custom makepkg.conf). The custom pacman.conf should be one that contains your own repo (for dependency resolution). This is how I manage my own automatic repo releases. Using docker may provide greater protection, but potentially malicious packages should be filtered before building, and anything that's potentially malicious shouldn't be released anyway.

  • Copying built packages to a target repo, running repo-add and uploading it all easily scriptable. You may find the following useful: pkg-clean_pkg_cache (pkg_scripts) for pruning old packages, db-check_sigs (pkg_scripts) for pre-upload signature verification as a sanity check, repo-add_and_sign for signing all packages and databases at once (if you don't want to leave your key loaded in gpg-agent).

If you decide to try using my code, let me know if you need help or additional functionality and I'll see what I can do.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#4 2017-02-03 09:51:54

RemoteAdmin
Member
Registered: 2017-02-03
Posts: 1

Re: Automatic building of packages - does it exist? interest?

tl;dr Using Self hosted Gitlab with LXC to automated building of core, extra, community and multilib packages. Optionally automatically adding and removing to repositories.

I know that this Topic is a bit old but I would like to revive the Discussion.

I'm currently testing gitlab in conjunction with automated Package building with the following settings:
* A Repository for each Package set to automatically mirror the Upstream from aur.archlinux.org.
* A Gitlab Runner with a LXC Executor and a LXC Machine which gets cloned before a build and destroyed afterwards. Ensuring that each Package is build with the same bare system consisting out of base and base-devel
* A Gitlab Runner on a Repository Machine to add the new build Package to the repository

The only downside is that I currently have to manually add the .gitlab-ci.yml after each update.

An Example of the Configuration can be found here:
* Google-Chrome Package: https://gitlab.com/aur-archlinux/google-chrome
* Repository: https://repo.arch-server.org/aur/

I've often heard that the Package Maintainers are using their own Machines to build packages for the various Arch Linux Repositories. The Advantages of using a System like this is that every PKGBUILD is ensured to run on a bare ArchLinux and it automates the building and releasing of Packages.

For hosting gitlab and the repository and its builder a simple VPS at about 10$ a month will do the trick. Also this solution is scalable as multiple lxc machines can run builds at the same time.

What steps would have to be performed:
- Setting up a Server with gitlab, the repositories and builders
- Migrating current Projects from one git (svn) to a git per Projekt

Advantages:
- Automated Package Building
- Assurances that a package can be build again under the same conditions
- (Optional) Automated Package Adding and Removing to repository (including testing)
- (Optional) Automated Package Adding to archive.archlinux.org
- No Developing to create a selfmade hosting and building solution (only one script at the builder to execute makepkg and one at the repo to add package to repository)
- Package don't share a git anymore like it currently is in the aur repository

Applying this would also be a step forward for adding better diff packages. By automated adding of packages to the repository the default options may be to add a diff to the last package. Further developing to pacman and or repo-add may allow us to add multiple diffs from older versions instead of only the last to the newest.
Also the builds can be integrated with gpg to automatically sign packages and repositories. With that we wouldn't cut down security on the signing side. The builder gpg key would be added to the archlinux-keyring and be signed by the Master Keys.

Is there interessant about a configuration like that?
I have read https://wiki.archlinux.org/index.php/De … ild_System but it doesn't come up what a good solution would be and where to talk about stuff like that.

Offline

#5 2017-02-03 11:33:34

phw
Member
Registered: 2013-05-27
Posts: 318

Re: Automatic building of packages - does it exist? interest?

Even if I might get burned as a heretic, here but why not use some AUR helper again for most of this?

I mean you stopped using an automation tool to do everything manually, and now you want automation again. Why not go back and use something like pacaur to check for updates and build the packages. Only thing missing is pushing the built packages to your repository so you other machines can benefit and don't need to rebuild, but that could be done easily with a script.

Personally I find the advise to remove existing AUR helpers and build your own NIH scripts stupid unless it is for one's self-education smile

EDIT: Sorry, did not notice this thread was risen from the dead until I had submitted my post smile

Last edited by phw (2017-02-03 11:34:08)

Offline

#6 2017-02-05 16:06:06

Chocopuff
Member
Registered: 2016-10-02
Posts: 26

Re: Automatic building of packages - does it exist? interest?

Since we're on the topic of this.....

I have heard from Allan on IRC that he is in fact looking to improve building from source and integrate it with pacman. In other words, do away with the ABS and have it baked into pacman.

I'm not sure exactly if I recall correctly in the details mentioned above. He told me as a result of my complaining about it on #archlinux.

Offline

#7 2017-02-05 16:24:25

JohnBobSmith
Member
From: Canada
Registered: 2014-11-29
Posts: 804

Re: Automatic building of packages - does it exist? interest?

Do away with the ABS? But, but... I use it regularly, almost to the point of relying on it exclusively to re-build my kernel at a version that actually works. I have at least a dozen AUR packages in use, even if I've since removed the download snapshot of it. The AUR and ABS are among the best parts of the Arch Linux distribution, in addition to pacman. It's half the reason I even switched to Arch in the first place! The other half being my profound love of The Arch Way, because it suits me. I have failed to understand, thus far, what is so bad/wrong/slow/clunky/broken about the current packaging system. That's not to say there are no faults (there are!) but it certainly does not need to undergo a massive overhaul as you seem to be describing, Chocopuff. Maybe it's my Arch/Linux experience that makes things easier, but really its only three simple actions: Download a snapshot, confirm the PKGBUILD, install with makepkg -si. Wait a small amount of time, and voila! So why change something that works soooo well...? Queue Allan? smile


I am diagnosed with bipolar disorder. As it turns out, what I thought was my greatest weakness is now my greatest strength.

Everyday, I make a conscious choice to overcome my challenges and my problems. It's not easy, but its better than the alternative...

Offline

#8 2017-02-05 16:28:24

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,461

Re: Automatic building of packages - does it exist? interest?

Look at what ABS actually does and how it works. It is pretty crappy and needs to be replaced. For your use-case, using SVN directly would probably be better.

Offline

#9 2017-02-05 16:30:56

Chocopuff
Member
Registered: 2016-10-02
Posts: 26

Re: Automatic building of packages - does it exist? interest?

I'm not sure exactly what the plan is, I don't remember. I think what was meant was just the tool "abs" in itself, not the AUR/being able to download the source easily. 

For me, the issue is, none of this is automated. I'd prefer it to be a lot more automated, especially checking for updates.

I hope I'm not spewing false recollections of my memory. I do remember being told this!

Offline

#10 2017-02-05 17:39:55

Spyhawk
Member
Registered: 2006-07-07
Posts: 485

Re: Automatic building of packages - does it exist? interest?

The plan is to integrate abs directly into pacman. There are some patches for a new "pacman -B" flag on the pacman-dev mailing list that would allow it to fetch PKGBUILDs directly, but afaik this feature is not a priority and is kinda in limbo for now.

Offline

#11 2017-02-05 17:41:05

firecat53
Member
From: Lake Stevens, WA, USA
Registered: 2007-05-14
Posts: 1,542
Website

Re: Automatic building of packages - does it exist? interest?

Asp is the tool that falconindy wrote a couple of years ago as an alternative/replacement for ABS.

Offline

#12 2017-02-05 20:44:16

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: Automatic building of packages - does it exist? interest?

I use a combination of aurchain (from Alads aurutils) and Graysky's clean-chroot-manager to automatically build all of the AUR packages that I use.

Last edited by Slithery (2017-02-05 20:44:50)


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#13 2017-02-06 01:17:25

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Automatic building of packages - does it exist? interest?

Chocopuff wrote:

I'm not sure exactly what the plan is, I don't remember. I think what was meant was just the tool "abs" in itself, not the AUR/being able to download the source easily. 

For me, the issue is, none of this is automated. I'd prefer it to be a lot more automated, especially checking for updates.

I hope I'm not spewing false recollections of my memory. I do remember being told this!

For clarity you should probably check the IRC logs to be posted here.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#14 2017-02-06 02:06:51

JohnBobSmith
Member
From: Canada
Registered: 2014-11-29
Posts: 804

Re: Automatic building of packages - does it exist? interest?

JohnBobSmith wrote:

...I have failed to understand, thus far, what is so bad/wrong/slow/clunky/broken about the current packaging system.

Scimmia wrote:

Look at what ABS actually does and how it works. It is pretty crappy and needs to be replaced. ...

Do you have any examples/evidence? What seems to be the mysterious underlying problem(s) with the system? I feel like I'm missing something really obvious and that it's going right over my head, because I don't have any huge problems with the system. What I am missing? What gives?? sad


I am diagnosed with bipolar disorder. As it turns out, what I thought was my greatest weakness is now my greatest strength.

Everyday, I make a conscious choice to overcome my challenges and my problems. It's not easy, but its better than the alternative...

Offline

#15 2017-02-06 16:54:16

Spyhawk
Member
Registered: 2006-07-07
Posts: 485

Re: Automatic building of packages - does it exist? interest?

Slow and doesn't leverage the power of version control system like git. Asp is the perfect replacement while we wait for the merge of the feature with pacman code.

Offline

#16 2017-02-06 17:24:04

Awebb
Member
Registered: 2010-05-06
Posts: 6,271

Re: Automatic building of packages - does it exist? interest?

JohnBobSmith wrote:
JohnBobSmith wrote:

...I have failed to understand, thus far, what is so bad/wrong/slow/clunky/broken about the current packaging system.

Scimmia wrote:

Look at what ABS actually does and how it works. It is pretty crappy and needs to be replaced. ...

Do you have any examples/evidence? What seems to be the mysterious underlying problem(s) with the system? I feel like I'm missing something really obvious and that it's going right over my head, because I don't have any huge problems with the system. What I am missing? What gives?? sad

0. Time to follow the different mailing lists and dig through the archives.
1. Try falconindy's asp for a while and compare it to abs. Now imagine you really like everything asp does that abs does not and simulate a virtual opinion on abs.
2. You're making a big fuzz for nothing here. If you're happy with abs, then enjoy yourself. It has been there for a long time and people have been complaining since the dawn of time, but it's still there. It sucks to some extend, but it doesn't suck hard enough to be instantly replaced by something else.
3. Read that mailing list yet?

Offline

#17 2017-02-06 18:49:27

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: Automatic building of packages - does it exist? interest?

Let's just hope it doesn't end up like pacman -F...


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#18 2017-02-21 09:15:24

rolling.robot
Member
Registered: 2013-02-14
Posts: 9

Re: Automatic building of packages - does it exist? interest?

Hi everyone.

Recently I was tinkering with ROS on virtual machine under Ubuntu and wanted to install in on Arch from aur. The thing that really stopped me is that arch is rolling-release and I would have to build everything again and again once the dependencies are updated. So it would be nice to have up-to date repo for that. There are some discussions on that topic on github and some users have a working aur installation.

There is quite a big infrastructure on Jenkins that keeps everything up to date for Ubuntu. Recently it was extended to Debian and there have been a discussion to add Arch to officially supported distributions when the code base is ready for that.

One of the issues here is that ROS has some dependencies that are not the part of official repositories of Arch and such tool proposed by jantman will help to keep them up to date with their dependencies. Also parts of the code could be used to build up code for official archlinux support by ROS buildfarm.

So I think the automation proposed by jantman is really useful for archlinux development and extension. I would like to help the development if needed and the approach seems very nice to me.

Links:
ROS archwiki page: https://wiki.archlinux.org/index.php/Ros
ROS official website: http://www.ros.org/
ROS buildfarm: http://build.ros.org/
Github discussion: https://github.com/bchretien/arch-ros-stacks/issues/57
Official ROS support discussion: https://discourse.ros.org/t/who-is-usin … inux/388/7
Aur installation instructions for previous release: http://wiki.ros.org/jade/Installation/Arch

Offline

#19 2017-03-14 09:42:13

opotonil
Member
Registered: 2009-09-12
Posts: 34

Re: Automatic building of packages - does it exist? interest?

I am using GitHub + Travis CI (Docker) + aurutils to automatizate the build of a pair of AUR packages that I use. To create the repository I am using Travis CI deploying to GitHub Releases.

Code repository: https://github.com/localnet/aurci
Package repository: https://github.com/localnet/aurci/releases
Travis CI build: https://travis-ci.org/localnet/aurci

If somebody is interested I can try to do a manual explaining next steps:
- Fork the GitHub repository
- Edit package list
- Configure Travis CI

Offline

#20 2017-05-07 15:03:45

jantman
Member
Registered: 2013-09-14
Posts: 7

Re: Automatic building of packages - does it exist? interest?

I guess I'm resurrecting an old thread again, but apparently I missed a bunch of replies.

All interesting comments. It looks like a few of you are already doing things that line up with my idea, albeit with some minor differences (i.e. RemoteAdmin using GitLab and LXC... where I'd rather use Docker, and something that I can just run on my local machine via cron).

phw wrote:

Even if I might get burned as a heretic, here but why not use some AUR helper again for most of this?

I mean you stopped using an automation tool to do everything manually, and now you want automation again. Why not go back and use something like pacaur to check for updates and build the packages. Only thing missing is pushing the built packages to your repository so you other machines can benefit and don't need to rebuild, but that could be done easily with a script.

Personally I find the advise to remove existing AUR helpers and build your own NIH scripts stupid unless it is for one's self-education smile

I first stopped using AUR helpers when the Puppet module for YAOURT broke. However, then I stumbled across all of the wiki information relating to this, and the stern advice to never build an AUR package without manually verifying the PKGBUILD file first. So that's where I'm at right now.

I think my biggest concern is how dependencies are managed. There are cases where I have an AUR package "A" that depends on AUR package "B" that depends on an official package "C". At the least, I'd want an automated build system to understand that it needs to install "C" in the build container, then build and install B, then build A. Ideally, I'd like that to end up being a directed graph of dependencies, so it can handle an arbitrary number and nesting level of packages to build.

I'm also sort of torn on the manual PKGBUILD review. On one hand, it certainly would be nice if my personal package repo was always, automatically, up to date, and rebuilt when something changed, and all I had to do was a simple pacman update. On the other hand, as a stopgap, I'd be perfectly happy if I got a nightly email listing all the updates in the last day to AUR packages I'm following along with the PKGBUILD diffs, and a simple way to pull in those changes.

Offline

#21 2017-05-07 17:07:53

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: Automatic building of packages - does it exist? interest?

I think my biggest concern is how dependencies are managed. There are cases where I have an AUR package "A" that depends on AUR package "B" that depends on an official package "C". At the least, I'd want an automated build system to understand that it needs to install "C" in the build container, then build and install B, then build A. Ideally, I'd like that to end up being a directed graph of dependencies, so it can handle an arbitrary number and nesting level of packages to build.

It's not that complicated since you can offload the work to tsort. Some things do work against you, mainly the existence of split packages and the serial nature of the AUR RPC. In Bash that's still solved in <100 lines and probably half that in Python. For repository dependencies just use makepkg -s, which runs pacman -T on make/depends and pacman -S --asdeps the output.

I'm also sort of torn on the manual PKGBUILD review. On one hand, it certainly would be nice if my personal package repo was always, automatically, up to date, and rebuilt when something changed, and all I had to do was a simple pacman update. On the other hand, as a stopgap, I'd be perfectly happy if I got a nightly email listing all the updates in the last day to AUR packages I'm following along with the PKGBUILD diffs, and a simple way to pull in those changes.

Seblu has a project where he basically creates a whitelist of trusted maintainers. Then you only have to check if the maintainer changed and cross fingers the trust you put in him is warranted. Xyne also has similar functionality in bauerbill.

https://github.com/seblu/aurbot

N.B. a while back I split off the diff logic from aursync to a separate script:

https://github.com/AladW/aurutils/blob/ … n/aurfetch

Then you can run that periodically and set the diff directory to something you watch with inotifywait (which in turn runs sendmail or whatever)

Last edited by Alad (2017-05-07 17:10:45)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#22 2017-11-07 00:11:20

Sebastian256
Member
Registered: 2017-11-06
Posts: 4

Re: Automatic building of packages - does it exist? interest?

@RemoteAdmin: Thanks for the suggestion to use GitLab CI for package builds. I managed to set it up such that everything happens on GitLab.com and the repository is published using GitLab Pages. Everything can be examined in the public build logs and nothing relies on private infrastructure.

If anyone is interested, the documentation about the setup is here: https://gitlab.com/archlinux-aur/support/wikis/home

Offline

#23 2019-08-20 12:00:08

ebal
Member
From: Athens, Greece
Registered: 2009-05-26
Posts: 224
Website

Re: Automatic building of packages - does it exist? interest?

Hi,

I wrote a personal blog post on this subject:

Building Archlinux Packages in Gitlab

It may not be perfect, but I think gets the job done.

I am using this for building my personal packages and I thought It may be helpful for someone else.
Thank you


https://balaskas.gr
Linux System Engineer - Registered Linux User #420129

Offline

#24 2019-08-20 13:17:40

2ManyDogs
Forum Moderator
Registered: 2012-01-15
Posts: 4,645

Re: Automatic building of packages - does it exist? interest?

Please do not necrobump.

Closing.


How to post. A sincere effort to use modest and proper language and grammar is a sign of respect toward the community.

Offline

Board footer

Powered by FluxBB