You are not logged in.

#1 2020-12-26 22:03:02

bionade24
Member
Registered: 2020-12-26
Posts: 3

ABS_CD - a CI/CD for Archlinux packages with a webinterface

Hello, I maintain a large amount of AUR packages (>300). Since I collaborate, I always faced problems how to check that they build properly and only push them to AUR if this condition was met. I had a command-line based solution and there are quite a few ones out in the wild sending mail on failed builds, but I decided a webinterface would suit my needs better and I couldn't find anything existing. (Later on I discovered some scripts for Jenkins but of course no optional AUR push and it's jenkins.) So I wrote it myself using Django and Docker. It's now ready for a first release and was tested with those 300 packages and a couple more for a half year and was improved during that time.

https://github.com/bionade24/abs_cd

Offline

#2 2020-12-28 12:16:32

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 8,785

Re: ABS_CD - a CI/CD for Archlinux packages with a webinterface

https://github.com/bionade24/abs_cd#abs-cd---cicd-for-the-arch-build-system wrote:

Packages get built in clean Docker containers,

I know very little about docker containers, how do those compare with building in a clean chroot[1] ?


[1] https://wiki.archlinux.org/index.php/De … ean_chroot


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.
Automounting / udisks2 : unnnecessary, prefer mounting manually with pmount
Aur helpers : not needed
favourite command : # systemctl set-default multi-user.target

Offline

#3 2020-12-28 14:53:32

Alad
Wiki Admin/IRC Op/TU
From: Bagelstan
Registered: 2014-05-04
Posts: 2,143
Website

Re: ABS_CD - a CI/CD for Archlinux packages with a webinterface

Docker works on multiple operating systems, devtools only works on Arch.

Last edited by Alad (2020-12-28 14:54:11)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby
Honest Alad's Package Emporium—Now with added bugs! (Grand reopening: December 1st 2018)

Offline

#4 2020-12-28 15:33:19

bionade24
Member
Registered: 2020-12-26
Posts: 3

Re: ABS_CD - a CI/CD for Archlinux packages with a webinterface

Lone_Wolf wrote:
https://github.com/bionade24/abs_cd#abs-cd---cicd-for-the-arch-build-system wrote:

Packages get built in clean Docker containers,

I know very little about docker containers, how do those compare with building in a clean chroot[1] ?


[1] https://wiki.archlinux.org/index.php/De … ean_chroot

I had used makechrootpkg and extra-x86_64_build before, but these are 2 things. One is that django, the web framwork and the repo are lying in a container. This avoid messing with the local system, makes installation very easy and allows to install this on servers running other Linux distros, too.

The second thing is that the build system uses docker instead of systemd-nspawn, which is also a container system and is used by mkchrootpkg. The extra-$arch-build script is just really slow, because it always creates new chroots, which is like always building a neww docker image. Using docker over systemd-nspawn has some advantages to me:
1. Makechrootpkg itself copy from the immutable root chroot to a mutable one. Copying more than one at once is afaik not covered. It's easily possible with a docker image.
2. Docker works nice if you run multiple container in parallel
3. It's easy to interact with docker via the socket, no shell commands on the host are needed when django runs in a container.
4. Docker doesn't mess with folders, sd-nspawn works like chroot, which also avoids file duplication
5. Docker is well-known for many people.

Note: This project is compatible with podman, too.

I still use sd-nspawn for my local build helper and to mess with linux installations and prefer it for that over chroot, VMs and docker. But I don't know anyone who would use it on servers, you would always go OCI containers or LXC, it doesn't make sense for longer deploy.

Offline

#5 2020-12-28 19:01:04

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,863

Re: ABS_CD - a CI/CD for Archlinux packages with a webinterface

Alad wrote:

Docker works on multiple operating systems, devtools only works on Arch.

Not really true... devtools works on Fedora too, if you dnf install arch-install-scripts...

Docker is merely a container technology for chrooting into an Arch operating system and running makepkg inside it. devtools is also a container technology for doing the same thing, but with systemd-nspawn. Multiple distros other than arch provide pacman/makepkg and all you need to do is be able to run makepkg --verifysource on the host, which doesn't require /var/lib/pacman/local/ to be the system package database.

bionade24 wrote:

The second thing is that the build system uses docker instead of systemd-nspawn, which is also a container system and is used by mkchrootpkg. The extra-$arch-build script is just really slow, because it always creates new chroots, which is like always building a neww docker image. Using docker over systemd-nspawn has some advantages to me:

Might depend on your filesystem... if you use btrfs it (makechrootpkg) creates subvolume copies, otherwise it uses rsync on all files.

bionade24 wrote:

1. Makechrootpkg itself copy from the immutable root chroot to a mutable one. Copying more than one at once is afaik not covered. It's easily possible with a docker image.
2. Docker works nice if you run multiple container in parallel

$ makechrootpkg -h
[...]
-l <copy>  The directory to use as the working copy of the chroot
           Useful for maintaining multiple copies
           Default: eschwartz
[...]
-T         Build in a temporary directory
[...]

-l secondcopy -> uses "secondcopy"
-l secondcopy -T -> uses "secondcopy-$PID" and deletes immediately upon completion

The latter is fairly well guaranteed to never conflict another copy.

I use -l all the time to build multiple packages at once in different terminal windows:

extra-x86_64-build
# switch tabs
extra-x86_64-build -- -l two

Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#6 2020-12-28 19:37:47

Alad
Wiki Admin/IRC Op/TU
From: Bagelstan
Registered: 2014-05-04
Posts: 2,143
Website

Re: ABS_CD - a CI/CD for Archlinux packages with a webinterface

Not really true... devtools works on Fedora too, if you dnf install arch-install-scripts...

Count me surprised, not only does Fedora's arch-install-scripts include pacstrap (unlike Debian...), they have a pacman package too.

Anyway, docker works on anything from any Linux distribution in the last decade up to Windows and OSX. (FreeBSD seems broken...). Something working on a single other Linux distribution* hardly compares.

* edit: I suppose if you're willing to build pacman and arch-install-scripts yourself, and have systemd as well, the number of Linux options increases. Seems like some effort just to use devtools though.

Last edited by Alad (2020-12-28 19:41:13)


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby
Honest Alad's Package Emporium—Now with added bugs! (Grand reopening: December 1st 2018)

Offline

#7 2020-12-28 19:50:33

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,863

Re: ABS_CD - a CI/CD for Archlinux packages with a webinterface

Implementing a custom builder on top of Docker is definitely the easy way, though equally one could implement a custom builder on top of any other container technology, including systemd-nspawn. The main benefit of devtools is the fact that it reuses packages from the host pacman cache and makepkg sources from the host $SRCDEST.

And yes, Fedora's arch-install-scripts is intended to actually be used for installing archlinux, the mkosi developers actually care about this use case. Meanwhile, Debian packages "arch-install-scripts" for the exclusive purpose of shipping "arch-chroot" as "scripts aimed at automating some menial tasks when debootstrapping". I wish they'd forked it to debian-chroot. Meh.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

Board footer

Powered by FluxBB