You are not logged in.

#1 2018-09-15 10:02:53

schard
Member
From: Hannover
Registered: 2016-05-06
Posts: 1,932
Website

[solved] AUR Packaging utility

For my own repository, I created an automated AUR package building utility.
It's purpose is to automatically build packages from AUR and add them to local repositories.
It uses two config files ~/packages.json and ~/repos.json, which can be changed by command line parameters.
These config files are expected to have a certain structure (below mine as example).
packages.json configures the packages to be built. Packages marked as not trusted (or not having the trusted flag set) are considered not trusted and will only be built when supplying --untrusted.
repos.json configures the respective repositories that shall be populated with the respective packages.

/usr/local/bin/autobuild

#! /usr/bin/env python3
#
#  (C) 2018 Richard Neumann <mail at richard dash neumann period de>
#
#  This program is free software: you can redistribute it and/or modify
#  it under the terms of the GNU General Public License as published by
#  the Free Software Foundation, either version 3 of the License, or
#  (at your option) any later version.
#
#  This program is distributed in the hope that it will be useful,
#  but WITHOUT ANY WARRANTY; without even the implied warranty of
#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#  GNU General Public License for more details.
#
#  You should have received a copy of the GNU General Public License
#  along with this program.  If not, see <http://www.gnu.org/licenses/>.
#
"""autobuild.py

Usage:
    autobuild [options]

Options:
    --pkg-config=<file>, -p     Specifies the packages config file to use.
                                [default: ~/packages.json].
    --repo-config=<file>, -r    Specifies the repositories config file to use.
                                [default: ~/repos.json].
    --untrusted, -u             Also built untrusted packages.
    --debug, -d                 Enable debug output.
    --help, -h                  Print this page.
"""
from json import loads
from logging import DEBUG, INFO, basicConfig, getLogger
from os import chdir
from pathlib import Path
from shutil import copyfile
from subprocess import check_call
from tempfile import TemporaryDirectory

from docopt import docopt


AUR_BASE = 'https://aur.archlinux.org/{}.git'
LOG_FORMAT = '[%(levelname)s] %(name)s: %(message)s'
LOGGER = getLogger(__file__)


def mtime(path):
    """Returns the mtime of the provided path."""

    return path.stat().st_mtime


def load_json(filename):
    """Loads a JSON file."""

    try:
        with open(filename, 'r') as file:
            text = file.read()
    except FileNotFoundError:
        return {}

    return loads(text)


def repoadd(name, package, sign=True):
    """Adds the respective package file."""

    command = ['/usr/bin/repo-add', f'{name}.db.tar.xz', str(package)]

    if sign:
        command.append('--sign')

    return check_call(command)


def makepkg(clean=True, sign=True):
    """Runs makepkg."""

    command = ['/usr/bin/makepkg']

    if clean:
        command.append('-C')

    if sign:
        command.append('--sign')

    return check_call(command)


def clone_package(name):
    """Clones a package from AUR."""

    command = ('/usr/bin/git', 'clone', '-o', 'aur', AUR_BASE.format(name))
    return check_call(command)


def build_packages(config, untrusted=False):
    """Builds packages from AUR and yields them."""

    for name, config in config.items():  # pylint: disable=R1704
        if config.get('trusted', False) or untrusted:
            clone_package(name)
            pkgdir = Path.cwd().joinpath(name)

            with WorkingDirectory(pkgdir):
                makepkg()
                yield Package.from_cwd()


def copy_package(package):
    """Copies the package to the current working directory."""

    cwd = Path.cwd()
    LOGGER.debug('Copying package to: %s.', cwd)
    pkgdst = cwd.joinpath(package.file.name)
    copyfile(package.file, pkgdst)

    if package.signature is not None:
        sigdst = cwd.joinpath(package.signature.name)
        copyfile(package.signature, sigdst)
    else:
        sigdst = None

    return Package(package.name, pkgdst, sigdst)


def update_repos(packages, config):
    """Updates the repositories."""

    for path, repos in config.items():
        with WorkingDirectory(path):
            LOGGER.debug('Updating %s.', Path.cwd())

            for name, pkgs in repos.items():
                for package in packages:
                    if package.name in pkgs:
                        package = copy_package(package)
                        repoadd(name, package.file.name)


def main(options):
    """Runs the script."""

    home = Path.home()
    untrusted = options['--untrusted']
    pkg_config = Path(options['--pkg-config'].replace('~', str(home)))
    repo_config = Path(options['--repo-config'].replace('~', str(home)))
    level = DEBUG if options['--debug'] else INFO
    basicConfig(level=level, format=LOG_FORMAT)
    LOGGER.debug('Using package config: %s.', pkg_config)
    LOGGER.debug('Using repo config: %s.', repo_config)
    pkg_config = load_json(pkg_config)
    repo_config = load_json(repo_config)

    with TemporaryDirectory() as tmpd:
        with WorkingDirectory(tmpd):
            packages = build_packages(config=pkg_config, untrusted=untrusted)
            update_repos(tuple(packages), config=repo_config)


class WorkingDirectory:
    """Context manager to temporarily change the working directory."""

    def __init__(self, dst):
        """Sets the destination directory."""
        self.dst = dst
        self.origin = Path.cwd()

    def __enter__(self):
        self.pushd()
        return self

    def __exit__(self, *_):
        self.popd()

    def pushd(self):
        """Changes to the working directory."""
        self.origin = Path.cwd()
        LOGGER.debug('Changing from %s to %s.', self.origin, self.dst)
        chdir(self.dst)

    def popd(self):
        """Changes back to the original directory."""
        LOGGER.debug('Changing from %s to %s.', self.dst, self.origin)
        chdir(self.origin)


class Package:  # pylint: disable=R0903
    """Represents a package."""

    __slots__ = ('name', 'file', 'signature')

    def __init__(self, name, file, signature):
        """Sets name, file and signature."""
        self.name = name
        self.file = file
        self.signature = signature

    @classmethod
    def from_cwd(cls):
        """Creates the package from the current working directory."""
        cwd = Path.cwd()
        name = cwd.name
        file, = cwd.glob('*.pkg.tar.xz')

        try:
            signature, = cwd.glob('*.pkg.tar.xz.sig')
        except ValueError:
            signature = None

        return cls(name, file, signature)


if __name__ == '__main__':
    main(docopt(__doc__))

~/packages.json

{
  "chromium-widevine": {
    "trusted": false
  },
  "python-argon2_cffi": {
    "trusted": false
  },
  "python-httpam-git": {
    "trusted": true
  },
  "python-magic-git": {
    "trusted": true
  },
  "python-mcipc-git": {
    "trusted": true
  },
  "python-peeweeplus-git": {
    "trusted": true
  },
  "python-timelib-git": {
    "trusted": true
  },
  "python-usernotify-git": {
    "trusted": true
  },
  "python-strflib-git": {
    "trusted": true
  },
  "python-functoolsplus-git": {
    "trusted": true
  }
}

~/repos.json

{
  "/srv/http/prop/mirror/pacman": {
    "rne-prop": [
      "chromium-widevine"
    ]
  },
  "/srv/http/pub/mirror/pacman": {
    "rne": [
      "python-argon2_cffi",
      "python-httpam-git",
      "python-magic-git",
      "python-mcipc-git",
      "python-peeweeplus-git",
      "python-timelib-git",
      "python-usernotify-git"
    ]
  }
}

Any feedback is welcome.

Solution:
The above project has been abandoned due to its complexity and the mentioned security issues in favor of repotool.

Last edited by schard (2019-07-04 10:45:21)

Offline

#2 2018-09-18 11:06:14

Alad
Wiki Admin/IRC Op
From: Bagelstan
Registered: 2014-05-04
Posts: 2,407
Website

Re: [solved] AUR Packaging utility

* I would trust maintainers, rather than packages, since packages can be orphaned and be taken over by random people (compare the acroread package)
* You should run repo-add with LANG=C, to avoid issues with unicode in metadata (e.g. shaman-git)
* People praise TOML for configuration formats, since it lets you easily add comments et al. Though a simple conf/tsv or even using the file system (directories) could work too.
* Resolve the .db symbolic link to the repo-add db, rather than hardcode .db.tar.xz
* Same for packages, use makepkg --packagelist instead of hardcoding extensions


Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Offline

#3 2018-09-24 14:03:40

schard
Member
From: Hannover
Registered: 2016-05-06
Posts: 1,932
Website

Re: [solved] AUR Packaging utility

@Alad
Thank you for your feedback.
I'd consider your first point an actual security issue and will definitely look into it.
Using JSON for configuration is just a personal preference since, the python library has a pretty sophisticated parser for it that automatically converts JSON objects into dicts and JSON types like lists, booleans, ints, floats, etc. into corresponding python data types.
Regarding makepkg --packagelist I learned something new today.
Thanks.

Last edited by schard (2018-09-24 14:04:18)

Offline

Board footer

Powered by FluxBB