[Solved] Using LLVM_DISTRIBUTION_COMPONENTS with GNU make

Lone_Wolf · 2023-07-22 13:12:38

This topic could be in atleast one of 4 sub-boards (newbie, aur, programming , this board) as it is deals with bash programming in PKGBUILDs, repo & aur packaging and I rate my programming skill as newbie.

archlinux llvm/clang builds use a 'trick' to reduce the number of components/targets actually being created during build() . The trick calls cmake, then retrieves the list of components to be created and removes things from it. cmake is then called a 2nd time with the edited target list to create build files with only those targets.

This speeds up the build, linking & checks a lot because only necessary things are build. The repo llvm/clang packages are built with cmake / ninja.

I want to to use that trick in aur llvm/clang packages that use cmake + make. Unfortunately I don't understand how the trick is implemented and need to dissect the code before I feel ok using it in PKGBUILDs.

LLVM_DISTRIBUTION_COMPONENTS:STRING
This variable can be set to a semi-colon separated list of LLVM build system components to install. All LLVM-based tools are components, as well as most of the libraries and runtimes. Component names match the names of the build system targets.

the repo code uses "ninja -t targets" to get a list of the components . By looking directly in the created makefile I found it had a list_install_components target (works only when cmake is present) that gives a somewhat similar list.

partial outputs of both lists :

$ ninja -t targets
ocaml_all: phony
srpm: phony
test-depends: phony
install-llvm-headers: phony
install-llvm-headers-stripped: phony
install-llvm-libraries: phony
install-llvm-libraries-stripped: phony
distribution: phony
install-distribution: phony
install-distribution-stripped: phony
package: phony
package_source: phony
edit_cache: phony
rebuild_cache: phony
list_install_components: phony
install: phony
install/local: phony
install/strip: phony
/home/panoramix/Temp/llvm-git/src/_build/CMakeFiles/ocaml_make_directory: CUSTOM_COMMAND
/home/panoramix/Temp/llvm-git/src/_build/CMakeFiles/srpm: CUSTOM_COMMAND
/home/panoramix/Temp/llvm-git/src/_build/CMakeFiles/check-all: CUSTOM_COMMAND
/home/panoramix/Temp/llvm-git/src/_build/CMakeFiles/test-depends: phony
/home/panoramix/Temp/llvm-git/src/_build/CMakeFiles/llvm-headers: phony
/home/panoramix/Temp/llvm-git/src/_build/CMakeFiles/install-llvm-headers: CUSTOM_COMMAND
/home/panoramix/Temp/llvm-git/src/_build/CMakeFiles/install-llvm-headers-stripped: CUSTOM_COMMAND
/home/panoramix/Temp/llvm-git/src/_build/CMakeFiles/install-llvm-libraries: CUSTOM_COMMAND
/home/panoramix/Temp/llvm-git/src/_build/CMakeFiles/install-llvm-libraries-stripped: CUSTOM_COMMAND
lib/Demangle/package: phony
lib/Demangle/package_source: phony
lib/Demangle/edit_cache: phony
lib/Demangle/rebuild_cache: phony

$ make list_install_components
Available install components are: "FileCheck" "LLVM" "LLVMAMDGPUAsmParser" "LLVMAMDGPUCodeGen" "LLVMAMDGPUDesc" "LLVMAMDGPUDisassembler"

the repo llvm puts the code that edits the targets in a custom function called _get_distribution_components() . see https://gitlab.archlinux.org/archlinux/ … n/PKGBUILD for the actual code.

I am still not sure if this is the right sub-board, but hope I have described the issue clearly so will stop here for now.
If moderators or posters feel another sub-board is a better fit, please state that (or act on it if you're a mod).

Last edited by Lone_Wolf (2023-08-19 14:07:39)

Lone_Wolf · 2023-07-24 12:15:53

Dissecting the used code

_get_distribution_components() {
  local target
  ninja -t targets | grep -Po 'install-\K.*(?=-stripped:)' | while read -r target; do
    case $target in
      llvm-libraries|distribution)
        continue
        ;;
      # shared libraries
      LLVM|LLVMgold)
        ;;
      # libraries needed for clang-tblgen
      LLVMDemangle|LLVMSupport|LLVMTableGen)
        ;;
      # exclude static libraries
      LLVM*)
        continue
        ;;
      # exclude llvm-exegesis (doesn't seem useful without libpfm)
      llvm-exegesis)
        continue
        ;;
    esac
    echo $target
  done
}

ninja -t targets : prints a list of all targets with each target on a separate line

grep -Po 'install-\K.*(?=-stripped:)'
-P / --perl-regexp Interpret PATTERNS as Perl-compatible regular expressions (PCREs)
o / --only-matching Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.

That appears to be a very powerful command.
Many lines of the ninja -t targets output with install also have a semi-duplicate with install and stripped in it , the \K.*(?=-stripped:) addition gets rid of those semi-duplcates.
The 'install- part seems to result in strippping out every line that doesn't include install- .

The output of make list_install_components doesn't include those parts at all, so the grep doesn't appear to be needed for that command .

The cleaned list is then piped into the while command as input where read -r processes it one line at a time .

In the case statement the clauses with continue signify unwanted components and execution jumps to esac .
Oddly the clauses without continue do nothing except descending further in the case statement and there's no catch-all clause.

The goal of the while do loop seems to be to remove some components from the list, but I fail to see how that is achieved.
I would probably list only unwanted cases in clauses and use *) as last clause to add everything wanted to the var with semi-colon separated list that holds the desired components.

The above analysis is my interpretation of what happens in the _get_distribution_components() and could very well be wrong.
Please comment / explain .

Added:

I misinterpreted continue, it doesn't jump to esac but to the end of the loop, done .
That indicates the echo $target line is what adds wanted components to the local target var.

If I'm correct with that it makes the structure of this codepiece clear.

Last edited by Lone_Wolf (2023-07-24 13:45:55)

Lone_Wolf · 2023-08-12 13:56:40

I have found a help target in the Makefile that creates a list that looks much closer to the one produced by ninja.

I just build llvm-minimal-git and ran some commands in $srcdir/_build .

$ make list_install_components > makecomponents.txt
$ wc -w makecomponents.txt
434 makecomponents.txt
$
$ make help > makehelp.txt
$ wc -l makehelp.txt 
3612 makehelp.txt
$

That suggests I'm much better of by trimming the components list.
However there are 260+ *.a static files in _build/lib which take up approx 1 GiB of space, so the potential gain is rather big.

Next : try to adjust the code to work with make help.

Lone_Wolf · 2023-08-12 15:21:19

reduce-targets-for-llvm-minimal-git test script.

#! /bin/sh
# FOR TESTING ONLY
# run from within llvm-minimal-git $srcdir/_build folder .


_get_distribution_components() {
local target
local include

make help | grep -Po 'install-\K.*(?=-stripped)' | while read -r target; do
    case $target in
        llvm-libraries|clang-libraries|clang-tidy-headers|distribution)
            include=0
            ;;
      # shared libraries
        LLVM|LLVMgold)
            include=1
            ;;
        # libraries needed for clang-tblgen
        LLVMDemangle|LLVMSupport|LLVMTableGen)
            include=1
            ;;
      # exclude static libraries
        LLVM*)
            include=0
            ;;
        # exclude llvm-exegesis (doesn't seem useful without libpfm)
        llvm-exegesis)
            include=0
            ;;
        clang|clangd|clang-*)
            include=1
            ;;
        clang*|findAllSymbols)
            include=0
            ;;
        # Anything not covered above is included
        *)
            include=1
    esac
    if [ include=1 ]; then echo $target
    fi
done
}

distribution_components=$(_get_distribution_components | paste -sd\;)
echo "$distribution_components" > distribution_components.list

Notes :
the script combines the logic from the _get_distribution_component() functions from llvm & clang since llvm-minimal-git builds both in the same package

I managed to get rid of continue and make the logical structure inside the loop easier to understand.

By setting include=1 before the case statement I can avoid using the catch-all *) but I feel having a catch-all condition in a case-statements should be mandatory .

Last edited by Lone_Wolf (2023-08-12 15:21:56)

Lone_Wolf · 2023-08-12 21:19:41

I've put the above function in the llvm-minimal-git PKGBUILD, made the other adjustments needed and tested.

build log still mentions linking cxx static library mutiple times , .a files are present in build folder etc .
I haven't done precise timings but didn't notice shorter compiletimes.

Same when I switched the PKGBUILD to ninja and even building repo llvm created & linked static librairies during buildtime.

At this point I feel the trick complicates the PKGBUILD for questionable gains.

If someone wants to try for themselves, here's the PKGBUILD I used.

# Maintainer: Lone_Wolf <lone_wolf@klaas-de-kat.nl>
# Contributor:	blackmoon3  <https://github.com/blacksky3>
# Contributor: yurikoles <root@yurikoles.com>
# Contributor: bearoso <bearoso@gmail.com>
# Contributor: Luchesar V. ILIEV <luchesar%2eiliev%40gmail%2ecom>
# Contributor: Anders Bergh <anders@archlinuxppc.org>
# Contributor: Armin K. <krejzi at email dot com>
# Contributor: Christian Babeux <christian.babeux@0x80.ca>
# Contributor: Jan "heftig" Steffens <jan.steffens@gmail.com>
# Contributor: Evangelos Foutras <evangelos@foutrelis.com>
# Contributor: Hesiod (https://github.com/hesiod)
# Contributor: Roberto Alsina <ralsina@kde.org>
# Contributor: Thomas Dziedzic < gostrc at gmail >
# Contributor: Tomas Lindquist Olsen <tomas@famolsen.dk>
# Contributor: Tomas Wilhelmsson <tomas.wilhelmsson@gmail.com>


pkgbase=llvm-minimal-git
pkgname=(llvm-minimal-git llvm-libs-minimal-git clang-minimal-git clang-libs-minimal-git)
pkgver=18.0.0_r471068.da56750f82a6
pkgrel=1
arch=('x86_64')
url="https://llvm.org/"
license=('custom:Apache 2.0 with LLVM Exception')
makedepends=(git cmake libffi libedit ncurses libxml2
             libxcrypt python python-setuptools ninja)
source=("git+https://github.com/llvm/llvm-project.git"
                'local://llvm-config.h'
)
                
md5sums=('SKIP'
         '295c343dcd457dc534662f011d7cff1a')
sha512sums=('SKIP'
            '75e743dea28b280943b3cc7f8bbb871b57d110a7f2b9da2e6845c1c36bf170dd883fca54e463f5f49e0c3effe07fbd0db0f8cf5a12a2469d3f792af21a73fcdd')
            

options=('!lto')
# explicitly disable lto to reduce number of build hangs / test failures

# LIT by default uses all available cores. this can lead to heavy stress on systems making them unresponsive.
# It can also happen that the kernel oom killer interferes and kills important tasks.
# A reasonable value to avoid these issues appears to be 75% of available cores.
# LITFLAGS is an env vars that can be used to achieve this. It should be set on command line or in files read by your shell on login (like .bashrc ) .
# example for systems with 24 cores
# LITFLAGS="-j 18"
# NOTE: It's your responbility to validate the value of LITFLAGS. If unsure, don't set it.

_get_distribution_components() {
local target
local include

ninja -t targets | grep -Po 'install-\K.*(?=-stripped:)' | while read -r target; do
    case $target in
        llvm-libraries|clang-libraries|clang-tidy-headers|distribution)
            include=0
            ;;
      # shared libraries
        LLVM|LLVMgold)
            include=1
            ;;
        # libraries needed for clang-tblgen
        LLVMDemangle|LLVMSupport|LLVMTableGen)
            include=1
            ;;
      # exclude static libraries
        LLVM*)
            include=0
            ;;
        # exclude llvm-exegesis (doesn't seem useful without libpfm)
        llvm-exegesis)
            include=0
            ;;
        clang|clangd|clang-*)
            include=1
            ;;
        clang*|findAllSymbols)
            include=0
            ;;
        # Anything not covered above is included
        *)
            include=1
    esac
    if [ include=1 ]; then echo $target
    fi
done
}


pkgver() {
    cd llvm-project/llvm

    # This will almost match the output of `llvm-config --version` when the
    # LLVM_APPEND_VC_REV cmake flag is turned on. The only difference is
    # dash being replaced with underscore because of Pacman requirements.
    local _pkgver=$(awk -F 'MAJOR |MINOR |PATCH |)' \
            'BEGIN { ORS="." ; i=0 } \
             /set\(LLVM_VERSION_/ { print $2 ; i++ ; if (i==2) ORS="" } \
             END { print "\n" }' \
             CMakeLists.txt)_r$(git rev-list --count HEAD).$(git rev-parse --short HEAD)
    echo "${_pkgver}"
}

build() {
    
    local cmake_args=(
        -G "Unix makefiles"
        -D CMAKE_BUILD_TYPE=Release
        -D CMAKE_INSTALL_PREFIX=/usr
        -D LLVM_BINUTILS_INCDIR=/usr/include
        -D FFI_INCLUDE_DIR=$(pkg-config --variable=includedir libffi)
        -D LLVM_VERSION_SUFFIX=""
        -D LLVM_APPEND_VC_REV=ON
        -D LLVM_HOST_TRIPLE=$CHOST
        -D LLVM_TARGETS_TO_BUILD="AMDGPU;X86"
        -D LLVM_BUILD_LLVM_DYLIB=ON
        -D LLVM_LINK_LLVM_DYLIB=ON
        -D CLANG_LINK_CLANG_DYLIB=ON
        -D LLVM_DYLIB_COMPONENTS="all"
        -D LLVM_INSTALL_UTILS=ON
        -D LLVM_ENABLE_RTTI=ON
        -D LLVM_ENABLE_FFI=ON
        -D LLVM_USE_PERF=ON
        -D LLVM_INCLUDE_BENCHMARKS=OFF
        -D LLVM_INCLUDE_EXAMPLES=OFF
        -D LLVM_BUILD_DOCS=OFF
        -D LLVM_INCLUDE_DOCS=OFF
        -D LLVM_ENABLE_OCAMLDOC=OFF
        -D LLVM_ENABLE_SPHINX=OFF
        -D LLVM_ENABLE_DOXYGEN=OFF
        -D LLVM_ENABLE_BINDINGS=OFF
        -D LLVM_ENABLE_PROJECTS="compiler-rt;clang-tools-extra;clang"
        -D LLVM_ENABLE_DUMP=ON
        -D LLVM_LIT_ARGS="$LITFLAGS"" -sv --ignore-fail"
  )


    cmake -B _tmpbuild -S "$srcdir"/llvm-project/llvm "${cmake_args[@]}" -Wno-dev
    
    pushd "$srcdir"/_build
    local distribution_components=$(_get_distribution_components | paste -sd\;)
    popd
    test -n "$distribution_components"
    cmake_args+=(-D LLVM_DISTRIBUTION_COMPONENTS="$distribution_components")
    
    cmake -B _build -S "$srcdir"/llvm-project/llvm "${cmake_args[@]}" -Wno-dev
    make  -C _build
}

check() {
    make -C _build check-llvm
    make -C _build check-clang
    make -C _build check-clang-tools
}

package_llvm-minimal-git() {
    pkgdesc="Collection of modular and reusable compiler and toolchain technologies"
    depends=(llvm-libs-minimal-git="$pkgver-$pkgrel" clang-libs-minimal-git="$pkgver-$pkgrel")
    provides=('llvm')
    conflicts=('llvm')
    optdepends=('python: for using lit (LLVM Integrated Tester)'
                          'python-setuptools: for using lit'
    )

    make -C _build DESTDIR="$pkgdir" install-distribution

    # Include lit for running lit-based tests in other projects
    pushd "$srcdir"/llvm-project/llvm/utils/lit 
    python3 setup.py install --root="$pkgdir" -O1
    # -O1 ensures the python files for lit will be optimized
    popd
    
    # Remove files which conflict with repo llvm-libs
    rm "$pkgdir"/usr/lib/{LLVMgold,lib{LLVM,LTO}}.so
    rm "$pkgdir"/usr/lib/libRemarks.so

    # prepare folders in srcdir to store files that are placed in other package_*() functions
    mkdir -p "$srcdir"{/llvm-libs/usr/lib,/clang-libs/usr/lib,/spirv-llvm-translator/usr/{bin,include/LLVMSPIRVLib/,lib/pkgconfig}}
    
    # The llvm runtime libraries go into llvm-libs-minimal-git
    mv -f "$pkgdir"/usr/lib/lib{LLVM-*.so,LTO.so.*,Remarks.so.*} "$srcdir"/llvm-libs/usr/lib

    # The clang runtime libraries go into clang-libs-minimal-git
    mv -f "$pkgdir"/usr/lib/libclang{,-cpp}.so* "$srcdir"/clang-libs/usr/lib

    # clang-minimal-git files go to a separate package
    mkdir -p "$srcdir"/clang/usr/{bin,include,lib,lib/cmake,share}
    mv -f "$pkgdir"/usr/lib/{libear,libscanbuild,libclang*.a,clang} "$srcdir"/clang/usr/lib
    mv -f "$pkgdir"/usr/lib/cmake/clang "$srcdir"/clang/usr/lib/cmake/
    mv -f "$pkgdir"/usr/include/{clang,clang-tidy,clang-c} "$srcdir"/clang/usr/include/
    # llvm project uses /usr/libexec and setting CMAKE_INSTALL_LIBEXECDIR doesn't affect that.
    # to comply with archlinux packaging standards we have to move some files manually
    mv -f "$pkgdir"/usr/libexec/* "$srcdir"/clang/usr/lib/clang
    rmdir "$pkgdir"/usr/libexec
    mv -f "$pkgdir"/usr/bin/{analyze-build,c-index-test,clang*,diagtool,find-all-symbols,git-clang-format,hmaptool,intercept-build,modularize,pp-trace,run-clang-tidy,scan-build,scan-build-py,scan-view} "$srcdir"/clang/usr/bin/
    mv -f "$pkgdir"/usr/share/{clang,man,opt-viewer,scan-build,scan-view} "$srcdir"/clang/usr/share/
    
#    sed -i 's|libexec|lib/clang|' "$pkgdir"/usr/bin/scan-build

  
    if [[ $CARCH == x86_64 ]]; then
        # Needed for multilib (https://bugs.archlinux.org/task/29951)
        # Header stub is taken from Fedora
        mv "$pkgdir"/usr/include/llvm/Config/llvm-config{,-64}.h
        install -Dm644 "$srcdir"/llvm-config.h "$pkgdir"/usr/include/llvm/Config/llvm-config.h
    fi

    install -Dm644 "$srcdir"/llvm-project/llvm/LICENSE.TXT "$pkgdir"/usr/share/licenses/"$pkgname"/LICENSE.TXT
    
}

package_llvm-libs-minimal-git() {
    pkgdesc="LLVM runtime libraries, trunk version"
    depends=('gcc-libs' 'zlib' 'libffi' 'libedit' 'ncurses' 'libxml2' 'llvm-libs')
    # some applications expect llvmgold.so to be present and fail badly if it isn't.
    # adding llvm-libs as depend is the easiest solution

    cp --preserve --recursive "$srcdir"/llvm-libs/* "$pkgdir"/

    install -Dm644 "$srcdir"/llvm-project/llvm/LICENSE.TXT "$pkgdir"/usr/share/licenses/$pkgname/LICENSE.TXT
}

_python_optimize() {
  python -m compileall "$@"
  python -O -m compileall "$@"
  python -OO -m compileall "$@"
}

package_clang-minimal-git(){
  pkgdesc='C language family frontend for LLVM (git version)'
  depends=(llvm-libs-minimal-git clang-libs-minimal-git gcc)
  optdepends=('openmp: OpenMP support in clang with -fopenmp'
              'python: for scan-view, scan-build, git-clang-format, clang-rename and python bindings'
              'llvm-minimal-git: referenced by some clang headers')
  conflicts=(compiler-rt clang)
  provides=(compiler-rt clang clang-analyzer clang-tools-extra)

    cp --preserve --recursive "$srcdir"/clang/* "$pkgdir"/

    # Move scanbuild-py into site-packages
    local site_packages=$(python -c "import site; print(site.getsitepackages()[0])")
    install -d "$pkgdir/$site_packages"
    mv "$pkgdir"/usr/lib/{libear,libscanbuild} "$pkgdir/$site_packages/"

    sed -i 's|libexec|lib/clang|' \
        "$pkgdir/usr/bin/scan-build" \
        "$pkgdir/$site_packages/libscanbuild/analyze.py"

    # Compile Python scripts
    _python_optimize "$pkgdir"/usr/share "$pkgdir"/$site_packages


    install -Dm644 "$srcdir"/llvm-project/compiler-rt/LICENSE.TXT "$pkgdir"/usr/share/licenses/$pkgname/compiler-rt-LICENSE.TXT
    install -Dm644 "$srcdir"/llvm-project/clang-tools-extra/LICENSE.TXT "$pkgdir"/usr/share/licenses/$pkgname/clang-tools-extra-LICENSE.TXT
    install -Dm644 "$srcdir"/llvm-project/clang/LICENSE.TXT "$pkgdir"/usr/share/licenses/$pkgname/clang-LICENSE.TXT

}

package_clang-libs-minimal-git() {
    pkgdesc="clang runtime libraries, trunk version"
    depends=(llvm-libs-minimal-git="$pkgver-$pkgrel")
    # clang-libs doesn't exist in repos at this time, but it's cleaner to provide & conflict it.
    # TODO: Once repo clang-libs comes into existence, verify if changes are needed to this package
    provides=('clang-libs')
    conflicts=('clang-libs')
    
    cp --preserve --recursive "$srcdir"/clang-libs/* "$pkgdir"/

    install -Dm644 "$srcdir"/llvm-project/llvm/LICENSE.TXT "$pkgdir"/usr/share/licenses/$pkgname/LICENSE.TXT
}


# vim:set ts=2 sw=2 et:

Lone_Wolf · 2023-08-15 12:25:18

After adding some debugging statements I figured out the condition I use in the if-statement doesn't do what I expected.

include=1 is not evaluated as a comparison and always reports true, thus nothing is removed from the targets to build .

I now use $include -eq 1 and have started a fresh build (with ccache cleared before starting).

Added:

Normal build took 62:49 , reduced targets build took 60:33 and was 3.6% faster.

The binary size differs a lot though :
normal 174.2
reduced 88.8 GiB

For now the conclusion is the build/compiletime benefit is small, but the decrease in binary size is huge .

Last edited by Lone_Wolf (2023-08-15 16:26:37)

Lone_Wolf · 2023-08-17 07:20:22

No issues while testing this new build, so I've uploaded it to aur, https://aur.archlinux.org/packages/llvm-minimal-git .

Looking back the biggest problem was finding a suitable list created by make. Next step is to apply this to the lib32 variant.

Lone_Wolf · 2023-08-19 14:07:14

I adjusted lib32-llvm-minimal-git without issues, also uploaded to aur.

Arch Linux

#1 2023-07-22 13:12:38

[Solved] Using LLVM_DISTRIBUTION_COMPONENTS with GNU make

#2 2023-07-24 12:15:53

Re: [Solved] Using LLVM_DISTRIBUTION_COMPONENTS with GNU make

#3 2023-08-12 13:56:40

Re: [Solved] Using LLVM_DISTRIBUTION_COMPONENTS with GNU make

#4 2023-08-12 15:21:19

Re: [Solved] Using LLVM_DISTRIBUTION_COMPONENTS with GNU make

#5 2023-08-12 21:19:41

Re: [Solved] Using LLVM_DISTRIBUTION_COMPONENTS with GNU make

#6 2023-08-15 12:25:18

Re: [Solved] Using LLVM_DISTRIBUTION_COMPONENTS with GNU make

#7 2023-08-17 07:20:22

Re: [Solved] Using LLVM_DISTRIBUTION_COMPONENTS with GNU make

#8 2023-08-19 14:07:14

Re: [Solved] Using LLVM_DISTRIBUTION_COMPONENTS with GNU make

Board footer