You are not logged in.

#1 2019-07-06 18:36:04

oinkl
Member
Registered: 2010-01-13
Posts: 47

Unrolling submodules into the source array

Reference: https://bbs.archlinux.org/viewtopic.php … 2#p1852912

Lone_Wolf wrote:
man 7 gitsubmodules wrote:

      A submodule is a repository embedded inside another repository. The submodule has its own history; the repository it is embedded in is
       called a superproject.

git submodule update --init --recursive

Putting that in the PKGBUILD means you're allowing the sourcecode to download things without your knowledge.
Worse, those extra repos can have their own submodules which bring in extra repos that can also have submodules etc.

Where will they be installed ?
Do they conflict with existing libraires ?
are you violating the licenses of those repos by using them this way ?
Just a few of the questions that can't be answered easily.


For  a single user that builds for their own use this command can be risky, but also very handy.

For PKGBUILDs intended for general public use it is a very bad idea.


https://wiki.archlinux.org/index.php/VC … Submodules shows an alternative that let's you control the submodules.

Look in the top git repo folder for a .gitmodules file to see what submodules are needed and where the superproject expects them..

I understand the point you're making, listing submodules gives us better control, visibility, security, etc. over what goes into the building of the package.
But slight rant - where does it end?

Case in point: aseprite depends on aseprite/skia (a forked and branched copy of google/skia). The build instructions for skia requires running $(python2 tools/git-sync-deps) which pulls in a bunch of dependencies from external git repos. Functionally, it's similar to git submodule update. If I'm already unrolling the dependencies for the top level project into the PKGBUILD source array, then ideologically, shouldn't I be doing the same for skia? Is it even feasible (I'm not even sure exactly what happens inside git-sync-deps).

DEPS files for skia:

deps = {
  "buildtools"                            : "https://chromium.googlesource.com/chromium/buildtools.git@505de88083136eefd056e5ee4ca0f01fe9b33de8",
  "common"                                : "https://skia.googlesource.com/common.git@9737551d7a52c3db3262db5856e6bcd62c462b92",
  "third_party/externals/angle2"          : "https://chromium.googlesource.com/angle/angle.git@956ab4d9fab36be9929e63829475d4d69b2c681c",
  "third_party/externals/dng_sdk"         : "https://android.googlesource.com/platform/external/dng_sdk.git@96443b262250c390b0caefbf3eed8463ba35ecae",
  "third_party/externals/egl-registry"    : "https://skia.googlesource.com/external/github.com/KhronosGroup/EGL-Registry@a0bca08de07c7d7651047bedc0b653cfaaa4f2ae",
  "third_party/externals/expat"           : "https://android.googlesource.com/platform/external/expat.git@android-6.0.1_r55",
  "third_party/externals/freetype"        : "https://skia.googlesource.com/third_party/freetype2.git@7edc937fe679d14d66f55cf6f7fa607925d38f3c",
  "third_party/externals/googletest"      : "https://android.googlesource.com/platform/external/googletest@dd43b9998e9a44a579a7aba6c1309407d1a5ed95",
  "third_party/externals/harfbuzz"        : "https://skia.googlesource.com/third_party/harfbuzz.git@1.4.2",
  "third_party/externals/icu"             : "https://chromium.googlesource.com/chromium/deps/icu.git@ec9c1133693148470ffe2e5e53576998e3650c1d",
  "third_party/externals/imgui"           : "https://skia.googlesource.com/external/github.com/ocornut/imgui.git@bc6ac8b2aee0614debd940e45bc9cd0d9b355c86",
  # TODO: remove jsoncpp after migrating clients to SkJSON
  "third_party/externals/jsoncpp"         : "https://chromium.googlesource.com/external/github.com/open-source-parsers/jsoncpp.git@1.0.0",
  "third_party/externals/libjpeg-turbo"   : "https://skia.googlesource.com/external/github.com/libjpeg-turbo/libjpeg-turbo.git@2.0.0",
  "third_party/externals/libpng"          : "https://skia.googlesource.com/third_party/libpng.git@v1.6.33",
  "third_party/externals/libwebp"         : "https://chromium.googlesource.com/webm/libwebp.git@v0.6.1",
  "third_party/externals/lua"             : "https://skia.googlesource.com/external/github.com/lua/lua.git@v5-3-4",
  "third_party/externals/microhttpd"      : "https://android.googlesource.com/platform/external/libmicrohttpd@748945ec6f1c67b7efc934ab0808e1d32f2fb98d",
  "third_party/externals/opencl-lib"      : "https://skia.googlesource.com/external/github.com/GPUOpen-Tools/common-lib-amd-APPSDK-3.0@4e6d30e406d2e5a65e1d65e404fe6df5f772a32b",
  "third_party/externals/opengl-registry" : "https://skia.googlesource.com/external/github.com/KhronosGroup/OpenGL-Registry@14b80ebeab022b2c78f84a573f01028c96075553",
  "third_party/externals/piex"            : "https://android.googlesource.com/platform/external/piex.git@bb217acdca1cc0c16b704669dd6f91a1b509c406",
  "third_party/externals/sdl"             : "https://skia.googlesource.com/third_party/sdl@5d7cfcca344034aff9327f77fc181ae3754e7a90",
  "third_party/externals/sfntly"          : "https://chromium.googlesource.com/external/github.com/googlei18n/sfntly.git@b18b09b6114b9b7fe6fc2f96d8b15e8a72f66916",
  "third_party/externals/spirv-headers"   : "https://skia.googlesource.com/external/github.com/KhronosGroup/SPIRV-Headers.git@661ad91124e6af2272afd00f804d8aa276e17107",
  "third_party/externals/spirv-tools"     : "https://skia.googlesource.com/external/github.com/KhronosGroup/SPIRV-Tools.git@e9e4393b1c5aad7553c05782acefbe32b42644bd",
  "third_party/externals/swiftshader"     : "https://swiftshader.googlesource.com/SwiftShader@d6126aea667e37d2d7b57b0a381b9f26912a21b7",
  #"third_party/externals/v8"              : "https://chromium.googlesource.com/v8/v8.git@5f1ae66d5634e43563b2d25ea652dfb94c31a3b4",
  "third_party/externals/zlib"            : "https://chromium.googlesource.com/chromium/src/third_party/zlib@ea3ba903faac98b64b2bf8de5e98cd97b335a474",
  "third_party/externals/Nima-Cpp"      : "https://github.com/2d-inc/Nima-Cpp.git@4bd02269d7d1d2e650950411325eafa15defb084",
  "third_party/externals/Nima-Math-Cpp" : "https://github.com/2d-inc/Nima-Math-Cpp.git@e0c12772093fa8860f55358274515b86885f0108",

  "../src": {
    "url": "https://chromium.googlesource.com/chromium/src.git@df35166fd2ae545e9d31701a2d9b9cb286dc5ad6",
    "condition": "checkout_chromium",
  },
}

On the other hand, the build instructions for aseprite recommends $(git submodule update --init --recursive). As the packager, I simply follow the build instructions as closely as possible. As a user, I'd expect the packager to do the absolute minimum hackery required to make the package work and insert as little of himself into the process as possible. As a user downloading the an AUR package, I'd expect him to know what goes into it. Would that be good enough?

Offline

#2 2019-07-06 18:43:23

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,463

Re: Unrolling submodules into the source array

Insane upstreams lead to insane PKGBUILDs. Having to have the source of all of those libs in-tree is insane, make no mistake.

Offline

#3 2019-07-06 18:50:23

oinkl
Member
Registered: 2010-01-13
Posts: 47

Re: Unrolling submodules into the source array

Pasting my first PKGBUILD which I wrote a two days ago for reference.
Tracing those submodules was not fun.

# Maintainer: Justin Wong <jusw85 at hotmail dot com>
# Contributor: Benoit Favre <benoit.favre@gmail.com>
# Contributor: Alexander Rødseth <rodseth@gmail.com>
# Contributor: Kamil Biduś <kamil.bidus@gmail.com>

pkgname=aseprite-git
_pkgname=aseprite
pkgver=1.2.13.r0.gaf4fd54c2
pkgrel=3
pkgdesc='Create animated sprites and pixel art'
arch=('x86_64' 'i686')
url='http://www.aseprite.org/'
license=('BSD' 'custom')
depends=('cmark' 'curl' 'libjpeg-turbo' 'giflib' 'tinyxml' 'pixman' 'libxcursor' 'fontconfig')
makedepends=('git' 'ninja' 'python2' 'clang')
conflicts=("${_pkgname}" "${_pkgname}-gpl"  )
source=("git+https://github.com/${_pkgname}/pixman.git"
        "git+https://github.com/${_pkgname}/simpleini.git"
        "git+https://github.com/${_pkgname}/gtest.git"
        "git+https://github.com/${_pkgname}/libwebp.git"
        "git+https://github.com/${_pkgname}/flic.git"
        "git+https://github.com/${_pkgname}/freetype2.git"
        "git+https://github.com/${_pkgname}/zlib.git"
        "git+https://github.com/${_pkgname}/libpng.git"
        "git+https://github.com/${_pkgname}/clip.git"
        "git+https://github.com/${_pkgname}/observable.git"
        "git+https://github.com/${_pkgname}/undo.git"
        "git+https://github.com/${_pkgname}/laf.git"
        "git+https://github.com/${_pkgname}/cmark.git"
        "git+https://github.com/${_pkgname}/harfbuzz.git"
        "git+https://github.com/${_pkgname}/libarchive.git"
        "git+https://github.com/${_pkgname}/json11.git"
        "git+https://github.com/${_pkgname}/benchmark.git"
        "git+https://github.com/${_pkgname}/giflib.git"
        "git+https://github.com/${_pkgname}/fmt.git"
        "git+https://github.com/${_pkgname}/tinyexpr.git"
        "git+https://github.com/${_pkgname}/lua"
        "git+https://github.com/${_pkgname}/stringencoders"
        "git+https://github.com/${_pkgname}/googletest"
        "git+https://github.com/${_pkgname}/skia.git#branch=aseprite-m71"
        "git+https://github.com/${_pkgname}/${_pkgname}.git"
        'desktop.patch')
sha256sums=('SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'SKIP'
            'bcb6229e42cef16a8a0273c2fce67ce81d243a085c90bd52ac15183e757ff875')
_submodules=(
    'pixman'
    'simpleini'
    'gtest'
    'libwebp'
    'flic'
    'freetype2'
    'zlib'
    'libpng'
    'clip'
    'observable'
    'undo'
    'laf'
    'cmark'
    'harfbuzz'
    'libarchive'
    'json11'
    'benchmark'
    'giflib'
    'fmt'
    'tinyexpr'
    'lua')
_submodules_path=(
    "third_party/pixman"
    "third_party/simpleini"
    "third_party/gtest"
    "third_party/libwebp"
    "src/flic"
    "third_party/freetype2"
    "third_party/zlib"
    "third_party/libpng"
    "src/clip"
    "src/observable"
    "src/undo"
    "laf"
    "third_party/cmark"
    "third_party/harfbuzz"
    "third_party/libarchive"
    "third_party/json11"
    "third_party/benchmark"
    "third_party/giflib"
    "third_party/fmt"
    "third_party/tinyexpr"
    "third_party/lua")

pkgver() {
    cd "${srcdir}/${_pkgname}"
    git describe --long --tags | sed 's/\([^-]*-g\)/r\1/;s/-/./g;s/^v//'
}

prepare() {
    cd "${srcdir}/${_pkgname}"
    git submodule init
    for (( i=0; i<${#_submodules[@]}; i++ )); do
        git config submodule.${_submodules_path[$i]}.url "${srcdir}/${_submodules[$i]}"
    done
    git submodule update

    cd laf
    git submodule init
    git config submodule.third_party/stringencoders.url "${srcdir}/stringencoders"
    git config submodule.third_party/googletest.url "${srcdir}/googletest"
    git submodule update

    cd "${srcdir}/${_pkgname}"
    patch --strip=1 --input="${srcdir}/desktop.patch"
    mkdir -p build

    cd "${srcdir}/skia"
    python2 tools/git-sync-deps
}

build() {
    cd "${srcdir}/skia"
    bin/gn gen out/Clang --args='is_debug=false is_official_build=true cc="clang" cxx="clang++" skia_use_system_expat=false skia_use_system_icu=false skia_use_system_libjpeg_turbo=false skia_use_system_libpng=false skia_use_system_libwebp=false skia_use_system_zlib=false'
    ninja -C out/Clang skia

    cd "${srcdir}/${_pkgname}/build"
    cmake \
        -DCMAKE_BUILD_TYPE=RelWithDebInfo \
        -DLAF_OS_BACKEND=skia \
        -DSKIA_DIR="$srcdir/skia" \
        -DSKIA_OUT_DIR="$srcdir/skia/out/Clang" \
        -DCMAKE_INSTALL_PREFIX=/usr \
        -DUSE_SHARED_CMARK=ON \
        -DUSE_SHARED_CURL=ON \
        -DUSE_SHARED_JPEGLIB=ON \
        -DUSE_SHARED_GIFLIB=ON \
        -DUSE_SHARED_ZLIB=ON \
        -DUSE_SHARED_LIBPNG=ON \
        -DUSE_SHARED_TINYXML=ON \
        -DUSE_SHARED_PIXMAN=ON \
        -DUSE_SHARED_FREETYPE=ON \
        -DUSE_SHARED_HARFBUZZ=ON \
        -G Ninja \
        ..
    ninja ${_pkgname}
}

package() {
    cd "${srcdir}/${_pkgname}/build"
    DESTDIR="${pkgdir}" ninja install

    # Remove extraneous files
    # https://github.com/aseprite/aseprite/issues/1574
    # https://github.com/aseprite/aseprite/issues/1602
    rm -f "${pkgdir}"/usr/bin/bsd*
    rm -f "${pkgdir}"/usr/lib/pkgconfig/libarchive.pc
    rm -f "${pkgdir}"/usr/share/man/man1/bsd*

    rm -f "${pkgdir}"/usr/bin/img2webp
    rm -fr "${pkgdir}"/usr/include/webp/
    rm -f "${pkgdir}"/usr/lib/libwebp*
    rm -fr "${pkgdir}"/usr/share/WebP/
    rm -f "${pkgdir}"/usr/share/man/man1/img2webp.1

    rm -f "${pkgdir}"/usr/include/json11.hpp
    rm -f "${pkgdir}"/usr/lib/libjson11.a
    rm -f "${pkgdir}"/usr/lib/pkgconfig/json11.pc

    find "${pkgdir}" -type d -empty -delete

    cd "${srcdir}/${_pkgname}"

    install -Dm644 "src/desktop/linux/${_pkgname}.desktop" "${pkgdir}/usr/share/applications/${_pkgname}.desktop"
    install -Dm644 "src/desktop/linux/mime/${_pkgname}.xml" "${pkgdir}/usr/share/mime/packages/${_pkgname}.xml"
    for i in {16,32,48,64,128,256}; do
        install -Dm644 "data/icons/ase${i}.png" "${pkgdir}/usr/share/icons/hicolor/${i}x${i}/apps/${_pkgname}.png"
        install -Dm644 "data/icons/doc${i}.png" "${pkgdir}/usr/share/icons/hicolor/${i}x${i}/mimetypes/image-x-${_pkgname}.png"
    done
    install -Dm644 "EULA.txt" "${pkgdir}/usr/share/licenses/${pkgname}/EULA"
    install -Dm644 "$srcdir/skia/LICENSE" "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE"
}

Offline

#4 2019-07-06 21:12:07

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,463

Re: Unrolling submodules into the source array

If you're using shared freetype, zlib, etc, why do you need the sources?

Offline

#5 2019-07-07 05:43:53

oinkl
Member
Registered: 2010-01-13
Posts: 47

Re: Unrolling submodules into the source array

I hadn't gotten around to testing which ones could be safely removed. Have updated accordingly. Thanks!

Offline

Board footer

Powered by FluxBB