[SOLVED] How to handle git lfs in PKGBUILD?

andrii_zymohliad · 2019-11-13 15:12:44

I want to make a (git-based) package for a project that uses git LFS. If I just add the repository link to the "source" variable in my PKGBUILD, makepkg would fail with the following error:

==> Extracting sources...
  -> Creating working copy of blablabla-git git repo...
Downloading assets/base.jpg (102 KB)
Error downloading object: assets/base.jpg (b982049): Smudge error: Error downloading assets/base.jpg (b982049cec5757587cf4c9fc2c54e8a0b11310934d415c23ff0b4a82d9294dc7): EOF

Errors logged to /bla/bla/bla/.git/lfs/logs/20191113T160731.301116428.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: assets/base.jpg: smudge filter lfs failed
==> ERROR: Failure while creating working copy of blablabla-git git repo
    Aborting...

I looked at makepkg scripts (/usr/share/makepkg/source/git.sh), and found that it fails doing "git clone -s" from already cloned local repo to another directory. Seems like it doesn't work with git LFS and it seems to be a git issue.

But are there any workarounds, or the only way would be to switch to archive-based package?

Last edited by andrii_zymohliad (2019-11-14 15:04:34)

Omar007 · 2019-11-13 22:20:46

Since you're not posting the PKGBUILD it's a bit hard to properly assist but my first guess is (since a lot of people forget this with VCS packages sadly...); do you have the appropriate VCS software in your makedepends array?
In this case, that would be git-lfs.

andrii_zymohliad · 2019-11-14 11:08:21

Sorry, unfortunately, I can't share the actual repository link, because it's proprietary software of the company I work at.
So here's a slightly modified version of the PKGBUILD:

# Maintainer: Andrii Zymohliad <azymohliad@pm.me>
pkgname=faceone-git
pkgver=0.0.0
pkgrel=1
pkgdesc='FaceOne library'
arch=(x86_64)
url='https://github.com/.../faceone'
license=('proprietary')
depends=()
makedepends=('cmake' 'gcc' 'git' 'git-lfs')
provides=('faceone')
source=("${pkgname}::git+ssh://git@github.com/.../faceone")
md5sums=('SKIP')

pkgver() {
    cd "${srcdir}/${pkgname}"
    printf "%s" "$(git describe --long | sed 's/\([^-]*-\)g/r\1/;s/-/./g')"
}

build() {
    mkdir -p "${srcdir}/build"
    cd "${srcdir}/build"

    cmake "../${pkgname}" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr
    cmake --build .
}

package() {
    cd "${srcdir}/build"
    cmake --build . --target install DESTDIR="${pkgdir}"
}

So yeah, I have git-lfs in makedepends, and I also have it installed already.

Is it generally possible to make packages relying on git LFS? I had a feeling that this should be some kind of common makepkg/git-lfs problem, but if it's not then I guess there's nothing anybody can help me with without an actual repository link...

Last edited by andrii_zymohliad (2019-11-14 11:09:15)

Lone_Wolf · 2019-11-14 12:02:48

As far as i know makepkg doesn't support git extensions/hooks like those used by git-lfs.

You have some options :

- Manually download the git-lfs source in the prepare() function
easy, but doesn't feel like a clean solution to me

- write a custom download agent for git-lfs and add that to makepg.conf (man makepkg.conf)
clean solution, but requires changes to config files

- convince pacman devs to add git-lfs support to makepkg
This is the 2nd time (first was in 2016) I've seen someone ask about using git-lfs in PKGBUILDS , so there doesn't seem to be much demand for it .

Omar007 · 2019-11-14 12:18:37

The odd thing is that when cloning a git-lfs repository you don't need to do anything special compared to a 'normal' git repository. Where things differ is when you're initially creating it.
For cloning you just need to have git-lfs installed and git itself handles the rest; `git clone ...` is all you'd have to do as usual.
I'm genuinely surprised that this would fail in combination with makepkg. It feels like you'd have to go out of your way to prevent this from working in the first place :S

You can decide to explicitly clone with Git LFS if you want to (`git lfs clone ...`), but there should really be no technical reason to do so.
In my eyes, the custom download agent route Lone_Wolf mentions would be the most appropriate if you wanted to implement an explicit Git LFS clone variant (e.g. to support a 'gitlfs+https://.....' source definition).
You don't even really need to change config files, you could do this within the PKGBUILD. This is not as clean but makes it a lot easier to share the PKGBUILD across systems without having to modify config files on each system first.

Last edited by Omar007 (2019-11-14 12:19:18)

andrii_zymohliad · 2019-11-14 13:25:07

Thanks to everyone for your responses!

Since the package is intended for internal use only, I would go for "easy" solution.

Lone_Wolf wrote:

- Manually download the git-lfs source in the prepare() function
easy, but doesn't feel like a clean solution to me

The problem is that it doesn't even reach prepare() function. Do you mean to remove source array at all and do the whole git clone in prepare() or is there any better way to get to prepare() (skip lfs somehow)?

andrii_zymohliad · 2019-11-14 15:03:53

So I removed sources array and did git clone in prepare() function. Now everything works And it actually made my PKGBUILD even simpler because I also had a bunch of submodules (I didn't include them in PKGBUILD above for simplicity because it reproduced the error without them too), but cloning in prepare() allowed me to handle everything with a single "git clone --recurse-submodules". Thank you, Lone_Wolf, for the idea and Omar007 for much other useful information!

Last edited by andrii_zymohliad (2019-11-14 15:05:12)

andrii_zymohliad · 2019-11-14 15:12:21

Omar007 wrote:

I'm genuinely surprised that this would fail in combination with makepkg.

From what I understand from /usr/share/makepkg/source/git.sh it did first git clone successfully, but then it tries to clone from this cloned local repo to another directory (git clone -s), end this is where it fails. It seems more like a git-lfs issue that it doesn't handle local clone properly.

Arch Linux

#1 2019-11-13 15:12:44

[SOLVED] How to handle git lfs in PKGBUILD?

#2 2019-11-13 22:20:46

Re: [SOLVED] How to handle git lfs in PKGBUILD?

#3 2019-11-14 11:08:21

Re: [SOLVED] How to handle git lfs in PKGBUILD?

#4 2019-11-14 12:02:48

Re: [SOLVED] How to handle git lfs in PKGBUILD?

#5 2019-11-14 12:18:37

Re: [SOLVED] How to handle git lfs in PKGBUILD?

#6 2019-11-14 13:25:07

Re: [SOLVED] How to handle git lfs in PKGBUILD?

#7 2019-11-14 15:03:53

Re: [SOLVED] How to handle git lfs in PKGBUILD?

#8 2019-11-14 15:12:21

Re: [SOLVED] How to handle git lfs in PKGBUILD?

Board footer