SOLVED -Best way to deal with large repos that you'd want to customize

Batou · 2017-04-03 19:17:51

Hi all,

I'm hoping one of you AUR masters could please tell me what the best way is to deal with Github repos that are large, change often, and have many forks that I'd like to try, customize and work on.

My plan was basically to use cower to fetch the PKGBUILD and then manually shallow-copy clone of a github repo (because running makepkg on its own results in gigabytes since it does a full clone), do some changes, grab and apply some patches and then run makepkg.

However, I immediately ran into an issue:

 $ makepkg -s
==> Making package: opencv-git r906.691db24-1 (Mon Apr  3 15:11:03 EDT 2017)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
==> ERROR: /tmp/opencv-git/opencv is not a clone of https://github.com/opencv/opencv
    Aborting...

So, makepkg doesn't like the manually downloaded repo for some reason (even though it's just a 'git clone --depth 1 ...' of the repo).

I did some googling and came across a StackExchange answer talking about custom DLAGENTs for makepkg. https://unix.stackexchange.com/question … llow-clone Apparently, you have to hard-code each repo and its custom DLAGENT into your /etc/makepkg.conf file.

The workflow I'm looking for is basically this: I'd like to have ~/Sources/ folder and inside of it I'd like to have several AUR packages for things I'm working on or for things that change often and are big. Then to upgrade, I'd like to just do a "git pull" for every package (Apply a patch if needed), do a 'makepkg -s' and 'pacman -S ..'. Does anyone do something like this? Is there a better way? Please share your tips.

Thank you!

Last edited by Batou (2017-04-03 20:21:05)

Trilby · 2017-04-03 19:51:25

makepkg -e

Also:

Batou wrote:

... I'd like to just do a "git pull" for every package (Apply a patch if needed), do a 'makepkg -s' and 'pacman -S ..'

You don't need the -s flag for makepkg on subsequent runs. You've already installed the dependencies and as you haven't changed the PKGBUILD, the deps could not have changed. Though there is no harm in this flag, is just not needed (edit: I suppose it might be if there are build deps that you've removed).

As for the next part, what would `pacman -S ...` be for? Do you mean `pacman -U`? If so, why are you not just using makepkg's -i flag?

So overall: `makepkg -ei`

Then: `man makepkg`

Batou · 2017-04-03 20:03:58

Thank you Trilby! I just tried it on a smaller package and ran into another issue. Maybe you or someone else can tell me what I'm doing wrong.

~/Sources $ mkcd Iosevka
~/S/Iosevka $ cower -dd ttf-iosevka-git
:: ttf-iosevka-git downloaded to /home/batou/Sources/Iosevka
~/S/Iosevka $ ll
total 4.0K
drwxrwxr-x 2 batou batou 4.0K Feb 23 16:12 ttf-iosevka-git
~/S/Iosevka $ cd ttf-iosevka-git
~/S/I/ttf-iosevka-git $ ll
total 8.0K
-rw-rw-r-- 1 batou batou  650 Feb 23 16:12 .SRCINFO
-rw-rw-r-- 1 batou batou 1.1K Feb 23 16:12 PKGBUILD
~/S/I/ttf-iosevka-git $ git clone --depth 1 https://github.com/be5invis/Iosevka.git
Cloning into 'Iosevka'...
remote: Counting objects: 78, done.
remote: Compressing objects: 100% (76/76), done.
remote: Total 78 (delta 2), reused 17 (delta 1), pack-reused 0
Unpacking objects: 100% (78/78), done.
~/S/I/ttf-iosevka-git $ ll
total 16K
-rw-rw-r--  1 batou batou  650 Feb 23 16:12 .SRCINFO
-rw-rw-r--  1 batou batou 1.1K Feb 23 16:12 PKGBUILD
drwxr-xr-x 11 batou batou 4.0K Apr  3 15:59 Iosevka
drwxr-xr-x  2 batou batou 4.0K Apr  3 15:59 src
~/S/I/ttf-iosevka-git $ makepkg -e
==> Making package: ttf-iosevka-git r906.695db24-1 (Mon Apr  3 15:59:16 EDT 2017)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> WARNING: Using existing $srcdir/ tree
==> Starting pkgver()...
/home/batou/Sources/Iosevka/ttf-iosevka-git/PKGBUILD: line 18: cd: Iosevka: No such file or directory
==> ERROR: pkgver is not allowed to be empty.
==> ERROR: pkgver() generated an invalid version:

and when you take a look PKGBUILD...

17 pkgver() {
18  cd "Iosevka"
19  printf "r%s.%s" "$(git rev-list --count HEAD)" "$(git rev-parse --short HEAD)"
20 }

Hmm... why can't it cd into the folder when it's there?

Last edited by Batou (2017-04-03 20:05:23)

Trilby · 2017-04-03 20:08:03

Clone into a src folder. Makepkg cannot cd into the "Iosevka" directory because it is not under the src directory. And - while I could decipher them this time - please use standard commands if you want help, not personal aliases:

cower -d some-package
cd /path/to/some-package
git clone https://some-url.tld/repo-name.git src/repo-name --depth 1
makepkg -esi

Alternatively, if you want to clone into the top level directory, then make the src directory and the child symlink:

cower -d some-package
cd /path/to/some-package
git clone https://some-url.tld/repo-name.git --depth 1
mkdir src
ln -srt src repo-name
makepkg -esi

This is closer to what makepkg itself does.

(editted as per below)

Last edited by Trilby (2017-04-03 20:24:27)

Batou · 2017-04-03 20:19:56

Thank you Trilby! I think it's just "src/repo-name" instead of "src/repo-name.git" in the git command above. Other than that, this works great.

PS: ll is just ls -l

Trilby · 2017-04-03 20:24:54

Yes, sorry - I was typing that a bit too fast.

I also learned, while testing this, that git will create parent directories for targets itself: that's handy (i.e., you don't need to create the directory "src" yourself if you clone into it).

Batou · 2017-04-03 23:03:12

Thanks Trilby! This will now be my workflow. I think it's great. I wonder why some AUR utility doesn't do something similar... it seems like a no-brainer for large packages.

Lone_Wolf · 2017-04-04 10:13:11

Maybe because there's a clean solution that only requires you to do a full clone once ?

man makepg.conf wrote:

SRCDEST="/path/to/directory"
If this value is not set, downloaded source files will only be stored in the current directory. Many people like to keep all
source files in a central location for easy cleanup, so this path can be set here.

Around 2012 I did a full clone of mesa git master, and it's been updated through makepkg hundreds of times.

Batou · 2017-04-04 17:08:29

Lone_Wolf wrote:

Maybe because there's a clean solution that only requires you to do a full clone once ?
man makepg.conf wrote:
SRCDEST="/path/to/directory"
If this value is not set, downloaded source files will only be stored in the current directory. Many people like to keep all
source files in a central location for easy cleanup, so this path can be set here.
Around 2012 I did a full clone of mesa git master, and it's been updated through makepkg hundreds of times.

That is awesome, Lone_Wolf! Thank you so much for this tip.

Do you know if some AUR utilities like pacaur or aura would work with maekpkg setup like this?

Lone_Wolf · 2017-04-05 11:07:09

The SRCDEST value in makepkg.conf can be overriden by setting an environment variable with the same name.

So it is possible for an AUR helper to override that folder.
If some aur helpers do that, their documentation should make it very clear they override pacman/makepkg behaviour.

A fast way to verify :
set srcdest folder to some pristine folder in makepkg.conf , build something (pref with small source download) with aur helper and check if the downloaded sources are present in the folder you set.

Last edited by Lone_Wolf (2017-04-05 11:08:09)

Batou · 2017-04-05 16:03:12

Thank you Lone_Wolf! Much appreciated!

Arch Linux

#1 2017-04-03 19:17:51

SOLVED -Best way to deal with large repos that you'd want to customize

#2 2017-04-03 19:51:25

Re: SOLVED -Best way to deal with large repos that you'd want to customize

#3 2017-04-03 20:03:58

Re: SOLVED -Best way to deal with large repos that you'd want to customize

#4 2017-04-03 20:08:03

Re: SOLVED -Best way to deal with large repos that you'd want to customize

#5 2017-04-03 20:19:56

Re: SOLVED -Best way to deal with large repos that you'd want to customize

#6 2017-04-03 20:24:54

Re: SOLVED -Best way to deal with large repos that you'd want to customize

#7 2017-04-03 23:03:12

Re: SOLVED -Best way to deal with large repos that you'd want to customize

#8 2017-04-04 10:13:11

Re: SOLVED -Best way to deal with large repos that you'd want to customize

#9 2017-04-04 17:08:29

Re: SOLVED -Best way to deal with large repos that you'd want to customize

#10 2017-04-05 11:07:09

Re: SOLVED -Best way to deal with large repos that you'd want to customize

#11 2017-04-05 16:03:12

Re: SOLVED -Best way to deal with large repos that you'd want to customize

Board footer