Can we make AUR using git clone && build to git shallow clone depth=1?

gwpl · 2024-08-18 10:01:38

Can we make AUR packages , that do git clone , to use git shallow cloning with depth = 1 by default?

Allan · 2024-08-18 10:21:19

No

gwpl · 2024-08-18 13:09:31

To me it sounds like optimization to make. I see no reason to clone everything for `makepkg` if depth=1 is all what is needed for build.

Does anyone know if :
* https://gitlab.archlinux.org/pacman/pac … epkg.sh.in
* https://gitlab.archlinux.org/pacman/pacman/-/issues

is right place to file such issue with short improvement proposal / feature request?

Trilby · 2024-08-18 13:21:16

How much difference does this really make? My aur repo clones seem to be from 4-6 KiB.

Scimmia · 2024-08-18 13:23:23

this has been brought up and rejected multiple times, including by the main pacman dev right in this thread. Opening an issue on gitlab won't do anything.

This breaks too many things.

WorMzy · 2024-08-18 13:36:43

https://wiki.archlinux.org/title/User:A … git_clones

gwpl · 2024-08-19 12:09:43

Hmm interesting! Thank you for link: https://wiki.archlinux.org/title/User:A … git_clones !

Ok, so if it may break things, then sure, deep clone as default.

However, what about "try_shallow_clone" option in PKGBUILD, so for packages with specified option it will try first shallow clone and if build fails, then fall back to default deep clone and try build again with deep clone?

That's sounds like safe "opt-in" option to introduce.

And regarding sizes: depends on repositories. Some are tiny and some are hundreds of MB (or even more in case of some Machine Learning / "AI models" related staff).

(May be less of an issue when installing ones on strong machine, but I was yesterday trying variety of combinations in dockerfile configurations, and then time multiplies.)

Last edited by gwpl (2024-08-19 12:10:27)

Awebb · 2024-08-19 12:22:36

Trilby wrote:

How much difference does this really make? My aur repo clones seem to be from 4-6 KiB.

I assume OP didn't mean cloning from aur.archlinux.org, but cloning from some git repo in the PKGBUILD itself.

Lone_Wolf · 2024-08-19 12:27:40

There are other ways to deal with large sources then shallow clones.

I build llvm trunk versions atleast once a week, and their sourcetree requires 5.6 GiB .

I set SRCDEST in makepg.conf to a location on my largest drive so everything makepkg downloads/clones is stored there.
This means I only need to download stuff once.
The next time I build the package only updates need to be downloaded.

https://wiki.archlinux.org/title/Makepk … tion_times for details .

Trilby · 2024-08-19 12:28:11

gwpl wrote:

... what about "try_shallow_clone" option in PKGBUILD

How can such an option take effect before you've retrieved the PKGBUILD? I gather from this that Awebb is correct, but I had been assuming that you were referring to cloning from the AUR as I don't see how asking about a shallow-clone of upstream content even makes any sense as 1) that would only apply to *-git packages, and 2) you can already configure that yourself for your own machine if that's what you want ... so what's the feature being requested?

Last edited by Trilby (2024-08-19 12:30:40)

Awebb · 2024-08-19 13:26:26

Trilby wrote:

that would only apply to *-git packages

To be fair, OP was explicitly asking about packages "that do git clone".

Trilby wrote:

you can already configure that yourself for your own machine if that's what you want

How'd I do that?

EDIT: There was the lingering question why anyone would want this. The Nextcloud server is ~250 MB shallow and ~7 GB full. Then again, the release tarball from the latest tag is only ~70 MB.

Last edited by Awebb (2024-08-19 13:31:28)

Trilby · 2024-08-19 13:31:40

Awebb wrote:

To be fair, OP was explicitly asking about packages "that do git clone".

True. But to be fairer, it was ambiguous - quite literally. The word "that" rather than "which" suggests a restrictive clause, however offsetting that clause with commas suggests it is a non-restrictive clause. The context initially seemed to favor the latter interpretation.

Awebb wrote:

How'd I do that?

Edit the PKGBUILD, or edit / replace /usr/share/makepkg/source/git.sh, or add another source script and add it to your makepkg.conf for the "git" protocol, or make a git wrapper script.

I'm sure there are yet other ways, but giving four different ways off the top of my head should highlight that it's quite doable.

EDIT: I forgot the fifth which was actually the first to come to my mind earlier in this thread: (shallow) clone manually in the appropriate directory next to the PKGBUILD, then just run makepkg (optionally with the -e flag) - this might be the easiest approach to turn into a simple script.

Last edited by Trilby (2024-08-19 13:40:50)

Awebb · 2024-08-19 13:37:06

Trilby wrote:

Edit the PKGBUILD, or edit / replace /usr/share/makepkg/source/git.sh, or add another source script and add it to your makepkg.conf for the "git" protocol, or make a git wrapper script.

Yes! That's what I first had in mind, but I thought there was some config flag in either makepkg.conf or "git-config" to globally force --depth, because you said "configure".

Trilby wrote:

But to be fairer, it was ambiguous - quite literally.

Indeed. I only guessed right what OP meant because it's not the first time I've seen this particular request.

Last edited by Awebb (2024-08-19 13:39:57)

daniel_shub · 2024-08-30 17:15:24

This answer on SE https://unix.stackexchange.com/a/778484/22724 suggests you can set

GITFLAGS="--mirror --depth 1 --single-branch"

No idea if that works. Based on what people other people are saying, even if it works, it is probably a bad idea.

Scimmia · 2024-08-30 17:24:11

And I believe that option gets removed in pacman/makepkg 7 anyway.

port19 · 2024-09-09 15:54:48

How unfortunate.
This makes the AUR a worse option than installing from source for any large project.
Just for reference, even with --depth 1 cloning emacs takes 82 seconds.

But with the general hostility towards the AUR by arch officials, this might one of those "It's not a bug, it's a feature" situations

Trilby · 2024-09-09 16:36:29

RE: Port19

Using the AUR is building from source (generally speaking, and pretty much in every case where this conversation would be relevant). If you want to just shallow clone the source, there are ample available ways to do so. That's not even what this topic is about. This topic is about what should be the default for makepkg.

What general hostility do you think there is towards the AUR and by which "arch officials"? You do realize the AUR is an official arch system. The individual packages are not officially supported, but the process of building from the AUR (or PKGBUILDs in general) certainly is. So who exactly do you think is being hostile to whom and how?

Last edited by Trilby (2024-09-09 16:36:57)

ddd · 2024-09-16 15:39:33

I have been happily using GITFLAGS until now, but that functionality has been removed: https://gitlab.archlinux.org/pacman/pac … quests/182

How do I set specific branch and a shallow clone of a local git repo as source without cloning the whole repo now?

Last edited by ddd (2024-09-17 10:03:34)

Lone_Wolf · 2024-09-17 10:54:04

There are (atleast) 2 viable options :

A
define a custom download agent with your preferred settings using the DLAGENTS env variable in PKGBUILD .

~~This works fine for local use, but is not very suitable for aur packages as each user needs to have the custom agent on their system..~~
EDIT: man makepkg.conf indicates DLAGENTS is an env variable that can be defined inside a PKGBUILD.

B
Don't put the sourcecode in source array, but fallback to downloading it in prepare() .

https://bbs.archlinux.org/viewtopic.php?id=298629 for an example.

Last edited by Lone_Wolf (2024-09-21 12:10:58)

Awebb · 2024-09-19 17:22:25

Will this really work? There is no git in DLAGENTS but in VSCLIENTS. Will adding a git::/usr/bin/git -whatever in DLAGENTS override whatever default git handler we have?

Lone_Wolf · 2024-09-21 12:13:03

While the example changes an existing protocol, man makepkg.conf states "any protocol can have a download agent" .

This suggests the new DLAGENT could have an arbitrary name like gitshallow to distinguish it from the regular git protocol .

Disclaimer: I have never needed to use/define DLAGENTS , so am working from documentation.

Arch Linux

#1 2024-08-18 10:01:38

Can we make AUR using git clone && build to git shallow clone depth=1?

#2 2024-08-18 10:21:19

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#3 2024-08-18 13:09:31

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#4 2024-08-18 13:21:16

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#5 2024-08-18 13:23:23

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#6 2024-08-18 13:36:43

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#7 2024-08-19 12:09:43

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#8 2024-08-19 12:22:36

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#9 2024-08-19 12:27:40

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#10 2024-08-19 12:28:11

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#11 2024-08-19 13:26:26

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#12 2024-08-19 13:31:40

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#13 2024-08-19 13:37:06

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#14 2024-08-30 17:15:24

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#15 2024-08-30 17:24:11

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#16 2024-09-09 15:54:48

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#17 2024-09-09 16:36:29

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#18 2024-09-16 15:39:33

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#19 2024-09-17 10:54:04

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#20 2024-09-19 17:22:25

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

#21 2024-09-21 12:13:03

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

Board footer