You are not logged in.

#1 2024-08-18 10:01:38

gwpl
Member
From: Warsaw, Poland
Registered: 2010-07-10
Posts: 35

Can we make AUR using git clone && build to git shallow clone depth=1?

Can we make AUR packages , that do git clone , to use git shallow cloning with depth = 1 by default?

Offline

#2 2024-08-18 10:21:19

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,463
Website

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

No

Offline

#3 2024-08-18 13:09:31

gwpl
Member
From: Warsaw, Poland
Registered: 2010-07-10
Posts: 35

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

To me it sounds like optimization to make. I see no reason to clone everything for `makepkg` if depth=1 is all what is needed for build.

Does anyone know if :
* https://gitlab.archlinux.org/pacman/pac … epkg.sh.in
* https://gitlab.archlinux.org/pacman/pacman/-/issues

is right place to file such issue with short improvement proposal / feature request?

Offline

#4 2024-08-18 13:21:16

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,234
Website

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

How much difference does this really make?  My aur repo clones seem to be from 4-6 KiB.


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#5 2024-08-18 13:23:23

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,959

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

this has been brought up and rejected multiple times, including by the main pacman dev right in this thread. Opening an issue on gitlab won't do anything.

This breaks too many things.

Offline

#6 2024-08-18 13:36:43

WorMzy
Administrator
From: Scotland
Registered: 2010-06-16
Posts: 12,327
Website

Re: Can we make AUR using git clone && build to git shallow clone depth=1?


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#7 2024-08-19 12:09:43

gwpl
Member
From: Warsaw, Poland
Registered: 2010-07-10
Posts: 35

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

Hmm interesting! Thank you for link: https://wiki.archlinux.org/title/User:A … git_clones !

Ok, so if it may break things, then sure, deep clone as default.

However, what about "try_shallow_clone" option in PKGBUILD, so for packages with specified option it will try first shallow clone and if build fails, then fall back to default deep clone and try build again with deep clone?

That's sounds like safe "opt-in" option to introduce.

And regarding sizes: depends on repositories. Some are tiny and some are hundreds of MB (or even more in case of some Machine Learning / "AI models" related staff).

(May be less of an issue when installing ones on strong machine, but I was yesterday trying variety of combinations in dockerfile configurations, and then time multiplies.)

Last edited by gwpl (2024-08-19 12:10:27)

Offline

#8 2024-08-19 12:22:36

Awebb
Member
Registered: 2010-05-06
Posts: 6,525

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

Trilby wrote:

How much difference does this really make?  My aur repo clones seem to be from 4-6 KiB.

I assume OP didn't mean cloning from aur.archlinux.org, but cloning from some git repo in the PKGBUILD itself.

Offline

#9 2024-08-19 12:27:40

Lone_Wolf
Administrator
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 12,789

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

There are other ways to deal with large sources then shallow clones.

I build llvm trunk versions atleast once a week, and their sourcetree requires 5.6 GiB .

I set SRCDEST in makepg.conf to a location on my largest drive so everything makepkg downloads/clones is stored there.
This means I only need to download stuff once.
The next time I build the package only updates need to be downloaded.

https://wiki.archlinux.org/title/Makepk … tion_times for details .


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Offline

#10 2024-08-19 12:28:11

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,234
Website

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

gwpl wrote:

... what about "try_shallow_clone" option in PKGBUILD

How can such an option take effect before you've retrieved the PKGBUILD?  I gather from this that Awebb is correct, but I had been assuming that you were referring to cloning from the AUR as I don't see how asking about a shallow-clone of upstream content even makes any sense as 1) that would only apply to *-git packages, and 2) you can already configure that yourself for your own machine if that's what you want ... so what's the feature being requested?

Last edited by Trilby (2024-08-19 12:30:40)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#11 2024-08-19 13:26:26

Awebb
Member
Registered: 2010-05-06
Posts: 6,525

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

Trilby wrote:

that would only apply to *-git packages

To be fair, OP was explicitly asking about packages "that do git clone".

Trilby wrote:

you can already configure that yourself for your own machine if that's what you want

How'd I do that?

EDIT: There was the lingering question why anyone would want this. The Nextcloud server is ~250 MB shallow and ~7 GB full. Then again, the release tarball from the latest tag is only ~70 MB.

Last edited by Awebb (2024-08-19 13:31:28)

Offline

#12 2024-08-19 13:31:40

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,234
Website

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

Awebb wrote:

To be fair, OP was explicitly asking about packages "that do git clone".

True.  But to be fairer, it was ambiguous - quite literally.  The word "that" rather than "which" suggests a restrictive clause, however offsetting that clause with commas suggests it is a non-restrictive clause.  The context initially seemed to favor the latter interpretation.

Awebb wrote:

How'd I do that?

Edit the PKGBUILD, or edit / replace /usr/share/makepkg/source/git.sh, or add another source script and add it to your makepkg.conf for the "git" protocol, or make a git wrapper script.

I'm sure there are yet other ways, but giving four different ways off the top of my head should highlight that it's quite doable.

EDIT: I forgot the fifth which was actually the first to come to my mind earlier in this thread: (shallow) clone manually in the appropriate directory next to the PKGBUILD, then just run makepkg (optionally with the -e flag) - this might be the easiest approach to turn into a simple script.

Last edited by Trilby (2024-08-19 13:40:50)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#13 2024-08-19 13:37:06

Awebb
Member
Registered: 2010-05-06
Posts: 6,525

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

Trilby wrote:

Edit the PKGBUILD, or edit / replace /usr/share/makepkg/source/git.sh, or add another source script and add it to your makepkg.conf for the "git" protocol, or make a git wrapper script.

Yes! That's what I first had in mind, but I thought there was some config flag in either makepkg.conf or "git-config" to globally force --depth, because you said "configure".

Trilby wrote:

But to be fairer, it was ambiguous - quite literally.

Indeed. I only guessed right what OP meant because it's not the first time I've seen this particular request.

Last edited by Awebb (2024-08-19 13:39:57)

Offline

#14 2024-08-30 17:15:24

daniel_shub
Member
Registered: 2012-06-21
Posts: 87

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

This answer on SE https://unix.stackexchange.com/a/778484/22724 suggests you can set

GITFLAGS="--mirror --depth 1 --single-branch"

No idea if that works. Based on what people other people are saying, even if it works, it is probably a bad idea.

Offline

#15 2024-08-30 17:24:11

Scimmia
Fellow
Registered: 2012-09-01
Posts: 11,959

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

And I believe that option gets removed in pacman/makepkg 7 anyway.

Offline

#16 2024-09-09 15:54:48

port19
Member
Registered: 2024-09-09
Posts: 1

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

How unfortunate.
This makes the AUR a worse option than installing from source for any large project.
Just for reference, even with --depth 1 cloning emacs takes 82 seconds.

But with the general hostility towards the AUR by arch officials, this might one of those "It's not a bug, it's a feature" situations

Offline

#17 2024-09-09 16:36:29

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,234
Website

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

RE: Port19

Using the AUR is building from source (generally speaking, and pretty much in every case where this conversation would be relevant).  If you want to just shallow clone the source, there are ample available ways to do so.  That's not even what this topic is about.  This topic is about what should be the default for makepkg.

What general hostility do you think there is towards the AUR and by which "arch officials"?  You do realize the AUR is an official arch system.  The individual packages are not officially supported, but the process of building from the AUR (or PKGBUILDs in general) certainly is.  So who exactly do you think is being hostile to whom and how?

Last edited by Trilby (2024-09-09 16:36:57)


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Offline

#18 2024-09-16 15:39:33

ddd
Member
Registered: 2021-11-21
Posts: 1

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

I have been happily using GITFLAGS until now, but that functionality has been removed: https://gitlab.archlinux.org/pacman/pac … quests/182

How do I set specific branch and a shallow clone of a local git repo as source without cloning the whole repo now?

Last edited by ddd (2024-09-17 10:03:34)

Offline

#19 2024-09-17 10:54:04

Lone_Wolf
Administrator
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 12,789

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

There are (atleast) 2 viable options :

A
define a custom download agent with your preferred settings using the DLAGENTS env variable in PKGBUILD .

This works fine for local use, but is not very suitable for aur packages as each user needs to have the custom agent on their system..
EDIT: man makepkg.conf indicates DLAGENTS is an env variable that can be defined inside a PKGBUILD.

B
Don't put the sourcecode in source array, but fallback to downloading it in prepare() .

https://bbs.archlinux.org/viewtopic.php?id=298629 for an example.

Last edited by Lone_Wolf (2024-09-21 12:10:58)


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Offline

#20 2024-09-19 17:22:25

Awebb
Member
Registered: 2010-05-06
Posts: 6,525

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

Will this really work? There is no git in DLAGENTS but in VSCLIENTS. Will adding a git::/usr/bin/git -whatever in DLAGENTS override whatever default git handler we have?

Offline

#21 2024-09-21 12:13:03

Lone_Wolf
Administrator
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 12,789

Re: Can we make AUR using git clone && build to git shallow clone depth=1?

While the example changes an existing protocol, man makepkg.conf states "any protocol can have a download agent" .

This suggests the new DLAGENT could have an arbitrary name like gitshallow to distinguish it from the regular git protocol .

Disclaimer: I have never needed to use/define DLAGENTS , so am working from documentation.


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Offline

Board footer

Powered by FluxBB