Pass argument to git before checking out a package

dmidge · 2018-08-26 22:41:55

Hi,

I have three questions:
1- When there is a package that requires some parameters for git, that needs to be set before checking out the package; how to do that within the PKGBUILD? For instance, I have a package that I try to build. But it seems too "big". The error message is:

error: RPC failed; curl 56 OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
fatal: The remote end hung up unexpectedly

I know I can solve this problem by passing this argument in git:

git config --global http.postBuffer 5242808000

But I need to do that, before makepkg tries to fetch anything... So, how to do that?

2- And how to check out a specific branch of a package? I guess I can change the branch in the "prepare()" part, but is there a cleaner way of doing it?

3- How to do the equivalent of?

git clone --recursive

Still in the PKGBUILD?

Cheers

Last edited by dmidge (2018-08-27 02:31:50)

eschwartz · 2018-08-27 04:32:11

dmidge wrote:

Hi,

I have three questions:
1- When there is a package that requires some parameters for git, that needs to be set before checking out the package; how to do that within the PKGBUILD? For instance, I have a package that I try to build. But it seems too "big". The error message is:
error: RPC failed; curl 56 OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
fatal: The remote end hung up unexpectedly
I know I can solve this problem by passing this argument in git:
git config --global http.postBuffer 5242808000
But I need to do that, before makepkg tries to fetch anything... So, how to do that?

That's an odd error, I've cloned repositories that were several GB without having any such issue.

2- And how to check out a specific branch of a package? I guess I can change the branch in the "prepare()" part, but is there a cleaner way of doing it?

See the PKGBUILD(5) manpage

Specifically, you want a #branch= fragment.

3- How to do the equivalent of?
git clone --recursive
Still in the PKGBUILD?

Cheers

You will want to do: https://wiki.archlinux.org/index.php/VC … Submodules

You could just git update init --recursive, but that would result in re-downloading all the sources every time you re-ran makepkg with a clean build.

dmidge · 2018-08-27 13:03:20

Hi Eschwartz,

Thanks for your reply!

Keeping the numbers for the one reading us:
1-

Eschwartz wrote:

That's an odd error, I've cloned repositories that were several GB without having any such issue.

I am not sure it is only because of the size of the repository. Because it is not that big. And to be entirely honest, I don't think I understood correctly what it is. I guess it is more related to the size of the HTTP POST requests. And I have no idea why the POST request are too big for the buffer on this repository. Anyway, it was suggested on some websites as the solution of the problem, and it worked. And I would like to pass that command before, otherwise the download just fails...
Otherwise, let say another situation that would requires the same kind of process - I used in the past some other packages in AUR, that failed because I needed to define a username with git config --global user.name "Mona Lisa or Whatever". And it needs to be passed before anything else. Thus, how to do that?

2-

Eschwartz wrote:

Specifically, you want a #branch= fragment.

Alright, so I should add that at the end of the git URL, and I guess that makepkg will remove that parameter from the URL and process it aside?

3-
Hm, okay. Submodules makes the process a bit more complicated. But I get why it is done like that. At least, the AUR updater knows exactly what is in it. Your other solution is a good alternative when the main package is not too big.

And maybe there is another question that I can add. Let's put the number 4: When I git clone the files myself in the same folder of the PKGBUILD, I have another folder that is created with all the source file. And then, running makepkg, I have the error "Error: /the/local/folder is not a clone of "https://github.com/distant/repo". But it is a clone, except that whenever I download the files with git, I go with "https://github.com/distant/repo.git", and the URL in PKGBUILD is "git+https://github.com/distant/repo". How to make sure that makepkg consider it is the same, and doesn't download again the same files?

Thanks!

eschwartz · 2018-08-27 13:28:53

dmidge wrote:

Hi Eschwartz,

Thanks for your reply!
Keeping the numbers for the one reading us:
1-
Eschwartz wrote:
That's an odd error, I've cloned repositories that were several GB without having any such issue.
I am not sure it is only because of the size of the repository. Because it is not that big. And to be entirely honest, I don't think I understood correctly what it is. I guess it is more related to the size of the HTTP POST requests. And I have no idea why the POST request are too big for the buffer on this repository. Anyway, it was suggested on some websites as the solution of the problem, and it worked. And I would like to pass that command before, otherwise the download just fails...
Otherwise, let say another situation that would requires the same kind of process - I used in the past some other packages in AUR, that failed because I needed to define a username with git config --global user.name "Mona Lisa or Whatever". And it needs to be passed before anything else. Thus, how to do that?

Which package requires you to define a user.name? This is completely wrong and should never happen.

I suspect what happened is they used a prepare function to run `git revert` or `git am`, and git wanted to create a new commit as a result. The proper approach is to use `git revert --no-commit`, or `patch`.

And regarding the http.postBuffer, some googling around tells me that this error probably happens when the git repository contains unusually large files, which cannot be broken up into smaller packs? and hits a limitation that is only in the http:// or https:// clone backend. Maybe it would be a better idea to try git:// in this case, though usually I recommend using https:// for the sake of TLS verification.

2-
Eschwartz wrote:
Specifically, you want a #branch= fragment.
Alright, so I should add that at the end of the git URL, and I guess that makepkg will remove that parameter from the URL and process it aside?

Correct.

3-
Hm, okay. Submodules makes the process a bit more complicated. But I get why it is done like that. At least, the AUR updater knows exactly what is in it. Your other solution is a good alternative when the main package is not too big.

Doing it right can be complicated.

And maybe there is another question that I can add. Let's put the number 4: When I git clone the files myself in the same folder of the PKGBUILD, I have another folder that is created with all the source file. And then, running makepkg, I have the error "Error: /the/local/folder is not a clone of "https://github.com/distant/repo". But it is a clone, except that whenever I download the files with git, I go with "https://github.com/distant/repo.git", and the URL in PKGBUILD is "git+https://github.com/distant/repo". How to make sure that makepkg consider it is the same, and doesn't download again the same files?

Thanks!

You can clone it without the .git at the end, or alternatively add the .git to the url in the PKGBUILD.

You can also use `git remote set-url origin https://...`

makepkg tries to make sure the url is the same, but it cannot tell when two different urls are resolved by the server as the same thing.

Trilby · 2018-08-27 14:12:38

RE #1, what git repository are you getting this error with? Help us replicate the problem so we can investigate what the best solution would be. You have one potential solution, but there might be others that'd be far more practical for use in a PKGBUILD.

dmidge · 2018-08-29 02:38:32

Hi guyz,

Thanks for your help!
Thanks to Eschwartz solution, on the #1 problem, I indeed managed to make the PKGBUILD work. It is not that proper, but it works. It is there: https://aur.archlinux.org/packages/trisycl-git/
I was trying to compile the triSYCL implementation that I found on github. But if you have any idea to make it better, any help is welcome! And if you have a better solution for the #1 problem, I am all hears!

Cheers!

Trilby · 2018-08-29 02:59:35

That git config in prepare should definitely not be there. Changing global git configs on the user's system without their knowledge or permission is very wrong and you will likely get some nasty responses to it (or you'd deserve to). But it's also not necessary, the PKGBUILD downloads fine without it. Yet even more importantly, if the download wouldn't work without it, having the git config in prepare will not help at all as that is run *after* the source is downloaded anyways.

On a larger picture note, I'm not sure why you asked for help here as you seem to have ignored all input and just went ahead and published something foolish to the AUR anyways.

You should drop the entire prepare function, and also remove the '-j4' from the make command: don't override user settings for no reason especially when on a large portion of systems it will make the build slower. EDIT: I see this last point was one I made on your last PKGBUILD as well. Wasted Breath.

Last edited by Trilby (2018-08-29 03:06:54)

a821 · 2018-08-29 06:34:14

In addition to the aforementioned issues, the `pkgver` function doesn't not work because the repo doesn't have tags. See https://wiki.archlinux.org/index.php/VC … 9_function

dmidge · 2018-08-29 12:22:14

Hi Guyz,

@Trilby: That is a harsher feedback than what I expected.... Though you're right, I did my best to contribute and have a working package, for guys like me that needed it, but couldn't find an available one...
The first question was because it is was not working and I was struggling to make it work. The second was because I had compilation issues and wondering if I could make it work by changing the release version. But finding a newer repo fixed the problem. The third was because I was asking myself that, and even if I didn't required it now, I was wondering how it was done. And someone that would find that post and need it would find it there... I don't think that it was "wasted breath". I am still learning, and probably won't be the only one from this post.
Anyway, I ended up forgetting to remove unnecessary parts when it was working.

@a821: Ok. So should it be?

pkgver() {
  cd "$pkgname"
  printf "r%s.%s" "$(git rev-list --count HEAD)" "$(git rev-parse --short HEAD)"
}

Cheers

Trilby · 2018-08-29 13:14:14

dmidge wrote:

I did my best to contribute and have a working package

A sincere thank you for that - I wouldn't want to discourage that motivation.

dmidge wrote:

I am still learning

In that case we should be able to patch this up quickly. I hope/suspect you'll find I'm as much as I am blunt when something is wrong, I am also quick to move on an appreciate good contributions when the little problems can be fixed up. I did react harshly because it seemed this thread could make progress on pinning down and fixing the issue, but instead of providing the information requested in order to do that, you published an ugly work around in the AUR.

We all make mistakes - I've made some big ones - but if you have people here waiting to help you, don't dive head first into whacky workarounds, work with us to get it right before publishing it to the AUR. You can definitely share drafts of PKGBUILDs here on the forums to catch any issues before it goes out to the general public.

I'm still curious about the actual issue for #1. As noted, I have no problem getting that source *without* adding that git config entry. Of course I believe it is failing for you, but that's what we should investigate: why did cloning the repo fail on your system, and is it just an issue on your end? A potential cause would be if you switched from "https" to "git" protocol. In that case, the proper fix would definitely just be to use the git protocol in the PKGBUILD. Any user can certainly opt to edit the source line to use https, and if it fails for them they can tweak any of their personal git configs to ensure it works. But that tweaking of the user system is definitely not something a PKGBUILD should do. And much the same logic is behind my objection to the -j4 flag for make. If your system is best served by using -j4, then you should have that in your own MAKEFLAGS set in /etc/makepkg.conf: don't override other user's settings to ones that just coincidentally work best on your hardware.

EDIT: and thanks for quickly pushing the touch ups to the AUR. Both of my concerns have been well addressed by the revision.

Last edited by Trilby (2018-08-29 14:30:23)

a821 · 2018-08-29 13:58:49

Given that you already prepended `1.2.1` to the `pkgver` function, which I guess it comes from the SYCL implementation version, it will be probably better to have something like

_pkgver=1.2.1
...

pkgver() {
   cd triSYCL
   printf "${_pkgver}.r%s.%s" "$(git rev-list --count HEAD)" "$(git rev-parse --short HEAD)"
}

So the PKBUILD's pkgver is consistent with what you have already published. In this case, the package version will look like `1.2.1.r1234.abcdef`.

You can leave the 1.2.1 out of pkgver, but then you need to add an `epoch` to the PKGBUILD, because `r1234.abcdef` < `1.2.1.r`. This is due to version comparison rules (see vercmp(8), pacman(8)).

Note that `cd $pkgname` won't work because `pkgname=trisycl-git`

Finally, the PKGBUILD installs to /usr/local, you need to change the prefix to /usr (if possible).

dmidge · 2018-08-30 01:27:47

Hi,

@Trilby: Indeed, for the #1 issue, I went from:

source=(git+https://github.com/triSYCL/triSYCL)

to:

source=(git+git://github.com/triSYCL/triSYCL)

The POST problem only came with the https protocol.
But that is good to know for the /etc/makepkg.conf! I was always overwriting myself the make parameter by hand all the time. Since it is quite long to compile at every try, I added it on the PKGBUILD. I forgot to remove it at the end before commiting, that is why it ended up out there. Sadly.

@a821: I tried to infer the package version from what I read. On the repo, they are talking about that. But I am not sure that the number is correct.
Anyway, before seeing your link, I thought that the update mechanism was only triggered when an increasing number for the version was encountered. Which is clearly not the case for hashes, where the hex number is quite random. Thus I thought it was quite mandatory to add the version number this way if the user wanted to be able to update their package.
And you're right, I changed the installation path to /usr. It should be live. I increased the pkgrel number.

Cheers.

eschwartz · 2018-08-30 02:22:09

Looking at your PKGBUILD...

You need to quote $srcdir/ in your package() function.

Also, I don't understand why boost and boost-libs are both makedepends and depends. boost depends on boost-libs, and you should only need boost as a makedepends. If the package is linked to boost-libs then you'll need to add boost-libs on its lonesome to depends, and if the package does *not* link to boost-libs then I'd encourage an investigation into why it is using static libraries and if that can be fixed.

Also consider matching the license field to the license field for https://www.archlinux.org/packages/?name=clang

dmidge wrote:

Hi,
@Trilby: Indeed, for the #1 issue, I went from:
source=(git+https://github.com/triSYCL/triSYCL)
to:
source=(git+git://github.com/triSYCL/triSYCL)

By the way, there's no need to repeat "git" twice.

The xxx+ part is just so that makepkg can tell the difference between an https:// file, and an https:// clone url for git. Because http, https, and ssh are very ambiguous urls and you need to prefix them with the version control method so it knows to use git, hg, bzr, svn, or whatever.

Last edited by eschwartz (2018-08-30 02:27:19)

dmidge · 2018-09-01 01:41:55

@Eschwartz:
Corrected for the "$srcdir".

To be honest, I don't know which version of boost is required, and neither do I know what is required in it. The only reason I added that is because it is said so on the readme file.
I guess it is more the headers (not the libs). But the library send in AUR seems to be header only. I am not sure though. But the only things that are compiled seems to be the examples and the tests. Thus boost would be required for the compilation, both for the compilation of the examples and the actual use by the user. I am not sure though.

Good point for the license! Fixed.

Alright, it is good to know that the second git is not required. But since I tend to reuse some of the previous pkgbuild that I used before when I try to create a new one, I guess that I would keep that extra information, since it doesn't harm anything. So it will help me for the next time.

Arch Linux

#1 2018-08-26 22:41:55

Pass argument to git before checking out a package

#2 2018-08-27 04:32:11

Re: Pass argument to git before checking out a package

#3 2018-08-27 13:03:20

Re: Pass argument to git before checking out a package

#4 2018-08-27 13:28:53

Re: Pass argument to git before checking out a package

#5 2018-08-27 14:12:38

Re: Pass argument to git before checking out a package

#6 2018-08-29 02:38:32

Re: Pass argument to git before checking out a package

#7 2018-08-29 02:59:35

Re: Pass argument to git before checking out a package

#8 2018-08-29 06:34:14

Re: Pass argument to git before checking out a package

#9 2018-08-29 12:22:14

Re: Pass argument to git before checking out a package

#10 2018-08-29 13:14:14

Re: Pass argument to git before checking out a package

#11 2018-08-29 13:58:49

Re: Pass argument to git before checking out a package

#12 2018-08-30 01:27:47

Re: Pass argument to git before checking out a package

#13 2018-08-30 02:22:09

Re: Pass argument to git before checking out a package

#14 2018-09-01 01:41:55

Re: Pass argument to git before checking out a package

Board footer