Non directly downloadable source file in PKGBUILD

Bermond · 2016-08-27 14:17:47

When making a PKGBUILD for the AUR there are some cases that the source file is not directly downloadable (no direct upstream download link).

For example, some upstream developers require registration at their website for downloading the source files. In this case, the download link will be given only after the website registration process and usually only after the user has agreed with the software license. In most cases it's a "hidden" address or some kind of link that expires over time, making it not suitable to placed in PKGBUILD. Examples of this situation includes some sdk files, like nvidia sdk and blackmagic decklink sdk, which upstream developers require manual website registration and license agreement prior to downloading.

To handle this situation, I've made PKGBUILD with the following approach:

source=('file://sourcefile.zip')

and placed a brief instruction in PKBGUILD first lines telling the user to manually download the source file after registration at upstream developer website. Also placed a note in package description. By doing this, 'makepkg' will not attempt to download the source file, but just expect it in the source directory and check for md5/sha sums.

But some people are complaining about this approach beucase 'makepkg' (and AUR helpers) will fail to build the package if the user has not manually downloaded the source file and placed it in source directory. Some are requesting me to use a source link from a person that hosted the source file in Dropbox, but I'm not comfortable with it due to licensing concerns and even due to security reasons.

Since this kind in situation is not covered in the Wiki, I'm asking for suggestions about how to handle this.

Is my PKGBUILD approach acceptable? Do you recommend any other solution? If possible, I would like also to hear some words about possible license and security concerns if the source file is hosted elsewhere over the internet like people are requesting.

Last edited by Bermond (2016-08-27 14:25:02)

Scimmia · 2016-08-27 14:26:11

Yes, your approach is fine. Pin a comment about it and put into in the PKGBUILD; people will still complain but it's their problem.

ewaller · 2016-08-27 14:41:04

...And your concerns about the legality of rehosting are most likely correct. I've not read the licence, but I doubt that it has the provisions that would allow you to redistribute the blob -- a very tenet of copyright.

WorMzy · 2016-08-27 14:54:56

Moving to Creating and Modifying Packages.

Bermond · 2016-08-27 17:05:35

Thank you for the answers.

Licensing concerns seems sufficient for having this approach.

Regarding security, is it reasonable to have security concerns about a source file downloaded from outside the upstream developer website like people are requesting? Or is it not a risk? Yes, we have md5/sha for checking source file integrity, but could it be faked/cracked/bypassed in terms of 'makepkg' checks?

Xyne · 2016-08-27 18:12:38

Security is not an issue if you trust the PKGBUILD maintainer and a strong checksum is used (sha256 or sha512). You can included both to make it even harder to spoof the checksum, but spoofing one is likely hard enough already. Of course, even then the packager should avoid hostile source hosts.

Licensing is the main concern. Packages that violate licenses will be removed from the AUR.

I don't think we have any explicit policy in place to forbid PKGBUILDs that require pre-emptive manual downloads but there is a tacit expectation that PKGBUILDs should be a complete recipe for building the package with makepkg. I think it would be worth bringing this up on aur-general which is the official mailing list for such discussions. It is followed by the TUs and the AUR devs (normally), whereas only some of us regularly follow the forum.

I expect that one of the arguments will be that if the project allows hosting the sources elsewhere then the packager should do that, and if it doesn't then it isn't open source and shouldn't be on the AUR. If the majority is ok with "incomplete" PKGBUILDs, then maybe we can discuss a way to mark them as such for AUR helpers.

Trilby · 2016-08-27 18:18:54

I'm surprised there isn't a policy for this. Requiring the user to download the file manually is the de facto standard in the AUR at the moment as there are many such packages already. All the matlab packages are just one variety of example.

Also I'm quite sure the AUR is not only for open source software. There are countless non-open source packages in the AUR including all the matlab packages, google chrome packages, and a laundry list of others. There are non-open-source packages in the main repos even. This isn't Parabola.

Xyne · 2016-08-27 18:42:33

I don't use any packages that require pre-emptive downloads before I run makepkg and I was simply not aware that they are that common. It's definitely not the de-facto standard from what I've seen and I would expect there to be far more complains from AUR-helper users if it were.

There is obviously no restriction to open source. There are e.g. games that require non-free content to work, freeware packages, etc. Those are all accepted and I fall in the camp of considering PKGBUILDs to be freely distributable recipes with the burden of respecting licenses falling on the user. I was just anticipating a discussion about whether AUR packages are expected to build "on their own" and a possible argument in favor of it.

As for documented policy, the AUR is still a little wild west and the TUs are a loose network of sheriffs that are mostly on the same page but not always.

Trilby · 2016-08-27 18:53:37

Xyne wrote:

It's definitely not the de-facto standard

Er... but it is. It may not be a formal standard, but de facto standard it is. All packages that require a user agree to something on a web form currently use a file:// in the source array. If they didn't they would be violating license terms and would be removed.

Scimmia · 2016-08-27 22:51:18

It most definitely is a de-facto standard. It's been done this way on many, many packages for many years. It has also been discussed previously in various places, including aur-general, and this is the recommended way of doing things.

Last edited by Scimmia (2016-08-27 22:52:08)

Bermond · 2016-08-28 02:04:35

Xyne wrote:

Security is not an issue if you trust the PKGBUILD maintainer and a strong checksum is used (sha256 or sha512). You can included both to make it even harder to spoof the checksum, but spoofing one is likely hard enough already.

I see. Currently I use sha256sums in my PKGBUILDs.
Good to know that it's OK to include more than one checksum for stronger source file verification.

Xyne wrote:

Of course, even then the packager should avoid hostile source hosts.

Xyne wrote:

Licensing is the main concern. Packages that violate licenses will be removed from the AUR.

Trilby wrote:

It may not be a formal standard, but de facto standard it is. All packages that require a user agree to something on a web form currently use a file:// in the source array. If they didn't they would be violating license terms and would be removed.

Scimmia wrote:

It most definitely is a de-facto standard. It's been done this way on many, many packages for many years. It has also been discussed previously in various places, including aur-general, and this is the recommended way of doing things.

These statements clarifies it all. Using 'file://' in such cases is the way to go.

I think it would be good if the use of 'file://' in the source array could be described in the Wiki entry for PKGBUILD for such type of situations, as like as its importance related to licensing issues for source files that require user agreement in a web form. It's just a suggestion since at this moment I could not find any information about it in the Wiki.

I will use this topic as an answer to people that complained about this approach.

Thanks to everyone that replied in this topic.

Xyne · 2016-08-28 18:34:49

@Trilby & Scimmia
Sorry, I misinterpreted the "de facto standard" to mean that most packages require manual downloads. Brainfart of a tired mind.

As for the wiki page, I agree that this convention should be included. Something like

In some cases, license restrictions or registration requirements may restrict (direct) access to source files. If a remote URL cannot be included in the source array, one may instead include a local "file://" URL along with commented instructions to the user for retrieving the sources manually. For example, some games require non-free game data to build which the user is expected to have. In such a case, the user should obtain the game data and then include an entry in the source array such as "file:///path/to/game_data" or just "game_data" if it has been copied into the build directory.
If the source files are freely distributable, it is preferable to host them online when possible to enable automatic building for other users.

I would edit it myself but my mind is buried in something else right now and I prefer to leave it up to someone with clearer thoughts.

Incidentally, is "file://" followed by a relative path officially supported by makepkg or is that just a consequence of non-standard parsing? My understanding is that "file" URIs only allow full paths.

eschwartz · 2016-08-28 22:09:23

It is a result of makepkg parsing, which has absolutely no idea about how to obtain files from file:// URIs and therefore doesn't know if they are valid or not, but does extract the last component of the URI to define the downloaded sources.

Essentially, it acts exactly the same as a file specified as a local-to-the-PKGBUILD file. Except that the AUR validation process considers it a remote file with an unknown protocol, rather than failing on a local file that isn't committed in git.

...

For this reason, I am thinking this might fit better in the Arch User Repository FAQ, since it is kind of AUR-specific.

Last edited by eschwartz (2016-08-28 22:14:58)

alive4ever · 2016-08-29 09:49:31

For locally downloaded source, I think 'file://' prefix is not needed.

Putting source file name is enough.

Trilby · 2016-08-29 10:59:38

Without file:// you will have trouble uploading it to the AUR as files without a uri scheme are expected to be included in the tarball.

Arch Linux

#1 2016-08-27 14:17:47

Non directly downloadable source file in PKGBUILD

#2 2016-08-27 14:26:11

Re: Non directly downloadable source file in PKGBUILD

#3 2016-08-27 14:41:04

Re: Non directly downloadable source file in PKGBUILD

#4 2016-08-27 14:54:56

Re: Non directly downloadable source file in PKGBUILD

#5 2016-08-27 17:05:35

Re: Non directly downloadable source file in PKGBUILD

#6 2016-08-27 18:12:38

Re: Non directly downloadable source file in PKGBUILD

#7 2016-08-27 18:18:54

Re: Non directly downloadable source file in PKGBUILD

#8 2016-08-27 18:42:33

Re: Non directly downloadable source file in PKGBUILD

#9 2016-08-27 18:53:37

Re: Non directly downloadable source file in PKGBUILD

#10 2016-08-27 22:51:18

Re: Non directly downloadable source file in PKGBUILD

#11 2016-08-28 02:04:35

Re: Non directly downloadable source file in PKGBUILD

#12 2016-08-28 18:34:49

Re: Non directly downloadable source file in PKGBUILD

#13 2016-08-28 22:09:23

Re: Non directly downloadable source file in PKGBUILD

#14 2016-08-29 09:49:31

Re: Non directly downloadable source file in PKGBUILD

#15 2016-08-29 10:59:38

Re: Non directly downloadable source file in PKGBUILD

Board footer