Add a `distfiles' option for makepkg

mytbk · 2013-07-20 02:02:34

Inspired by Gentoo distfiles in Gentoo mirrors, I've added a `distfiles' option in makepkg, so that makepkg can download source files from Gentoo mirrors if possible. This is good for some users who have limitations in their network to help them download source files faster and easier from mirrors.

However, my patch is very simple and therefore very buggy, just as follows.

diff -rupN orig/makepkg mod/makepkg
--- orig/makepkg	2013-07-20 09:47:42.608789577 +0800
+++ mod/makepkg	2013-07-20 09:38:10.861886996 +0800
@@ -48,7 +48,7 @@ declare -r startdir="$PWD"
 
 packaging_options=('strip' 'docs' 'libtool' 'staticlibs' 'emptydirs' 'zipman' \
                    'purge' 'upx' 'debug')
-other_options=('ccache' 'distcc' 'buildflags' 'makeflags')
+other_options=('ccache' 'distcc' 'buildflags' 'makeflags' 'distfiles')
 splitpkg_overrides=('pkgver' 'pkgrel' 'epoch' 'pkgdesc' 'arch' 'url' 'license' \
                     'groups' 'depends' 'optdepends' 'provides' 'conflicts' \
                     'replaces' 'backup' 'options' 'install' 'changelog')
@@ -330,6 +330,11 @@ download_local() {
 download_file() {
 	local netfile=$1
 
+	if check_buildenv "distfiles" "y" && ! check_option "distfiles" "n"; then
+		distfile=${DISTURL}/$(basename $netfile)
+		wget ${distfile}
+	fi 
+
 	local filepath=$(get_filepath "$netfile")
 	if [[ -n "$filepath" ]]; then
 		msg2 "$(gettext "Found %s")" "${filepath##*/}"

To use this option, just add 'distfiles' in BUILDENV and add a line like

DISTURL="http://mirrors.ustc.edu.cn/gentoo/distfiles/"

.

karol · 2013-07-20 02:09:38

I'm not familiar with gentoo and distfiles, can you please explain in more detail? What would be a use case for this?

mytbk · 2013-07-20 02:23:12

karol wrote:

I'm not familiar with gentoo and distfiles, can you please explain in more detail? What would be a use case for this?

There are many source tarballs in Gentoo mirror sites, and the package manager can download source files from GENTOO_MIRRORS defined in make.conf prior to the upstream url in the ebuild files.
I'm in a school network, and I only have access to IPv4 addresses in a certain IP list or IPv6 addresses. If this option is possible, downloading source files is easier for me. Moreover, It's usually faster to download files from a good mirror than the upstream.

karol · 2013-07-20 09:56:00

mytbk wrote:

It's usually faster to download files from a good mirror than the upstream.

That's what I understood from reading about distfiles, but the download speed was never a problem for me - I saturate my connection anyway ;P

mytbk wrote:

I'm in a school network, and I only have access to IPv4 addresses in a certain IP list or IPv6 addresses.

Ah, I get it.

lolilolicon · 2013-07-20 10:25:13

mytbk wrote:

.

+	if check_buildenv "distfiles" "y" && ! check_option "distfiles" "n"; then
+		distfile=${DISTURL}/$(basename $netfile)
+		wget ${distfile}
+	fi 
+

Shouldn't force wget here. Better change the download_sources function, e.g.

-				download_file "$netfile"
+				download_file "${DISTURL%/}/${netfile##*/}"

Add in the ifs and elses where appropriate.

mytbk · 2013-07-20 16:18:17

lolilolicon wrote:

mytbk wrote:
.
+	if check_buildenv "distfiles" "y" && ! check_option "distfiles" "n"; then
+		distfile=${DISTURL}/$(basename $netfile)
+		wget ${distfile}
+	fi 
+
Shouldn't force wget here. Better change the download_sources function, e.g.
-				download_file "$netfile"
+				download_file "${DISTURL%/}/${netfile##*/}"
Add in the ifs and elses where appropriate.

Yes, so I need someone better at scripting to help have this complicated job accomplished.
Edit: I think it's a better idea, I'll have a try then.

Last edited by mytbk (2013-07-20 16:20:00)

lolilolicon · 2013-07-20 17:29:46

mytbk wrote:

Yes, so I need someone better at scripting to help have this complicated job accomplished.
Edit: I think it's a better idea, I'll have a try then.

Having taken a closer look at the code, I notice that my example didn't cover the case where the source url is prefixed with `filename::', for the sake of which here is an updated example:

if [[ -z $DISTURL ]]; then
  download_file "$netfile"
elif [[ $netfile = *::* ]]; then
  download_file "${netfile%%::*}::${DISTURL%/}/${netfile##*/}"
else
  download_file "${DISTURL%/}/${netfile##*/}"
fi

With this modification, you can run `DISTURL=http://foo.bar makepkg` for your purpose.
There might be some other pitfalls, but I think it should be fine for most cases.

progandy · 2013-07-20 17:45:44

Maybe it would be a better idea to keep that functionality out of makepkg and let users use a custom DLAGENT?

lolilolicon · 2013-07-20 17:55:29

@progandy, kudos, you're absolutely right.

mytbk · 2013-07-21 00:53:39

progandy wrote:

Maybe it would be a better idea to keep that functionality out of makepkg and let users use a custom DLAGENT?

Oh, so you're going to use a wrapper to do this.

Arch Linux

#1 2013-07-20 02:02:34

Add a `distfiles' option for makepkg

#2 2013-07-20 02:09:38

Re: Add a `distfiles' option for makepkg

#3 2013-07-20 02:23:12

Re: Add a `distfiles' option for makepkg

#4 2013-07-20 09:56:00

Re: Add a `distfiles' option for makepkg

#5 2013-07-20 10:25:13

Re: Add a `distfiles' option for makepkg

#6 2013-07-20 16:18:17

Re: Add a `distfiles' option for makepkg

#7 2013-07-20 17:29:46

Re: Add a `distfiles' option for makepkg

#8 2013-07-20 17:45:44

Re: Add a `distfiles' option for makepkg

#9 2013-07-20 17:55:29

Re: Add a `distfiles' option for makepkg

#10 2013-07-21 00:53:39

Re: Add a `distfiles' option for makepkg

Board footer