problems downloading from Sourceforge mirrors - workaround

Lone_Wolf · 2006-08-26 22:13:54

Today i tried to make a package that needed to download the sources from sourceforge.

The trick in this thread http://bbs.archlinux.org/viewtopic.php?t=15542 , adding the ip of a good sf mirror to /etc/hosts didn't work with heanet, surfnet and belnet.

using an url like this in the pkgbuild did work

http://heanet.dl.sourceforge.net/sourceforge/vdrift/vdrift-2006-07-08-src.tar.bz2

The problem with this is that you need to specifiy a certain mirror, and the mirror the pkgbuilder chooses might be slow for other users.

I've checked download urls from sourceforge mirrors and it seems they use this schema :

http://<mirrorname>.dl.sourceforge.net/sourceforge/<projectname>/<filename>

Imo a possible solution would be to add a new envrionment variable to /etc/makepkg.conf, called something like sfmirror.
Users would edit that variable to point to a sourceforge mirror that is close to them and pkg builders could use this variable in the source url.

Before i submit this idea to the bugtracker as a change to makepkg, i'd like to hear some comments from other package builders.

arooaroo · 2006-08-26 22:21:29

I actually don't see a problem with specifying a mirror in the pkgbuild. I think that pkgbuilds should *work* and whilst in an ideal world being able to use the optimal mirror is best, it's hardly the end of the world.

Dusty · 2006-08-26 22:55:40

I say post the suggested fix to the bugtracker. Link back here for discussion. Its an ugly hack, but sourceforge is a big provider, and I'm also tired of trying to fix broken PKGBUILDs by manual editing. Using specific mirrors is a reasonable solution too, especially if Arch users decide to choose one mirror and use it uniformly (makes it easier to write sed scripts to replace all of /var/abs or AUR packages downloaded). I mean, lots of app sources are mirrored (example: the kernel), but we don't worry about selecting mirrors for those. Its just that sf is such a large provider, you notice the irritation frequently.

The ideal solution, though, is to have sourceforge fix their download.sf.net URL so it works!

Dusty

tomk · 2006-08-26 23:03:36

The effect of putting the ip address in /etc/hosts and specifying the mirror in the PKGBUILD is exactly the same - it's just two different ways to achieve the same result. If /etc/hosts didn't work for me, I'd be inclined to suspect a bad configuration somewhere along the line. Your proposed feature request for makepkg is a third way to get the same result. IME, it's not necessary, but others may disagree.

Re the bigger issue, it seems perfectly obvious to me that PKGBUILDs submitted for use by users spread around the globe should not be tied to one mirror - that is, after all, the underlying purpose of mirroring, right?

T-Dawg · 2006-08-27 00:22:45

tomk has got it right. Everyone's sf mirror will be different around the globe.
If anything this variable should be introduced into makepkg.conf because it is user specific. Kind of hackish though. The best solution would be for sourceforge to select mirrors faster.

T-Dawg · 2006-08-27 01:20:15

ZOMG! I made a patch

--- /usr/bin/makepkg.old        2006-08-26 20:40:50.000000000 -0400
+++ /usr/bin/makepkg    2006-08-27 09:12:05.000000000 -0400
@@ -480,6 +480,11 @@
                        exit 1
                fi
                msg "    Downloading $file"
+               if [ "`echo $netfile |grep /dl.sourceforge.net`" ]; then
+                       if [ ! -z $sfmirror ]; then
+                               netfile=`echo $netfile |sed "s|dl.sourceforge.net|$sfmirror|"`
+                       fi
+               fi
                $FTPAGENT $netfile 2>&1
                if [ ! -f $file ]; then
                        error "Failed to download $file"

then you just add your sourceforge mirror as sfmirror="bla.dl.sourceforge.net" in /etc/makepkg.conf

Dusty · 2006-08-27 02:27:52

anybody considered contacting sf to discover if they are aware of or care about the problem?

Dusty

dtw · 2006-08-27 07:15:25

Lone_Wolf wrote:

Imo a possible solution would be to add a new envrionment variable to /etc/makepkg.conf, called something like sfmirror.
Users would edit that variable to point to a sourceforge mirror that is close to them and pkg builders could use this variable in the source url.

That was the solution I had in mind for a while too, but I imagine that people may not lke it. Would be better for sourceforge to "fix" it

tomk · 2006-08-27 09:33:29

SF Support Request # 1547432.

Lone_Wolf · 2006-08-27 12:30:30

Submitted to Flyspray : 5294

Lone_Wolf · 2006-10-09 08:22:53

I've just checked, the only thing that changed in the sf bugreport is the name of the guy who is responsible for solving it.
Also not much repsonse on the flyspray request.

The problem still exists, so i copied makepkg to makepkg.sf , added the patch made by Penguin and now build packages with makepkg.sf .
It's a hackish solution, but it works.

Lone_Wolf · 2006-10-28 11:11:41

As there appear to be people that have trouble getting penguins patch to work, i've made an aur package for it.

makepkg-sf in AUR

tomk · 2006-11-14 16:33:12

tomk wrote:

SF Support Request # 1547432.

The (leisurely) response from sourceforge:

Please use download links provided in the File Release
System of each project, rather than handcrafting URLs to
retrieve specific files.
If you are basing this off of a section of sf.net
documentation, please update this ticket with a URL to the
offending documentation and it will be corrected.

Of course, that is correct. Their system is set up for optimum results when downloading via the selected project's web page, using a http://prdownloads.soureceforge.net/.. link. The way to pursue this further with them would be by way of a new feature request to accommodate ABS and other such build systems, but I wouldn't expect a positive response.

Lone_Wolf · 2006-11-14 19:33:57

Lone_Wolf wrote:

Submitted to Flyspray : 5294

Phrakture has closed the above bug report :

Using /etc/hosts is not a workaround. Defining a variable to do that would be a workaround.
I personally think the /etc/hosts solution is the best and most succinct, as it allows you to click the link and download it too. - i.e. if our web interface shows "dl.sourceforge.net" for the source, and we used the hackish $sfmirror, you would be unable to click the link. Adding it to /etc/hosts would allow you to do this.
I cannot think of any reason why the /etc/hosts solution is _not_ a good one.
The most I would be willing to do is provide a /etc/hosts line when makepkg detects a dl.sourceforge.net URL... but I think /etc/hosts is ideal.

He misunderstood the way the patch T-dawg made works, but as the bug report is closed, we can't commend on that anymore.

Aur users who have trouble with sf mirrors now have 2 choices :

Use the "official" /etc/hosts solution or the workaround of using a patched version of makepkg.

Since i haven't been able to get the /etc/hosts solution working on 4 different archlinux installations (2 desktops, a laptop and a vmware VM) i'm sticking with the patched version.

Combined with sourceforges response, it means we stand at the same point as when this thread was started :
Individual users will have to change their settings by either tweaking /etc/hosts or using an unofficial patched version of makepkg to be able to build sourceforge packages from aur.

Not the kind of solution i was hoping for ..

T-Dawg · 2006-11-14 21:15:59

I still don't understand how the hosts method is supposed to work either. You can't simply put

<bla>.dl.sourceforge.net      dl.sourceforge

in hosts and expect it to resolve. It must be the actual ip address. Using the ip address of your preferred sourceforge mirror is not good because that IP ADDRESS can and will change at any given time...

tomk · 2006-11-14 23:47:00

Your experience is different to mine so, T-Dawg. I've had heanet's ip address in /etc/hosts since this problem first came to light, and it hasn't changed once. I would have though it would be the same for any of the sf mirrors.

dtw · 2006-11-14 23:55:18

Same for me, tomk, same mirror though

Arch Linux

#1 2006-08-26 22:13:54

problems downloading from Sourceforge mirrors - workaround

#2 2006-08-26 22:21:29

Re: problems downloading from Sourceforge mirrors - workaround

#3 2006-08-26 22:55:40

Re: problems downloading from Sourceforge mirrors - workaround

#4 2006-08-26 23:03:36

Re: problems downloading from Sourceforge mirrors - workaround

#5 2006-08-27 00:22:45

Re: problems downloading from Sourceforge mirrors - workaround

#6 2006-08-27 01:20:15

Re: problems downloading from Sourceforge mirrors - workaround

#7 2006-08-27 02:27:52

Re: problems downloading from Sourceforge mirrors - workaround

#8 2006-08-27 07:15:25

Re: problems downloading from Sourceforge mirrors - workaround

#9 2006-08-27 09:33:29

Re: problems downloading from Sourceforge mirrors - workaround

#10 2006-08-27 12:30:30

Re: problems downloading from Sourceforge mirrors - workaround

#11 2006-10-09 08:22:53

Re: problems downloading from Sourceforge mirrors - workaround

#12 2006-10-28 11:11:41

Re: problems downloading from Sourceforge mirrors - workaround

#13 2006-11-14 16:33:12

Re: problems downloading from Sourceforge mirrors - workaround

#14 2006-11-14 19:33:57

Re: problems downloading from Sourceforge mirrors - workaround

#15 2006-11-14 21:15:59

Re: problems downloading from Sourceforge mirrors - workaround

#16 2006-11-14 23:47:00

Re: problems downloading from Sourceforge mirrors - workaround

#17 2006-11-14 23:55:18

Re: problems downloading from Sourceforge mirrors - workaround

Board footer