contagious - download anything concurrently from multiple mirrors

lolilolicon · 2010-12-18 16:43:03

So I was bothered how slow that 49MB firefox source was downloaded.
It took like half an hour, it was like slower than ftp.archlinux.org

One thing the open source community have taught me is the concept of mirrors.
I love mirrors.

I'd like to share here, this little script I wrote to really make use of mirrors:

#!/bin/bash
# contagious: for your crappy internet connection.

profiles_dir=${HOME}/bin/contagious.d

# Generate the metalink XML
# @global filename hashtype hashsum uris
gen_metalink()
{
  local metalink uri urltype
  metalink='<?xml version="1.0" encoding="utf-8"?>'
  metalink+=$'\n''<metalink version="3.0" xmlns="http://www.metalinker.org/">'
  metalink+=$'\n''<files>'

  metalink+=$'\n'"  <file name=\"${filename}\">"
  if [ -n "${hashtype}" ] && [ -n "${hashsum}" ]; then
    metalink+=$'\n'"    <verification>"
    metalink+=$'\n'"      <hash type=\"${hashtype}\">${hashsum}</hash>"
    metalink+=$'\n'"    </verification>"
  fi
  metalink+=$'\n'"    <resources>"
  for uri in "${uris[@]}"; do
    urltype=${uri%%://*}
    metalink+=$'\n'"      <url type=\"${urltype}\">${uri}</url>"
  done
  metalink+=$'\n'"    </resources>"
  metalink+=$'\n'"  </file>"

  metalink+=$'\n''</files>'
  metalink+=$'\n''</metalink>'
  echo "${metalink}"
}
# @global: uris mirrors filepath filename
# Note when filepath is "/", mirrors _are_ uris.
populate_uris()
{
  local m
  uris=()
  if [ "${filepath}" == '/' ]; then
    uris=("${mirrors[@]}")
  elif [ -z "${filepath}" ]; then
    for m in "${mirrors[@]}"; do
      uris+=(${m%/}/${filename})
    done
  else
    for m in "${mirrors[@]}"; do
      uris+=(${m%/}/${filepath%/}/${filename})
    done
  fi
}
check_profile_env()
{
  if [ ${#mirrors[@]} -eq 0 ]; then
    echo 'You must provide at least one mirror!' >&2
    return 1
  elif [ -z "${filename}" ]; then
    echo 'filename is empty!' >&2
    return 1
  fi
}
check_profile_file()
{
  if [ -z "${profile}" ]; then
    echo "You must specify a profile under directory \`${profiles_dir}'" >&2
    return 250
  elif [ ! -r "${profiles_dir}/${profile}" ]; then
    echo "File nonexistent or unreadable: \`${profiles_dir}/${profile}'" >&2
    return 250
  fi
}

profile=$1; check_profile_file || exit $?
filename= filepath= hashtype= hashsum= mirrors=()
source "${profiles_dir%/}/${profile}" && check_profile_env || exit $?
populate_uris
gen_metalink | aria2c --metalink-file=- --metalink-servers=40 --min-split-size=1M --lowest-speed-limit=10K

# vim:ts=2 sw=2 ft=sh:

To test it out, you need some profile under your ${profiles_dir}.
For example, put the following in ~/bin/contagious.d/firefox

# This is a profile for contagious.
# @profile firefox
# @variables filename filepath hashsum hashtype mirrors

# NOTE:
# 1. Set filepath to "/" if URIs in mirrors are complete links to the file.
# 2. Otherwise, the final URI will be: mirror + filepath + filename
# 3. hashsum and hashtype are optional, but needed for integrity check.
# 4. hashtype examples: md5 sha1 sha256

pkgver=3.6.13
filename=firefox-${pkgver}.source.tar.bz2
filepath=firefox/releases/${pkgver}/source
hashsum='4b90775c0f29cb7e170a80894311d8c7a2cd794c50e2124b70d1b83011c45f63'
hashtype=sha256
unset pkgver

# See: http://www.mozilla.org/community/mirrors.html
mirrors=(
  "http://ftp.mozilla.org/pub/mozilla.org/"
  "http://releases.mozilla.org/pub/mozilla.org/"
  "http://3347-mozilla.voxcdn.com/pub/mozilla.org/"
  "http://daimajin.mirror.aarnet.edu.au/pub/mozilla/"
  "http://fastspeedtest.net/mirrors/mozilla.org/"
  "http://ftp.ankara.edu.tr/mozilla/"
  "http://ftp.cs.pu.edu.tw/pub/Mozilla/"
  "http://ftp.cvut.cz/mozilla/"
  "http://ftp.daum.net/mozilla/"
  "http://ftp.df.lth.se/mozilla/"
  "http://ftp.halifax.rwth-aachen.de/mozilla/"
  "http://ftp.heanet.ie/mirrors/ftp.mozilla.org/pub/mozilla.org/"
  "http://ftp.isu.edu.tw/pub/Mozilla/"
  "http://ftp.jaist.ac.jp/pub/mozilla.org/"
  "http://ftp.kaist.ac.kr/mozilla/"
  # Add more mirrors here...
)

Now try it (Make sure you have aria2 installed):

$  contagious firefox
[#1 SIZE:2.5MiB/49.0MiB(5%) CN:40 SPD:971.0KiBs ETA:49s]

Tell me what you think

Edit:
I consider this thread a good place for people to post lists of mirrors of everything.
Example:

Mozilla mirrors: http://www.mozilla.org/community/mirrors.html
A compiled list: http://dpaste.com/288068/

Last edited by lolilolicon (2010-12-18 18:00:43)

superchango · 2010-12-18 17:56:42

Woah, that sound very interesting. I'm going to try it. Thnxs!!

lolilolicon · 2010-12-19 05:36:00

OK. Figured github would be a nice place to host _your_ profiles
Here: https://github.com/lolilolicon/contagious
PKGBUILD

Arch Linux

#1 2010-12-18 16:43:03

contagious - download anything concurrently from multiple mirrors

#2 2010-12-18 17:56:42

Re: contagious - download anything concurrently from multiple mirrors

#3 2010-12-19 05:36:00

Re: contagious - download anything concurrently from multiple mirrors

Board footer