Fully functioning pacget script for pacman 3?

vacant · 2008-01-17 11:42:13

The pacget script works but always connects to only one server. $repo is always set to "custom".

Using the new /etc/pacman.d/mirrorlist, for downloading (non-custom) packages, pacget needs to:

1) extract the third from last field in the server url, delimited by "/" i.e. extract one of exta, core, community etc
2) for each line appended to $url, replace $repo with the extracted field

My scripting skills are basic so I'd probably end up bodging it by recreating temporary repo lists or something.

Anyone help?

shining · 2008-01-17 14:32:11

vacant wrote:

My scripting skills are basic so I'd probably end up bodging it by recreating temporary repo lists or something.

You could always do that.
But I don't really get the point of connecting to multiple servers. Isn't one enough? How slow is it for you?

vacant · 2008-01-17 14:43:48

Well I warned you I would bodge it Just need to check REPOFIELD is the correct field number (containing $repo) of the first mirror. Also I haven't tested "custom" repos as I don't use them.

#!/bin/bash

# ------------ Begin Configuration ------------ #

# The log file
LOG=/var/log/pacget.log
# Number of connections per server (don't go beyond 2, please)
CONNECTIONS=2
# Max number of servers to download from (follows ordering in include file)
SERVERS=4
# Maximum download speed (0 = unrestricted)
MAX_SPEED=0
# Minimum download speed (0 = don't care)
MIN_SPEED=0
# Maximum tries per download
MAX_TRIES=2
# Server timeout period
TIMEOUT=15
# Passive FTP or not: 'yes' or 'no'
FTP_PASV="no"
# 'none' or 'prealloc'
FILE_ALLOC="none"
# Use color in messages
USE_COLOR="yes"
# Pacman 3 mirrors now all in one file with $repo placeholder
MIRRORLIST="/etc/pacman.d/mirrorlist"
# This is the FIELD NUMBER (delimited by "/") of the $repo placeholder of the FIRST entry in MIRRORLIST
REPOFIELD=6

# ------------- End Configuration ------------- #

msg() {
  echo ""
  if [ "${USE_COLOR}" = "yes" ]; then
    echo -ne "   \033[1;34m->\033[1;0m \033[1;1m${1}\033[1;0m" >&2
  else
    echo -n "   -> ${1}" >&2
  fi
}

error() {
  if [ "$USE_COLOR" = "yes" ]; then
    echo -e "\033[1;31m==> ERROR:\033[1;0m \033[1;1m$1\033[1;0m" >&2
  else
    echo "==> ERROR: $1" >&2
  fi
}

ARIA2_BIN=$(which aria2c 2> /dev/null)

# ----- do some checks first -----
if [ ! -x "$ARIA2_BIN" ]; then
  error "aria2c was not found or isn't executable."
  exit 1
fi

if [ $# -ne 2 ]; then
  error "Incorrect number of arguments"
  exit 1
fi

filename=$(basename ${1})
server=${1%/${filename}}

repo=$( echo ${server} | cut -d/ -f${REPOFIELD} )

msg "Fetching from repo ${repo}"

# Override number of connections for db files
if [[ ${filename} = *.db.tar.gz ]]; then
  CONNECTIONS=1
fi

# For db files, or when using a custom repo (which most likely doesn't have any mirror),
# use only the URL passed by pacman; Otherwise, extract the list of servers (from the include file of the repo) to download from
if [[ ${filename} = *.db.tar.gz ]]; then
  url="${1}"
else
  url=$( sed s"/Server = //" ${MIRRORLIST} | sed s"/\$repo/${repo}/" | sed s"/$/\/${filename}/" | head -n ${SERVERS} )
fi

# Passive FTP or not?
[ "${FTP_PASV}" = "yes" ] && OPT_FTP_PASV="--ftp-pasv" || OPT_FTP_PASV=""

msg "Downloading ${filename} \n"

cd /var/cache/pacman/pkg/

${ARIA2_BIN} --log=${LOG} --timeout=${TIMEOUT} --max-tries=${MAX_TRIES} --allow-overwrite=true \
             --split=${CONNECTIONS} ${OPT_FTP_PASV} --file-allocation=${FILE_ALLOC} \
             --lowest-speed-limit=${MIN_SPEED} --max-download-limit=${MAX_SPEED} \
             ${url} --out=${filename}.pacget && [ ! -f ${filename}.pacget.aria2 ] && mv ${filename}.pacget ${2} && chmod 644 ${2}

exit $?

vacant · 2008-01-17 15:46:20

shining wrote:

Isn't one enough? How slow is it for you?

I'm on quite a cheap ISP package and one server tends to get choked to 2Mb while I can max out my connection in the mornings if I use concurrent servers. Using aria2c fully is a "nice have" rather than a "must have".

vacant · 2008-01-18 16:54:10

This pacget is fully automatic

#!/bin/bash

# ------------ Begin Configuration ------------ #

# The log file
LOG=/var/log/pacget.log
# Number of connections per server (don't go beyond 2, please)
CONNECTIONS=2
# Max number of servers to download from (follows ordering in include file)
SERVERS=4
# Maximum download speed (0 = unrestricted)
MAX_SPEED=0
# Minimum download speed (0 = don't care)
MIN_SPEED=0
# Maximum tries per download
MAX_TRIES=2
# Server timeout period
TIMEOUT=15
# Passive FTP or not: 'yes' or 'no'
FTP_PASV="no"
# 'none' or 'prealloc'
FILE_ALLOC="none"
# Use color in messages
USE_COLOR="yes"
# Pacman 3 mirrors now all in one file with $repo placeholder
MIRRORLIST="/etc/pacman.d/mirrorlist"

# ------------- End Configuration ------------- #

msg() {
  echo ""
  if [ "${USE_COLOR}" = "yes" ]; then
    echo -ne "   \033[1;34m->\033[1;0m \033[1;1m${1}\033[1;0m" >&2
  else
    echo -n "   -> ${1}" >&2
  fi
}

error() {
  if [ "$USE_COLOR" = "yes" ]; then
    echo -e "\033[1;31m==> ERROR:\033[1;0m \033[1;1m$1\033[1;0m" >&2
  else
    echo "==> ERROR: $1" >&2
  fi
}

ARIA2_BIN=$(which aria2c 2> /dev/null)

# ----- do some checks first -----
if [ ! -x "$ARIA2_BIN" ]; then
  error "aria2c was not found or isn't executable."
  exit 1
fi

if [ $# -ne 2 ]; then
  error "Incorrect number of arguments"
  exit 1
fi

filename=$(basename ${1})
server=${1%/${filename}}

# count the number of "/" in the server parameter, subtract 1 to give the field number of the repo

repofield=`expr $( echo ${server} | tr -cd "/" | wc -c ) - 1`

# extract the repo (core, extra etc)

repo=$( echo ${server} | cut -d/ -f${repofield} )

msg "Fetching from repo ${repo}"

# Override number of connections for db files
if [[ ${filename} = *.db.tar.gz ]]; then
  CONNECTIONS=1
fi

# For db files, or when using a custom repo (which most likely doesn't have any mirror),
# use only the URL passed by pacman; Otherwise, extract the list of servers (from the include file of the repo) to download from
if [[ ${filename} = *.db.tar.gz ]]; then
  url="${1}"
else
  url=$( sed s"/Server = //" ${MIRRORLIST} | sed s"/\$repo/${repo}/" | sed s"/$/\/${filename}/" | head -n ${SERVERS} )  
fi

# Passive FTP or not?
[ "${FTP_PASV}" = "yes" ] && OPT_FTP_PASV="--ftp-pasv" || OPT_FTP_PASV=""

msg "Downloading ${filename} \n"

cd /var/cache/pacman/pkg/

${ARIA2_BIN} --log=${LOG} --timeout=${TIMEOUT} --max-tries=${MAX_TRIES} --allow-overwrite=true \
             --split=${CONNECTIONS} ${OPT_FTP_PASV} --file-allocation=${FILE_ALLOC} \
             --lowest-speed-limit=${MIN_SPEED} --max-download-limit=${MAX_SPEED} \
             ${url} --out=${filename}.pacget && [ ! -f ${filename}.pacget.aria2 ] && mv ${filename}.pacget ${2} && chmod 644 ${2}

exit $?

sabooky · 2008-01-19 09:03:30

If you feel your script is stable enough please update the wiki.
http://wiki.archlinux.org/index.php/Imp … ror_Script

Sorry I haven't really contributed much to the script since writing it, mainly because I don't use it anymore, but also because I suck with bash :\
Anyways, if no one fixes it up in the next few days I'll give it a shot.

vacant · 2008-01-19 10:23:21

I've updated the wiki with a version that should also work with a custom repo.

sabooky · 2008-01-20 01:58:35

vacant wrote:

I've updated the wiki with a version that should also work with a custom repo.

I updated the script, I think your mirrorlist file is different than mine, because yours didn't parse the mirrors correctly for me. (comments and such threw it off)

Anyways, here are the changes I made:
- Check /etc/pacman.conf for the appropriate mirror file (figured since technically it's possible to use other files, might as well support it, also makes it backwards compatible)
- Changed the sed code to be more lenient in parsing mirrorlist. (comments and differences in whitespace)
- Cleaned up the code here and there. (or broke it, guess we'll have to wait and see)

Like I mentioned earlier, I don't use the pacget script, so there may be some problems with it. If there are any issues, please let me know.

vacant · 2008-01-20 09:40:47

Thanks, that worked fine with my mirrorlist (rankmirrors, only european servers) downloading 123MB of updates this morning.

Arch Linux

#1 2008-01-17 11:42:13

Fully functioning pacget script for pacman 3?

#2 2008-01-17 14:32:11

Re: Fully functioning pacget script for pacman 3?

#3 2008-01-17 14:43:48

Re: Fully functioning pacget script for pacman 3?

#4 2008-01-17 15:46:20

Re: Fully functioning pacget script for pacman 3?

#5 2008-01-18 16:54:10

Re: Fully functioning pacget script for pacman 3?

#6 2008-01-19 09:03:30

Re: Fully functioning pacget script for pacman 3?

#7 2008-01-19 10:23:21

Re: Fully functioning pacget script for pacman 3?

#8 2008-01-20 01:58:35

Re: Fully functioning pacget script for pacman 3?

#9 2008-01-20 09:40:47

Re: Fully functioning pacget script for pacman 3?

Board footer