Pacman2Aria2 & Powerpill-light

Xyne · 2011-04-04 04:01:46

Info page: http://xyne.archlinux.ca/projects/pacman2aria2/

Read the info page for up-to-date information. Basically, it can be used in pipes between pacman and aria2c to emulate powerpill downloads (with reflector-augmented mirrorlists). It's not really an end-user application, but rather a component for user scripts.*

The package includes a default script named "powerpill-light" than can be used as a replacement for powerpill for simple upgrades, e.g.

powerpill-light -u kde

Note the absence of "-S". Copy the file somewhere and edit it to suite your needs. There are some commented examples in there to get you started.

To be completely clear, this is not a full powerpill replacement, but it should be useful for speeding up downloads.

I just threw this together, so feel free to suggest improvements. One idea that I have in mind is to make "powerpill-light" a script that just source a bash file at /etc/powerpill-light.conf. Then the user could edit that file while still using /usr/bin/powerpill-light.

Also, post any useful scripts that you create with this so others might benefit and be inspired.

* Really, basic scripting will make it much more flexible than it would be if I tried to wrap it in in an app with fixed options.

4javier · 2011-04-08 10:02:04

Hi xyne
Into the italian community some of us are working on a bash script wrapping pacman that allow multiple and segmented downloads of the packages by aria2. I didn't think that you should be releasing updated tools so soon, I think our project has the same goal of your pacman2aria2
Anyway, we are making that also for didactical reasons, so we'll keep going on.
I'd like to know if in your program you are able to caught some pacman exit statuses, or if you like us are forced to parse the simple string output of pacman with all the controindications that this system carry on:
1) localized output
2) particular behaviour (SyncFirst packages, package substitution...etc) that make pacman output not the classic and simple list of packages (after some manipulations, of course)
3) many others actually I ignore...
Thanks in advance and sorry for disturb.

Xyne · 2011-04-08 19:07:35

Hi 4javier,

pacman2aria2 just parses the URL list from pacman when run with "pacman -p --print-format '%r %l'. The list is piped into pacman2aria2, so it does not run pacman directly and it does now know or care about the exit status of the program that generates the output.

Powerpill-light uses pipes to move data around so it will fail if any component in the pipe fails. In that sense, it is aware of the exit status of each component and will halt if any component fails.

Localization is not an issue either because the recognized output does not contain any localized strings, only repo names and package URLs.

Both the pacman2aria2 and powerpill-light scripts are very simple. Just open them in an editor and take a look at what they do. The first is written in Python3, the second is a very simple Bash script.

4javier · 2011-04-08 19:22:36

# pacman -Sup --print-format '%r %l'
:: Aggiornamento del sistema in corso...
testing ftp://ftp5.gwdg.de/pub/linux/archlinux/ … pkg.tar.xz
testing ftp://ftp5.gwdg.de/pub/linux/archlinux/ … pkg.tar.xz
testing ftp://ftp5.gwdg.de/pub/linux/archlinux/ … pkg.tar.xz
testing ftp://ftp5.gwdg.de/pub/linux/archlinux/ … pkg.tar.xz
testing ftp://ftp5.gwdg.de/pub/linux/archlinux/ … pkg.tar.xz
testing ftp://ftp5.gwdg.de/pub/linux/archlinux/ … pkg.tar.xz
testing ftp://ftp5.gwdg.de/pub/linux/archlinux/ … pkg.tar.xz

Bold part is localized, I strip it by sed '/::/d' into my script. But there are many other possible output from pacman, not always prepended by ::, and of various legths. i.e. when a package is proposed to substitute an existing one. Did you find a way to generalize your method, or have we to modify our sed manipulation for every possible output?

Xyne · 2011-04-08 20:07:06

Just ignore that output. You can grep for ".pkg." for example, or more complicated regexes such as a word followed by a space then a URL.

Please take a look at the code for pacman2aria2. You can see exactly how I parse the input on lines 44-49. It's dead simple.

Last edited by Xyne (2011-04-08 20:07:34)

4javier · 2011-04-08 20:50:46

Ok. I read your code. Thus neither you found a way to manage unexpected pacman output. I think I won't be able too, if pacman doesn't expose different exit status for different cases.
Thanks a lot and sorry to bother you

Xyne · 2011-04-09 00:38:35

If you're only worried about internationalization, you can invoke pacman with "LC_ALL=C pacman ...".

Beyond that, you should only need to check whether it exited successfully.

lolilolicon · 2011-04-09 08:10:01

Hi, Xyne. Powerpill was great. Nice to see a replacement from you

I have written a similar script in bash, which does multi-threaded
downloading using metalink and aria2c. For anyone who's interested to test it
out, here it is on github:
https://github.com/lolilolicon/pacmaria2

Last edited by lolilolicon (2011-04-09 08:11:40)

4javier · 2011-04-09 08:36:37

@xyne
I just wanna know exactly if pacman throws some exit status replying to query. But I think it exposes one just when it ends. ie:
actual behaviour

pacman -Syu
"there are these pkgs to upgrade"
....
....
....
do you wanna do it?Y/n
Y
exit status 0

other case

pacman -Syu
"hey, wait a moment. you should upgrade pacman first"
do you wanna do it?[S/n]
[my reply]
"now you can re-execute pacman -Syu
exit status 0

How I'd like it beahaves

pacman -Syu
"there are these pkgs to upgrade"
[b]control code = 0[/b]
....
....
....
do you wanna do it?Y/n
Y
exit status 0

pacman -Syu
"hey, wait a moment. you should upgrade pacman first"
[b]control code = 1[/b]
do you wanna do it?[Y/n]
Y
"now you can re-execute pacman -Syu
exit status 0

A control code based on query outcome. Hope I have been able to get it clear, sorry but my english is poor.

graysky · 2011-05-30 08:05:17

Comment: Thanks xyne. Powerpill was one of your greatest hits in my book I really like the idea of sourcing /etc/powerpill-light.conf in the script btw.

Question: I'm using the default /usr/bin/powerpill-light and am little confused by the comments in regarding the -y switch. If I use powerpill-light as my pacman wrapper, and I wish to do a complete system update (equivalent to an old "powerpill -Syu"), how should I do it? Is this right: "powerpill-light -u" or...? If I understand the bash right, that would omit the -y switch entirely yet in the pacman man page:

-y, --refresh
Download a fresh copy of the master package list from the server(s) defined in pacman.conf(5). This should
typically be used each time you use --sysupgrade or -u. Passing two --refresh or -y flags will force a
refresh of all package lists even if they appear to be up to date.

Should I add a line to call a "pacman -Sy" then invoke the rest of the wrapper like:

#!/bin/bash
pacman-color -Sy && pacman-color -Sp --print-format '%r %l' "$@" | \
pacman2aria2 -l 50 | \
aria2c --lowest-speed-limit=50K --continue --log-level=warn --max-concurrent-downloads=50 --input-file - --dir=/var/cac$
pacman-color -S "$@"

Last edited by graysky (2011-05-30 08:07:48)

sausageandeggs · 2011-05-30 09:24:57

You just leave off the 'S' so a full refresh/update is ' powerpill-light -yu '

Xyne · 2011-06-09 18:36:52

Sorry for the late reply.

The problem is that powerpill-light invokes pacman twice with the same arguments, first to get the download list by appending '-p', and then after the packages have been downloaded to install them. If you include '-y' then pacman will check for database updates twice. It doesn't really matter because all it does is send a few redundant HTTP HEAD requests to check for new databases, but it's unnecessary.

Ideally powerpill-light should parse and filter the command-line arguments, but it's really only meant to be a very simple temporary substitute until I get the time to write a worthy powerpill replacement.

I usually just use "powerpill-light -u" because I still have cron running "pacman -Sy".

Bah, I'm tired and probably rambling.

Quickguide
For any pacman operation with "-S", remove "-S" and pass the rest of the command to powerpill-light, e.g.
pacman -Syu -> powerpill-light -yu
pacman -S xorg -> powerpill-light xorg

Yannick_LM · 2011-06-10 20:03:59

completely off-topic, but this comment from the source code made me laugh
(pacma2aria2 is written in Python, powerpill was written in Perl. ...)

# Perlish... so what?
          print( '\t'.join( map(lambda m: MIRROR_URL_FORMAT.format(m, repo, arch) + '/' + os.path.basename(url), mirrors) ) )

Keep up good work !

Xyne · 2011-06-15 21:19:25

Yannick_LM wrote:

completely off-topic, but this comment from the source code made me laugh
(pacma2aria2 is written in Python, powerpill was written in Perl. ...)
# Perlish... so what?
          print( '\t'.join( map(lambda m: MIRROR_URL_FORMAT.format(m, repo, arch) + '/' + os.path.basename(url), mirrors) ) )
Keep up good work !

I forget that some people actually read my code.

Stebalien · 2011-06-16 01:04:01

Thanks for another great tool.
If you want it to be more pythonic/easier to read, try using list comprehensions instead of maps and lambdas:

-    mirrors = list( map(lambda m: m['url'], mirrors) )
+    mirrors = [ m['url'] for m in mirrors ]

-    print( '\t'.join( map(lambda m: MIRROR_URL_FORMAT.format(m, repo, arch) + '/' + os.path.basename(url), mirrors) ) )
+    urlbase = os.path.basename(url)
+    print( '\t'.join( [ MIRROR_URL_FORMAT.format(m, repo, arch) + "/" + urlbase for m in mirrors ] ) )

Also, there is an easier way to fill a dictionary:

-    arches = {}
-    for arch in ARCHITECTURES:
-        arches[arch] = 0
+    arches = dict.fromkeys(ARCHITECTURES, 0)

Xyne · 2011-06-16 04:58:59

@Stebalien
Good tips, thanks. I've updated the script.

Stebalien · 2011-06-16 06:38:33

Xyne wrote:

@Stebalien
Good tips, thanks. I've updated the script.

Just FYI, you forgot to remove lines 39-41 .

Last edited by Stebalien (2011-06-16 06:39:51)

Xyne · 2011-06-16 14:10:17

Um, I meant to do that, cuz the script needs to practice... yeah, that's my story.

Removed, thanks.

gee · 2011-09-05 00:40:16

Hey there, thanks for these scripts!

In case that helps anyone I have modified the powerpill-light script to be usable from yaourt.

Here is the section (may be ugly code, I am no bash dev)

input="$@"

if [[ $input == -S* ]]; then
        input=${input:2}

        input=${input/--force -/-f}

        if [[ ${input:0:1} == " " ]]; then
                input=${input:1}
        fi

        if [[ ${input:0:1} != "-" ]]; then
                input="-"$input
        fi

        pacman -Sp --print-format '%r %l' "$input" | \
        pacman2aria2 -l 50 | \
        aria2c --allow-overwrite=true -c --file-allocation=none --max-connection-per-server=6 --max-file-not-found=5 --min-split-size=1M --no-conf --lowest-speed-limit=50K --log-level=warn $

        input=${input/-yy/-}
        input=${input/-y/-}

        if [[ -n $input ]] && [[ $input != "-" ]]; then
                pacman-color -S "$input"
        fi
else
        pacman-color $input
fi

Last edited by gee (2011-09-05 01:41:36)

gee · 2011-09-05 09:42:58

ok that version is actually not fully bullet proof, and finishing it in pure bash is too much for me

TheSaint · 2011-09-05 12:29:16

Try this>>>>

#! /usr/bin/env python
# -*- coding: utf-8 -*-
# TAB size 3

#            utilities to allow multiple downloads for pacman by aria2c
# ###############################by _______ Fulvio##############################
#                    email fulvio AT pc DOT jaring dot my

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import os, sys, re, time, socket, shutil
from subprocess import Popen, PIPE
from subprocess import getstatusoutput as chkout
from tempfile import NamedTemporaryFile
from configparser import ConfigParser as ConfPar
#from optparse import OptionParser as Opar
from random import shuffle
from xmlrpc import client as xmlrpclib
from code import InteractiveConsole as IC

# Set variable to default
BIN = '/usr/bin/aria2c'
DIR = '/tmp/paq'
PACDIR = '/var/lib/pacman'
Path= os.path
RE = re.compile('.*(ht|f)tp://.*',re.IGNORECASE)
PAC_CONF = '/etc/pacman.conf'
update= ''
CMD='pacman -S'+ update+ 'up --print-format '+ "'%r %l %s'"
REPOSITORIES = ['community', 'community-staging', 'community-testing', 'core',
	'extra', 'gnome-unstable', 'kde-unstable', 'multilib', 'multilib-testing'
	'staging', 'testing']
EOL= os.linesep
#PACDESTDIR = '/var/cache/pacman/pkg'
PACDESTDIR = '/tmp/pkg'
C_MAIN = '\033[1;37;40m'			# main text
C_OTHER = '\033[1;34;40m'			# prefix & brackets
C_BUSY = '\033[0;33;40m'			# busy
C_FAIL = '\033[1;31;40m'			# failed
C_DONE = '\033[1;37;40m'			# completed
C_CLEAR = '\033[1;0m'


#Opar.add_option('-m', '--multi', dest='multi',  default=False, action='store_true',
#help='Stand-alone function to download all packages and store them into normal pacman cache (/var/cache/pacman/pkg)')

#Opar.add_option('-d', '--dir',dest='PACDESTDIR',  default='/var/lib/pacman', 
#action='store_true', type='string',
#help='Directory where to move the downloaded packages. If none is given then they will remain in /tmp/paq')

def one_pkg(uri, outfile):
	# non existing dir will be made
	if not Path.isdir(DIR): os.mkdir(DIR)
	outdir = Path.dirname(outfile)
	tempfile = Path.basename(outfile) + '.paq'
	# searching what's the repo in the given uri
	used_repos= [r for r in uri.split('/') if r in REPOSITORIES]
	if len(used_repos) > 0:
		mirr_D= get_mirrors(PAC_CONF, used_repos)
		uris= aria_one(uri, mirr_D[str(used_repos[0])])
	else: uris= uri
	with NamedTemporaryFile(mode='w', prefix='paq-', delete=False) as tmp:
		tmp.writelines(uris)
		# following options must start the line with a space !!! Like " dir=/tmp/xxx"
		tmp.write(EOL + ' dir='+ DIR+ EOL+ ' out='+ tempfile + EOL)
	aria2= Popen([BIN, '--input-file='+ tmp.name], stdout=PIPE)
	name = Path.basename(uri).rsplit('.', 3)[0]; recurse= True; width= term_width()
	while recurse:
		for lin in aria2.stdout:
			data = str(lin.decode()).strip()
			if data.startswith('[#1'):
				log(width, name, data.strip('[#1 ]'), e='\r')
			elif data.startswith('(OK):'):
				log(width, name, 'DONE')
				recurse= False
				break
			elif data.startswith('(ERR):'): log(width, name, 'FAIL')
		aria2.wait()
	if aria2.returncode == 0:
		# it got it right and save it to destination
		shutil.copy2(DIR+ '/'+ tempfile, outdir)
		# removing temporary debris
		os.rename(tempfile, outfile)
		os.remove(DIR+ '/'+ tempfile)
		os.remove(tmp.name)

class aria_handler():
	''' class to activate aria2 and send RPC commands to it'''

	def __init__(self, port= 6800, passwd= '', user= ''):
		self.handle= self.passwd= self.port= self.user= ''
		if passwd: self.passwd= ' --rpc-passwd='+ passwd
		self.numport= port
		if port != 6800: self.port= ' --rpc-listen-port='+ str(port)
		if user: self.user= ' --rpc-user='+ user
		self.ARIA_CMD= 'aria2c -D --enable-rpc'+ \
							self.passwd+ self.port+ self.user
		self.handle= \
		xmlrpclib.ServerProxy('http://localhost:%s/rpc' %int(self.numport))
		self.multi_call= xmlrpclib.MultiCall(self.handle)
		try:
			if self.handle.aria2.getVersion() != '': return
		except socket.error:
			pass
		# when aria2c is running, then don't try it again
		try: chkout(self.ARIA_CMD)
		except: raise SystemExit('aria2c is not working as daemon')
		# everything is good, it will return an handle
		return

	def addUri(self, uriList, extra_options):
		return self.handle.aria2.addUri(uriList, extra_options)

	def read_status(self, DL_ID):
		'''Function to report download statuses and some other extra'''
		return self.handle.aria2.tellStatus(str(DL_ID))

	def active_status(self, showed):
		'''Function to report actual downloads statuses. The showed should give
		particular data, see aria2c manual'''
		return self.handle.aria2.tellActive(showed)

	def shutDown(self):
		'''Trying to complete and kill aria daemon. the method will return OK'''
		if self.handle:
			return self.handle.aria2.shutdown()
		return None

	def forceShutDown(self):
		'''Trying to complete and kill aria daemon, the method return OK, but all
		the download might be suddenly stopped'''
		if salf.handle:
			return self.handle.aria2.forceShutdown()
		return None

def log(width, label, msg, e= EOL):

	'''Pretty one line terminal logger. Label would be put to the left side in
	white boldface and msg would be to the extremeright side, according to
	width given value'''
	size= width - len(label) - len(msg) - 4
	#when ecceding the width, then label will be truncated
	if size < 0: label= label[:(size-2)]
	spaces = ' ' * size; status = C_BUSY
	if msg == 'DONE': status = C_DONE
	elif msg == 'FAIL': status = C_FAIL
	#import pdb; pdb.set_trace()
	print(' %s%s%s%s%s[%s%s%s]%s' % (C_MAIN, label, C_CLEAR, spaces,
	C_OTHER, status, msg, C_OTHER, C_CLEAR), end=e)
	sys.stdout.flush()

def term_width():

	''' A easy way to know the terminal window width'''
	return int(Popen(['stty', 'size'],
					stdout=PIPE).communicate()[0].split()[1])

def get_mirrors(config, which_repos):

	'''retrieving a mirror list from a given config file. The file should comply
	to standard ConfigParser parsing method
	If a list of repositories is included, then only those will be returned'''

	cp= ConfPar(); cp.read(config); itms= {}
	# getting common repositories, if none is provided
	if isinstance(which_repos,list) and len(which_repos) > 0:
		std_repos=  which_repos
	else:
		std_repos= set(cp.sections()) & set(REPOSITORIES)
		# checking if  was added a particular repository
		new_repos= set(cp.sections()) - set(REPOSITORIES)
	# remove options from pacman.conf which aren't referring to a server
		for srvr in new_repos:
			if 'server' in cp.options(srvr):
				std_repos.add(srvr)
	# here it gets the mirrors for every listed repository
	for repo in std_repos:
		if 'include' in cp.options(repo):
			f = cp.get(repo, 'include')
			itms[repo]= read_mirr_list(f, repo)
		elif 'server' in cp.options(repo):
			itms[repo]= cp.get(repo, 'server')
		else: itms[repo]= ''
	# return a repositories dictionary
	return itms

def read_mirr_list(_file, repo):

	'''reading all mirrors from included file. A shuffled list is returned '''
	mirrors= []
	with open(_file, 'r') as f:
		for line in f:
			line = line.strip()
			if line.startswith('Server'):
				path= line.split('=')[-1].lstrip().replace('$repo', repo)
				mirrors.append(path.replace('/$arch',''))
	shuffle(mirrors)
	return mirrors
	
def get_pkgList(CMD, EOL = ''):
	''' Try to get the package list updated from pacman output or saved in a file'''
	if isinstance(CMD, str):
		a= chkout(CMD)[1]
		# Convert the buffer into a string
		b = a.split('\n')
	else:
		a = CMD; b = []
		for l in CMD:
			l = l.rstrip('\n')
			if not RE.match(l): continue
			if l.split(' ',2)[2] == '0':
				continue
			b.append(l)
	# it doesn't convert properly, return empty list
	if type(b) is not list: return []
	repo_list= []; pkgs = {}; total_size= 0
	for pkg_listed in b:
		#skip non-url sentences
		if not RE.match(pkg_listed): continue
		repo, url, size= pkg_listed.strip().split(' ', 2)
		total_size += int(size)
		try:
			pkgs[repo].append(url)
		except KeyError:
			pkgs[repo] = [url]
	return pkgs, total_size

def aria_all(pkg_D, mirr_D, retList= False):

	'''A composed message to be send to aria2c for the download
	This method will download all the packages from the supplied list'''
	aria_str= []
	for item in pkg_D.keys():
		for pkg in pkg_D[item]:
			if not RE.match(pkg):
				raise SystemExit('String is '+ pkg)
			aria_single= aria_one(pkg, mirr_D[item], '', retList)
			aria_str.append(aria_single)
	return aria_str

def aria_one(url, mrList= '', sep= '\t', retList= False):

	'''A composed message to be send to aria2c for the download
	This method will compose for a single package with multiple mirrors
	retList determine if the data will pack either in a string or in a list
	The choice is made because of sending command via xmlrpc or CLI'''
	tmpList= []
	if mrList == '': return url
	dlstring= ''
	for mrr in mrList:
		uri= mrr + '/' + url_strip(url)
		if retList:
			tmpList.append(uri)
		else:
			dlstring += (uri + sep)
	if retList: return tmpList
	return dlstring.rstrip(sep)

def url_strip(url):

	'''small reformat url and architecture'''
	arch= url.rsplit('/', 2)[-2]
	url= os.path.basename(url)
	return '%s/%s' %(arch,url)

def query_yes_no(question, default="yes"):

	"""Ask a yes/no question via raw_input() and return their answer.
	"question" is a string that is presented to the user.
	"default" is the presumed answer if the user just hits <Enter>.
	It must be "yes" (the default), "no" or None (meaning
	an answer is required of the user).
	The "answer" return value is one of "yes" or "no"."""
	valid = {"yes":"yes",   "y":"yes",  "ye":"yes", "no":"no", "n":"no"}
	if default == None:
		prompt = " [y/n] "
	elif default == "yes":
		prompt = " [Y/n] "
	elif default == "no":
		prompt = " [y/N] "
	else:
		raise ValueError("invalid default answer: '%s'" % default)

	while 1:
		sys.stdout.write(question + prompt)
		choice = IC.raw_input('').lower()
		if default is not None and choice == '':
			return default
		elif choice in valid.keys():
			return valid[choice]
		else:
			sys.stdout.write("Please respond with 'yes' or 'no' "\
							 "(or 'y' or 'n').\n")

def multi_pkg():
	root = (os.geteuid() == 0); update= ''
	try:
		CMD= open(infile)
	except (NameError, IOError):
		if root:
			if	query_yes_no('do you want update pacman database ') == 'yes':
				update= 'y'
		CMD= 'pacman -S'+ update+ 'up --print-format '+ "'%r %l %s'"
	# non existing dir will be made
	if not Path.isdir(PACDESTDIR): os.mkdir(PACDESTDIR)
	print('Reading packages database, it might take a while', end= '\r')
	pkg_D, totalsize= get_pkgList(CMD)
	options={'dir': DIR}
	print(' ' * (term_width()-2), end= '\r')
	if len(pkg_D) < 1 or totalsize == 0:
		sys.exit(' No new packages to download')
	if	query_yes_no('Want you download {:,} Kbytes '.format(int(totalsize/1024))) == 'no':
		sys.exit(0)
	mirr_D= get_mirrors(PAC_CONF, list(pkg_D.keys()))
	aria_multi= aria_all( pkg_D, mirr_D, retList= True)
	a= aria_handler(); pkg_D= {}
	for aria_single in aria_multi:
		DL_ID= a.addUri(aria_single, options)
		pkg_D[DL_ID]=''
	try:
		tdloaded= 1
		while len(pkg_D) > 0:
			for pkgID in pkg_D:
				b= a.read_status(pkgID)
				if 'complete' in b['status']:
					tdloaded += int(b['totalLength'])
					try:
						pkg_D.pop(b['gid'])
						f= b['files'][0]['path']
						shutil.move(f,PACDESTDIR)
					except IndexError:
						pass
					except shutil.Error:
						if Path.isfile(PACDESTDIR+ '/'+ Path.basename(f)):
							os.remove(f)
					break
				rcvd_byte= max_speed= 0
				b= a.active_status(['downloadSpeed', 'completedLength'])
				for di in b:
					max_speed += int(di['downloadSpeed'])
					rcvd_byte += int(di['completedLength'])
				percent= (tdloaded+ rcvd_byte)/ totalsize* 100
				max_speed= max_speed/ 1024; sp= 'K'
				if max_speed > 1024:
					max_speed= max_speed/ 1024; sp= 'M'
				log(term_width(),
				'Downloading {0} files. {1:.0f}% . Total speed>>'.format(len(b), percent),
				'{:07.2f} {:s}b/s'.format(max_speed, sp), e= '\r')
	except KeyboardInterrupt:
		print('Stopped process')
	finally:
		shutil.rmtree(options['dir'],ignore_errors= True)
		a.shutDown()
		if root:
			if	query_yes_no('\nDo you want install the downloaded packages? ') == 'no':
				sys.exit(0)
			a= chkout('pacman -Su --noconfirm')
			if a[0]:
				print('Error while updating, please read pacman log')

if __name__ == "__main__":
	#import pdb; pdb.set_trace()
	if '--single' in sys.argv:
		try: uri, outfile, method = sys.argv[1:]
		except ValueError: raise SystemExit('incorrect number of arguments')
		one_pkg(uri, outfile)
		sys.exit(0)
	if len(sys.argv)== 1:
		multi_pkg()
	if '--multi' in sys.argv:
		try: infile = sys.argv[2]
		except (IndexError,ValueError): raise SystemExit('No file')
		multi_pkg()

Usage:
First otion
-------------
It can be used in conjunction with pacman as per XferCommand = /usr/bin/ap %u %o --single in pacman.conf. This will do multiple server and split packages for many servers as listed on mirrorlist or any include in pacman.conf. Therefore care should be taken to reduce to minimum the server number, which it could be done by reflector. Reflector isn't called here, but it might be imported and made operational.

Second option
-------------------
It can be use as stand-alone and it will download al packages in PACDESTDIR which is by default /var/cache/pacman/pkg/. If necessary can modify it for somewhere else which won't need root permission.
There's a chance to download a from a list that is a bash command like

pacman -Sup --print-format '%r %l %s'

This can be issued as normal user, if root is used, then also -Syup is possible.
Supposed the name is ap then if it's used with a list the command would be

ap --multi dl-list.txt

OR

su -c ap

Examples:

ap -- It will download the packages into a public folder, but must set PACDESTDIR to some accessible folder

su -c ap -- it will download all new packages and if PACDESTDIR is the default it will be possible to proceed to update the new packages without visible report

ap --multi download_list-- download from another list as explained above. IF superuser does it, won't do much difference all the packages going according to PACDESTDIR

TODO - add some option parser to allow non positional option to be taken and allow to choose the packages destination folder.

Some small issues are there. In particular the refreshing on the downloading speed. Some time switching from pacman/yaourt use to standalone will cause an error. Just ignore and retry.

Testers and critics are welcome

Last edited by TheSaint (2011-09-05 12:33:34)

gee · 2011-09-05 21:09:52

If you do it as part of XferCommand then you cannot download multiple packages in parallel no?

TheSaint · 2011-09-05 21:42:24

gee wrote:

If you do it as part of XferCommand then you cannot download multiple packages in parallel no?

Yup, it does a package at time.
I mean, it's invoked by pacman, it will prepare a list of link from several mirrors for the requested package and pass that list to aria2c to download. Aria2 should have a version => 1.12. Even the databases are arranged in such a way. The benefit isn't that great for small packages.

One of the point I left out is regarding the aria2c configuration.
Here is a proposed one:

# The log file
#log-level=error
#log=/tmp/log/aria2.log
# Server timeout period
timeout=5
# 'none' or 'falloc'
file-allocation=none
max-file-not-found=5
uri-selector=adaptive
remote-time=true
max-concurrent-downloads=8
continue=true
split=4
max-connection-per-server=4
max-tries=2
min-split-size=1M
allow-overwrite=true
summary-interval=0
timeout=5
#quiet=true

I've put it in /root/.aria2/aria2.conf because I'm commonly using as root. I wouldn't have it for general purpose and set it into /etc.
If used as stand-alone program, then it would be faster. Just to compare with pacman/yaourt the speed will go at 400%. Without any limit on download it will go for the maximum modem speed, when it's clear from traffic bottlenecks

Xyne · 2012-03-20 18:13:08

Anyone using pacman2aria2 will probably be interested in pm2ml.

Arch Linux

#1 2011-04-04 04:01:46

Pacman2Aria2 & Powerpill-light

#2 2011-04-08 10:02:04

Re: Pacman2Aria2 & Powerpill-light

#3 2011-04-08 19:07:35

Re: Pacman2Aria2 & Powerpill-light

#4 2011-04-08 19:22:36

Re: Pacman2Aria2 & Powerpill-light

#5 2011-04-08 20:07:06

Re: Pacman2Aria2 & Powerpill-light

#6 2011-04-08 20:50:46

Re: Pacman2Aria2 & Powerpill-light

#7 2011-04-09 00:38:35

Re: Pacman2Aria2 & Powerpill-light

#8 2011-04-09 08:10:01

Re: Pacman2Aria2 & Powerpill-light

#9 2011-04-09 08:36:37

Re: Pacman2Aria2 & Powerpill-light

#10 2011-05-30 08:05:17

Re: Pacman2Aria2 & Powerpill-light

#11 2011-05-30 09:24:57

Re: Pacman2Aria2 & Powerpill-light

#12 2011-06-09 18:36:52

Re: Pacman2Aria2 & Powerpill-light

#13 2011-06-10 20:03:59

Re: Pacman2Aria2 & Powerpill-light

#14 2011-06-15 21:19:25

Re: Pacman2Aria2 & Powerpill-light

#15 2011-06-16 01:04:01

Re: Pacman2Aria2 & Powerpill-light

#16 2011-06-16 04:58:59

Re: Pacman2Aria2 & Powerpill-light

#17 2011-06-16 06:38:33

Re: Pacman2Aria2 & Powerpill-light

#18 2011-06-16 14:10:17

Re: Pacman2Aria2 & Powerpill-light

#19 2011-09-05 00:40:16

Re: Pacman2Aria2 & Powerpill-light

#20 2011-09-05 09:42:58

Re: Pacman2Aria2 & Powerpill-light

#21 2011-09-05 12:29:16

Re: Pacman2Aria2 & Powerpill-light

#22 2011-09-05 21:09:52

Re: Pacman2Aria2 & Powerpill-light

#23 2011-09-05 21:42:24

Re: Pacman2Aria2 & Powerpill-light

#24 2012-03-20 18:13:08

Re: Pacman2Aria2 & Powerpill-light

Board footer