You are not logged in.

#1 2015-01-10 17:27:13

amish
Member
Registered: 2014-05-10
Posts: 333

pacman and aria2 - better configuration?

Today I used some tips from:
https://wiki.archlinux.org/index.php/im … figuration

And configured pacman to use aria2.

However I found existing wiki instructions bit complicated. So I made my own config which I would like to share (and also get reviews/opinions).

Wiki entry has too many config parameters fit in one single line which made things look too complex.

First thing I did is that I moved everything to a config file and change XferCommand to:

XferCommand = echo Downloading %u ... && /usr/bin/aria2c --conf-path=/root/.aria2/pacman-aria2.conf %u

Config file /root/.aria2/pacman-aria2.conf contains this:

# error handling
timeout=60
connect-timeout=30
max-tries=5
retry-wait=10
max-file-not-found=1

# downloading
split=3
max-connection-per-server=3
min-split-size=1M
max-concurrent-downloads=1
file-allocation=none
remote-time=true
conditional-get=true
no-netrc=true

# resuming
continue=true
allow-overwrite=true
always-resume=false

# proxy
#http-proxy=127.0.0.1:8080
#https-proxy=127.0.0.1:8080
#ftp-proxy=127.0.0.1:8080

# console
#quiet=true
console-log-level=warn
summary-interval=0
#enable-color=false
#human-readable=false
#show-console-readout=false
#truncate-console-readout=false

# logging
log-level=warn
log=/var/log/pacman-aria2.log

Above config adds better error logging and better display on console. And also proxy config for aria2

This also adds support for resuming download based on timestamp+size using conditional-get option of aria2.

Major change about this configuration (from one in wiki) is that, this config does not write to .part files but directly writes to the destination file.

This change has one advantage and one limitation because of the way pacman calls XferCommand.

Advantage is that repository DB files are not downloaded everytime the "pacman -Syu" is run. They are downloaded only if there is change.

Limitation is that if for some reason the PKG file is not downloaded completely (say due to internet connectivity issues) then when next time you run "pacman -Syu", pacman will report file integrity error and prompts you if to delete the file or not?

This is because when pacman runs the next time, it assumes that if PKG file exists it does not need to be downloaded and does not call XferCommand at all for that file. And hence XferCommand does not get opportunity to complete the broken download. Pacman does not realize that file is incompletely downloaded.

In my opinion when XferCommand is specified, pacman should ALWAYS call it and let XferCommand decide if file is downloaded completely or not. Afterall XferCommand are normally supposed to be "smarter downloaders" than pacman's own internal downloading mechanism.

NOTE:
If you have frequent connectivity issues, then above can be reversed by use of .part file i.e. adding -d / -o %o to XferCommand (not tested thoroughly)

XferCommand = echo Downloading %u ... && /usr/bin/aria2c --conf-path=/root/.aria2/pacman-aria2.conf -d / -o %o %u

But in that case repository DB files will be re-downloaded everytime you run "pacman -Syu" (even if they are not changed)


Please give your opinions. Thank you smile

Last edited by amish (2015-01-10 17:32:48)


Forum signature: I discuss. I put my thoughts strongly. But I definitely respect all developers and time they put in.

Offline

#2 2015-01-10 20:24:26

TheSaint
Member
From: my computer
Registered: 2007-08-19
Posts: 1,411

Re: pacman and aria2 - better configuration?

I'm using this for pacman and makepkg.conf.

max-concurrent-downloads=8
continue
split=4
max-connection-per-server=4
max-tries=3
min-split-size=1M
allow-overwrite=true
log-level=error
#log=-
remote-time=true
timeout=5
uri-selector=inorder
summary-interval=0

My downloader is slightly different because I'm using my program (It's a python script)
So the pacman.conf line is

XferCommand = /home/user/mypgm/4pacman/pfu.py --single %u %o

Definitely one may put the program under system wide path


do it good first, it will be faster than do it twice the saint wink

Offline

#3 2015-05-09 01:01:00

thoffmeyer
Member
From: /dev/zero
Registered: 2006-07-27
Posts: 90
Website

Re: pacman and aria2 - better configuration?

TheSaint wrote:

I'm using this for pacman and makepkg.conf.

max-concurrent-downloads=8
continue
split=4
max-connection-per-server=4
max-tries=3
min-split-size=1M
allow-overwrite=true
log-level=error
#log=-
remote-time=true
timeout=5
uri-selector=inorder
summary-interval=0

My downloader is slightly different because I'm using my program (It's a python script)
So the pacman.conf line is

XferCommand = /home/user/mypgm/4pacman/pfu.py --single %u %o

Definitely one may put the program under system wide path

I have your script setup but it just stalls, then eventually outputs this and fails:

┌[tim@muramasa][16:58:02]
└[~/x/pfu]> pacman -Syu
:: Synchronizing package databases...
core                                                                    [DONE]
core                                                                    [FAIL]
> /home/tim/x/pfu/pfu.py(250)__call__()
-> if int(b['errorCode']) > 0:
(Pdb)

It does not download anything.I don't know if it needs updating or if i'm missing something but I do have aria2 installed, along with configs in ~/.aria2/aria2.conf and /root/.aria2/aria2.conf`

More goodness, I don't know python so I'm not much help but from what I see it seems that it's having trouble downloading the sigs..

┌[tim@muramasa][21:15:25]
└[~/x/pfu]> pacman -Syu
:: Synchronizing package databases...
core                                                                    [DONE]
core                                                                    [FAIL]
> /home/tim/x/pfu/pfu.py(250)__call__()
-> if int(b['errorCode']) > 0:
(Pdb)
(Pdb) exit
Traceback (most recent call last):
  File "/home/tim/x/pfu/pfu.py", line 246, in __call__
    shutil.copy2(tmpfile, outdir)
  File "/usr/lib/python3.4/shutil.py", line 245, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.4/shutil.py", line 108, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/paq/core.db.sig.part'

Last edited by thoffmeyer (2015-05-09 01:17:09)

Offline

#4 2015-05-09 12:11:46

TheSaint
Member
From: my computer
Registered: 2007-08-19
Posts: 1,411

Re: pacman and aria2 - better configuration?

Sorry there are some debugging routines included in the file. So, when it shows

(Pdb)

You may issue a command like help to understand what to do. Else you may just issue quit to leave the debugger or you may give continue to proceed the program.
I'm going to have a look for better results
Meanwhile you may have the following options

  • return to pacman defaults

  • you may try out the multi options, which is the standalone program that will fill the cache up to the latest packages. Usually I do

    # pfu -Vyn7 -r
    # pacman -Su
  • correct the program by commenting (or remove) line 248 and 249. Also the line 251 (before removing lines) to substitute pass with die(255).
    By commenting I mean to put a # a the line beginning

A new link will substitute the current one, once I'll find a better solution

Last edited by TheSaint (2015-05-09 12:32:49)


do it good first, it will be faster than do it twice the saint wink

Offline

#5 2015-05-09 12:22:00

tomk
Forum Fellow
From: Ireland
Registered: 2004-07-21
Posts: 9,839

Re: pacman and aria2 - better configuration?

amish wrote:

Advantage is that repository DB files are not downloaded everytime the "pacman -Syu" is run. They are downloaded only if there is change.

Just FYI, this is the way pacman works anyway, regardless of the downloader used.

Offline

#6 2015-05-09 12:38:14

TheSaint
Member
From: my computer
Registered: 2007-08-19
Posts: 1,411

Re: pacman and aria2 - better configuration?

In my case is not happening. They are downloaded anyway. I don't know how pacman uses libcurl to detect the file properties and skip the download.


do it good first, it will be faster than do it twice the saint wink

Offline

#7 2015-05-09 13:30:15

tomk
Forum Fellow
From: Ireland
Registered: 2004-07-21
Posts: 9,839

Re: pacman and aria2 - better configuration?

I don't know how it does it either smile, but this is what it does here on two consecutive -Sy operations:

$ pacman -Sy
:: Synchronizing package databases...
 testing is up to date
 core is up to date
 extra                                              1735.9 KiB   262K/s 00:07 [------------------------------------------------------------------------------] 100%
 community-testing is up to date
 community                                            2.6 MiB   288K/s 00:09 [------------------------------------------------------------------------------] 100%
 multilib-testing is up to date
 multilib is up to date
$ pacman -Sy
:: Synchronizing package databases...
 testing is up to date
 core is up to date
 extra is up to date
 community-testing is up to date
 community is up to date
 multilib-testing is up to date
 multilib is up to date

I guess you could look at the source code to figure it out.

Anyway, not entirely relevant, please continue your work.

Offline

#8 2015-05-09 14:31:36

amish
Member
Registered: 2014-05-10
Posts: 333

Re: pacman and aria2 - better configuration?

In my case too pacman downloads DB file everytime.

It seems its server dependent. For  some server it autodetects if file is changed or not and for others it downloads everytime.


Forum signature: I discuss. I put my thoughts strongly. But I definitely respect all developers and time they put in.

Offline

#9 2015-05-10 00:08:05

TheSaint
Member
From: my computer
Registered: 2007-08-19
Posts: 1,411

Re: pacman and aria2 - better configuration?

That is what mine doing.
Here's the new version, but it's rather a beta testing (which hasn't testers big_smile)
There's a little idle time if there're no download, but it's a problem  mostly related with aria2c. Some times it goes to time-out.

Any comment is welcome, but no promises implied that I'll fix bugs in a short time. Let me say it goes as fast as the feedbacks are coming in .


do it good first, it will be faster than do it twice the saint wink

Offline

Board footer

Powered by FluxBB