pacman and aria2 - better configuration?

amish · 2015-01-10 17:27:13

Today I used some tips from:
https://wiki.archlinux.org/index.php/im … figuration

And configured pacman to use aria2.

However I found existing wiki instructions bit complicated. So I made my own config which I would like to share (and also get reviews/opinions).

Wiki entry has too many config parameters fit in one single line which made things look too complex.

First thing I did is that I moved everything to a config file and change XferCommand to:

XferCommand = echo Downloading %u ... && /usr/bin/aria2c --conf-path=/root/.aria2/pacman-aria2.conf %u

Config file /root/.aria2/pacman-aria2.conf contains this:

# error handling
timeout=60
connect-timeout=30
max-tries=5
retry-wait=10
max-file-not-found=1

# downloading
split=3
max-connection-per-server=3
min-split-size=1M
max-concurrent-downloads=1
file-allocation=none
remote-time=true
conditional-get=true
no-netrc=true

# resuming
continue=true
allow-overwrite=true
always-resume=false

# proxy
#http-proxy=127.0.0.1:8080
#https-proxy=127.0.0.1:8080
#ftp-proxy=127.0.0.1:8080

# console
#quiet=true
console-log-level=warn
summary-interval=0
#enable-color=false
#human-readable=false
#show-console-readout=false
#truncate-console-readout=false

# logging
log-level=warn
log=/var/log/pacman-aria2.log

Above config adds better error logging and better display on console. And also proxy config for aria2

This also adds support for resuming download based on timestamp+size using conditional-get option of aria2.

Major change about this configuration (from one in wiki) is that, this config does not write to .part files but directly writes to the destination file.

This change has one advantage and one limitation because of the way pacman calls XferCommand.

Advantage is that repository DB files are not downloaded everytime the "pacman -Syu" is run. They are downloaded only if there is change.

Limitation is that if for some reason the PKG file is not downloaded completely (say due to internet connectivity issues) then when next time you run "pacman -Syu", pacman will report file integrity error and prompts you if to delete the file or not?

This is because when pacman runs the next time, it assumes that if PKG file exists it does not need to be downloaded and does not call XferCommand at all for that file. And hence XferCommand does not get opportunity to complete the broken download. Pacman does not realize that file is incompletely downloaded.

In my opinion when XferCommand is specified, pacman should ALWAYS call it and let XferCommand decide if file is downloaded completely or not. Afterall XferCommand are normally supposed to be "smarter downloaders" than pacman's own internal downloading mechanism.

NOTE:
If you have frequent connectivity issues, then above can be reversed by use of .part file i.e. adding -d / -o %o to XferCommand (not tested thoroughly)

XferCommand = echo Downloading %u ... && /usr/bin/aria2c --conf-path=/root/.aria2/pacman-aria2.conf -d / -o %o %u

But in that case repository DB files will be re-downloaded everytime you run "pacman -Syu" (even if they are not changed)

Please give your opinions. Thank you

Last edited by amish (2015-01-10 17:32:48)

TheSaint · 2015-01-10 20:24:26

I'm using this for pacman and makepkg.conf.

max-concurrent-downloads=8
continue
split=4
max-connection-per-server=4
max-tries=3
min-split-size=1M
allow-overwrite=true
log-level=error
#log=-
remote-time=true
timeout=5
uri-selector=inorder
summary-interval=0

My downloader is slightly different because I'm using my program (It's a python script)
So the pacman.conf line is

XferCommand = /home/user/mypgm/4pacman/pfu.py --single %u %o

Definitely one may put the program under system wide path

thoffmeyer · 2015-05-09 01:01:00

TheSaint wrote:

I'm using this for pacman and makepkg.conf.
max-concurrent-downloads=8
continue
split=4
max-connection-per-server=4
max-tries=3
min-split-size=1M
allow-overwrite=true
log-level=error
#log=-
remote-time=true
timeout=5
uri-selector=inorder
summary-interval=0
My downloader is slightly different because I'm using my program (It's a python script)
So the pacman.conf line is
XferCommand = /home/user/mypgm/4pacman/pfu.py --single %u %o
Definitely one may put the program under system wide path

I have your script setup but it just stalls, then eventually outputs this and fails:

┌[tim@muramasa][16:58:02]
└[~/x/pfu]> pacman -Syu
:: Synchronizing package databases...
core                                                                    [DONE]
core                                                                    [FAIL]
> /home/tim/x/pfu/pfu.py(250)__call__()
-> if int(b['errorCode']) > 0:
(Pdb)

It does not download anything.I don't know if it needs updating or if i'm missing something but I do have aria2 installed, along with configs in ~/.aria2/aria2.conf and /root/.aria2/aria2.conf`

More goodness, I don't know python so I'm not much help but from what I see it seems that it's having trouble downloading the sigs..

┌[tim@muramasa][21:15:25]
└[~/x/pfu]> pacman -Syu
:: Synchronizing package databases...
core                                                                    [DONE]
core                                                                    [FAIL]
> /home/tim/x/pfu/pfu.py(250)__call__()
-> if int(b['errorCode']) > 0:
(Pdb)
(Pdb) exit
Traceback (most recent call last):
  File "/home/tim/x/pfu/pfu.py", line 246, in __call__
    shutil.copy2(tmpfile, outdir)
  File "/usr/lib/python3.4/shutil.py", line 245, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/usr/lib/python3.4/shutil.py", line 108, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/paq/core.db.sig.part'

Last edited by thoffmeyer (2015-05-09 01:17:09)

TheSaint · 2015-05-09 12:11:46

Sorry there are some debugging routines included in the file. So, when it shows

(Pdb)

You may issue a command like help to understand what to do. Else you may just issue quit to leave the debugger or you may give continue to proceed the program.
I'm going to have a look for better results
Meanwhile you may have the following options

return to pacman defaults
you may try out the multi options, which is the standalone program that will fill the cache up to the latest packages. Usually I do
```
# pfu -Vyn7 -r
# pacman -Su
```
correct the program by commenting (or remove) line 248 and 249. Also the line 251 (before removing lines) to substitute pass with die(255).
By commenting I mean to put a # a the line beginning

A new link will substitute the current one, once I'll find a better solution

Last edited by TheSaint (2015-05-09 12:32:49)

tomk · 2015-05-09 12:22:00

amish wrote:

Advantage is that repository DB files are not downloaded everytime the "pacman -Syu" is run. They are downloaded only if there is change.

Just FYI, this is the way pacman works anyway, regardless of the downloader used.

TheSaint · 2015-05-09 12:38:14

In my case is not happening. They are downloaded anyway. I don't know how pacman uses libcurl to detect the file properties and skip the download.

tomk · 2015-05-09 13:30:15

I don't know how it does it either , but this is what it does here on two consecutive -Sy operations:

$ pacman -Sy
:: Synchronizing package databases...
 testing is up to date
 core is up to date
 extra                                              1735.9 KiB   262K/s 00:07 [------------------------------------------------------------------------------] 100%
 community-testing is up to date
 community                                            2.6 MiB   288K/s 00:09 [------------------------------------------------------------------------------] 100%
 multilib-testing is up to date
 multilib is up to date
$ pacman -Sy
:: Synchronizing package databases...
 testing is up to date
 core is up to date
 extra is up to date
 community-testing is up to date
 community is up to date
 multilib-testing is up to date
 multilib is up to date

I guess you could look at the source code to figure it out.

Anyway, not entirely relevant, please continue your work.

amish · 2015-05-09 14:31:36

In my case too pacman downloads DB file everytime.

It seems its server dependent. For some server it autodetects if file is changed or not and for others it downloads everytime.

TheSaint · 2015-05-10 00:08:05

That is what mine doing.
Here's the new version, but it's rather a beta testing (which hasn't testers )
There's a little idle time if there're no download, but it's a problem mostly related with aria2c. Some times it goes to time-out.

Any comment is welcome, but no promises implied that I'll fix bugs in a short time. Let me say it goes as fast as the feedbacks are coming in .

citrusagent · 2024-12-11 16:07:31

Thanks for your contribution. Can you provide a permalink?

V1del · 2024-12-11 18:21:48

You are responding to a 9 year old post, none of these files and options are relevant to today.

The reason to use aria was parallel download support, which is part of baseline pacman now.

Closing this old thread, please don't necrobump: https://wiki.archlinux.org/title/Genera … bumping%22

Arch Linux

#1 2015-01-10 17:27:13

pacman and aria2 - better configuration?

#2 2015-01-10 20:24:26

Re: pacman and aria2 - better configuration?

#3 2015-05-09 01:01:00

Re: pacman and aria2 - better configuration?

#4 2015-05-09 12:11:46

Re: pacman and aria2 - better configuration?

A new link will substitute the current one, once I'll find a better solution

#5 2015-05-09 12:22:00

Re: pacman and aria2 - better configuration?

#6 2015-05-09 12:38:14

Re: pacman and aria2 - better configuration?

#7 2015-05-09 13:30:15

Re: pacman and aria2 - better configuration?

#8 2015-05-09 14:31:36

Re: pacman and aria2 - better configuration?

#9 2015-05-10 00:08:05

Re: pacman and aria2 - better configuration?

#10 2024-12-11 16:07:31

Re: pacman and aria2 - better configuration?

#11 2024-12-11 18:21:48

Re: pacman and aria2 - better configuration?

Board footer