powerpill reborn: pacman wrapper for faster downloads

Xyne · 2013-03-13 22:10:45

@dapolinario
It should be fixed in powerpill-2013.3. Thanks for reporting it.

dapolinario · 2013-03-14 00:12:26

Solved. Thanks!

dapolinario · 2013-03-18 03:05:07

Xyne, aria2c has been updated, but still the files are downloaded every time I send packages to install. Do you know if the problem is fixed?

Xyne · 2013-03-18 08:02:10

That problem is fixed but now I have found another bug in aria2c

Basically, the "--continue" option is required to avoid redundant downloads, but enabling it with "--split" generates invalid range headers from most servers. For now I have disabled splitting and enabled continue. Update the package and merge the aria2/args changes from the new powerpill.json file. Let me know if it works as expected.

I'll contact the dev again.

dapolinario · 2013-03-18 09:23:42

It works as expected. Thanks!

Xyne · 2013-03-18 22:02:53

The range header errors were due to a regression upstream. The dev sent me a patch. You can find an updated PKGBUILD with the patch here if you do not want to wait for the upstream release. Rebuild and edit powerpill.json to re-enable splitting.

dapolinario · 2013-03-19 03:18:35

Thank you! It worked perfectly.

dapolinario · 2013-03-22 10:35:12

@Xyne
Why -Syy option don't work?

Xyne · 2013-03-22 19:58:48

dapolinario wrote:

@Xyne
Why -Syy option don't work?

It should be fixed in powerpill-2013.3.22. Please test it and let me know.

dapolinario · 2013-03-23 01:23:40

Works fine!

dapolinario · 2013-03-24 22:36:24

@Xyne
Have you noticed that when run -Sy, normally break any repository? Then I have to run -Syy to correct. It's a problem with aria2? Have you thought about leaving the pacman update repositories and powerpill only download them? I prefer powerpill doing all the work.

Last edited by dapolinario (2013-03-24 22:36:42)

Xyne · 2013-03-25 18:28:57

dapolinario wrote:

@Xyne
Have you noticed that when run -Sy, normally break any repository? Then I have to run -Syy to correct. It's a problem with aria2?

Yes, I have noticed and yes, it is an aria2c bug. I'll try to figure out the right set of options for database downloads.

Have you thought about leaving the pacman update repositories and powerpill only download them? I prefer powerpill doing all the work.

I might add an option to use pacman internally for database downloads but I would rather just fix the current bug. If you prefer to use pacman then you can create an alias that runs "pacman -Sy; powerpill..".

Xyne · 2013-03-25 19:23:30

Let me know if the problem persists in powerpill-2013.3.25.

dapolinario · 2013-03-25 23:49:07

There is no more corruptin, just downloading every time I run command. Rather than downloading corrupted. Thanks.

Last edited by dapolinario (2013-03-27 00:03:36)

Xyne · 2013-03-26 21:19:54

dapolinario wrote:

There is corrupting more, just downloading every time I run command. Rather than downloading corrupted. Thanks.

After more testing it seems that it depends on the mirror. According to the Aria2 man page, it uses the "If-Modified-Since" HTTP header. Not all mirrors appear to support this.

You have 3 choices:

use a different mirror
use an rsync server for database downloads
create a script or alias to use pacman for database downloads

dapolinario · 2013-04-05 12:17:52

Xyne, the powerpill will survive the upgrade pacman?

Xyne · 2013-04-05 17:53:44

dapolinario wrote:

Xyne, the powerpill will survive the upgrade pacman?

Why wouldn't it?

dapolinario · 2013-04-05 18:04:17

After updating to version 4.1 pacman, the powerpill not work correctly. I thought it might be something more complex.

Xyne · 2013-04-05 19:26:02

dapolinario wrote:

After updating to version 4.1 pacman, the powerpill not work correctly. I thought it might be something more complex.

Please post the error message and/or an explanation of the problem instead of vague remarks that force me to play a cat-and-mouse game to get the information I need.

dapolinario · 2013-04-05 20:36:36

Sorry.

daniel ~ $ cat /etc/pacman.conf 
#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives

#
# GENERAL OPTIONS
#
[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir     = /
#DBPath      = /var/lib/pacman/
#CacheDir    = /var/cache/pacman/pkg/
#LogFile     = /var/log/pacman.log
#GPGDir      = /etc/pacman.d/gnupg/
HoldPkg     = pacman glibc
#XferCommand = /usr/bin/curl -C - -f %u > %o
#XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#CleanMethod = KeepInstalled
#UseDelta    = 0.7
Architecture = auto

# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
#IgnorePkg   =
#IgnoreGroup =

#NoUpgrade   =
#NoExtract   =

# Misc options
#UseSyslog
Color
#TotalDownload
CheckSpace
#VerbosePkgLists

# By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages.
SigLevel    = Required DatabaseOptional
LocalFileSigLevel = Optional
#RemoteFileSigLevel = Required

# NOTE: You must run `pacman-key --init` before first using pacman; the local
# keyring can then be populated with the keys of all official Arch Linux
# packagers with `pacman-key --populate archlinux`.

#
# REPOSITORIES
#   - can be defined here or included from another file
#   - pacman will search repositories in the order defined here
#   - local/custom mirrors can be added here or in separate files
#   - repositories listed first will take precedence when packages
#     have identical names, regardless of version number
#   - URLs will have $repo replaced by the name of the current repo
#   - URLs will have $arch replaced by the name of the architecture
#
# Repository entries are of the format:
#       [repo-name]
#       Server = ServerName
#       Include = IncludePath
#
# The header [repo-name] is crucial - it must be present and
# uncommented to enable the repo.
#

# The testing repositories are disabled by default. To enable, uncomment the
# repo name header and Include lines. You can add preferred servers immediately
# after the header, and they will be used before the default mirrors.

#[testing]
#Include = /etc/pacman.d/mirrorlist

[core]
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /etc/pacman.d/mirrorlist

#[community-testing]
#Include = /etc/pacman.d/mirrorlist

[community]
Include = /etc/pacman.d/mirrorlist

# If you want to run 32 bit applications on your x86_64 system,
# enable the multilib repositories as required here.

#[multilib-testing]
#Include = /etc/pacman.d/mirrorlist

[multilib]
Include = /etc/pacman.d/mirrorlist

[repo-ck]
SigLevel = PackageRequired
Server = http://repo-ck.com/$arch

[xyne-x86_64]
SigLevel = Required
Server = http://xyne.archlinux.ca/repos/xyne

# An example of a custom package repository.  See the pacman manpage for
# tips on creating your own repositories.
#[custom]
#SigLevel = Optional TrustAll
#Server = file:///home/custompkgs

daniel ~ $ sudo powerpill -Syu
[sudo] password for daniel: 
/usr/lib/python3.3/site-packages/pycman/config.py:130: InvalidSyntax: unable to parse /etc/pacman.conf, unrecognized option: 'Color'
  warnings.warn(InvalidSyntax(f.name, 'unrecognized option', key))
/usr/lib/python3.3/site-packages/pycman/config.py:125: InvalidSyntax: unable to parse /etc/pacman.conf, unrecognized option: 'LocalFileSigLevel'
  warnings.warn(InvalidSyntax(f.name, 'unrecognized option', key))

Resultados do Download:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
d7cf3c|OK  |       0B/s|/var/lib/pacman/sync/multilib.db
121b63|OK  |        n/a|/var/lib/pacman/sync/xyne-x86_64.db.sig
acef60|OK  |    41KiB/s|/var/lib/pacman/sync/xyne-x86_64.db
050c3b|OK  |   110KiB/s|/var/lib/pacman/sync/repo-ck.db
b99fb4|OK  |       0B/s|/var/lib/pacman/sync/community.db
ead835|OK  |       0B/s|/var/lib/pacman/sync/core.db
18c340|OK  |   243KiB/s|/var/lib/pacman/sync/extra.db
6eb8eb|ERR |       0B/s|/var/lib/pacman/sync/core.db.sig
b47364|ERR |       0B/s|/var/lib/pacman/sync/multilib.db.sig
2f02c8|ERR |       0B/s|/var/lib/pacman/sync/extra.db.sig
58d48a|ERR |       0B/s|/var/lib/pacman/sync/community.db.sig

Legenda de estados:
(OK):download completo.(ERR):ocorreu um erro.

aria2 continuará o download se a transferência for reiniciada.
Caso tenha havido algum erro, veja o arquivo de log. Veja a opção '-l' na ajuda/página de manual para detalhes.

Last edited by dapolinario (2013-04-05 20:43:45)

Xyne · 2013-04-05 21:04:50

dapolinario wrote:

SigLevel    = Required DatabaseOptional
...
[core]
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /etc/pacman.d/mirrorlist

[community]
Include = /etc/pacman.d/mirrorlist

Resultados do Download:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
d7cf3c|OK  |       0B/s|/var/lib/pacman/sync/multilib.db
121b63|OK  |        n/a|/var/lib/pacman/sync/xyne-x86_64.db.sig
acef60|OK  |    41KiB/s|/var/lib/pacman/sync/xyne-x86_64.db
050c3b|OK  |   110KiB/s|/var/lib/pacman/sync/repo-ck.db
b99fb4|OK  |       0B/s|/var/lib/pacman/sync/community.db
ead835|OK  |       0B/s|/var/lib/pacman/sync/core.db
18c340|OK  |   243KiB/s|/var/lib/pacman/sync/extra.db
6eb8eb|ERR |       0B/s|/var/lib/pacman/sync/core.db.sig
b47364|ERR |       0B/s|/var/lib/pacman/sync/multilib.db.sig
2f02c8|ERR |       0B/s|/var/lib/pacman/sync/extra.db.sig
58d48a|ERR |       0B/s|/var/lib/pacman/sync/community.db.sig

Legenda de estados:
(OK):download completo.(ERR):ocorreu um erro.

aria2 continuará o download se a transferência for reiniciada.
Caso tenha havido algum erro, veja o arquivo de log. Veja a opção '-l' na ajuda/página de manual para detalhes.

You are telling it to look for signature files:

SigLevel    = Required DatabaseOptional

but they do not exist. When I tested it, aria2c reported the same error on my system, but it continued to download packages. You should be able to ignore the error. Otherwise, change the SigLevel to "PackageRequired" for the official repos.

[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

dapolinario wrote:

daniel ~ $ sudo powerpill -Syu
[sudo] password for daniel: 
/usr/lib/python3.3/site-packages/pycman/config.py:130: InvalidSyntax: unable to parse /etc/pacman.conf, unrecognized option: 'Color'
  warnings.warn(InvalidSyntax(f.name, 'unrecognized option', key))
/usr/lib/python3.3/site-packages/pycman/config.py:125: InvalidSyntax: unable to parse /etc/pacman.conf, unrecognized option: 'LocalFileSigLevel'
  warnings.warn(InvalidSyntax(f.name, 'unrecognized option', key))

This is a pyalpm error. It should not affect powerpill's operation. You can file a bug report about it on the tracker.

silmaa · 2013-04-11 12:29:59

Hi,

I just installed powerpill (version 2013.3.26), but I have problems using it :

Traceback (most recent call last):
  File "/usr/lib/python3.3/urllib/request.py", line 1252, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.3/http/client.py", line 1061, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.3/http/client.py", line 1099, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.3/http/client.py", line 1057, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.3/http/client.py", line 902, in _send_output
    self.send(msg)
  File "/usr/lib/python3.3/http/client.py", line 840, in send
    self.connect()
  File "/usr/lib/python3.3/http/client.py", line 1202, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python3.3/ssl.py", line 210, in wrap_socket
    _context=self)
  File "/usr/lib/python3.3/ssl.py", line 310, in __init__
    raise x
  File "/usr/lib/python3.3/ssl.py", line 306, in __init__
    self.do_handshake()
  File "/usr/lib/python3.3/ssl.py", line 513, in do_handshake
    self._sslobj.do_handshake()
socket.timeout: _ssl.c:531: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.3/site-packages/Reflector.py", line 171, in retrieve
    with urllib.request.urlopen(MirrorStatus.URL, None, self.connection_timeout) as f:
  File "/usr/lib/python3.3/urllib/request.py", line 160, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.3/urllib/request.py", line 473, in open
    response = self._open(req, data)
  File "/usr/lib/python3.3/urllib/request.py", line 491, in _open
    '_open', req)
  File "/usr/lib/python3.3/urllib/request.py", line 451, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.3/urllib/request.py", line 1287, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib/python3.3/urllib/request.py", line 1255, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error _ssl.c:531: The handshake operation timed out>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/powerpill", line 822, in <module>
    exit(run_main())
  File "/usr/bin/powerpill", line 815, in run_main
    return main(args)
  File "/usr/bin/powerpill", line 802, in main
    powerpill.download(pm2ml_args)
  File "/usr/bin/powerpill", line 507, in download
    build_download_queue(pm2ml_args, conf=self.pacman_conf)
  File "/usr/lib/python3.3/site-packages/pm2ml.py", line 734, in build_download_queue
    mirrorstatus, mirrors = process_options(reflector_options)
  File "/usr/lib/python3.3/site-packages/Reflector.py", line 568, in process_options
    mirrors = ms.get_mirrors()
  File "/usr/lib/python3.3/site-packages/Reflector.py", line 217, in get_mirrors
    return self.get_obj()['urls']
  File "/usr/lib/python3.3/site-packages/Reflector.py", line 210, in get_obj
    self.retrieve()
  File "/usr/lib/python3.3/site-packages/Reflector.py", line 176, in retrieve
    raise MirrorStatusError('failed to retrieve mirror data: %s' % (e))
Reflector.MirrorStatusError: 'failed to retrieve mirror data: <urlopen error _ssl.c:531: The handshake operation timed out>'

My pacman.conf :

#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives

#
# GENERAL OPTIONS
#
[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir     = /
#DBPath      = /var/lib/pacman/
#CacheDir    = /var/cache/pacman/pkg/
#LogFile     = /var/log/pacman.log
#GPGDir      = /etc/pacman.d/gnupg/
HoldPkg     = pacman glibc
# If upgrades are available for these packages they will be asked for first
# SyncFirst   = pacman
#XferCommand = /usr/bin/curl -C - -f %u > %o
#XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#CleanMethod = KeepInstalled
Architecture = auto

# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
#IgnorePkg   =
#IgnoreGroup =

#NoUpgrade   =
#NoExtract   =

# Misc options
#UseSyslog
#UseDelta
#TotalDownload
CheckSpace
#VerbosePkgLists

# By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages.
#SigLevel = Optional TrustedOnly

# NOTE: You must run `pacman-key --init` before first using pacman; the local
# keyring can then be populated with the keys of all official Arch Linux
# packagers with `pacman-key --populate archlinux`.

#
# REPOSITORIES
#   - can be defined here or included from another file
#   - pacman will search repositories in the order defined here
#   - local/custom mirrors can be added here or in separate files
#   - repositories listed first will take precedence when packages
#     have identical names, regardless of version number
#   - URLs will have $repo replaced by the name of the current repo
#   - URLs will have $arch replaced by the name of the architecture
#
# Repository entries are of the format:
#       [repo-name]
#       Server = ServerName
#       Include = IncludePath
#
# The header [repo-name] is crucial - it must be present and
# uncommented to enable the repo.
#

# The testing repositories are disabled by default. To enable, uncomment the
# repo name header and Include lines. You can add preferred servers immediately
# after the header, and they will be used before the default mirrors.

#[testing]
#SigLevel = PackageRequired
#Include = /etc/pacman.d/mirrorlist

[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[extra]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

#[community-testing]
#SigLevel = PackageRequired
#Include = /etc/pacman.d/mirrorlist

[community]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[multilib]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

[archlinuxfr]
Server = http://repo.archlinux.fr/$arch
SigLevel = Never


# If you want to run 32 bit applications on your x86_64 system,
# enable the multilib repositories as required here.

#[multilib-testing]
#SigLevel = PackageRequired
#Include = /etc/pacman.d/mirrorlist


# An example of a custom package repository.  See the pacman manpage for
# tips on creating your own repositories.
#[custom]
#SigLevel = Optional TrustAll
#Server = file:///home/custompkgs

I didn't modify the powerpill.conf

{
  "aria2": {
    "args": [
      "--allow-overwrite=true",
      "--always-resume=false",
      "--auto-file-renaming=false",
      "--check-integrity=true",
      "--conditional-get=true",
      "--continue=true",
      "--file-allocation=none",
      "--log-level=error",
      "--max-concurrent-downloads=100",
      "--max-connection-per-server=5",
      "--min-split-size=5M",
      "--show-console-readout=true",
      "--split=10"
    ],
    "path": "/usr/bin/aria2c"
  },
  "pacman": {
    "config": "/etc/pacman.conf",
    "path": "/usr/bin/pacman"
  },
  "pacserve": {
    "server": null
  },
  "powerpill": {
    "ask": true,
    "reflect databases": false
  },
  "reflector": {
    "args": [
      "--protocol",
      "http",
      "--latest",
      "50"
    ]
  },
  "rsync": {
    "args": [
      "--no-motd",
      "--progress"
    ],
    "db only": true,
    "path": "/usr/bin/rsync",
    "servers": []
  }
}

Thanks in advance

Xyne · 2013-04-11 18:16:47

Hi silmaa,

It looks like a transient error. Either the Mirrorstatus server was not responding when you ran the command, or there was a problem with your internet connection. I have updated pm2ml to catch this error and print a warning instead of exiting, but the underlying problem is beyond my control. If you see the error often then you should check your connection.

Yaro · 2013-04-18 16:48:26

I am getting absolutely flooded with messages like this one:

04/18 11:45:50 [ERROR] CUID#394 - Download aborted. URI=http://mirror.nexcess.net/archlinux/extra/os/x86_64/extra.db.sig
Exception: [AbstractCommand.cc:304] errorCode=3 URI=http://mirror.nexcess.net/archlinux/extra/os/x86_64/extra.db.sig
  -> [HttpSkipResponseCommand.cc:211] errorCode=3 Resource not found
[DL:0B UL:0B][#cd5f69 0B/0B][#34276f 0B/0B][#877752 0B/0B][#a3f5a0 0B/0B]                                                                                                                                                                              
04/18 11:45:51 [ERROR] CUID#395 - Download aborted. URI=http://ftp.df.lth.se/pub/archlinux/extra/os/x86_64/extra.db.sig
Exception: [AbstractCommand.cc:304] errorCode=3 URI=http://ftp.df.lth.se/pub/archlinux/extra/os/x86_64/extra.db.sig
  -> [HttpSkipResponseCommand.cc:211] errorCode=3 Resource not found

It goes on like this for several minutes, then sits around until I press enter, and I get this:

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
7d040c|OK  |    65KiB/s|/var/lib/pacman/sync/core.db
5d1cd0|OK  |    50KiB/s|/var/lib/pacman/sync/multilib.db
481b6e|OK  |   184KiB/s|/var/lib/pacman/sync/extra.db
2bb171|OK  |   119KiB/s|/var/lib/pacman/sync/community.db
cd5f69|ERR |       0B/s|/var/lib/pacman/sync/core.db.sig
34276f|ERR |       0B/s|/var/lib/pacman/sync/extra.db.sig
877752|ERR |       0B/s|/var/lib/pacman/sync/community.db.sig
a3f5a0|ERR |       0B/s|/var/lib/pacman/sync/multilib.db.sig

I haven't changed the default settings for powerpill at all.

Xyne · 2013-04-18 21:20:33

Your mirror(s) in /etc/pacman.conf are likely out of sync with the mirrors retrieved by reflector. You can do one of the following:

* Look at the Mirror Status page and select a mirror that matches the default Reflector options in /etc/powerpill/powerpill.json
* Adjust the Reflector options to try to match your main mirrors.
* Switch to an Rsync mirror.

Read the output of "reflector --help" to see the available Reflector options.

Arch Linux

#76 2013-03-13 22:10:45

Re: powerpill reborn: pacman wrapper for faster downloads

#77 2013-03-14 00:12:26

Re: powerpill reborn: pacman wrapper for faster downloads

#78 2013-03-18 03:05:07

Re: powerpill reborn: pacman wrapper for faster downloads

#79 2013-03-18 08:02:10

Re: powerpill reborn: pacman wrapper for faster downloads

#80 2013-03-18 09:23:42

Re: powerpill reborn: pacman wrapper for faster downloads

#81 2013-03-18 22:02:53

Re: powerpill reborn: pacman wrapper for faster downloads

#82 2013-03-19 03:18:35

Re: powerpill reborn: pacman wrapper for faster downloads

#83 2013-03-22 10:35:12

Re: powerpill reborn: pacman wrapper for faster downloads

#84 2013-03-22 19:58:48

Re: powerpill reborn: pacman wrapper for faster downloads

#85 2013-03-23 01:23:40

Re: powerpill reborn: pacman wrapper for faster downloads

#86 2013-03-24 22:36:24

Re: powerpill reborn: pacman wrapper for faster downloads

#87 2013-03-25 18:28:57

Re: powerpill reborn: pacman wrapper for faster downloads

#88 2013-03-25 19:23:30

Re: powerpill reborn: pacman wrapper for faster downloads

#89 2013-03-25 23:49:07

Re: powerpill reborn: pacman wrapper for faster downloads

#90 2013-03-26 21:19:54

Re: powerpill reborn: pacman wrapper for faster downloads

#91 2013-04-05 12:17:52

Re: powerpill reborn: pacman wrapper for faster downloads

#92 2013-04-05 17:53:44

Re: powerpill reborn: pacman wrapper for faster downloads

#93 2013-04-05 18:04:17

Re: powerpill reborn: pacman wrapper for faster downloads

#94 2013-04-05 19:26:02

Re: powerpill reborn: pacman wrapper for faster downloads

#95 2013-04-05 20:36:36

Re: powerpill reborn: pacman wrapper for faster downloads

#96 2013-04-05 21:04:50

Re: powerpill reborn: pacman wrapper for faster downloads

#97 2013-04-11 12:29:59

Re: powerpill reborn: pacman wrapper for faster downloads

#98 2013-04-11 18:16:47

Re: powerpill reborn: pacman wrapper for faster downloads

#99 2013-04-18 16:48:26

Re: powerpill reborn: pacman wrapper for faster downloads

#100 2013-04-18 21:20:33

Re: powerpill reborn: pacman wrapper for faster downloads

Board footer