You are not logged in.
It seems that rsync in powerpill is broken. I did the modifications to the json file as described in previous posts, and got this:
$ sudo powerpill -Su
Traceback (most recent call last):
File "/usr/bin/powerpill", line 794, in <module>
sys.exit(run_main())
File "/usr/bin/powerpill", line 787, in run_main
return main(args)
File "/usr/bin/powerpill", line 704, in main
pacman_conf = pm2ml.PacmanConfig(pargs['pacman_config'])
File "/usr/lib/python3.3/site-packages/pm2ml.py", line 72, in __init__
self.load_from_file(conf)
File "/usr/lib/python3.3/site-packages/pm2ml.py", line 90, in load_from_file
self.repos[key] = self[key]['Server']
KeyError: 'Server'
Anyway, what is the advantage of using rsync servers over http/https servers?
Offline
It seems that rsync in powerpill is broken. I did the modifications to the json file as described in previous posts
Please post the actual file. I suspect that you have made a mistake. Post your pacman.conf and mirrorlist as well.
Anyway, what is the advantage of using rsync servers over http/https servers?
rsync is a much more efficient protocol for transferring files because that is what it was designed to do. It can handle multiple files in a single connection and it can use efficient compression and existing data to reduce the amount of data transferred.
HTTP is a verbose and less efficient protocol (all of the metadata is sent as plain text) and most servers and clients require multiple connections for multiple files even if the current standard supports multiple transfers within a single connection. That means that on top of all the hypertext babble, multiple tcp connections must be established too. It takes longer and the connection is not saturated at all times.
For most users I would expect the best rsync mirror's connection to be faster than their own internet connection. For users with bandwidths that surpass the mirrors (either due to great speed or mirror throttling), parallel connections via https will be faster. Powerpill doesn't try to manage parallel rsync because that would just create unnecessary complexity with limited use.
If you need more than that, read up on the rsync and HTTP protocols.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
There's is part of the json where i have done modification to.
"rsync": {
"args": [
"--no-motd",
"--progress"
],
"db only": true,
"path": "/usr/bin/rsync",
"servers": ["rsync://ftp.swin.edu.au/archlinux/$repo/os/$arch"]
}
My mirrorlist is empty. I want to make powerpill use an rsync server.
OK, here is the problem, when I fill up mirrorlist with a http server powerpill worked, but it will not download the packages from the rsync server.
Thanks for the detailed explanation on rsync vs https/http. Didn't get a good answer from Dr. Google.
Offline
"db only": true
That limits rsync to updating the database. It still requires HTTP mirrors for downloading packages. Set it to false if you want to download packages via rsync as well. Please read the powerpill.json man page ("man powerpill.json").
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Thanks Xyne. Setting that to false worked. The mirrorlist must contain at least one http server though.
Offline
Hi.
I just installed powerpill, did the update but it fails when trying to download the .sig files
b5d7d7|ERR | 0B/s|/var/lib/pacman/sync/extra.db.sig
899e91|ERR | 0B/s|/var/lib/pacman/sync/multilib.db.sig
8fcc32|ERR | 0B/s|/var/lib/pacman/sync/core.db.sig
85eb3d|ERR | 0B/s|/var/lib/pacman/sync/community.db.sig
I have a few servers in /etc/pacman.d/mirrorlist, and one rsync server. I've tried with "db only" false and true.
Last edited by Viper_Scull (2014-05-14 10:17:22)
Athlon II X4 620 + Gigabyte 785GPM-UD2H + 4GB DDR3 + SSD OCZ Vertex2 60GB
Archlinux x86_64 + Openbox
Offline
IIRC the databases are not signed, *.db.sig files don't exist.
How exactly did you try to update and did it go OK apart from the database signatures errors? It worked for me.
Offline
I did 'powerpill -Syu' and just in case 'powerpill -Syy'. Both gave the same errors.
This is the pastebin:
http://pastebin.com/gtDAbsJQ
Athlon II X4 620 + Gigabyte 785GPM-UD2H + 4GB DDR3 + SSD OCZ Vertex2 60GB
Archlinux x86_64 + Openbox
Offline
I've done a program, similar to powerpill. Just doesn't pass all argument like pacman one.
I included a call to Reflector, so the program can use the freshest mirrors on-the-fly. Moreover it includes pyalpm and aria2 XLMRPC.
If anybody out there likes to try it, just MP to me, I appreciate if some would give me some test about it.
do it good first, it will be faster than do it twice the saint
Offline
@Viper_Scull
You have to set the correct SigLevel in /etc/pacman.conf. The official databases are not signed so you should use "PackageRequired", e.g.
[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Hi there. Today I got an issue with doing an upgrade using aura, which is a wrapper for powerpill (and pacman). It looks like powerpill is pulling in an old version of the databases for some reason.
See details here: https://github.com/aurapm/aura/issues/276.
Offline
Can you reproduce this issue with powerpill? Are you using the same config for pacman and powerpill / aura?
Offline
Yes. Behaviour is the same with and without aura.
I haven't explicitly set different configs that I'm aware of. How would I check that?
Offline
Post your /etc/powerpill/powerpill.json file. Maybe you're using a different server for syncing the databases (e.g. via rsync).
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Ok, so it looks like I probably did set up powerpill to use rsync.
$ cat /etc/powerpill/powerpill.json
{
"aria2": {
"args": [
"--allow-overwrite=true",
"--always-resume=false",
"--auto-file-renaming=false",
"--check-integrity=true",
"--conditional-get=true",
"--continue=true",
"--file-allocation=none",
"--log-level=error",
"--max-concurrent-downloads=100",
"--max-connection-per-server=5",
"--min-split-size=5M",
"--show-console-readout=true",
"--split=10"
],
"path": "/usr/bin/aria2c"
},
"pacman": {
"config": "/etc/pacman.conf",
"path": "/usr/bin/pacman"
},
"pacserve": {
"server": null
},
"powerpill": {
"ask": true,
"reflect databases": false
},
"reflector": {
"args": [
"--latest",
"50",
"--sort",
"score"
]
},
"rsync": {
"args": [
"--no-motd",
"--progress"
],
"db only": true,
"path": "/usr/bin/rsync",
"servers": [
"rsync://mirror.aarnet.edu.au/archlinux/$repo/os/$arch",
"rsync://ftp.eenet.ee/archlinux/$repo/os/$arch",
"rsync://ftp.nluug.nl/archlinux/$repo/os/$arch",
"rsync://ftp5.gwdg.de/pub/linux/archlinux/$repo/os/$arch",
"rsync://rsync.gtlib.gatech.edu/archlinux/$repo/os/$arch",
"rsync://rsync.las.ic.unicamp.br/pub/archlinux/$repo/os/$arch",
"rsync://ftp.uni-kl.de/pub/linux/archlinux/$repo/os/$arch",
"rsync://mirrors.uk2.net/archlinux/$repo/os/$arch",
"rsync://mirror.csclub.uwaterloo.ca/archlinux/$repo/os/$arch",
"rsync://mirror.umd.edu/archlinux/$repo/os/$arch",
"rsync://ftp.iinet.net.au/archlinux/$repo/os/$arch",
"rsync://ftp-stud.hs-esslingen.de/archlinux/$repo/os/$arch",
"rsync://mirror.internode.on.net/archlinux/$repo/os/$arch",
"rsync://ftp.osuosl.org/archlinux/$repo/os/$arch",
"rsync://archlinux.c3sl.ufpr.br/archlinux/$repo/os/$arch",
"rsync://rsync.spline.inf.fu-berlin.de/archlinux/$repo/os/$arch",
"rsync://mirror.selfnet.de/archlinux/$repo/os/$arch",
"rsync://lug.mtu.edu/archlinux/$repo/os/$arch",
"rsync://mirror.archlinux.fi/archlinux/archlinux/$repo/os/$arch",
"rsync://archlinux.nautile.nc/archlinux/$repo/os/$arch",
"rsync://ftp.df.lth.se/pub/archlinux/$repo/os/$arch",
"rsync://ftp.tku.edu.tw/archlinux/$repo/os/$arch",
"rsync://mirrors.kernel.org/archlinux/$repo/os/$arch",
"rsync://ftp.rnl.ist.utl.pt/pub/archlinux/$repo/os/$arch",
"rsync://mirror.its.dal.ca/archlinux/$repo/os/$arch",
"rsync://ftp.jaist.ac.jp/pub/Linux/ArchLinux/$repo/os/$arch",
"rsync://ftp.halifax.rwth-aachen.de/archlinux/$repo/os/$arch",
"rsync://mirror.datacenter.by/archlinux/$repo/os/$arch",
"rsync://mirrors.cat.pdx.edu/archlinux/$repo/os/$arch",
"rsync://linux.cs.nctu.edu.tw/archlinux/$repo/os/$arch",
"rsync://ftp.byfly.by/archlinux/$repo/os/$arch",
"rsync://cake.lib.fit.edu/archlinux/$repo/os/$arch",
"rsync://ftp.ntua.gr/archlinux/$repo/os/$arch",
"rsync://mirrors.mithril.org.ua/archlinux/$repo/os/$arch",
"rsync://mirrors.ustc.edu.cn/archlinux/$repo/os/$arch",
"rsync://ftp.lysator.liu.se/pub/archlinux/$repo/os/$arch",
"rsync://mirror.de.leaseweb.net/archlinux/$repo/os/$arch",
"rsync://mirror.nl.leaseweb.net/archlinux/$repo/os/$arch",
"rsync://ftp.tuxdroid.org/archlinux/$repo/os/$arch",
"rsync://ftp.tsukuba.wide.ad.jp/archlinux/$repo/os/$arch",
"rsync://mirror.neolabs.kz/archlinux/$repo/os/$arch",
"rsync://mirror.bytemark.co.uk/archlinux/$repo/os/$arch",
"rsync://mirrors.liquidweb.com/archlinux/$repo/os/$arch",
"rsync://ftp.wa.co.za/pub/archlinux/$repo/os/$arch",
"rsync://mirror.i3d.net/archlinux/$repo/os/$arch",
"rsync://rsync.archlinux.ikoula.com/archlinux/$repo/os/$arch",
"rsync://ftp.swin.edu.au/archlinux/$repo/os/$arch",
"rsync://archlinux.mirror.rafal.ca/archlinux/$repo/os/$arch",
"rsync://mirror.michael-eckert.net/archlinux/$repo/os/$arch",
"rsync://mirror.fluxent.de/archlinux/$repo/os/$arch",
"rsync://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch",
"rsync://mirror.cc.columbia.edu/archlinux/$repo/os/$arch",
"rsync://ftp.neutrinopower.de/archlinux/$repo/os/$arch",
"rsync://rsync.mirrorservice.org/ftp.archlinux.org/$repo/os/$arch",
"rsync://ftp.snt.utwente.nl/archlinux/$repo/os/$arch",
"rsync://mirror.vexxhost.com/archlinux/$repo/os/$arch",
"rsync://mirror.unitedcolo.de/archlinux/$repo/os/$arch",
"rsync://ftp.vectranet.pl/archlinux/$repo/os/$arch",
"rsync://mirrors.aggregate.org/archlinux/$repo/os/$arch",
"rsync://archlinux.vi-di.fr/archlinux/$repo/os/$arch",
"rsync://archlinux.aubrac-medical.fr/archlinux/$repo/os/$arch",
"rsync://ftp.linux.ro/archlinux/$repo/os/$arch",
"rsync://mirror.system.is/arch/$repo/os/$arch",
"rsync://ftp.hawo.stw.uni-erlangen.de/archlinux/$repo/os/$arch",
"rsync://archlinux.cu.be/archlinux/$repo/os/$arch",
"rsync://mirrors.hustunique.com/archlinux/$repo/os/$arch",
"rsync://dfw.mirror.rackspace.com/archlinux/$repo/os/$arch",
"rsync://hkg.mirror.rackspace.com/archlinux/$repo/os/$arch",
"rsync://iad.mirror.rackspace.com/archlinux/$repo/os/$arch",
"rsync://ord.mirror.rackspace.com/archlinux/$repo/os/$arch",
"rsync://syd.mirror.rackspace.com/archlinux/$repo/os/$arch",
"rsync://rsync.piotrkosoft.net/archlinux/$repo/os/$arch",
"rsync://mirrors.hust.edu.cn/archlinux/$repo/os/$arch",
"rsync://mirror.clibre.uqam.ca/archlinux/$repo/os/$arch",
"rsync://mirrors.ispros.com.bd/archlinux/$repo/os/$arch",
"rsync://mirror.rackcentral.com.au/archlinux/$repo/os/$arch",
"rsync://mirrors.manchester.m247.com/arch-linux/$repo/os/$arch",
"rsync://arch.serverspace.co.uk/Arch/$repo/os/$arch",
"rsync://mirror.js-webcoding.de/pub/archlinux/$repo/os/$arch",
"rsync://ftp.fau.de/archlinux/$repo/os/$arch",
"rsync://mirror.lagoon.nc/pub/archlinux/$repo/os/$arch",
"rsync://mirrors.advancedhosters.com/archlinux/$repo/os/$arch",
"rsync://mirror.pregi.net/pub/Linux/archlinux/$repo/os/$arch",
"rsync://gluttony.sin.cvut.cz/arch/$repo/os/$arch",
"rsync://mirror.cedia.org.ec/archlinux/$repo/os/$arch",
"rsync://mir.art-software.fr/archlinux/$repo/os/$arch",
"rsync://mirrors.nic.cz/archlinux/$repo/os/$arch"
]
}
}
Offline
Ok, so it looks like I probably did set up powerpill to use rsync.
Yep. Either use a more reliable rsync server or disable rsync support to use the mirrors in your mirrorlist via http. The problem that you seem have now is that your main rsync mirror is lagging behind your main http mirror. I recommend using the same mirror for both rsync and http to avoid this issue entirely.
Please point this out to the other aura users who are claiming that powerpill exhibits weird behavior. It's just a configuration error.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
I have been reviewing the synchronization code sporadically since this last issue came up because I have also noticed some inconsistencies in the way it detects updates. In particular, I use paconky to monitor available updates and sometimes powerpill -Syu fails to detect these.
The issue is now solved with two tweaks:
database metalinks are now include preference attributes to ensure that databases are downloaded preferentially from the primary mirror
the addition of --remote-time=true to the list of Aria2 arguments in powerpill.json
If you have edited your powerpill.json file then you will need to manually add --remote-time=true to the Aria2 argument list.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
I have "--remote-time=true" included in the .json (it's a default, and I recently installed), but I still am not receiving all of the updates and am constantly forced to issue a pacman -Syu to get the last few updates as if I try again with powerpill it says there's nothing to do
config: http://codepad.org/Oo1janUZ
Offline
It seems that the metalink preferences are not always respected which leads to different databases being downloaded from different servers randomly. I do not know if metalink clients are required by the specification to strictly respect preferences so this may be an Aria2 bug.
The simplest solution is to use only 1 mirror in your pacman mirrorlist to ensure that all databases are downloaded from the same server. Otherwise you should use servers that remain synchronized according to the official mirrorstatus page.
Incidentally, reflector's "--sort rate" option has no effect other than wasting bandwidth when used in powerpill.json. All of the retrieved mirrors are included in the internally generated metalink without any preferences so the only thing that option does here is download the core database 50 times for no reason.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
I was autogenerating the mirrorlist with this script I made
#/bin/bash
cd /etc/pacman.d/
sed 's/^#Server/Server/' mirrorlist.pacnew > mirrorlist.rankmirrors && \
rankmirrors -n 14 mirrorlist.rankmirrors > mirrorlist
and running it every so often, usually when there's a new pacnew file, or I notice a lot of misses with pacman
Offline
When I run a powerpill -Syy, I am getting this:
10/25 16:05:18 [ERROR] CUID#7 - Download aborted. URI=http://repo-ck.com/x86_64/repo-ck.db.sig
Exception: [AbstractCommand.cc:350] errorCode=3 URI=http://repo-ck.com/x86_64/repo-ck.db.sig
-> [HttpSkipResponseCommand.cc:215] errorCode=3 Resource not found
and then this
aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
receiving incremental file list
rsync: link_stat "/core/os/x86_64/core.db.sig" (in archlinux) failed: No such file or directory (2)
rsync: link_stat "/extra/os/x86_64/extra.db.sig" (in archlinux) failed: No such file or directory (2)
rsync: link_stat "/community/os/x86_64/community.db.sig" (in archlinux) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1637) [generator=3.1.1]
receiving incremental file list
rsync: link_stat "/core/os/x86_64/core.db.sig" (in archlinux) failed: No such file or directory (2)
rsync: link_stat "/extra/os/x86_64/extra.db.sig" (in archlinux) failed: No such file or directory (2)
rsync: link_stat "/community/os/x86_64/community.db.sig" (in archlinux) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1637) [generator=3.1.1]
receiving incremental file list
rsync: link_stat "/core/os/x86_64/core.db.sig" (in archlinux) failed: No such file or directory (2)
rsync: link_stat "/extra/os/x86_64/extra.db.sig" (in archlinux) failed: No such file or directory (2)
rsync: link_stat "/community/os/x86_64/community.db.sig" (in archlinux) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1637) [generator=3.1.1]
receiving incremental file list
rsync: link_stat "/core/os/x86_64/core.db.sig" (in archlinux) failed: No such file or directory (2)
rsync: link_stat "/extra/os/x86_64/extra.db.sig" (in archlinux) failed: No such file or directory (2)
rsync: link_stat "/community/os/x86_64/community.db.sig" (in archlinux) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1637) [generator=3.1.1]
receiving incremental file list
rsync: link_stat "/core/os/x86_64/core.db.sig" (in archlinux) failed: No such file or directory (2)
rsync: link_stat "/extra/os/x86_64/extra.db.sig" (in archlinux) failed: No such file or directory (2)
rsync: link_stat "/community/os/x86_64/community.db.sig" (in archlinux) failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1637) [generator=3.1.1]
any ideas?
Brottweiler@#archlinux
Offline
Repo databases are not signed. Post your pacman.conf.
Offline
Repo databases are not signed. Post your pacman.conf.
#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives
#
# GENERAL OPTIONS
#
[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir = /
#DBPath = /var/lib/pacman/
#CacheDir = /var/cache/pacman/pkg/
#LogFile = /var/log/pacman.log
#GPGDir = /etc/pacman.d/gnupg/
HoldPkg = pacman glibc
#XferCommand = /usr/bin/curl -C - -f %u > %o
#XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#CleanMethod = KeepInstalled
#UseDelta = 0.7
Architecture = auto
# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
#IgnorePkg =
#IgnoreGroup =
#NoUpgrade =
#NoExtract =
# Misc options
#UseSyslog
Color
#TotalDownload
CheckSpace
#VerbosePkgLists
ILoveCandy
# By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages.
SigLevel = Required DatabaseOptional
LocalFileSigLevel = Optional
#RemoteFileSigLevel = Required
# NOTE: You must run `pacman-key --init` before first using pacman; the local
# keyring can then be populated with the keys of all official Arch Linux
# packagers with `pacman-key --populate archlinux`.
#
# REPOSITORIES
# - can be defined here or included from another file
# - pacman will search repositories in the order defined here
# - local/custom mirrors can be added here or in separate files
# - repositories listed first will take precedence when packages
# have identical names, regardless of version number
# - URLs will have $repo replaced by the name of the current repo
# - URLs will have $arch replaced by the name of the architecture
#
# Repository entries are of the format:
# [repo-name]
# Server = ServerName
# Include = IncludePath
#
# The header [repo-name] is crucial - it must be present and
# uncommented to enable the repo.
#
# The testing repositories are disabled by default. To enable, uncomment the
# repo name header and Include lines. You can add preferred servers immediately
# after the header, and they will be used before the default mirrors.
#[testing]
#Include = /etc/pacman.d/mirrorlist
[core]
Include = /etc/pacman.d/mirrorlist
[extra]
Include = /etc/pacman.d/mirrorlist
#[community-testing]
#Include = /etc/pacman.d/mirrorlist
[community]
Include = /etc/pacman.d/mirrorlist
# If you want to run 32 bit applications on your x86_64 system,
# enable the multilib repositories as required here.
#[multilib-testing]
#Include = /etc/pacman.d/mirrorlist
#[multilib]
#Include = /etc/pacman.d/mirrorlist
# An example of a custom package repository. See the pacman manpage for
# tips on creating your own repositories.
#[custom]
#SigLevel = Optional TrustAll
#Server = file:///home/custompkgs
[repo-ck]
Server = http://repo-ck.com/$arch
[infinality-bundle]
Server = http://bohoomil.com/repo/$arch
[xyne-x86_64]
# A repo for Xyne's own projects: http://xyne.archlinux.ca/projects/
# Packages for the "x86_64" architecture.
# Note that this includes all packages in [xyne-any].
SigLevel = Required
Server = http://xyne.archlinux.ca/repos/xyne
[lxqt-git]
Server = http://repo.stobbstechnical.com/$arch
Edit: Solved it my adding
SigLevel = DatabaseNever
Last edited by raggerv8 (2014-10-26 11:10:59)
Brottweiler@#archlinux
Offline
I am trying powerpill for the first time and also got the error about repo database signatures. But, why does SigLevel have to be DatabaseNever. Why doesn't DatabaseOptional work? It works that way for pacman. Seems like this is a bug in powerpill.
Offline
Rsync requires an explicit list of paths to sync when syncing a subset. That list is generated from the operation and the SigLevel setting in pacman.conf. If you know in advance that there is no signature file for the database then there is no reason to try to download one. I could add try-catch logic to the rsync code in powerpill but I have yet to see a repo that only occasionally signs the database. Even if such a server exists, there is no point in downloading the signature if the user plans to trust the database even in its absence.
I consider it irrelevant that a completely different protocol (HTTP) handles this case differently. The rsync option of powerpill uses a separate list of servers and is thus an exclusive feature of powerpill and not a wrapped function of pacman. It works correctly if you adapt your configuration but you can use powerpill without it.
There may also be some rsync options to work around (e.g. ignore missing files) that you could add to the powerpill configuration file.
Btw, I use
[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist
etc.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline