You are not logged in.
I'm not sure what exactly is wrong with mine.
/etc/powerpill/powerpill.json
{
"aria2": {
"args": [
"--allow-overwrite=true",
"--always-resume=false",
"--auto-file-renaming=false",
"--check-integrity=true",
"--conditional-get=true",
"--continue=true",
"--file-allocation=none",
"--log-level=error",
"--max-concurrent-downloads=100",
"--max-connection-per-server=5",
"--min-split-size=5M",
"--show-console-readout=true",
"--split=10"
],
"path": "/usr/bin/aria2c"
},
"pacman": {
"config": "/etc/pacman.conf",
"path": "/usr/bin/pacman"
},
"pacserve": {
"server": null
},
"powerpill": {
"ask": true,
"reflect databases": false
},
"reflector": {
"args": [
"--protocol",
"http",
"--latest",
"50"
]
},
"rsync": {
"args": [
"--no-motd",
"--progress"
],
"db only": true,
"path": "/usr/bin/rsync",
"servers": [rsync://mirror.rit.edu/archlinux/$repo/os/$arch]
}
}
When I run a powerpill command, I get this:
[cody@cody-laptop ~]$ powerpill -S python-gobject
error: failed to load /etc/powerpill/powerpill.json [No JSON object could be decoded]
Check the file for syntax errors.[cody@cody-laptop ~]$
What did I do wrong in the "servers" section? It's the only thing I changed. Sorry if it's something simple, I have never written in json.
Offline
"servers": [rsync://mirror.rit.edu/archlinux/$repo/os/$arch]
You need to quote all strings in JSON. That line should be
"servers": ["rsync://mirror.rit.edu/archlinux/$repo/os/$arch"]
I will try to make the error message more explicit.
The JSON error message usually indicates the exact location of the error, e.g.
Expecting ':' delimiter: line 3 column 12 (char 26)
Some errors, however, simply generate
No JSON object could be decoded
for some reason.
The same errors also confuse the JSON parsers from the yajl package (e.g. json_verify). There does not seem to be any way to easily provide clearer messages in these cases.
Last edited by Xyne (2013-05-08 17:28:04)
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
The same errors also confuse the JSON parsers from the yajl package (e.g. json_verify). There does not seem to be any way to easily provide clearer messages in these cases.
http://jsonlint.com/ detects this error. Maybe mention this validator in the error message and let the user do it manually
| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |
Online
Whenever I attempt to use yaourt/powerpill together, I get the error in the below output. However, if I use yaourt by itself or powerpill by itself, there are no problems. Has anyone ran into this and know how to fix it? I'm currently using the default configs, aside from changing the pacman binary to powerpill in /etc/yaourtrc. Maybe I'm doing something wrong?
Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
8a9490|OK | 0B/s|/var/lib/pacman/sync/testing.db
13bf1e|OK | 0B/s|/var/lib/pacman/sync/core.db
b93ed6|OK | 0B/s|/var/lib/pacman/sync/extra.db
f36466|OK | 0B/s|/var/lib/pacman/sync/multilib-testing.db
7b472f|OK | 0B/s|/var/lib/pacman/sync/multilib.db
31d2e5|OK | 0B/s|/var/lib/pacman/sync/community-testing.db
bbef66|OK | 0B/s|/var/lib/pacman/sync/community.db
44b710|OK | 0B/s|/var/lib/pacman/sync/repo-ck.db
779a44|OK | 0B/s|/var/lib/pacman/sync/infinality-bundle.db
4771ee|OK | 0B/s|/var/lib/pacman/sync/infinality-bundle-multilib.db
Status Legend:
(OK):download completed.
error: no targets specified (use -h for help)
Edit: I am running the following versions
powerpill 2013.5-1
yaourt-git 1.3.4.g6a7a258-1
Last edited by brenix (2013-05-09 00:10:57)
Offline
@progandy
That link is useful, thanks.
@brenix
That doesn't give me much to go on. You could try setting the following script as the pacman binary in /etc/yaourtrc to display the arguments that yaourt is passing to powerpill:
#!/bin/bash
echo /usr/bin/powerpill "$@"
/usr/bin/powerpill "$@"
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
I also encountered this problem. Following your method and execute yaourt -Syu --aur, I got below output:
/usr/bin/powerpill --color auto -Sy
[#f2908f 0B/0B CN:1 DL:0B]
05/09 19:03:58 [NOTICE] GID#f2908fe98833b720 - Download has already completed: /var/lib/pacman/sync/xorg113.db
05/09 19:03:58 [NOTICE] Download complete: /var/lib/pacman/sync/xorg113.db
Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
f2908f|OK | 0B/s|/var/lib/pacman/sync/xorg113.db
Status Legend:
(OK):download completed.
receiving incremental file list
sent 31 bytes received 1079 bytes 317.14 bytes/sec
total size is 3669384 speedup is 3305.75
error: no targets specified (use -h for help)
Offline
Powerpill 2013.5.9 should solve the problem. Please upgrade and confirm if it does.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Powerpill 2013.5.9 should solve the problem. Please upgrade and confirm if it does.
This fixed my issue... Thanks!
Offline
2013.5.10 supports the new Python 3 version of Pacserve and should be noticeably faster now due to both more efficient pacserve querying and better pacserve peer detection.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Recently I've been getting a error when updating with powerpill. Output results in:
Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
495660|OK | 0B/s|/var/lib/pacman/sync/xyne-i686.db
f9c8dd|OK | 0B/s|/var/lib/pacman/sync/xyne-i686.db.sig
1ab9c1|OK | 0B/s|/var/lib/pacman/sync/community.db
49acf8|OK | 0B/s|/var/lib/pacman/sync/core.db
e1ed03|OK | 604KiB/s|/var/lib/pacman/sync/extra.db
5b95eb|ERR | 0B/s|/var/lib/pacman/sync/community.db.sig
f16fff|ERR | 0B/s|/var/lib/pacman/sync/extra.db.sig
Status Legend:
(OK):download completed.(ERR):error occurred.
aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
And then at the final output indicates"
error: GPGME error: No data
error: GPGME error: No data
error: database 'extra' is not valid (invalid or corrupted database (PGP signature))
error: database 'community' is not valid (invalid or corrupted database (PGP signature))
It seems to download the packages, so that when I run "pacman -Syyu" it continues the installation without having to download anything.
Any ideas?
Thanks,
Offline
Check your SigLevel settings in /etc/pacman.conf. It seems that it's trying to download signatures for extra and community, but not core, so use the settings of the latter for the former.
The official repos do not (yet) provide database signatures, so you can change the SigLevel setting to PackageRequired to avoid the errors. For example,
[extra]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist
If you need more help than that, post your pacman.conf file.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Check your SigLevel settings in /etc/pacman.conf. It seems that it's trying to download signatures for extra and community, but not core, so use the settings of the latter for the former.
Thanks, that was it.
However, now I get:
Traceback (most recent call last):
File "/usr/bin/powerpill", line 777, in <module>
exit(run_main())
File "/usr/bin/powerpill", line 770, in run_main
return main(args)
File "/usr/bin/powerpill", line 757, in main
powerpill.download(pm2ml_args)
File "/usr/bin/powerpill", line 591, in download
urls = [found[pkg.filename]]
TypeError: 'NoneType' object is not subscriptable
Offline
uploading fix now, should be available in a few minutes
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
That fixed it. Working great. Thanks a lot!
Offline
Hi Xyne,
I'm glad of your Powerpill, it's the best that I found.
The only annoyance is the way aria2c logging on the console, excuse my paranoia.
I wrote a python3 script that I filter the aria2 output by setting aria2 as daemon and querying it for results, via rpc calls.
I kill aria2 after use, but someone would like to keep it alive, which will take some correction
Now I'd like to make a class/function for that filtering, but I'm rather confused about which implementation to use.
Maybe it is easy to add as verbosity option in Powerpill.
I'm trying this. Here are the snippets:
from xmlrpc import client as rpcCln
from subprocess import Popen, PIPE, getstatusoutput as chkout
class aria_handler():
''' class to activate aria2 and send RPC commands to it'''
def __init__(self, port= 6800, passwd= '', user= ''):
self.handle= self.passwd= self.port= self.user= ''
if passwd: self.passwd= ' --rpc-passwd='+ passwd
self.numport= port
if port != 6800: self.port= ' --rpc-listen-port='+ str(port)
if user: self.user= ' --rpc-user='+ user
self.ARIA_CMD= 'aria2c -D --enable-rpc'+ \
self.passwd+ self.port+ self.user
self.handle= \
rpcCln.ServerProxy('http://localhost:%s/rpc' %int(self.numport))
self.multi_call= rpcCln.MultiCall(self.handle)
try:
if self.handle.aria2.getVersion() != '': return
except socket.error:
pass
# when aria2c is running, then don't try it again
a= chkout(self.ARIA_CMD)
if a[0] : sys.exit('aria2c is not working as daemon')
# everything is good, it will return an handle
time.sleep(1) #allow some time to settle
return
def mcl_addUri(self, uriList, extra_options):
return self.multi_call.aria2.addUri(uriList, extra_options)
def addUri(self, uriList, extra_options):
return self.handle.aria2.addUri(uriList, extra_options)
def read_status(self, DL_ID):
'''Function to report download statuses and some other extra'''
return self.handle.aria2.tellStatus(str(DL_ID))
def active_status(self, showed):
'''Function to report actual downloads statuses. The showed should give
particular data, see aria2c manual'''
return self.handle.aria2.tellActive(showed)
def ChangeGopt(self,options):
'''Function to alter the standard global aria2 options like:
download-result
log
log-level
max-concurrent-downloads
max-download-result
max-overall-download-limit
max-overall-upload-limit
save-cookies
save-session
server-stat-of'''
return self.handle.aria2.changeGlobalOption(options)
def dlInfo(self):
''' It returns global statistics such as overall download and
upload speed. The response is of type struct and contains following
keys. The value type is string '''
if self.handle:
return self.handle.aria2.getGlobalStat()
def shutDown(self):
'''Trying to complete and kill aria daemon. the method will return OK'''
if self.handle:
return self.handle.aria2.shutdown()
return None
def forceShutDown(self):
'''Trying to complete and kill aria daemon, the method return OK, but all
the download might be suddenly stopped'''
if self.handle:
return self.handle.aria2.forceShutdown()
return None
def log(width, label, msg, e= EOL):
'''Pretty one line terminal logger. Label would be put to the left side in
white boldface and msg would be to the extremeright side, according to
width given value'''
size= width - len(label) - len(msg) - 4
#when ecceding the width, then label will be truncated
if size < 0: label= label[:(size-2)]
spaces = ' ' * size; status = C_BUSY
if msg == 'DONE': status = C_DONE
elif msg == 'FAIL': status = C_FAIL
print('{0}{1}{2}{3}{4}{5}[{6}]{4}{2}'.format(C_MAIN, label, C_CLEAR, spaces,
C_OTHER, status, msg), end=e)
sys.stdout.flush()
def term_width():
''' A easy way to know the terminal window width'''
return int(Popen(['stty', 'size'],
stdout=PIPE).communicate()[0].split()[1])
AND
URL= re.compile('.*(ht|f)tp://.*',re.IGNORECASE)
#Some wrong indentation, just posted as example to show what it would
#be loaded into a dictionary.
for aria_single in aria_multi:
#aria_single is a list of uris
for pkg in aria_single:
# any error it will stop and track it down
if not URL.match(pkg):
sys.exit(EOL+ 'Getting trouble to guess the url >'+ pkg)
#options is a dict that is like CLI options without hyphen(s)
# like: DIR = '/some/path/dir'
# options={'dir': DIR, 'max-overall-download-limit': max_speed}
DL_ID= aria2.addUri(aria_single, options)
pkg_D[DL_ID]=''; before = time.time()
AND
import time
TIMEOUT = 60 # Second of waiting some download
def ariaNiceOut(pkg_D):
'''pkg_D is a dict which contains the aria2 downloading GID.
For every completed download it will pop out the request from the dictionary untill
all are done'''
before = time.time()
try:
totaldld= 1
while len(pkg_D) > 0:
for pkgID in pkg_D:
time.sleep(0.5) # Let us don't choke aria2
if int(time.time() - before) > TIMEOUT:
sys.exit(EOL+ 'Timed Out!')
b= aria2.read_status(pkgID)
if 'complete' in b['status']:
totaldld += int(b['totalLength'])
try:
pkg_D.pop(b['gid'])
except IndexError:
pass
break
rcvd_byte= max_speed= 0
b= aria2.active_status(['downloadSpeed', 'completedLength'])
max_speed= int(aria2.dlInfo()['downloadSpeed'])
for di in b:
rcvd_byte += int(di['completedLength'])
percent= (totaldld+ rcvd_byte)/ totalsize
log(term_width(),
'Downloading {0} files. {1:.1%} . Total speed >>'.format(len(b),
percent), convert(max_speed)+'/s', e= '\r')
# if some bytes are downloaded it can reset the timeout
if max_speed > 0: before = time.time()
except KeyboardInterrupt:
sys.exit(EOL+ 'Stopped process')
finally:
aria2.shutDown()
print()
Here you may find my file with a couple of working versions.
Any suggestion is welcome
Last edited by TheSaint (2013-06-04 13:48:33)
do it good first, it will be faster than do it twice the saint
Offline
This is not something that I would want to include directly in Powerpill because I do not want to have to update the code if/when aria2c output changes. I think the best way to do this would be to write a wrapper around aria2c that accepts all of the same command-line arguments. You can then set your wrapper as the aria2c executable in /etc/powerpill/powerpill.json. You can even pass in custom arguments to your script in that case via the "args" list of the "aria2" section of that file.
Incidentally, I recommend using argument lists for command invocation instead of single strings such as
self.ARIA_CMD= 'aria2c -D --enable-rpc'+ \
self.passwd+ self.port+ self.user
Using a list will avoid all issues with shell escaping in the future, especially if you ever build commands from user input.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
This is not something that I would want to include directly in Powerpill because I do not want to have to update the code if/when aria2c output changes
I'm using this kind of program across 3 or 4 aria2 upgrades, and I found just one issue around aria2 version 1.2.x. However that's the risk when calling third party programs.
Clearly the wrapper would be in a separate module, callable by a CLI option. Failure may drop the option instantly.
Basically I would just share some idea, if you feel uncomfortable with that, I'll try it on my own, but I need to know more things to do so. Then at completion one can fill free to try it out.
I think the best way to do this would be to write a wrapper around aria2c that accepts all of the same command line arguments. You can then set your wrapper as the aria2c executable in /etc/powerpill/powerpill.json.
The manual express that we can pass almost all the commands via rpc calls like those command line arguments (CLAs), except no hyphens as separators and we may modify some option on the fly, while aria2 is running. Options are some kind of JSON-RPC/XML-RPC structure build.
What I shown here are just a figure how it would do the filter. Furthermore I mentioned snippets, which means a long away to be working.
I thought to implement the aria2 wrapper as small server or pipe, see multiprocessing documentations. I'm completely stuck to find how to write the functions.
You can even pass in custom arguments to your script in that case via the "args" list of the "aria2" section of that file.
I read and modified that file. Just I wonder why it is send as CLA. Even an aria2 configuration file should be possible in the same manner, as per man page
--conf-path=<PATH>
Change the configuration file path to PATH. Default:
$HOME/.aria2/aria2.conf
For that case the hyphens are avoided and the rule is for one option per line.
Your way allows to extend CLAs, but may open some security risk.
Incidentally, I recommend using argument lists for command invocation instead of single strings such as
self.ARIA_CMD= 'aria2c -D --enable-rpc'+ \ self.passwd+ self.port+ self.user
Using a list will avoid all issues with shell escaping in the future, especially if you ever build commands from user input.
AFAIK, the subprocess.getstatusoutput() doesn't implement the way you said.
>>> from subprocess import getstatusoutput as gso
>>> gso(['ls','la'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.3/subprocess.py", line 681, in getstatusoutput
with os.popen('{ ' + cmd + '; } 2>&1', 'r') as pipe:
TypeError: Can't convert 'list' object to str implicitly
I'm concerned that sending commands from within script may have some risk impact, I don't see solution else than that, to accomplish the task.
Last edited by TheSaint (2013-06-05 08:17:26)
do it good first, it will be faster than do it twice the saint
Offline
Sorry, after posting another long post in another thread, I'm not in the mood for selective quoting.
internal vs external invocation
Given that the aria2 executable is called at most twice in a given powerpill invocation and that the download itself is the bottleneck by several orders of magnitude, there really isn't much of a reason to include this in the powerpill codebase itself. A sensible wrapper, especially one that uses an aria2 daemon via the rpc interface, would have general utility beyond powerpill anyway, so I think it's a good idea to make a separate, stand-alone wrapper.
CLA vs RPC interface vs configuration file
CLA allows powepill to easily add more arguments as necessary for different invocations. There is no inherent risk in this vs calls to an RPC interface of passing a configuration file. The RPC interface would require a daemon to be running though, which most users neither need nor want, and using a configuration file would require extra disk IO, or it would be passed in via STDIN with the subprocess call, which ultimately is not much different than using CLAs to begin with.
subprocess.getstatusoutput()
The call is deprecated and should no longer be used (it's in the "17.5.5. Legacy Shell Invocation Functions" section of the current official documentation), in part because of what I said about the need for shell escapes. Use subprocess.Popen and the "communicate" method instead, or some other supported method from the subprocess module. As for the risk, if you are controlling the arguments via your program then there is no inherent risk. Besides, any user who can run that command locally can run any malicious commands directly themselves.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
OK, final.
aria2 as daemon, it's one's taste. If I'll get the wrapper done, i think it won't matter how many calls are sent to aria2.
Mostly I'll setup a class which can talk in "parallel" to aria2 (multiprocessing.Managers). The overhead of setting up a new deamon it won't harm much the system (as aria2 the aria2 author mention) and at the powerpill completion it would be easy to tell aria2 to shutdown.
CLAs, Your preference having the most priority, for powerpill. I just pose an idea, but as you said would make too muck impact on powerpill.
getstatusoutput(), Maybe you can help me on a couple of things about pyalpm, because I'd like to remove these calls from my program, as far as it concern with pacman. I'm choosing pyalpm. The aria2 startup, I think I will put into bash script follow by my program or change the subprocess call, as you suggest.For these points, you may PM to me, if you like.
do it good first, it will be faster than do it twice the saint
Offline
aria2 as daemon, it's one's taste. If I'll get the wrapper done, i think it won't matter how many calls are sent to aria2.
Mostly I'll setup a class which can talk in "parallel" to aria2 (multiprocessing.Managers). The overhead of setting up a new deamon it won't harm much the system (as aria2 the aria2 author mention) and at the powerpill completion it would be easy to tell aria2 to shutdown.
I don't think there is any point in running a daemon for a single operation. If you write your wrapper then you should have the daemon running all the time for it to make sense. The daemon would run in the background and the script would add files to it's input queue. Personally I think the greatest utility of this would be for those that use aria2c to manage torrents as it will always be running then.
getstatusoutput(), Maybe you can help me on a couple of things about pyalpm, because I'd like to remove these calls from my program, as far as it concern with pacman. I'm choosing pyalpm. The aria2 startup, I think I will put into bash script follow by my program or change the subprocess call, as you suggest.For these points, you may PM to me, if you like.
I don't understand. Do you want me to PM you so that we can discuss pyalpm?
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Aria2 as daemon is like any other program in multitasking as you put threading in reflector. In a particular case, it's possible to instruct aria2 to quit when the parent process disappear.
Have it background or foreground in both case will do what we want aria2 to do. For whom using it as long term jobs, just leave it alive, else we can tell go away when the jobs are finish, like it does with powerpill.
In deep it's nothing wrong on your great work, the problem is from aria2. Actually I stop all its output, but it's hard to guess at which point is the process.
If you fill fine, I'll wait for your PM. What I'm studying now, is to get a list from pyalpm, like it does "pacman -Sup --print-format %r %l %s" (%l or %n).
do it good first, it will be faster than do it twice the saint
Offline
I haven't learnt about zsync until it hit the "Recent Updates" corner of the Arch Linux homepage today. How can we make use of it in Powerpill?
Offline
zsync is only useful if people offer zsync downloads
Until mirrors start offering zsync downloads there is not much point in trying to support it. When the time comes, I will try to implement some sort of (pseudo-)plugin system for supporting third-party downloaders.
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline
Eh heh, I just found a bug. I'll see about submitting a patch once I figure out where the code that handles this is... Make it a bit more robust.
09/17 20:05:41 [ERROR] CUID#25 - Download aborted. URI=http://repo.mate-desktop.org/archlinux/x86_64%20#auto/mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz
Exception: [AbstractCommand.cc:314] errorCode=3 URI=http://repo.mate-desktop.org/archlinux/x86_64%20
-> [HttpSkipResponseCommand.cc:215] errorCode=3 Resource not found
Of course, pacman handles this just fine.
Edit: Hmmm... if I understand this correctly, the problem might be in pm2ml
Edit 2: Nevermind, ignore this post. Seems to be an issue with pyalpm.
Last edited by frostyfrog (2013-09-18 04:38:20)
{arch32} {subtlewm}{Acer Aspire One AO532h}
{arch64} {Headless Server}
Grrr! 400 char limit
Offline
It's working here with
pacman 4.1.2-1
pm2ml 2013.6-1
powerpill 2013.7.25-1
pyalpm 0.6.2-1
$ powerpill -Swd --cachedir . mate-notification-daemon
09/18 04:51:01 [NOTICE] Verification finished successfully. file=/tmp/pkg/mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz
09/18 04:51:01 [NOTICE] Download complete: /tmp/pkg/mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz
Download Results:
gid |stat|avg speed |path/URI
======+====+===========+=======================================================
5807cd|OK | 368KiB/s|/tmp/pkg/mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz
Status Legend:
(OK):download completed.
$ ls -l mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz
-rw-r--r-- 1 xyne mx 71992 2013-09-18 04:51:01 mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz
My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone
Offline