You are not logged in.

#101 2013-05-08 16:11:14

pivotraze
Member
From: Missoula, MT
Registered: 2011-08-21
Posts: 113

Re: powerpill reborn: pacman wrapper for faster downloads

I'm not sure what exactly is wrong with mine.

/etc/powerpill/powerpill.json

{
  "aria2": {
    "args": [
      "--allow-overwrite=true",
      "--always-resume=false",
      "--auto-file-renaming=false",
      "--check-integrity=true",
      "--conditional-get=true",
      "--continue=true",
      "--file-allocation=none",
      "--log-level=error",
      "--max-concurrent-downloads=100",
      "--max-connection-per-server=5",
      "--min-split-size=5M",
      "--show-console-readout=true",
      "--split=10"
    ],
    "path": "/usr/bin/aria2c"
  },
  "pacman": {
    "config": "/etc/pacman.conf",
    "path": "/usr/bin/pacman"
  },
  "pacserve": {
    "server": null
  },
  "powerpill": {
    "ask": true,
    "reflect databases": false
  },
  "reflector": {
    "args": [
      "--protocol",
      "http",
      "--latest",
      "50"
    ]
  },
  "rsync": {
    "args": [
      "--no-motd",
      "--progress"
    ],
    "db only": true,
    "path": "/usr/bin/rsync",
    "servers": [rsync://mirror.rit.edu/archlinux/$repo/os/$arch]
  }
}

When I run a powerpill command, I get this:

[cody@cody-laptop ~]$ powerpill -S python-gobject
error: failed to load /etc/powerpill/powerpill.json [No JSON object could be decoded]
Check the file for syntax errors.[cody@cody-laptop ~]$ 

What did I do wrong in the "servers" section? It's the only thing I changed. Sorry if it's something simple, I have never written in json.

Offline

#102 2013-05-08 17:09:59

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

pivotraze wrote:
"servers": [rsync://mirror.rit.edu/archlinux/$repo/os/$arch]

You need to quote all strings in JSON. That line should be

"servers": ["rsync://mirror.rit.edu/archlinux/$repo/os/$arch"]

I will try to make the error message more explicit.


Edit

The JSON error message usually indicates the exact location of the error, e.g.

Expecting ':' delimiter: line 3 column 12 (char 26)

Some errors, however, simply generate

No JSON object could be decoded

for some reason.

The same errors also confuse the JSON parsers from the yajl package (e.g. json_verify). There does not seem to be any way to easily provide clearer messages in these cases.

Last edited by Xyne (2013-05-08 17:28:04)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#103 2013-05-08 18:00:13

progandy
Member
Registered: 2012-05-17
Posts: 5,184

Re: powerpill reborn: pacman wrapper for faster downloads

Xyne wrote:

The same errors also confuse the JSON parsers from the yajl package (e.g. json_verify). There does not seem to be any way to easily provide clearer messages in these cases.

http://jsonlint.com/ detects this error. Maybe mention this validator in the error message and let the user do it manually wink


| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

Offline

#104 2013-05-09 00:03:25

brenix
Member
From: California
Registered: 2008-03-05
Posts: 185

Re: powerpill reborn: pacman wrapper for faster downloads

Whenever I attempt to use yaourt/powerpill together, I get the error in the below output. However, if I use yaourt by itself or powerpill by itself, there are no problems. Has anyone ran into this and know how to fix it? I'm currently using the default configs, aside from changing the pacman binary to powerpill in /etc/yaourtrc. Maybe I'm doing something wrong?

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
8a9490|OK  |       0B/s|/var/lib/pacman/sync/testing.db
13bf1e|OK  |       0B/s|/var/lib/pacman/sync/core.db
b93ed6|OK  |       0B/s|/var/lib/pacman/sync/extra.db
f36466|OK  |       0B/s|/var/lib/pacman/sync/multilib-testing.db
7b472f|OK  |       0B/s|/var/lib/pacman/sync/multilib.db
31d2e5|OK  |       0B/s|/var/lib/pacman/sync/community-testing.db
bbef66|OK  |       0B/s|/var/lib/pacman/sync/community.db
44b710|OK  |       0B/s|/var/lib/pacman/sync/repo-ck.db
779a44|OK  |       0B/s|/var/lib/pacman/sync/infinality-bundle.db
4771ee|OK  |       0B/s|/var/lib/pacman/sync/infinality-bundle-multilib.db

Status Legend:
(OK):download completed.
error: no targets specified (use -h for help)

Edit: I am running the following versions
powerpill 2013.5-1
yaourt-git 1.3.4.g6a7a258-1

Last edited by brenix (2013-05-09 00:10:57)

Offline

#105 2013-05-09 05:07:01

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

@progandy
That link is useful, thanks.

@brenix
That doesn't give me much to go on. You could try setting the following script as the pacman binary in /etc/yaourtrc to display the arguments that yaourt is passing to powerpill:

#!/bin/bash
echo /usr/bin/powerpill "$@"
/usr/bin/powerpill "$@"

My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#106 2013-05-09 11:08:10

budgerigar
Member
Registered: 2012-09-14
Posts: 1

Re: powerpill reborn: pacman wrapper for faster downloads

I also encountered this problem. Following your method and execute yaourt -Syu --aur, I got below output:

/usr/bin/powerpill --color auto -Sy
[#f2908f 0B/0B CN:1 DL:0B]                                                                                                                                  
05/09 19:03:58 [NOTICE] GID#f2908fe98833b720 - Download has already completed: /var/lib/pacman/sync/xorg113.db

05/09 19:03:58 [NOTICE] Download complete: /var/lib/pacman/sync/xorg113.db

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
f2908f|OK  |       0B/s|/var/lib/pacman/sync/xorg113.db

Status Legend:
(OK):download completed.
receiving incremental file list

sent 31 bytes  received 1079 bytes  317.14 bytes/sec
total size is 3669384  speedup is 3305.75
error: no targets specified (use -h for help)

Offline

#107 2013-05-09 18:12:20

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

Powerpill 2013.5.9 should solve the problem. Please upgrade and confirm if it does.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#108 2013-05-09 21:34:08

brenix
Member
From: California
Registered: 2008-03-05
Posts: 185

Re: powerpill reborn: pacman wrapper for faster downloads

Xyne wrote:

Powerpill 2013.5.9 should solve the problem. Please upgrade and confirm if it does.

This fixed my issue... Thanks!

Offline

#109 2013-05-10 20:19:24

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

2013.5.10 supports the new Python 3 version of Pacserve and should be noticeably faster now due to both more efficient pacserve querying and better pacserve peer detection.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#110 2013-05-10 21:47:55

damacas
Member
From: Ontario, Canada
Registered: 2011-05-27
Posts: 50

Re: powerpill reborn: pacman wrapper for faster downloads

Recently I've been getting a error when updating with powerpill.  Output results in:

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
495660|OK  |       0B/s|/var/lib/pacman/sync/xyne-i686.db
f9c8dd|OK  |       0B/s|/var/lib/pacman/sync/xyne-i686.db.sig
1ab9c1|OK  |       0B/s|/var/lib/pacman/sync/community.db
49acf8|OK  |       0B/s|/var/lib/pacman/sync/core.db
e1ed03|OK  |   604KiB/s|/var/lib/pacman/sync/extra.db
5b95eb|ERR |       0B/s|/var/lib/pacman/sync/community.db.sig
f16fff|ERR |       0B/s|/var/lib/pacman/sync/extra.db.sig

Status Legend:
(OK):download completed.(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.

And then at the final output indicates"

error: GPGME error: No data
error: GPGME error: No data
error: database 'extra' is not valid (invalid or corrupted database (PGP signature))
error: database 'community' is not valid (invalid or corrupted database (PGP signature))

It seems to download the packages, so that when I run "pacman -Syyu" it continues the installation without having to download anything.

Any ideas?

Thanks,

Offline

#111 2013-05-10 22:07:49

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

Check your SigLevel settings in /etc/pacman.conf. It seems that it's trying to download signatures for extra and community, but not core, so use the settings of the latter for the former.

The official repos do not (yet) provide database signatures, so you can change the SigLevel setting to PackageRequired to avoid the errors. For example,

[extra]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

If you need more help than that, post your pacman.conf file.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#112 2013-05-10 22:57:04

damacas
Member
From: Ontario, Canada
Registered: 2011-05-27
Posts: 50

Re: powerpill reborn: pacman wrapper for faster downloads

Xyne wrote:

Check your SigLevel settings in /etc/pacman.conf. It seems that it's trying to download signatures for extra and community, but not core, so use the settings of the latter for the former.

Thanks, that was it.

However, now I get:

Traceback (most recent call last):
  File "/usr/bin/powerpill", line 777, in <module>
    exit(run_main())
  File "/usr/bin/powerpill", line 770, in run_main
    return main(args)
  File "/usr/bin/powerpill", line 757, in main
    powerpill.download(pm2ml_args)
  File "/usr/bin/powerpill", line 591, in download
    urls = [found[pkg.filename]]
TypeError: 'NoneType' object is not subscriptable

Offline

#113 2013-05-10 23:02:02

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

uploading fix now, should be available in a few minutes


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#114 2013-05-11 01:18:00

damacas
Member
From: Ontario, Canada
Registered: 2011-05-27
Posts: 50

Re: powerpill reborn: pacman wrapper for faster downloads

That fixed it.  Working great. Thanks a lot!

Offline

#115 2013-06-04 08:32:53

TheSaint
Member
From: my computer
Registered: 2007-08-19
Posts: 1,523

Re: powerpill reborn: pacman wrapper for faster downloads

Hi Xyne,
I'm glad of your Powerpill, it's the best that I found.
The only annoyance is the way aria2c logging on the console, excuse my paranoia.
I wrote a python3 script that I filter the aria2 output by setting aria2 as daemon and querying it for results, via rpc calls.
I kill aria2 after use, but someone would like to keep it alive, which will take some correction wink

Now I'd like to make a class/function for that filtering, but I'm rather confused about which implementation to use.
Maybe it is easy to add as verbosity option in Powerpill.

I'm trying this. Here are the snippets:

from xmlrpc import client as rpcCln
from subprocess import Popen, PIPE, getstatusoutput as chkout

class aria_handler():
    ''' class to activate aria2 and send RPC commands to it'''

    def __init__(self, port= 6800, passwd= '', user= ''):
        self.handle= self.passwd= self.port= self.user= ''
        if passwd: self.passwd= ' --rpc-passwd='+ passwd
        self.numport= port
        if port != 6800: self.port= ' --rpc-listen-port='+ str(port)
        if user: self.user= ' --rpc-user='+ user
        self.ARIA_CMD= 'aria2c -D --enable-rpc'+ \
                            self.passwd+ self.port+ self.user
        self.handle= \
        rpcCln.ServerProxy('http://localhost:%s/rpc' %int(self.numport))
        self.multi_call= rpcCln.MultiCall(self.handle)
        try:
            if self.handle.aria2.getVersion() != '': return
        except socket.error:
            pass
        # when aria2c is running, then don't try it again
        a= chkout(self.ARIA_CMD)
        if a[0] : sys.exit('aria2c is not working as daemon')
        # everything is good, it will return an handle
        time.sleep(1) #allow some time to settle
        return

    def mcl_addUri(self, uriList, extra_options):
        return self.multi_call.aria2.addUri(uriList, extra_options)

    def addUri(self, uriList, extra_options):
        return self.handle.aria2.addUri(uriList, extra_options)

    def read_status(self, DL_ID):
        '''Function to report download statuses and some other extra'''
        return self.handle.aria2.tellStatus(str(DL_ID))

    def active_status(self, showed):
        '''Function to report actual downloads statuses. The showed should give
        particular data, see aria2c manual'''
        return self.handle.aria2.tellActive(showed)

    def ChangeGopt(self,options):

        '''Function to alter the standard global aria2 options like:
            download-result
            log
            log-level
            max-concurrent-downloads
            max-download-result
            max-overall-download-limit
            max-overall-upload-limit
            save-cookies
            save-session
            server-stat-of'''
        return self.handle.aria2.changeGlobalOption(options)

    def dlInfo(self):

        ''' It returns global statistics such as overall download and
        upload speed. The response is of type struct and contains following
        keys. The value type is string '''
        if self.handle:
            return self.handle.aria2.getGlobalStat()

    def shutDown(self):

        '''Trying to complete and kill aria daemon. the method will return OK'''
        if self.handle:
            return self.handle.aria2.shutdown()
        return None

    def forceShutDown(self):

        '''Trying to complete and kill aria daemon, the method return OK, but all
        the download might be suddenly stopped'''
        if self.handle:
            return self.handle.aria2.forceShutdown()
        return None

def log(width, label, msg, e= EOL):

    '''Pretty one line terminal logger. Label would be put to the left side in
    white boldface and msg would be to the extremeright side, according to
    width given value'''
    size= width - len(label) - len(msg) - 4
    #when ecceding the width, then label will be truncated
    if size < 0: label= label[:(size-2)]
    spaces = ' ' * size; status = C_BUSY
    if msg == 'DONE': status = C_DONE
    elif msg == 'FAIL': status = C_FAIL
    print('{0}{1}{2}{3}{4}{5}[{6}]{4}{2}'.format(C_MAIN, label, C_CLEAR, spaces,
    C_OTHER, status, msg), end=e)
    sys.stdout.flush()

def term_width():

    ''' A easy way to know the terminal window width'''
    return int(Popen(['stty', 'size'],
                    stdout=PIPE).communicate()[0].split()[1])

AND

URL= re.compile('.*(ht|f)tp://.*',re.IGNORECASE)
#Some wrong indentation, just posted as example to show what it would
#be loaded into a dictionary.
	for aria_single in aria_multi:
                #aria_single is a list of uris
		for pkg in aria_single:
			# any error it will stop and track it down
			if not URL.match(pkg):
				sys.exit(EOL+ 'Getting trouble to guess the url >'+ pkg)
                        #options is a dict that is like CLI options without hyphen(s)
                        # like: DIR = '/some/path/dir'
                        # options={'dir': DIR, 'max-overall-download-limit': max_speed}
	                DL_ID= aria2.addUri(aria_single, options)
		pkg_D[DL_ID]=''; before = time.time()

AND

import time
TIMEOUT = 60 # Second of waiting some download

def ariaNiceOut(pkg_D):
    '''pkg_D is a dict which contains the aria2 downloading GID.
    For every completed download it will pop out the request from the dictionary untill
    all are done'''
    before = time.time()
    try:
        totaldld= 1
        while len(pkg_D) > 0:
            for pkgID in pkg_D:
                time.sleep(0.5) # Let us don't choke aria2
                if int(time.time() - before) > TIMEOUT:
                    sys.exit(EOL+ 'Timed Out!')
                b= aria2.read_status(pkgID)
                if 'complete' in b['status']:
                    totaldld += int(b['totalLength'])
                    try:
                        pkg_D.pop(b['gid'])
                    except IndexError:
                        pass
                    break
                rcvd_byte= max_speed= 0
                b= aria2.active_status(['downloadSpeed', 'completedLength'])
                max_speed= int(aria2.dlInfo()['downloadSpeed'])
                for di in b:
                    rcvd_byte += int(di['completedLength'])
                percent= (totaldld+ rcvd_byte)/ totalsize
                log(term_width(),
                'Downloading {0} files. {1:.1%} . Total speed >>'.format(len(b),
                percent), convert(max_speed)+'/s', e= '\r')
                # if some bytes are downloaded it can reset the timeout
                if max_speed > 0: before = time.time()
    except KeyboardInterrupt:
        sys.exit(EOL+ 'Stopped process')
    finally:
        aria2.shutDown()
        print()

Here you may find my file with a couple of working versions.
Any suggestion is welcome

Last edited by TheSaint (2013-06-04 13:48:33)


do it good first, it will be faster than do it twice the saint wink

Offline

#116 2013-06-04 20:35:58

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

This is not something that I would want to include directly in Powerpill because I do not want to have to update the code if/when aria2c output changes. I think the best way to do this would be to write a wrapper around aria2c that accepts all of the same command-line arguments. You can then set your wrapper as the aria2c executable in /etc/powerpill/powerpill.json. You can even pass in custom arguments to your script in that case via the "args" list of the "aria2" section of that file.

Incidentally, I recommend using argument lists for command invocation instead of single strings such as

        self.ARIA_CMD= 'aria2c -D --enable-rpc'+ \
                            self.passwd+ self.port+ self.user

Using a list will avoid all issues with shell escaping in the future, especially if you ever build commands from user input.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#117 2013-06-05 08:14:02

TheSaint
Member
From: my computer
Registered: 2007-08-19
Posts: 1,523

Re: powerpill reborn: pacman wrapper for faster downloads

Xyne wrote:

This is not something that I would want to include directly in Powerpill because I do not want to have to update the code if/when aria2c output changes

I'm using this kind of program across 3 or 4 aria2 upgrades, and I  found just one issue around aria2 version 1.2.x. However that's the risk when calling third party programs.
Clearly the wrapper would be in a separate module, callable by a CLI option. Failure may drop the option instantly.
Basically I would just share some idea, if you feel uncomfortable with that, I'll try it on my own, but I need to know more things to do so. Then at completion one can fill free to try it out. smile

Xyne wrote:

I think the best way to do this would be to write a wrapper around aria2c that accepts all of the same command line arguments. You can then set your wrapper as the aria2c executable in /etc/powerpill/powerpill.json.

The manual express that we can pass almost all the commands via rpc calls like those command line arguments (CLAs), except no hyphens as separators and we may modify some option on the fly, while aria2 is running. Options are some kind of JSON-RPC/XML-RPC structure build.
What I shown here are just a figure how it would do the filter. Furthermore I mentioned snippets, which means a long away to be working.
I thought to implement the aria2 wrapper as small server or pipe, see multiprocessing documentations. I'm completely stuck to find how to write the functions.

Xyne wrote:

You can even pass in custom arguments to your script in that case via the "args" list of the "aria2" section of that file.

I read and modified that file. Just I wonder why it is send as CLA. Even an aria2 configuration file should be possible in the same manner, as per man page

--conf-path=<PATH>
              Change    the    configuration    file   path   to   PATH.    Default:
              $HOME/.aria2/aria2.conf

For that case the hyphens are avoided and the rule is for one option per line.
Your way allows to extend CLAs, but may open some security risk.

Xyne wrote:

Incidentally, I recommend using argument lists for command invocation instead of single strings such as

        self.ARIA_CMD= 'aria2c -D --enable-rpc'+ \
                            self.passwd+ self.port+ self.user

Using a list will avoid all issues with shell escaping in the future, especially if you ever build commands from user input.

AFAIK, the subprocess.getstatusoutput() doesn't implement the way you said.

>>> from subprocess import getstatusoutput as gso
>>> gso(['ls','la'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.3/subprocess.py", line 681, in getstatusoutput
    with os.popen('{ ' + cmd + '; } 2>&1', 'r') as pipe:
TypeError: Can't convert 'list' object to str implicitly

I'm concerned that sending commands from within script may have some risk impact, I don't see solution else than that, to accomplish the task.

Last edited by TheSaint (2013-06-05 08:17:26)


do it good first, it will be faster than do it twice the saint wink

Offline

#118 2013-06-05 18:37:32

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

Sorry, after posting another long post in another thread, I'm not in the mood for selective quoting.

internal vs external invocation
Given that the aria2 executable is called at most twice in a given powerpill invocation and that the download itself is the bottleneck by several orders of magnitude, there really isn't much of a reason to include this in the powerpill codebase itself. A sensible wrapper, especially one that uses an aria2 daemon via the rpc interface, would have general utility beyond powerpill anyway, so I think it's a good idea to make a separate, stand-alone wrapper.

CLA vs RPC interface vs configuration file
CLA allows powepill to easily add more arguments as necessary for different invocations. There is no inherent risk in this vs calls to an RPC interface of passing a configuration file. The RPC interface would require a daemon to be running though, which most users neither need nor want, and using a configuration file would require extra disk IO, or it would be passed in via STDIN with the subprocess call, which ultimately is not much different than using CLAs to begin with.

subprocess.getstatusoutput()
The call is deprecated and should no longer be used (it's in the "17.5.5. Legacy Shell Invocation Functions" section of the current official documentation), in part because of what I said about the need for shell escapes. Use subprocess.Popen and the "communicate" method instead, or some other supported method from the subprocess module. As for the risk, if you are controlling the arguments via your program then there is no inherent risk. Besides, any user who can run that command locally can run any malicious commands directly themselves.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#119 2013-06-06 15:41:11

TheSaint
Member
From: my computer
Registered: 2007-08-19
Posts: 1,523

Re: powerpill reborn: pacman wrapper for faster downloads

OK, final.

  • aria2 as daemon, it's one's taste. If I'll get the wrapper done, i think it won't matter how many calls are sent to aria2.
    Mostly I'll setup a class which can talk in "parallel" to aria2 (multiprocessing.Managers). The overhead of setting up a new deamon it won't harm much the system (as aria2 the aria2 author mention) and at the powerpill completion it would be easy to tell aria2 to shutdown.

  • CLAs, Your preference having the most priority, for powerpill. I just pose an idea, but as you said would make too muck impact on powerpill.

  • getstatusoutput(), Maybe you can help me on a couple of things about pyalpm, because I'd like to remove these calls from my program, as far as it concern with pacman. I'm choosing pyalpm. The aria2 startup, I think I will put into bash script follow by my program or change the subprocess call, as you suggest.For these points, you may PM to me, if you like.


do it good first, it will be faster than do it twice the saint wink

Offline

#120 2013-06-06 20:31:36

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

TheSaint wrote:
  • aria2 as daemon, it's one's taste. If I'll get the wrapper done, i think it won't matter how many calls are sent to aria2.
    Mostly I'll setup a class which can talk in "parallel" to aria2 (multiprocessing.Managers). The overhead of setting up a new deamon it won't harm much the system (as aria2 the aria2 author mention) and at the powerpill completion it would be easy to tell aria2 to shutdown.

I don't think there is any point in running a daemon for a single operation. If you write your wrapper then you should have the daemon running all the time for it to make sense. The daemon would run in the background and the script would add files to it's input queue. Personally I think the greatest utility of this would be for those that use aria2c to manage torrents as it will always be running then.

TheSaint wrote:
  • getstatusoutput(), Maybe you can help me on a couple of things about pyalpm, because I'd like to remove these calls from my program, as far as it concern with pacman. I'm choosing pyalpm. The aria2 startup, I think I will put into bash script follow by my program or change the subprocess call, as you suggest.For these points, you may PM to me, if you like.

I don't understand. Do you want me to PM you so that we can discuss pyalpm?


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#121 2013-06-08 17:44:02

TheSaint
Member
From: my computer
Registered: 2007-08-19
Posts: 1,523

Re: powerpill reborn: pacman wrapper for faster downloads

Aria2 as daemon is like any other program in multitasking as you put threading in reflector. In a particular case, it's possible to instruct aria2 to quit when the parent process disappear.
Have it background or foreground in both case will do what we want aria2 to do. For whom using it as long term jobs, just leave it alive, else we can tell  go away when the jobs are finish, like it does with powerpill.
In deep it's nothing wrong on your great work, the problem is from aria2. Actually I stop all its output, but it's hard to guess at which point is the process.

If you fill fine, I'll wait for your PM. What I'm studying now, is to get a list from pyalpm, like it does "pacman -Sup --print-format %r %l %s" (%l or %n).


do it good first, it will be faster than do it twice the saint wink

Offline

#122 2013-08-24 10:37:47

Vrantheo
Member
Registered: 2012-07-20
Posts: 33

Re: powerpill reborn: pacman wrapper for faster downloads

I haven't learnt about zsync until it hit the "Recent Updates" corner of the Arch Linux homepage today. How can we make use of it in Powerpill?

Offline

#123 2013-08-25 19:53:29

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

zsync author wrote:

zsync is only useful if people offer zsync downloads

Until mirrors start offering zsync downloads there is not much point in trying to support it. When the time comes, I will try to implement some sort of (pseudo-)plugin system for supporting third-party downloaders.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#124 2013-09-18 04:02:16

frostyfrog
Member
From: Utah, USA
Registered: 2011-03-27
Posts: 42

Re: powerpill reborn: pacman wrapper for faster downloads

Eh heh, I just found a bug. smile I'll see about submitting a patch once I figure out where the code that handles this is... Make it a bit more robust.

09/17 20:05:41 [ERROR] CUID#25 - Download aborted. URI=http://repo.mate-desktop.org/archlinux/x86_64%20#auto/mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz
Exception: [AbstractCommand.cc:314] errorCode=3 URI=http://repo.mate-desktop.org/archlinux/x86_64%20
  -> [HttpSkipResponseCommand.cc:215] errorCode=3 Resource not found

Of course, pacman handles this just fine.

Edit: Hmmm... if I understand this correctly, the problem might be in pm2ml
Edit 2: Nevermind, ignore this post. Seems to be an issue with pyalpm. hmm

Last edited by frostyfrog (2013-09-18 04:38:20)


{arch32} {subtlewm}{Acer Aspire One AO532h}
{arch64} {Headless Server}
Grrr! 400 char limit sad

Offline

#125 2013-09-18 04:57:59

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: powerpill reborn: pacman wrapper for faster downloads

It's working here with

  • pacman 4.1.2-1

  • pm2ml 2013.6-1

  • powerpill 2013.7.25-1

  • pyalpm 0.6.2-1

$ powerpill -Swd --cachedir . mate-notification-daemon
09/18 04:51:01 [NOTICE] Verification finished successfully. file=/tmp/pkg/mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz

09/18 04:51:01 [NOTICE] Download complete: /tmp/pkg/mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
5807cd|OK  |   368KiB/s|/tmp/pkg/mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz

Status Legend:
(OK):download completed.

$ ls -l mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz
-rw-r--r-- 1 xyne mx 71992 2013-09-18 04:51:01 mate-notification-daemon-1.6.0-2-x86_64.pkg.tar.xz

My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

Board footer

Powered by FluxBB