powerpill reborn: pacman wrapper for faster downloads

cradlemann · 2018-04-30 12:24:39

Xyne wrote:

cradlemann wrote:
During my searches I found couple of bugs, using flake8 tool and want to fix them.
Which bugs in which files? All I see are cosmetic errors.

return platform.machine() - missing import platform in Powerpill.py file

cradlemann · 2018-06-01 07:43:36

Hi Xyne,
Is it possible to ask confirmation before download everything in powerpill and bauerbill? And add to bauerbill posibility to edit PKGBUILD files? Used to do it very often, when I used yaourt.

Xyne · 2018-06-01 19:50:39

@cradlemann
I've added it to the todo list.

raggerv8 · 2018-10-20 01:53:44

error: xyne-x86_64: missing required signature
error: database 'xyne-x86_64' is not valid (invalid or corrupted database (PGP signature))

Error on your end?

Edit: seems to be fixed

Last edited by raggerv8 (2018-10-21 00:12:06)

x4fyr · 2018-11-02 14:10:25

There is a problem, when there are many packages to download via rsync, as rsync has hard coded arg limit of 1000. Maybe this should be considered in powerpill, so that so many packages are not downloaded with a single command.

Xyne · 2018-11-04 02:21:10

@x4fyr
Are you hitting a 1000 package limit on a regular upgrade? I'm just curious because you're the first person to report the issue.

I've added the dealing with the limit to my todo list and will try to get to it soonish.

x4fyr · 2018-11-04 14:36:16

It was a regular upgrade on a system I haven't used some months. I dont think too many people use powerpill+rsync and don't upgrade their system with many packages that infrequent. Imho it is a very special case, so no hurry needed.
I circumvented it, by first upgrading just core and then the rest.

The limit in rsync is actually not 1000 files but 1000 arguments. So it will be a bit less, as the options will count into the limit, too.

Xyne · 2018-11-04 16:03:44

Rsync commands are now split and run in parallel to handle the limit.

Ropid · 2018-12-20 22:42:04

I just noticed that when I try to install "nvidia-utils", powerpill tries to download a different package named "nvidia-340xx-utils" that has that nvidia-utils name in its "provides=..." PKGBUILD setting. Here's the output when I stop it with Ctrl-C:

$ sudo powerpill -S nvidia-utils
[#9dff34 2.0MiB/22MiB(9%) CN:4 DL:692KiB ETA:30s]^C                                                                                                                                                                                          
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
9dff34|INPR|   535KiB/s|/var/cache/pacman/pkg/nvidia-340xx-utils-340.107-3-x86_64.pkg.tar.xz

Status Legend:
(INPR):download in-progress.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.

The powerpill version:

$ pacman -Qi powerpill
Name            : powerpill
Version         : 2018.11-1
...

Last edited by Ropid (2018-12-20 22:45:41)

Xyne · 2018-12-24 01:48:54

Fixed in python3-xcpf-2018.12-1. Thanks for reporting the error.

zerophase · 2019-01-10 08:05:21

@cradlemann
This won't help you edit packages manually, but I believe it is possible to set up a script bauerbill uses to modify specific packages. There's a section in the json for per package scripts, I believe.

@xyne
With the pacman update for delta3 should we still be using rsync, or both rsync and delta3 for the best download times?

eschwartz · 2019-01-10 13:52:39

There was no pacman update for xdelta3. xdelta3 has *always* been an optional dependency of the pacman binary, *if* you have a repository which provides delta packages, which the official ones do not.

Recently the pacman PKGBUILD was updated to reflect this longstanding fact, but that is merely advertisement, not implementation of a feature.

zerophase · 2019-01-10 19:42:11

@eschwartz

Ah, alright. xdelta3 was just, recently, split off from pacman to a separate package, or updated to be listed as an optional dependency? Yeah, there's just that one guy advertising on the forums about having an xdelta3 server setup.

@xyne
Is my hooks array setup correctly? I have the script in /etc/bauerbill/hooks/linux-ck/prebuild.

  "hooks" : {
    "directory" : ["/etc/bauerbill/hooks"],
    "arguments" : {
      "default": ["{PackageBase}", "{LastModified}"]
    },
    "commands" : {
      "prebuild" :  {
	"linux-ck" : ["linux-ck_modify.sh"],
        "default" : []
      }
    }
  }

Last edited by zerophase (2019-01-10 19:42:35)

raggerv8 · 2019-01-18 19:33:55

I started getting this today...

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.7/site-packages/Powerpill.py", line 43, in <module>
    import pm2ml
  File "/usr/lib/python3.7/site-packages/pm2ml.py", line 38, in <module>
    import XCPF.PacmanConfig
  File "/usr/lib/python3.7/site-packages/XCPF/PacmanConfig.py", line 28, in <module>
    from pycman.config import pacman_conf_enumerator, _logmask, cb_log, LIST_OPTIONS, BOOLEAN_OPTIONS
ImportError: cannot import name 'pacman_conf_enumerator' from 'pycman.config' (/usr/lib/python3.7/site-packages/pycman/config.py)

I noticed another user had the same issue and they wrote about it on the AUR page, and you replied.

Xyne @ AUR wrote:

Can you compare the installed file at /usr/lib/python3.7/site-packages/Powerpill.py to Powerpill.py in the source archive?

My /usr/lib/python3.7/site-packages/Powerpill.py and the source archive Powerpill.py are identical.

Last edited by raggerv8 (2019-01-18 19:34:52)

crazymanjinn · 2019-01-19 02:49:07

I'm seeing the stack trace same thing as raggerv8 above.

It appears to be due to an update in pyalpm package. If I downgrade to version 0.8.4-2, then powerpill works without a problem. However, upgrading to 0.8.5-1 or newer, causes this to break with that same import error.

edit: looking into this more, it looks like pyalpm completely changed the format of pycman/config.py, just looking at relevant part of the diff to the import error in the stack trace:

diff --git a/pycman/config.py b/pycman/config.py
index f62a946..84f520e 100644
--- a/pycman/config.py
+++ b/pycman/config.py
@@ -63,65 +76,105 @@ BOOLEAN_OPTIONS = (
    'ShowSize',
    'UseDelta',
    'TotalDownload',
-   'CheckSpace'
+   'CheckSpace',
+   'VerbosePkgLists',
+   'ILoveCandy',
+   'Color',
+   'DisableDownloadTimeout',
 )
 
-def pacman_conf_enumerator(path):
-   filestack = []
-   current_section = None
-   filestack.append(open(path))
+class PacmanConfEnumeratorSession():
+
+   def __init__(self, path):
+       self.path = path

I guess python3-xcpf needs to be updated to handle all the changes in pyalpm?

Last edited by crazymanjinn (2019-01-19 03:03:50)

zerophase · 2019-01-19 03:04:57

It's from all of the free floating config functions being placed in a class in pyalpm, PacmanConfig.

barraponto · 2019-01-22 20:43:16

Is there a repository I can make pull request to?

txtsd · 2019-06-09 14:58:40

Does not work with pacman-git.

λ powerpill -Syu
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.7/site-packages/Powerpill.py", line 41, in <module>
    import pyalpm
ImportError: /usr/lib/python3.7/site-packages/pyalpm.cpython-37m-x86_64-linux-gnu.so: undefined symbol: alpm_option_set_deltaratio

Xyne · 2019-06-15 03:09:01

Thanks for the heads-up, but the error is in pyalpm. Please test pyalpm-git. If the error has not been fixed there, please report it to the pyalpm devs.

SolarAquarion · 2019-10-23 02:22:39

/usr/share/makepkg/buildenv/buildflags.sh: line 28: build_options: readonly variable
/usr/share/makepkg/buildenv/compiler.sh: line 30: build_options: readonly variable
/usr/share/makepkg/buildenv/makeflags.sh: line 28: build_options: readonly variable

That's what i get when i do build.sh

Xyne · 2019-10-25 05:25:45

@SolarAquarion
Thanks for notifying me of the errors, which were due to changes in makepkg in pacman 5.2.0. These have been fixed in bauerbill-2019.

For future bauerbill errors and feedback, please post in the Bauerbill thread.

FirstAirBender · 2020-12-12 03:22:11

Please use the full path to python when launching the application.

In the powerpill application, I see

python3 -m Powerpill "$@"

Today I got the error: /home/c/.pyenv/versions/3.9.1/bin/python3: No module named Powerpill
This because I manage some local python using pyenv, so I had to run

pyenv global system

for powerpill to work

I use yay as my AUR helper.

Last edited by FirstAirBender (2020-12-12 03:22:26)

V1del · 2020-12-12 10:21:56

It really isn't the job of the single application/utility to workaround your environment. Why are you running a system package within your local build/pyenv environment?

FirstAirBender · 2020-12-13 08:05:49

V1del wrote:

It really isn't the job of the single application/utility to workaround your environment. Why are you running a system package within your local build/pyenv environment?

I wasn't trying to run it within the pyenv environment. I use pyenv for other python stuff. Yay always displays a warning when run with sudo, so I don't always use sudo to run it, which is why it picks up stuff from my user's environment.

Anyways after searching a bit, I created the following alias for running yay.

alias yay='env --ignore-environment sudo --login --user="$USER" yay '

Feel free to judge

Last edited by FirstAirBender (2020-12-13 08:11:00)

eschwartz · 2020-12-13 13:29:58

Python software really should be getting launched via wrapper scripts looking like this:

#!/usr/bin/python3

import sys
from Powerpill import main

sys.exit(main())

There is zero reason to wrap everything in bash just to interpose another process -- unless wasting a few CPU cycles is a reason. Setuptools entry points handle this seamlessly and get installed during "python3 setup.py install", reducing the number of lines in a PKGBUILD.
Setuptools entry points also guarantee the #! shebang points to the absolute path of the python executable used to perform "python3 setup.py install", meaning no env lookup, meaning it is guaranteed to never run in a virtualenv, regardless of your interactive shell environment (unless it was installed from a virtualenv, in which case it will *always* run in that virtualenv).

Nevertheless, the bash entry point (and the one-file command line tool installed as a module to begin with) is a classic feature of Xyne software.

Anyway. Yeah. There's ways to fix this, and even ways to simplify packaging *and* be more correct about it which just fix this by accident.

Arch Linux

#276 2018-04-30 12:24:39

Re: powerpill reborn: pacman wrapper for faster downloads

#277 2018-06-01 07:43:36

Re: powerpill reborn: pacman wrapper for faster downloads

#278 2018-06-01 19:50:39

Re: powerpill reborn: pacman wrapper for faster downloads

#279 2018-10-20 01:53:44

Re: powerpill reborn: pacman wrapper for faster downloads

#280 2018-11-02 14:10:25

Re: powerpill reborn: pacman wrapper for faster downloads

#281 2018-11-04 02:21:10

Re: powerpill reborn: pacman wrapper for faster downloads

#282 2018-11-04 14:36:16

Re: powerpill reborn: pacman wrapper for faster downloads

#283 2018-11-04 16:03:44

Re: powerpill reborn: pacman wrapper for faster downloads

#284 2018-12-20 22:42:04

Re: powerpill reborn: pacman wrapper for faster downloads

#285 2018-12-24 01:48:54

Re: powerpill reborn: pacman wrapper for faster downloads

#286 2019-01-10 08:05:21

Re: powerpill reborn: pacman wrapper for faster downloads

#287 2019-01-10 13:52:39

Re: powerpill reborn: pacman wrapper for faster downloads

#288 2019-01-10 19:42:11

Re: powerpill reborn: pacman wrapper for faster downloads

#289 2019-01-18 19:33:55

Re: powerpill reborn: pacman wrapper for faster downloads

#290 2019-01-19 02:49:07

Re: powerpill reborn: pacman wrapper for faster downloads

#291 2019-01-19 03:04:57

Re: powerpill reborn: pacman wrapper for faster downloads

#292 2019-01-22 20:43:16

Re: powerpill reborn: pacman wrapper for faster downloads

#293 2019-06-09 14:58:40

Re: powerpill reborn: pacman wrapper for faster downloads

#294 2019-06-15 03:09:01

Re: powerpill reborn: pacman wrapper for faster downloads

#295 2019-10-23 02:22:39

Re: powerpill reborn: pacman wrapper for faster downloads

#296 2019-10-25 05:25:45

Re: powerpill reborn: pacman wrapper for faster downloads

#297 2020-12-12 03:22:11

Re: powerpill reborn: pacman wrapper for faster downloads

#298 2020-12-12 10:21:56

Re: powerpill reborn: pacman wrapper for faster downloads

#299 2020-12-13 08:05:49

Re: powerpill reborn: pacman wrapper for faster downloads

#300 2020-12-13 13:29:58

Re: powerpill reborn: pacman wrapper for faster downloads

Board footer