powerpill reborn: pacman wrapper for faster downloads

Xyne · 2020-12-13 16:23:37

I'll gladly change all of my launcher scripts to use best practices. I appreciate having this shortcoming pointed out, but I would appreciate it even more without the snark. I get that it's part of your personality, but it's irksome and unnecessary. Rather than assume all of the mistakes in someone else's code are due to deliberate disregard for various standards, try assuming that maybe, just maybe, those choices were made, possibly a long time ago, based on limited knowledge or before current best practices were established. Many such vestiges persist until attention is brought to them, as is the case here.

As for installing "command line tools" as modules, they are actually used as modules by other packages. If you have further suggestions to make without the layer of unmistakable disdain, please feel free to share.

eschwartz · 2020-12-13 18:22:03

Setuptools entry points are a nice-to-have, but not everyone needs, wants, or uses them (especially if your code is not on PyPI and you don't distribute via any PyPA method).

There's no specific standard to point at for "don't use bash just to execute python immediately, when just using python in the first place would do equally well", but it would be simpler and introduce fewer moving parts to write a python script in /usr/bin rather than a bash script...
(It would also avoid the --help text printing the module name, e.g. "usage: Reflector.py", and instead print the script name, "usage: reflector". argparse can't know that it was launched from "/usr/bin/reflector" if all it sees is "python3 -m Reflector" running "/usr/lib/python3.9/site-packages/Reflector.py" as a module)

It's the same reason why we don't install "cp" to /usr/lib/coreutils/cp and create a /usr/bin/cp wrapper script:

#!/bin/bash

cp "$@"

Xyne wrote:

Rather than assume all of the mistakes in someone else's code are due to deliberate disregard for various standards

Speaking of "best practices", how is the migration from tarballs to some form of version control system (e.g. git) as was promised in 2011 and again in 2015 coming along? IIRC the last news on the subject was "undefined concerns about git".
I dislike working with or looking at code where anything other than the latest version disappears (and occasionally not even that is available, so the AUR package cannot build) and it's impossible to see the differences between versions unless you have meticulously maintained local copies of a berjillion tarballs, and even then, fairly awkwardly due to the need to extract and run `diff` repeatedly between versions. Even the https://xyne.archlinux.ca/projects/powerpill/#changelog is unmaintained for the last 5 years, as it still claims the latest version is 2016-01-15!

Your stuff is a bit of a black box, and it's inconvenient to read or reason about the code in order to offer improvements. You seem to prefer it that way. For all I know, bash entry points is more of the same. I mostly stopped caring or using, years ago. (I think this was mostly due to the disappointment about refusing to use version control, and partly due to getting sick and tired of trying to fill in the blanks from the changelogs by diffing tarballs.)

I don't know where it might be getting used as a module vs. a script, but mea culpa -- I shouldn't have assumed that in particular.

eschwartz · 2020-12-13 18:22:35

https://python-packaging.readthedocs.io … ripts.html

The distutils approach:
https://docs.python.org/3/distutils/set … ng-scripts

The setuptools approach:
https://setuptools.readthedocs.io/en/la … point.html

The distutils script approach is probably fine, it just handles the "best practice" of taking your arbitrary script and installing it with shebang fixups, as part of setup.py install rather than manually "install -Dm755 src dest".

The setuptools entry point would let you declare it all in setup.py and drop a file from the sources (take a look at e.g. /usr/bin/pip to see the script it generates), but add a build dependency on setuptools. (Very recent setuptools versions use importlib.metadata if available, rather than needing a runtime dependency on setuptools too.)
OTOH, using setuptools would correctly handle Windows, where you need to have a .exe because that's how Windows rolls... which would matter so much more if your Arch-centric software was ever expected to run from Windows cmd.exe... still useful knowledge in case you decide to write interesting cross-platform programs that do, in fact, need to run on Windows.

Last edited by eschwartz (2020-12-13 18:23:30)

FirstAirBender · 2020-12-13 20:55:12

Xyne wrote:

I'll gladly change all of my launcher scripts to use best practices.

Awesome!

Xyne · 2020-12-13 23:14:52

eschwartz wrote:

https://python-packaging.readthedocs.io … ripts.html
The distutils approach:
https://docs.python.org/3/distutils/set … ng-scripts
The setuptools approach:
https://setuptools.readthedocs.io/en/la … point.html
The distutils script approach is probably fine, it just handles the "best practice" of taking your arbitrary script and installing it with shebang fixups, as part of setup.py install rather than manually "install -Dm755 src dest".
The setuptools entry point would let you declare it all in setup.py and drop a file from the sources (take a look at e.g. /usr/bin/pip to see the script it generates), but add a build dependency on setuptools. (Very recent setuptools versions use importlib.metadata if available, rather than needing a runtime dependency on setuptools too.)
OTOH, using setuptools would correctly handle Windows, where you need to have a .exe because that's how Windows rolls... which would matter so much more if your Arch-centric software was ever expected to run from Windows cmd.exe... still useful knowledge in case you decide to write interesting cross-platform programs that do, in fact, need to run on Windows.

Thanks.

edit
The powerpill executable has been switched to a distutils script and will be uploaded shortly. I'll convert all of the other packages as I update them.

Last edited by Xyne (2020-12-13 23:53:33)

txtsd · 2022-07-22 13:55:58

paru:

:: keys need to be imported:
     D89FAAEB4CECAFD199A2F5E612C6F735F7A9A519 wanted by: python3-xcgf-2021-4
:: import? [Y/n]: y
gpg: key 0x12C6F735F7A9A519: no user ID
gpg: Total number processed: 1
error: failed to run: gpg --recv-keys D89FAAEB4CECAFD199A2F5E612C6F735F7A9A519:

yay:

:: PGP keys need importing:
 -> D89FAAEB4CECAFD199A2F5E612C6F735F7A9A519, required by: python3-xcgf  python3-memoizedb  python3-xcpf  pm2ml  powerpill
:: Importing keys with gpg...
gpg: key 0x12C6F735F7A9A519: no user ID
gpg: Total number processed: 1
 -> problem importing keys

Is this a problem on my end? gpg imports seem to work fine otherwise.

Xyne · 2022-07-22 23:57:47

Which keyservers do you use?

You can also export it with "pacman-key -e D89FAAEB4CECAFD199A2F5E612C6F735F7A9A519" and then manually import it with gpg.

JaydenDev · 2022-07-23 01:31:01

The original (first) post has an outdated link with an expired domain. Could you update the link? Also will you soon add AUR support to powerpill?

As well, powerpill -Syu doesn't work

    ~  sudo powerpill -Syu                                                                                                                                     ✔ 

07/22 20:33:57 [NOTICE] Downloading 9 item(s)

07/22 20:33:58 [NOTICE] GID#38c035e82ddef909 - Download has already completed: /var/lib/pacman/sync/community.db

07/22 20:33:58 [NOTICE] GID#a2b243a7e1bcffca - Download has already completed: /var/lib/pacman/sync/xyne-any.db

07/22 20:33:58 [NOTICE] Download complete: /var/lib/pacman/sync/xyne-any.db

07/22 20:33:58 [NOTICE] GID#e6008a97744cb625 - Download has already completed: /var/lib/pacman/sync/extra.db

07/22 20:33:58 [NOTICE] GID#19eb23438705c994 - Download has already completed: /var/lib/pacman/sync/xyne-any.db.sig

07/22 20:33:58 [NOTICE] Download complete: /var/lib/pacman/sync/xyne-any.db.sig

07/22 20:33:58 [ERROR] CUID#12 - Download aborted. URI=https://america.mirror.pkgbuild.com/extra/os/x86_64/extra.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://america.mirror.pkgbuild.com/extra/os/x86_64/extra.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [ERROR] CUID#22 - Download aborted. URI=https://mirror.lty.me/archlinux/extra/os/x86_64/extra.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://mirror.lty.me/archlinux/extra/os/x86_64/extra.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [NOTICE] GID#b2b3a34282022598 - Download has already completed: /var/lib/pacman/sync/multilib.db

07/22 20:33:58 [NOTICE] GID#fcaeceaee5273443 - Download has already completed: /var/lib/pacman/sync/core.db

07/22 20:33:58 [ERROR] CUID#8 - Download aborted. URI=https://archlinux.thaller.ws/community/os/x86_64/community.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://archlinux.thaller.ws/community/os/x86_64/community.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [ERROR] CUID#10 - Download aborted. URI=https://mirror.f4st.host/archlinux/core/os/x86_64/core.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://mirror.f4st.host/archlinux/core/os/x86_64/core.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [ERROR] CUID#26 - Download aborted. URI=https://america.mirror.pkgbuild.com/core/os/x86_64/core.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://america.mirror.pkgbuild.com/core/os/x86_64/core.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [ERROR] CUID#27 - Download aborted. URI=https://archlinux.thaller.ws/core/os/x86_64/core.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://archlinux.thaller.ws/core/os/x86_64/core.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [ERROR] CUID#25 - Download aborted. URI=https://mirror.lty.me/archlinux/community/os/x86_64/community.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://mirror.lty.me/archlinux/community/os/x86_64/community.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [ERROR] CUID#29 - Download aborted. URI=https://america.mirror.pkgbuild.com/community/os/x86_64/community.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://america.mirror.pkgbuild.com/community/os/x86_64/community.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [ERROR] CUID#28 - Download aborted. URI=https://europe.mirror.pkgbuild.com/core/os/x86_64/core.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://europe.mirror.pkgbuild.com/core/os/x86_64/core.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [ERROR] CUID#31 - Download aborted. URI=https://mirror.lty.me/archlinux/core/os/x86_64/core.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://mirror.lty.me/archlinux/core/os/x86_64/core.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [NOTICE] Download GID#2a18f04c906c63e4 not complete: /var/lib/pacman/sync/core.db.sig

07/22 20:33:58 [ERROR] CUID#23 - Download aborted. URI=https://europe.mirror.pkgbuild.com/extra/os/x86_64/extra.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://europe.mirror.pkgbuild.com/extra/os/x86_64/extra.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:58 [ERROR] CUID#30 - Download aborted. URI=https://mirror.f4st.host/archlinux/community/os/x86_64/community.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://mirror.f4st.host/archlinux/community/os/x86_64/community.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found
[DL:0B][#38c035 0B/0B][#f95684 0B/0B][#fcaece 0B/0B][#e6008a 0B/0B][#7bce1e 0B/0B](+1)                                                                                
07/22 20:33:58 [NOTICE] Download complete: /var/lib/pacman/sync/community.db

07/22 20:33:58 [NOTICE] Download complete: /var/lib/pacman/sync/core.db

07/22 20:33:58 [NOTICE] Download complete: /var/lib/pacman/sync/extra.db

07/22 20:33:58 [NOTICE] Download complete: /var/lib/pacman/sync/multilib.db

07/22 20:33:58 [ERROR] CUID#32 - Download aborted. URI=https://archlinux.thaller.ws/extra/os/x86_64/extra.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://archlinux.thaller.ws/extra/os/x86_64/extra.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:59 [ERROR] CUID#33 - Download aborted. URI=https://europe.mirror.pkgbuild.com/community/os/x86_64/community.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://europe.mirror.pkgbuild.com/community/os/x86_64/community.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:59 [NOTICE] Download GID#f95684fd0c9c8b75 not complete: /var/lib/pacman/sync/community.db.sig

07/22 20:33:59 [ERROR] CUID#34 - Download aborted. URI=https://mirror.f4st.host/archlinux/extra/os/x86_64/extra.db.sig
Exception: [AbstractCommand.cc:351] errorCode=3 URI=https://mirror.f4st.host/archlinux/extra/os/x86_64/extra.db.sig
  -> [HttpSkipResponseCommand.cc:218] errorCode=3 Resource not found

07/22 20:33:59 [NOTICE] Download GID#7bce1e4fdd8bdf34 not complete: /var/lib/pacman/sync/extra.db.sig

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
a2b243|OK  |       0B/s|/var/lib/pacman/sync/xyne-any.db
19eb23|OK  |       0B/s|/var/lib/pacman/sync/xyne-any.db.sig
2a18f0|ERR |       0B/s|/var/lib/pacman/sync/core.db.sig
38c035|OK  |       0B/s|/var/lib/pacman/sync/community.db
fcaece|OK  |       0B/s|/var/lib/pacman/sync/core.db
e6008a|OK  |       0B/s|/var/lib/pacman/sync/extra.db
b2b3a3|OK  |       0B/s|/var/lib/pacman/sync/multilib.db
f95684|ERR |       0B/s|/var/lib/pacman/sync/community.db.sig
7bce1e|ERR |       0B/s|/var/lib/pacman/sync/extra.db.sig

Status Legend:
(OK):download completed.(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
:: Starting full system upgrade...
 there is nothing to do

Last edited by JaydenDev (2022-07-23 01:34:28)

txtsd · 2022-07-23 05:17:16

Xyne wrote:

Which keyservers do you use?
You can also export it with "pacman-key -e D89FAAEB4CECAFD199A2F5E612C6F735F7A9A519" and then manually import it with gpg.

This is in my `dirmngr.conf`

keyserver hkps://keys.openpgp.org
keyserver hkp://pgp.mit.edu
keyserver hkps://hkps.pool.sks-keyservers.net:443
keyserver hkp://ipv4.pool.sks-keyservers.net:11371

EDIT: Okay I checked the wiki and put `hkps://keys.mailvelope.com` above the openpgp one, and it works well now.
I might as well remove the openpgp one if it causes problems like this.

Last edited by txtsd (2022-07-23 05:26:20)

Xyne · 2022-07-23 21:45:49

@JaydenDev
Bauerbill is already just powerpill with AUR support.

The errors that you're getting are due to a misconfigured pacman.conf. The official repo databases are not signed, only the packages. You need to use SigLevel = PackageRequired.

The link in the first post has been updated.

@txtsd
I'm glad that you've got it working.

Cebtenzzre · 2024-02-10 20:17:55

I tried out powerpill again today, and immediately ran up against this code in pm2ml.py:

try:
    siglevel = self.pacman_conf[db.name]['SigLevel'].split()[0]
except KeyError:
    siglevel = None
download_sig = XCPF.PacmanConfig.needs_sig(siglevel, pargs.sigs, 'Database')

When the section's SigLevel is "Required DatabaseNever", it should not attempt to download database sigs for a repo. But it does, because it barely attempts to parse the SigLevel - it just sees "Required", ignoring the rest of the line. It doesn't even look at the global SigLevel AFAICT.

Arch Linux

#301 2020-12-13 16:23:37

Re: powerpill reborn: pacman wrapper for faster downloads

#302 2020-12-13 18:22:03

Re: powerpill reborn: pacman wrapper for faster downloads

#303 2020-12-13 18:22:35

Re: powerpill reborn: pacman wrapper for faster downloads

#304 2020-12-13 20:55:12

Re: powerpill reborn: pacman wrapper for faster downloads

#305 2020-12-13 23:14:52

Re: powerpill reborn: pacman wrapper for faster downloads

#306 2022-07-22 13:55:58

Re: powerpill reborn: pacman wrapper for faster downloads

#307 2022-07-22 23:57:47

Re: powerpill reborn: pacman wrapper for faster downloads

#308 2022-07-23 01:31:01

Re: powerpill reborn: pacman wrapper for faster downloads

#309 2022-07-23 05:17:16

Re: powerpill reborn: pacman wrapper for faster downloads

#310 2022-07-23 21:45:49

Re: powerpill reborn: pacman wrapper for faster downloads

#311 2024-02-10 20:17:55

Re: powerpill reborn: pacman wrapper for faster downloads

Board footer