powerpill maybe duplicate download

czg · 2023-03-29 11:22:47

when I tried parellel download using powerpill , it seems to redownload the same packages

$ sudo powerpill --log ~/ppill_log.log -Syyu rime-double-pinyin rime-emoji
03/29 19:10:39 [NOTICE] Download complete: /var/lib/pacman/sync/core.db
[DL:3.3MiB][#f6245d 5.7MiB/7.2MiB(79%)][#e08099 48KiB/1.7MiB(2%)] 
# this show parellel work
...
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
b66195|OK  |       0B/s|/var/cache/pacman/pkg/rime-double-pinyin-0.0.0.20190120-1-x86_64.pkg.tar.zst
ff7db5|OK  |       0B/s|/var/cache/pacman/pkg/rime-emoji-0.0.0.20230308-1-any.pkg.tar.zst
# here 0B/s because I had downloaded them before 
...
Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
ea1bd8|OK  |   579KiB/s|//var/cache/pacman/pkg/rime-double-pinyin-0.0.0.20190120-1-x86_64.pkg.tar.zst.part

Status Legend:
(OK):download completed.

03/29 19:20:46 [NOTICE] Downloading 1 item(s)

03/29 19:20:47 [NOTICE] Download complete: //var/cache/pacman/pkg/rime-emoji-0.0.0.20230308-1-any.pkg.tar.zst.part

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
256327|OK  |   551KiB/s|//var/cache/pacman/pkg/rime-emoji-0.0.0.20230308-1-any.pkg.tar.zst.part
# weird maybe redownload the same packages

my env

$ pacman --version
Pacman v6.0.2 - libalpm v13.0.2
# install powerpill by 
$ yay -S powerpill --noconfirm
$ pacman -Qi powerpill 
Name            : powerpill
Version         : 2021.11-4
Description     : Pacman wrapper for faster downloads.
Architecture    : any
URL             : https://xyne.dev/projects/powerpill
Licenses        : GPL
Groups          : None
Provides        : None
Depends On      : aria2  pm2ml>2012.12.12  pyalpm  python3  python3-xcgf  python3-xcpf
Optional Deps   : python3-threaded_servers: internal Pacserve support
                  reflector: Reflector and Rsync support [installed]
                  rsync: Rsync download support [installed]
....

here is my powerpill.json

{
  "aria2": {
    "args": [
      "-",
      "--allow-overwrite=true",
      "--always-resume=false",
      "--auto-file-renaming=false",
      "--check-integrity=true",
      "--conditional-get=true",
      "--continue=true",
      "--file-allocation=none",
      "--log-level=error",
      "--max-concurrent-downloads=100",
      "--max-connection-per-server=16",
      "--min-split-size=1M",
      "--split=20",
      "--remote-time=true",
      "--show-console-readout=true",
      "--conf-path=/etc/powerpill/aria2.conf"
    ],
    "path": "/usr/bin/aria2c"
  },
  "pacman": {
    "config": "/etc/pacman_powermill.conf",
    "path": "/usr/bin/pacman"
  },
  "pacserve": {
    "server": null
  },
  "powerpill": {
    "select": false,
    "reflect databases": false
  },
  "rsync": {
    "args": [
      "--no-motd",
      "--progress",
      "--verbose"
    ],
    "db only": true,
    "path": "/usr/bin/rsync",
    "servers": []
  }
}

/etc/pacman_powermill.conf

#
# /etc/pacman.conf
#
# See the pacman.conf(5) manpage for option and repository directives

#
# GENERAL OPTIONS
#
[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir     = /
#DBPath      = /var/lib/pacman/
#CacheDir    = /var/cache/pacman/pkg/
#LogFile     = /var/log/pacman.log
#GPGDir      = /etc/pacman.d/gnupg/
#HookDir     = /etc/pacman.d/hooks/
HoldPkg     = pacman glibc
#XferCommand = /usr/bin/curl -L -C - -f -o %o %u
#XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
XferCommand =/usr/bin/aria2c --allow-overwrite=true --continue=true --file-allocation=none --log-level=error --max-tries=2 --max-connection-per-server=16 --split=20 --max-file-not-found=5 --min-split-size=1M --no-conf --remote-time=true --summary-interval=60 --timeout=5 --dir=/ --out %o %u
#CleanMethod = KeepInstalled
Architecture = auto

# Pacman won't upgrade packages listed in IgnorePkg and members of IgnoreGroup
#IgnorePkg   =
#IgnoreGroup =

#NoUpgrade   =
#NoExtract   =

# Misc options
#UseSyslog
Color
#NoProgressBar
CheckSpace
#VerbosePkgLists
#ParallelDownloads = 5

# By default, pacman accepts packages signed by keys that its local keyring
# trusts (see pacman-key and its man page), as well as unsigned packages.
#SigLevel    = Required DatabaseNever
SigLevel    = PackageRequired
LocalFileSigLevel = Optional
#RemoteFileSigLevel = Required

# NOTE: You must run `pacman-key --init` before first using pacman; the local
# keyring can then be populated with the keys of all official Arch Linux
# packagers with `pacman-key --populate archlinux`.

#
# REPOSITORIES
#   - can be defined here or included from another file
#   - pacman will search repositories in the order defined here
#   - local/custom mirrors can be added here or in separate files
#   - repositories listed first will take precedence when packages
#     have identical names, regardless of version number
#   - URLs will have $repo replaced by the name of the current repo
#   - URLs will have $arch replaced by the name of the architecture
#
# Repository entries are of the format:
#       [repo-name]
#       Server = ServerName
#       Include = IncludePath
#
# The header [repo-name] is crucial - it must be present and
# uncommented to enable the repo.
#

# The testing repositories are disabled by default. To enable, uncomment the
# repo name header and Include lines. You can add preferred servers immediately
# after the header, and they will be used before the default mirrors.

#[testing]
#Include = /etc/pacman.d/mirrorlist

[core]
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /etc/pacman.d/mirrorlist

#[community-testing]
#Include = /etc/pacman.d/mirrorlist

[community]
Include = /etc/pacman.d/mirrorlist

# If you want to run 32 bit applications on your x86_64 system,
# enable the multilib repositories as required here.

#[multilib-testing]
#Include = /etc/pacman.d/mirrorlist

#[multilib]
#Include = /etc/pacman.d/mirrorlist

# An example of a custom package repository.  See the pacman manpage for
# tips on creating your own repositories.
#[custom]
#SigLevel = Optional TrustAll
#Server = file:///home/custompkgs

'/etc/powerpill/aria2.conf' contents left blank

Last edited by czg (2023-03-29 16:21:10)

Xyne · 2023-04-10 20:27:41

Powerpill uses aria2c to download files, which always displays messages to indicate that it's downloading the files even when it just verifies existing files. You can test this by running a download command for a large package twice (e.g. "powerpill -Sw 0ad-data").

I don't know where the double slash in

//var

is coming from. Does it appear when using powerpill directly instead of through yay?

czg · 2023-04-11 05:28:54

My fault, I found the '//var' comes from '/etc/pacman_powermill.conf' used in my '/etc/powerpill/powerpill.json',
comment it and then no duplicate.

Thanks for your replies.

And one more question, sometimes I use yay to upgrade packages with powerpill (download package no problem) , it will download to one 'ir=' directory (Sorry, I temporarily forget what command drop this weird behavior), what may result this?

'/etc/pacman_powermill.conf'

...
XferCommand =/usr/bin/aria2c --allow-overwrite=true --continue=true --auto-file-renaming=false --file-allocation=falloc --log-level=error --max-tries=2 --max-connection-per-server=16 --split=20 --max-file-not-found=5 --min-split-size=1M --no-conf --remote-time=true --summary-interval=60 --timeout=5 --dir=/ --out %o %u
...

and the context around '//var' with 'XferCommand' in conf

:: Retrieving packages...

04/11 13:16:54 [NOTICE] Downloading 1 item(s)
[#fd5f3d 16KiB/41KiB(38%) CN:1 DL:42KiB]                                                                                                                            
04/11 13:16:55 [NOTICE] Download complete: //var/cache/pacman/pkg/rime-emoji-0.0.0.20230308-1-any.pkg.tar.zst.part

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
fd5f3d|OK  |    62KiB/s|//var/cache/pacman/pkg/rime-emoji-0.0.0.20230308-1-any.pkg.tar.zst.part

Status Legend:
(OK):download completed.
(1/1) checking keys in keyring                                                                      [##########################################################] 100%

Arch Linux

#1 2023-03-29 11:22:47

powerpill maybe duplicate download

#2 2023-04-10 20:27:41

Re: powerpill maybe duplicate download

#3 2023-04-11 05:28:54

Re: powerpill maybe duplicate download

Board footer