Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

Alad · 2018-05-01 14:17:07

@Alad & Spyhawk
The dependency graph resolution algorithm was too inefficient to handle the behemoth that is ros-indigo-desktop-full in a reasonable amount of time. I have tweaked the algorithm to avoid redundant recursion and it should be very fast now (update to python3-xcpf-2017.2.10). Bauerbill may still appear to hang while retrieving data from the AUR but that is due to the number of packages that it has to query. You can use "--debug" if you want to monitor those queries.

Now that x86 is unsupported, how about returning to the RPC for metadata instead of .SRCINFO, or to preserve existing logic, use an optional flag like --with-srcinfo?

SolarAquarion · 2018-05-28 22:10:32

bauerbill needs to be updated for pacman 5.1

Alad · 2018-05-28 22:15:08

I guess pyalpm needs to be fixed first...

SolarAquarion · 2018-05-29 03:34:08

it was fixed, lol. I didn't get the system at first

Last edited by SolarAquarion (2018-05-29 03:34:31)

Alad · 2018-06-04 10:15:16

Alad wrote:

@Alad & Spyhawk
The dependency graph resolution algorithm was too inefficient to handle the behemoth that is ros-indigo-desktop-full in a reasonable amount of time. I have tweaked the algorithm to avoid redundant recursion and it should be very fast now (update to python3-xcpf-2017.2.10). Bauerbill may still appear to hang while retrieving data from the AUR but that is due to the number of packages that it has to query. You can use "--debug" if you want to monitor those queries.
Now that x86 is unsupported, how about returning to the RPC for metadata instead of .SRCINFO, or to preserve existing logic, use an optional flag like --with-srcinfo?

Alternatively you could use the RPC for initial dependency resolution, then use the resulting .SRCINFO files for the "actual" dependency resolution.

Something else, probably a useful thing to have is git diff. I looked at python3-aur and it seems to either do git clone or git pull, losing diffs in the process. FWIW most of the current active helpers support this: https://wiki.archlinux.org/index.php/AUR_helpers

Last edited by Alad (2018-06-04 10:16:07)

SolarAquarion · 2018-06-07 14:28:05

I'm getting a failure in Bauerbill

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 260, in _aur_query_wrapper
    for r in _aur_query(typ, url):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 284, in _aur_query
    with urllib.request.urlopen(url, data=post_data) as f:
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 414: Request-URI Too Large

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 260, in _aur_query_wrapper
    for r in _aur_query(typ, url):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 284, in _aur_query
    with urllib.request.urlopen(url, data=post_data) as f:
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 414: Request-URI Too Large

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/http/client.py", line 566, in _readall_chunked
    value.append(self._safe_read(chunk_left))
  File "/usr/lib/python3.6/http/client.py", line 614, in _safe_read
    raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(3921 bytes read, 4263 more expected)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/site-packages/Bauerbill.py", line 1151, in <module>
    sys.exit(run_main())
  File "/usr/lib/python3.6/site-packages/Bauerbill.py", line 1132, in run_main
    return main(args)
  File "/usr/lib/python3.6/site-packages/Bauerbill.py", line 1103, in main
    sync_pkgs, sync_deps, build_pkgs, build_deps = bb.get_sync_and_build_targets()
  File "/usr/lib/python3.6/site-packages/Bauerbill.py", line 534, in get_sync_and_build_targets
    pm2ml_pargs,
  File "/usr/lib/python3.6/site-packages/pm2ml.py", line 610, in resolve_targets_from_arguments
    only_needed=pargs.needed,
  File "/usr/lib/python3.6/site-packages/pm2ml.py", line 539, in resolve_targets
    a, b, orphans = self.determine_upgradable(downgrades=(upgrade > 1))
  File "/usr/lib/python3.6/site-packages/pm2ml.py", line 313, in determine_upgradable
    for aur_pkg in found_aur_pkgs:
  File "/usr/lib/python3.6/site-packages/pm2ml.py", line 214, in search_aur
    for pkg in AUR.RPC.insert_full_urls(self.aur.info(names)):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 177, in insert_full_urls
    for pkg in pkgs:
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 380, in get
    for pkg in self._get(*args, **kwargs):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 415, in _get
    for pkg in self.mdb.get_nth_field_many(table, format_rpc_args(names)):
  File "/usr/lib/python3.6/site-packages/MemoizeDB.py", line 579, in get_nth_field_many
    for vs in self.get_many(table, keys):
  File "/usr/lib/python3.6/site-packages/MemoizeDB.py", line 549, in get_many
    for key, values in func(uncached_keys):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 226, in rpc_info
    for pkg in aur_query('info', args):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 248, in aur_query
    for r in _aur_query_wrapper(typ, format_rpc_args(args), by=by):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 268, in _aur_query_wrapper
    for r in _aur_query_wrapper(typ, args[:i], by=by):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 268, in _aur_query_wrapper
    for r in _aur_query_wrapper(typ, args[:i], by=by):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 260, in _aur_query_wrapper
    for r in _aur_query(typ, url):
  File "/usr/lib/python3.6/site-packages/AUR/RPC.py", line 285, in _aur_query
    response = json.loads(f.read().decode(CODING))
  File "/usr/lib/python3.6/http/client.py", line 456, in read
    return self._readall_chunked()
  File "/usr/lib/python3.6/http/client.py", line 570, in _readall_chunked
    raise IncompleteRead(b''.join(value))
http.client.IncompleteRead: IncompleteRead(61232 bytes read)

Xyne · 2018-06-12 23:11:50

@Alad
Noted.

@SolarAquarion
Does the error persist? It looks like it could be due to a transitory server error. Do you know which packages it is querying?

SolarAquarion · 2018-06-13 03:03:13

it's transistory

SolarAquarion · 2018-08-02 19:19:46

/usr/lib/python3.7/site-packages/Bauerbill.py:549: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
logging.warn(msg)

Xyne · 2018-08-07 00:36:07

SolarAquarion wrote:

/usr/lib/python3.7/site-packages/Bauerbill.py:549: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
logging.warn(msg)

Fixed with today's update.

Pse · 2018-08-09 22:19:05

It would appear the PKGBUILD (https://aur.archlinux.org/packages/bauerbill/) for Bauerbill is pointing to a package that no longer exists (points to 2017-10, but only 2018 can be found upstream).

Xyne · 2018-08-09 22:30:38

@Pse
fixed, thanks

SolarAquarion · 2018-08-16 10:34:42

When I do bauerbill -Syu it looks like this

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
253c61|OK  |       0B/s|/var/lib/pacman/sync/multilib-testing.db
820859|OK  |       0B/s|/var/lib/pacman/sync/multilib.db
864dac|OK  |   871KiB/s|/var/lib/pacman/sync/heftig.db
2ea63d|OK  |       0B/s|/var/lib/pacman/sync/mesa-git.db
064f8a|OK  |    69KiB/s|/var/lib/pacman/sync/quarry.db.sig
951d62|OK  |   546KiB/s|/var/lib/pacman/sync/testing.db
aad1e4|OK  |   209KiB/s|/var/lib/pacman/sync/community-testing.db
b95675|ERR |       0B/s|/var/lib/pacman/sync/herecura.db.sig
fbb22b|OK  |       0B/s|/var/lib/pacman/sync/herecura.db
195c25|OK  |       0B/s|/var/lib/pacman/sync/unity8.db
39708a|OK  |   1.3MiB/s|/var/lib/pacman/sync/extra.db
31877c|OK  |       0B/s|/var/lib/pacman/sync/unity8.db.sig
92500c|OK  |   695KiB/s|/var/lib/pacman/sync/quarry.db
87ff45|OK  |   709KiB/s|/var/lib/pacman/sync/community.db
3adcb2|ERR |       0B/s|/var/lib/pacman/sync/community.db.sig
36f431|ERR |       0B/s|/var/lib/pacman/sync/community-testing.db.sig
58ae9b|INPR|       0B/s|/var/lib/pacman/sync/core.db
668c83|ERR |       0B/s|/var/lib/pacman/sync/core.db.sig
c51d5b|ERR |       0B/s|/var/lib/pacman/sync/extra.db.sig
ed3a94|ERR |       0B/s|/var/lib/pacman/sync/multilib.db.sig
ac53d1|ERR |       0B/s|/var/lib/pacman/sync/multilib-testing.db.sig

Xyne · 2018-08-22 17:24:08

Try a different mirror for the official repos then.

SolarAquarion · 2018-09-04 15:26:28

It always fail when it's trying to download the .sig

Xyne · 2018-09-08 16:09:26

@SolarAquarion
For the official repos that do not provide database signatures, set the appropriate SigLevel, e.g.:

[core]
SigLevel = PackageRequired
Include = /etc/pacman.d/mirrorlist

SolarAquarion · 2018-09-08 18:16:54

==> ERROR: Failure while downloading https://xyne.archlinux.ca/projects/powe … 9.8.tar.xz

eschwartz · 2018-09-09 08:14:28

SolarAquarion wrote:

It always fail when it's trying to download the .sig

Powerpill now explicitly documents, that it does not implement pacman's "Optional" level for signatures and you *must* explicitly set the expected signature level, since the official pacman.conf makes use of said "Optional" level.

Pse · 2018-09-13 19:35:41

It would appear the powerpill PKGBUILD is now also pointing to an invalid link:

==> Making package: powerpill 2018.9.8-1 (Thu 13 Sep 2018 04:06:42 PM -03)
==> WARNING: Skipping dependency checks.
==> Retrieving sources...
  -> Downloading powerpill-2018.9.8.tar.xz...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (22) The requested URL returned error: 404 Not Found
==> ERROR: Failure while downloading https://xyne.archlinux.ca/projects/powerpill/src/powerpill-2018.9.8.tar.xz

Alad · 2018-09-14 11:16:55

Can confirm

a821 · 2018-09-14 14:22:04

See https://aur.archlinux.org/packages/powerpill/. The correct links are in the first comment (on 2018-09-11)

Edit: see comment below

Last edited by a821 (2018-09-14 18:17:00)

eschwartz · 2018-09-14 15:28:21

You what.

How are the binary compiled packages, equivalent to the sources?

a821 · 2018-09-14 18:20:39

Eschwartz wrote:

How are the binary compiled packages equivalent to the sources?

You're right. Obviously I didn't pay attention

SolarAquarion · 2019-01-21 21:27:20

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.7/site-packages/Bauerbill.py", line 42, in <module>
    import pm2ml
  File "/usr/lib/python3.7/site-packages/pm2ml.py", line 38, in <module>
    import XCPF.PacmanConfig
  File "/usr/lib/python3.7/site-packages/XCPF/PacmanConfig.py", line 28, in <module>
    from pycman.config import pacman_conf_enumerator, _logmask, cb_log, LIST_OPTIONS, BOOLEAN_OPTIONS
ImportError: cannot import name 'pacman_conf_enumerator' from 'pycman.config' (/usr/lib/python3.7/site-packages/pycman/config.py)

SolarAquarion · 2020-12-10 13:16:09

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.9/site-packages/pbget.py", line 244, in <module>
    main()
  File "/usr/lib/python3.9/site-packages/pbget.py", line 232, in main
    pkg['Name'],
KeyError: 'Name'

I'm getting this when i do ./download.sh

Last edited by SolarAquarion (2020-12-10 13:16:48)

Arch Linux

#226 2018-05-01 14:17:07

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#227 2018-05-28 22:10:32

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#228 2018-05-28 22:15:08

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#229 2018-05-29 03:34:08

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#230 2018-06-04 10:15:16

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#231 2018-06-07 14:28:05

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#232 2018-06-12 23:11:50

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#233 2018-06-13 03:03:13

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#234 2018-08-02 19:19:46

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#235 2018-08-07 00:36:07

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#236 2018-08-09 22:19:05

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#237 2018-08-09 22:30:38

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#238 2018-08-16 10:34:42

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#239 2018-08-22 17:24:08

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#240 2018-09-04 15:26:28

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#241 2018-09-08 16:09:26

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#242 2018-09-08 18:16:54

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#243 2018-09-09 08:14:28

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#244 2018-09-13 19:35:41

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#245 2018-09-14 11:16:55

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#246 2018-09-14 14:22:04

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#247 2018-09-14 15:28:21

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#248 2018-09-14 18:20:39

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#249 2019-01-21 21:27:20

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#250 2020-12-10 13:16:09

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

Board footer