Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

Xyne · 2017-03-10 23:37:59

The previous error (No such file or directory: '/var/lib/pacman/db.lck') indicates that something else removed the lock while you were running bauerbill. I suppose Bauerbill could just silently ignore it as it is meant to unlock the database at that point anyway, but it nevertheless indicates an issue with multiple programs accessing the database at the same time. What else were you running?

For the second error, check the permissions on the databases in ~/.cache/AUR/ (or whatever $XDG_CACHE_HOME/AUR may be set to). Those files should be owned by your user. I'll change MemoizeDB to catch the exception and thrown another with the path to the database.

steventheevilz · 2017-03-11 18:44:11

K, Thank you very much!

steventheevilz · 2017-03-11 18:46:08

Update: The perms are correct, but it still gives the same error.

Xyne · 2017-03-19 19:59:58

Remove the databases and rerun the script to make sure that the script is recreating them (and thus using the paths that you think it is). Post the permissions and paths of the recreated databases.

steventheevilz · 2017-03-25 17:58:17

After reinstalling the program, it works!!! TYVM!

GSF1200S · 2017-03-27 04:59:03

I see that Bauerbill has ABS support, but can it selectively rebuild packages? So say:

bb-wrapper -Syu --aur

updates everything, updates AUR packages by building them (of course), and also builds (as an example:) vim, geany, and xfce4-panel from ABS if an update is present (but updates everything else from the repos)?

Basically I have a list of packages that I build to gain the hardening from hardening-wrapper. Its a small list, and I'd like to be able to have any packages from that small list built from source if updates are available for those packages.

yaourt is the only AUR helper I know of that can do this- you can use customizepkg. Any file in /etc/customizepkg.d/ denotes that package needs to be built from ABS automatically. So, if an empty file at /etc/customizepkg.d/xfce4-panel exists, xfce4-panel will be built from ABS automatically whenever an update is available.

Of course yaourt sucks for many reasons (and its dangerous) so I dont use it and have just been painfully doing all this manually. Is there a way to get Bauerbill to fill this role? A line in bauerbill.json perhaps or something?

I could write a script in bash to fill this roll, but no sense reinventing the wheel if you have a better idea.

Xyne · 2017-03-29 01:00:27

For now I have added a new field to bauerbill.json named build patterns. It should be a list of glob patterns. Any pattern with a slash ("/") will be matched against a qualified package name (<repo>/<pkgname>). Any pattern without a slash will be matched against the unqualified package name (<pkgname>). All packages that match will be automatically added to the build queue if possible. I have only done some quick testing so let me know if you find a bug.

For example, to build all packages from the core repo that start with "p" and glibc, use

"build patterns" : ["core/p*", "glibc"]

This will be more cleanly handled with the pacboy backend. That already supports forcing builds via the [+] prefix and excluding builds with the [-] prefix (which will likely become configurable). The configuration file will have automatic build and nobuild lists (if it doesn't already... I don't remember if they've already been included) and much easier automated per-package customized builds.

GSF1200S · 2017-03-29 06:43:58

Xyne wrote:

For now I have added a new field to bauerbill.json named build patterns. It should be a list of glob patterns. Any pattern with a slash ("/") will be matched against a qualified package name (<repo>/<pkgname>). Any pattern without a slash will be matched against the unqualified package name (<pkgname>). All packages that match will be automatically added to the build queue if possible. I have only done some quick testing so let me know if you find a bug.
For example, to build all packages from the core repo that start with "p" and glibc, use
"build patterns" : ["core/p*", "glibc"]
This will be more cleanly handled with the pacboy backend. That already supports forcing builds via the [+] prefix and excluding builds with the [-] prefix (which will likely become configurable). The configuration file will have automatic build and nobuild lists (if it doesn't already... I don't remember if they've already been included) and much easier automated per-package customized builds.

Before I got all elaborate, I figured I'd start with a simple test. Checked to see if I had updates, and found lpsolve to be the only package needing one besides kernels and headers. Added this package to bauerbill.json as follows:

"build patterns" : ["lpsolve"],

I figured I would mess with qualified packages after this test. As a control, I also manually downgraded youtube-dl, but didnt add it to "build patterns" above. Bug noticed: lpsolve was still downloaded from the repos (in binary form), and then after the update occurred I got this error message:

: Processing package changes...
(1/6) upgrading libinput                           [######################] 100%
(2/6) upgrading linux                              [######################] 100%
>>> Updating module dependencies. Please wait ...
(3/6) upgrading linux-headers                      [######################] 100%
(4/6) upgrading linux-lts                          [######################] 100%
>>> Updating module dependencies. Please wait ...
(5/6) upgrading youtube-dl                         [######################] 100%
(6/6) upgrading lpsolve                            [######################] 100%
:: Running post-transaction hooks...
(1/4) Updating linux-lts initcpios
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'default'
  -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts.img
==> Starting build: 4.9.18-1-lts
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [encrypt]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux-lts.preset: 'fallback'
  -> -k /boot/vmlinuz-linux-lts -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-lts-fallback.img -S autodetect
==> Starting build: 4.9.18-1-lts
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: wd719x
  -> Running build hook: [encrypt]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-lts-fallback.img
==> Image generation successful
(2/4) Updating linux initcpios
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 4.10.6-1-ARCH
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [encrypt]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 4.10.6-1-ARCH
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: wd719x
==> WARNING: Possibly missing firmware for module: aic94xx
  -> Running build hook: [encrypt]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful
(3/4) Updating udev hardware database...
(4/4) Arming ConditionNeedsUpdate...
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/MemoizeDB.py", line 235, in db_insert
    c.execute(query, args)
sqlite3.OperationalError: attempt to write a readonly database

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/Bauerbill.py", line 1134, in run_main
    return main(args)
  File "/usr/lib/python3.6/site-packages/Bauerbill.py", line 1119, in main
    bb.generate_build_scripts(build_pkgs, build_deps)
  File "/usr/lib/python3.6/site-packages/Bauerbill.py", line 648, in generate_build_scripts
    scripts['download'] = self.download_script(build_pkgs)
  File "/usr/lib/python3.6/site-packages/Bauerbill.py", line 670, in download_script
    pbs = XCPF.ArchPkg.collect_pkgbases(build_pkgs)
  File "/usr/lib/python3.6/site-packages/XCPF/ArchPkg.py", line 216, in collect_pkgbases
    pkgbases[pkg.pkgbase()].append(pkg)
  File "/usr/lib/python3.6/site-packages/XCPF/ArchPkg.py", line 481, in pkgbase
    return self.get_pkginfo()['pkgbase']
  File "/usr/lib/python3.6/site-packages/XCPF/ArchPkg.py", line 441, in get_pkginfo
    self.pkg.db.name, self.pkg.arch, self.pkg.name
  File "/usr/lib/python3.6/site-packages/XCPF/common.py", line 159, in memoized_archlinux_org_pkg_info
    txt = mdb.get_one('request', url)
  File "/usr/lib/python3.6/site-packages/MemoizeDB.py", line 561, in get_one
    return self.get_nth_field(table, key, n=n)
  File "/usr/lib/python3.6/site-packages/MemoizeDB.py", line 569, in get_nth_field
    vs = self.get(table, key)
  File "/usr/lib/python3.6/site-packages/MemoizeDB.py", line 530, in get
    self.db_insert(table, key, values)
  File "/usr/lib/python3.6/site-packages/MemoizeDB.py", line 244, in db_insert
    self.raise_mdb_error('row insertion(s) failed', error=e)
  File "/usr/lib/python3.6/site-packages/MemoizeDB.py", line 181, in raise_mdb_error
    raise MDBError(self.conn, *args, **kwargs)
MemoizeDB.MDBError: <exception str() failed>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/site-packages/Bauerbill.py", line 1152, in <module>
    run_main()
  File "/usr/lib/python3.6/site-packages/Bauerbill.py", line 1145, in run_main
    logging.error(str(e))
  File "/usr/lib/python3.6/site-packages/MemoizeDB.py", line 79, in __str__
    display_msg = '{} ({})'.format(db_names)
IndexError: tuple index out of range

I tried it again using "youtube-dl" in build patterns but same deal. I dont know if something is up with my system, if I've misinterpreted what you intended to say, or if this is a bug but thats what I've got right now. I understand your response and in theory your solution in bauerbill.json would be more than enough for me (let alone pacboy). Let me know what to try/what I need to fix...

Xyne · 2017-03-29 12:07:07

The error in catching the permission error should be fixed now so it will print out the affected path, but the error itself will persist and is unrelated to the recent update. Please let me know what the permissions on the database are. There is some use case in which permissions are not handled correctly, but I have not yet been able to reproduce it.

I tested the build patterns again and the target package was not downloaded from the repo here.

GSF1200S · 2017-03-29 20:11:15

Xyne wrote:

The error in catching the permission error should be fixed now so it will print out the affected path, but the error itself will persist and is unrelated to the recent update. Please let me know what the permissions on the database are. There is some use case in which permissions are not handled correctly, but I have not yet been able to reproduce it.

I get:

[2017-03-29 14:54:51] ERROR: MDBError ("/root/.cache/XCPF/pkginfo.sqlite3") [attempt to write a readonly database]

but when I attempt to ls /root/.cache/XCPF (as root) I dont see database or file contained within. **EDIT** ok, the database shows up when I run bb-wrapper -Syu, but only for a brief period of time before it is deleted. I caught it and this is the result:

-rw-r--r-- 1 root root 12288 Mar 29 20:54 pkginfo.sqlite3

Xyne wrote:

I tested the build patterns again and the target package was not downloaded from the repo here.

Im not sure if something is up with my system or what then, because it downloads it before printing out the above error. FWIW my system is completely up-to-date, I've been using it for years, and I have no known issues. Im still unfamiliar with aria output so maybe I'm missing something:

...
03/29 14:54:24 [NOTICE] Downloading 1 item(s)

03/29 14:54:24 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/youtube-dl-2017.03.24-1-any.pkg.tar.xz

03/29 14:54:24 [NOTICE] Download complete: /var/cache/pacman/pkg/youtube-dl-2017.03.24-1-any.pkg.tar.xz

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
7f6943|OK  |       0B/s|/var/cache/pacman/pkg/youtube-dl-2017.03.24-1-any.pkg.tar.xz

Status Legend:
(OK):download completed.
:: Starting full system upgrade...
warning: wine: ignoring package upgrade (2.1-1 => 2.4-1)
resolving dependencies...
looking for conflicting packages...

Packages (1) youtube-dl-2017.03.24-1

Total Installed Size:  10.58 MiB
Net Upgrade Size:      -0.01 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                     [######################] 100%
(1/1) checking package integrity                   [######################] 100%
(1/1) loading package files                        [######################] 100%
(1/1) checking for file conflicts                  [######################] 100%
(1/1) checking available disk space                [######################] 100%
:: Processing package changes...
(1/1) upgrading youtube-dl                         [######################] 100%
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
[2017-03-29 14:54:51] ERROR: MDBError ("/root/.cache/XCPF/pkginfo.sqlite3") [attempt to write a readonly database]

It seems based on the output the wrapper is passing to pacman before trying to build? If I do:

bb-wrapper -Syu --build

I just get the readonly database error above (but it doesnt try to download the package from the repos) [same permissions as above]. I want to make sure I understand your initial response adding "build patterns" correctly: I should be able to have "youtube-dl" in build patterns (as I did above), issue bb-wrapper -Syu and have youtube-dl built from source while all other package updates are installed from the repos, correct?

Just so we know we're on the same page...

Last edited by GSF1200S (2017-03-30 01:58:03)

Xyne · 2017-03-30 14:17:07

GSF1200S wrote:

I just get the readonly database error above (but it doesnt try to download the package from the repos) [same permissions as above]. I want to make sure I understand your initial response adding "build patterns" correctly: I should be able to have "youtube-dl" in build patterns (as I did above), issue bb-wrapper -Syu and have youtube-dl built from source while all other package updates are installed from the repos, correct?
Just so we know we're on the same page...

Yep. For testing, I have added "build patterns":["aria2"] to bauerbill.json. When I issue bb-wrapper -S --cachedir foo nano aria2, for example, only nano is downloaded to the cache directory then pacman is used to install nano before proceeding to build and install aria2. I get no permission errors either.

Are you using an alias or running the wrapper script as root?

GSF1200S · 2017-03-30 17:49:42

Xyne wrote:

GSF1200S wrote:
I just get the readonly database error above (but it doesnt try to download the package from the repos) [same permissions as above]. I want to make sure I understand your initial response adding "build patterns" correctly: I should be able to have "youtube-dl" in build patterns (as I did above), issue bb-wrapper -Syu and have youtube-dl built from source while all other package updates are installed from the repos, correct?
Just so we know we're on the same page...
Yep. For testing, I have added "build patterns":["aria2"] to bauerbill.json. When I issue bb-wrapper -S --cachedir foo nano aria2, for example, only nano is downloaded to the cache directory then pacman is used to install nano before proceeding to build and install aria2. I get no permission errors either.
Are you using an alias or running the wrapper script as root?

No alias, and im not running the wrapper as root. Pacaur works, I dont have any issues with makepkg/ABS manually, I redid all pacman-key just to make sure I had no issues there (though I dont think its remotely related), and im not calling bb-wrapper from another script or anything like that.

What sucks about the issue is that its not consistent enough for me to give you an exact bead on what my problem exactly is. Your last post caused me to try -S (I had been just trying to -Syu prior), and I have some more info that might help.

First, as root I completely deleted /root/.cache. Permissions looked right to me prior (when I could catch the database popping up) but what the hell. I -Scc'd my package cache (i have btrfs so no loss), and I made sure /home/user/build was empty. Nano did end up back in there with my experiments, but its inconsequential. Then:

...
loading packages...
warning: downgrading package youtube-dl (2017.03.24-1 => 2017.03.22-1)
resolving dependencies...
looking for conflicting packages...

Packages (1) youtube-dl-2017.03.22-1

Total Installed Size:  10.59 MiB
Net Upgrade Size:       0.12 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                     [######################] 100%
(1/1) checking package integrity                   [######################] 100%
(1/1) loading package files                        [######################] 100%
(1/1) checking for file conflicts                  [######################] 100%
(1/1) checking available disk space                [######################] 100%
:: Processing package changes...
(1/1) downgrading youtube-dl                       [######################] 100%
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...

I downgraded youtube-dl to test. Onto the wrapper script (but using -S instead of doing -Syu) [gonna cut out build spam]:

[gsf1200s@comp ~]$ bb-wrapper -S --cachedir /tmp/foo nano youtube-dl
created build/download.sh
created build/build.sh
~/build ~
[2017-03-30 12:03:18] INFO: searching ABS Git interface
[2017-03-30 12:03:31] INFO: found nano at https://projects.archlinux.org/svntogit/packages.git/plain/nano/trunk
[2017-03-30 12:03:44] INFO: found youtube-dl at https://projects.archlinux.org/svntogit/community.git/plain/youtube-dl/trunk
~/build/nano ~/build
nano : 4719
~/build
~/build/youtube-dl ~/build
youtube-dl : 4720
~/build
==> Making package: youtube-dl 2017.03.24-1 (Thu Mar 30 12:03:45 CDT 2017)
==> WARNING: Skipping dependency checks.
==> Retrieving sources...
  -> Downloading youtube-dl-2017.03.24.tar.gz...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0==> ERROR: A package has already been built. (use -f to overwrite)
  0     3    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
100   603    0   603    0     0    131      0 --:--:--  0:00:04 --:--:--   335
100 2581k  100 2581k    0     0   162k      0  0:00:15  0:00:15 --:--:--  444k
  -> Downloading youtube-dl-2017.03.24.tar.gz.sig...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     3    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100   608    0   608    0     0    214      0 --:--:--  0:00:02 --:--:--   656
100   566  100   566    0     0    117      0  0:00:04  0:00:04 --:--:--   649
==> Validating source files with sha256sums...
    youtube-dl-2017.03.24.tar.gz ... Passed
    youtube-dl-2017.03.24.tar.gz.sig ... Skipped
==> Verifying source file signatures with gpg...
    youtube-dl-2017.03.24.tar.gz ... Passed
~/build/youtube-dl ~/build
==> Making package: youtube-dl 2017.03.24-1 (Thu Mar 30 12:04:06 CDT 2017)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Found youtube-dl-2017.03.24.tar.gz
  -> Found youtube-dl-2017.03.24.tar.gz.sig
==> Validating source files with sha256sums...
    youtube-dl-2017.03.24.tar.gz ... Passed
    youtube-dl-2017.03.24.tar.gz.sig ... Skipped
==> Verifying source file signatures with gpg...
    youtube-dl-2017.03.24.tar.gz ... Passed
==> Extracting sources...
  -> Extracting youtube-dl-2017.03.24.tar.gz with bsdtar
==> Starting prepare()...
==> Entering fakeroot environment...
==> Starting package()...
... 
...
==>Tidying install...
  -> Removing libtool files...
  -> Purging unwanted files...
  -> Removing static library files...
  -> Stripping unneeded symbols from binaries and libraries...
  -> Compressing man and info pages...
==> Checking for packaging issue...
==> Creating package "youtube-dl"...
  -> Generating .PKGINFO file...
  -> Generating .BUILDINFO file...
  -> Generating .MTREE file...
  -> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: youtube-dl 2017.03.24-1 (Thu Mar 30 12:04:27 CDT 2017)
==> Installing package youtube-dl with pacman -U...
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) youtube-dl-2017.03.24-1

Total Installed Size:  10.46 MiB
Net Upgrade Size:      -0.12 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                     [######################] 100%
(1/1) checking package integrity                   [######################] 100%
(1/1) loading package files                        [######################] 100%
(1/1) checking for file conflicts                  [######################] 100%
(1/1) checking available disk space                [######################] 100%
:: Processing package changes...
(1/1) upgrading youtube-dl                         [######################] 100%
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
~/build
~/build/nano ~/build
==> WARNING: A package has already been built, installing existing package...
==> Installing package nano with pacman -U...
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) nano-2.7.5-1

Total Installed Size:  1.89 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                     [######################] 100%
(1/1) checking package integrity                   [######################] 100%
(1/1) loading package files                        [######################] 100%
(1/1) checking for file conflicts                  [######################] 100%
(1/1) checking available disk space                [######################] 100%
:: Processing package changes...
(1/1) installing nano                              [######################] 100%
:: Running post-transaction hooks...
(1/2) Arming ConditionNeedsUpdate...
(2/2) Updating the info directory file...
~/build
~

Pulls nano from cache, builds youtube-dl, installs as expected, and no readonly database error. Then I downgrade youtube-dl again (and delete youtube-dl from /home/user/build and /var/cache/pacman/pkg) just as above. Then I try -Syu:

[gsf1200s@comp ~]$ bb-wrapper -Syu --cachedir /tmp/foo

03/30 12:13:05 [NOTICE] Downloading 6 item(s)

03/30 12:13:05 [NOTICE] GID#f1bec0371359c9c9 - Download has already completed: /var/lib/pacman/sync/xyne-x86_64.db.sig

03/30 12:13:05 [NOTICE] Download complete: /var/lib/pacman/sync/xyne-x86_64.db.sig

03/30 12:13:05 [NOTICE] GID#a3e6634e4a0b7d80 - Download has already completed: /var/lib/pacman/sync/xyne-x86_64.db

03/30 12:13:05 [NOTICE] Download complete: /var/lib/pacman/sync/xyne-x86_64.db
[DL:0B][#6a5118 0B/0B][#5e62e2 0B/0B][#aa8d97 0B/0B][#de011e 0B/0B]            
03/30 12:13:09 [NOTICE] GID#de011e90481edd5b - Download has already completed: /var/lib/pacman/sync/multilib.db

03/30 12:13:09 [NOTICE] Download complete: /var/lib/pacman/sync/multilib.db

03/30 12:13:09 [NOTICE] GID#6a5118a334c977a4 - Download has already completed: /var/lib/pacman/sync/community.db

03/30 12:13:09 [NOTICE] Download complete: /var/lib/pacman/sync/community.db

03/30 12:13:09 [NOTICE] GID#5e62e2772dd817b8 - Download has already completed: /var/lib/pacman/sync/core.db

03/30 12:13:09 [NOTICE] Download complete: /var/lib/pacman/sync/core.db

03/30 12:13:09 [NOTICE] GID#aa8d973e9c6f0599 - Download has already completed: /var/lib/pacman/sync/extra.db

03/30 12:13:09 [NOTICE] Download complete: /var/lib/pacman/sync/extra.db

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
f1bec0|OK  |       0B/s|/var/lib/pacman/sync/xyne-x86_64.db.sig
a3e663|OK  |       0B/s|/var/lib/pacman/sync/xyne-x86_64.db
de011e|OK  |       0B/s|/var/lib/pacman/sync/multilib.db
6a5118|OK  |       0B/s|/var/lib/pacman/sync/community.db
5e62e2|OK  |       0B/s|/var/lib/pacman/sync/core.db
aa8d97|OK  |       0B/s|/var/lib/pacman/sync/extra.db

Status Legend:
(OK):download completed.

03/30 12:13:09 [NOTICE] Downloading 1 item(s)
[#79982f 1.7MiB/1.8MiB(98%) CN:1 DL:282KiB]                                    
03/30 12:13:20 [NOTICE] Verification finished successfully. file=/tmp/foo/youtube-dl-2017.03.24-1-any.pkg.tar.xz

03/30 12:13:20 [NOTICE] Download complete: /tmp/foo/youtube-dl-2017.03.24-1-any.pkg.tar.xz

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
79982f|OK  |   167KiB/s|/tmp/foo/youtube-dl-2017.03.24-1-any.pkg.tar.xz

Status Legend:
(OK):download completed.
:: Starting full system upgrade...
warning: wine: ignoring package upgrade (2.1-1 => 2.4-1)
resolving dependencies...
looking for conflicting packages...

Packages (1) youtube-dl-2017.03.24-1

Total Installed Size:  10.58 MiB
Net Upgrade Size:      -0.01 MiB

:: Proceed with installation? [Y/n] n
[2017-03-30 12:13:38] ERROR: MDBError ("/root/.cache/XCPF/pkginfo.sqlite3") [attempt to write a readonly database]
~/build ~
~

It tries to download the package from the repos, and gives me the readonly database error again. It does the same thing if I select 'y' and let it install the repo package first... sometimes. Take this example of where it downloaded the package from the repos, and then:

[poeticrpm@codething ~]$ bb-wrapper -Syu

03/30 12:06:58 [NOTICE] Downloading 6 item(s)

03/30 12:06:58 [NOTICE] GID#4d828d5836dfca1d - Download has already completed: /var/lib/pacman/sync/xyne-x86_64.db.sig

03/30 12:06:58 [NOTICE] Download complete: /var/lib/pacman/sync/xyne-x86_64.db.sig

03/30 12:06:58 [NOTICE] GID#124af4c10b3dc6e4 - Download has already completed: /var/lib/pacman/sync/xyne-x86_64.db

03/30 12:06:58 [NOTICE] Download complete: /var/lib/pacman/sync/xyne-x86_64.db
[DL:0B][#861d8d 0B/0B][#a1b261 0B/0B][#d368f8 0B/0B][#1489fa 0B/0B]            
03/30 12:07:02 [NOTICE] GID#861d8d116e7ab7d1 - Download has already completed: /var/lib/pacman/sync/community.db

03/30 12:07:02 [NOTICE] Download complete: /var/lib/pacman/sync/community.db

03/30 12:07:02 [NOTICE] GID#a1b2619ae96a705b - Download has already completed: /var/lib/pacman/sync/core.db

03/30 12:07:02 [NOTICE] Download complete: /var/lib/pacman/sync/core.db

03/30 12:07:02 [NOTICE] GID#1489fa6ad6ac7180 - Download has already completed: /var/lib/pacman/sync/multilib.db

03/30 12:07:02 [NOTICE] Download complete: /var/lib/pacman/sync/multilib.db

03/30 12:07:02 [NOTICE] GID#d368f884ddc28c51 - Download has already completed: /var/lib/pacman/sync/extra.db

03/30 12:07:02 [NOTICE] Download complete: /var/lib/pacman/sync/extra.db

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
4d828d|OK  |       0B/s|/var/lib/pacman/sync/xyne-x86_64.db.sig
124af4|OK  |       0B/s|/var/lib/pacman/sync/xyne-x86_64.db
861d8d|OK  |       0B/s|/var/lib/pacman/sync/community.db
a1b261|OK  |       0B/s|/var/lib/pacman/sync/core.db
1489fa|OK  |       0B/s|/var/lib/pacman/sync/multilib.db
d368f8|OK  |       0B/s|/var/lib/pacman/sync/extra.db

Status Legend:
(OK):download completed.

03/30 12:07:02 [NOTICE] Downloading 1 item(s)
[#47ceb9 1.7MiB/1.8MiB(95%) CN:1 DL:274KiB]                                    
03/30 12:07:13 [NOTICE] Verification finished successfully. file=/var/cache/pacman/pkg/youtube-dl-2017.03.24-1-any.pkg.tar.xz

03/30 12:07:13 [NOTICE] Download complete: /var/cache/pacman/pkg/youtube-dl-2017.03.24-1-any.pkg.tar.xz

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
47ceb9|OK  |   166KiB/s|/var/cache/pacman/pkg/youtube-dl-2017.03.24-1-any.pkg.tar.xz

Status Legend:
(OK):download completed.
:: Starting full system upgrade...
warning: wine: ignoring package upgrade (2.1-1 => 2.4-1)
resolving dependencies...
looking for conflicting packages...

Packages (1) youtube-dl-2017.03.24-1

Total Installed Size:  10.58 MiB
Net Upgrade Size:      -0.01 MiB

:: Proceed with installation? [Y/n] n
created build/download.sh
created build/build.sh
~/build ~
[2017-03-30 12:07:21] INFO: searching ABS Git interface
[2017-03-30 12:07:35] INFO: found youtube-dl at https://projects.archlinux.org/svntogit/community.git/plain/youtube-dl/trunk
~/build/youtube-dl ~/build
youtube-dl : 8570
~/build
==> Making package: youtube-dl 2017.03.24-1 (Thu Mar 30 12:07:36 CDT 2017)
==> WARNING: Skipping dependency checks.
==> Retrieving sources...
  -> Downloading youtube-dl-2017.03.24.tar.gz...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     3    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100   603    0   603    0     0    195      0 --:--:--  0:00:03 --:--:--   663
100 2581k  100 2581k    0     0   139k      0  0:00:18  0:00:18 --:--:--  398k
  -> Downloading youtube-dl-2017.03.24.tar.gz.sig...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     3    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100   608    0   608    0     0    214      0 --:--:--  0:00:02 --:--:--   377
100   566  100   566    0     0    117      0  0:00:04  0:00:04 --:--:--   369
==> Validating source files with sha256sums...
    youtube-dl-2017.03.24.tar.gz ... Passed
    youtube-dl-2017.03.24.tar.gz.sig ... Skipped
==> Verifying source file signatures with gpg...
    youtube-dl-2017.03.24.tar.gz ... Passed
~/build/youtube-dl ~/build
==> Making package: youtube-dl 2017.03.24-1 (Thu Mar 30 12:08:00 CDT 2017)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Found youtube-dl-2017.03.24.tar.gz
  -> Found youtube-dl-2017.03.24.tar.gz.sig
==> Validating source files with sha256sums...
    youtube-dl-2017.03.24.tar.gz ... Passed
    youtube-dl-2017.03.24.tar.gz.sig ... Skipped
==> Verifying source file signatures with gpg...
    youtube-dl-2017.03.24.tar.gz ... Passed
==> Extracting sources...
  -> Extracting youtube-dl-2017.03.24.tar.gz with bsdtar
==> Starting prepare()...
==> Entering fakeroot environment...
==> Starting package()...
...
...
==> Tidying install...
  -> Removing libtool files...
  -> Purging unwanted files...
  -> Removing static library files...
  -> Stripping unneeded symbols from binaries and libraries...
  -> Compressing man and info pages...
==> Checking for packaging issue...
==> Creating package "youtube-dl"...
  -> Generating .PKGINFO file...
  -> Generating .BUILDINFO file...
  -> Generating .MTREE file...
  -> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: youtube-dl 2017.03.24-1 (Thu Mar 30 12:08:21 CDT 2017)
==> Installing package youtube-dl with pacman -U...
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) youtube-dl-2017.03.24-1

Total Installed Size:  10.46 MiB
Net Upgrade Size:      -0.12 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                     [######################] 100%
(1/1) checking package integrity                   [######################] 100%
(1/1) loading package files                        [######################] 100%
(1/1) checking for file conflicts                  [######################] 100%
(1/1) checking available disk space                [######################] 100%
:: Processing package changes...
(1/1) upgrading youtube-dl                         [######################] 100%
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
~/build
~

In this case, it tried again to install from the repos, I told it no, and then it went on to build the package.

The thing is, I know that a few times -Syu has worked fine, but I cant seem to reproduce it working correctly with any consistency. It works for everyone else so I dont know wtf is up on my end

Last edited by GSF1200S (2017-03-30 17:52:15)

Xyne · 2017-03-30 22:54:41

I found the bug responsible for the package download. Strangely, by fixing that I can now reproduce the database permission error. I'm still scratching my head on that. I have some suspicions but I need to take the time to debug it. I'll post an update once I figure it out (and hopefully release a fix).

edit
It should be fixed now. Please update bauerbill, powerpill and pm2ml and try again.

Last edited by Xyne (2017-03-30 23:43:20)

GSF1200S · 2017-03-31 13:38:46

Xyne wrote:

I found the bug responsible for the package download. Strangely, by fixing that I can now reproduce the database permission error. I'm still scratching my head on that. I have some suspicions but I need to take the time to debug it. I'll post an update once I figure it out (and hopefully release a fix).
edit
It should be fixed now. Please update bauerbill, powerpill and pm2ml and try again.

I wanted to make sure I didnt jump the gun before giving the all clear; after using it extensively it seems the latest version works perfect. No database errors and it seems to honor "build packages" as expected now.

Thanks for the fast turnaround with the feature add and fixes

dapolinario · 2017-03-31 14:32:24

But, for me in the last update (powerpill 2017-1 and other packages) occurs the problem:

https://gist.github.com/anonymous/64739 … 77d3098570

Xyne · 2017-04-01 00:07:54

dapolinario wrote:

But, for me in the last update (powerpill 2017-1 and other packages) occurs the problem:
https://gist.github.com/anonymous/64739 … 77d3098570

That should be fixed with pm2ml-2017.3-1.

dapolinario · 2017-04-01 02:43:58

Thanks!

GSF1200S · 2017-04-01 10:19:18

Xyne wrote:

zerophase wrote:
I noticed trying to add commands for editing PKGBUILDS on the prebuild hook adds a space between each letter.
Are you trying to use a single string as a command? Bauerbill expects each command to be a JSON list, e.g. ["cmd", "--foo", "--bar", "arg1", "arg2"].

FYI, I've noticed this too and I have it as you mention. Relevant from /etc/bauerbill.json:

"hooks" : {
    "directory" : null,
    "arguments" : {
      "default": ["{PackageBase}", "{LastModified}"]
    },
    "commands" : {
      "predownload" : {
        "default" : ["bash", "bbloop.sh"] 
      }
    }
  }

Result in download.sh:

########## Download sources for package base xfce4-calculator-plugin. ##########
# Maintainer(s): oberon2007
# Last modified: 2017-01-19 13:17:24 CST
# Repository: AUR
# Packages: xfce4-calculator-plugin
pushd xfce4-calculator-plugin
b a s h
b b l o o p . s h
makepkg --verifysource --nodeps &
pids+=($!)
echo xfce4-calculator-plugin ": $!"
popd

I just used xfce4-calculator-plugin here as an example. It also seems to newline though maybe I'm not getting the syntax right in bauerbill.json. For now I have my update script use sed to insert my commands before makepkg which works fine. I rarely build new stuff from the AUR so I'd just manually edit it there. Thought I'd let you know (I dont need a fix)...

Xyne · 2017-04-03 23:59:19

GSF1200S wrote:

 "default" : ["bash", "bbloop.sh"]

It supports multiple commands and expects a list of lists. Use

GSF1200S wrote:

 "default" : [["bash", "bbloop.sh"]]

GSF1200S · 2017-04-04 20:08:49

Xyne wrote:

GSF1200S wrote:
 "default" : ["bash", "bbloop.sh"] 
It supports multiple commands and expects a list of lists. Use
GSF1200S wrote:
 "default" : [["bash", "bbloop.sh"]] 

Ahh indeed, that works fine

Been using Bauerbill quite a bit and no issues.

SolarAquarion · 2017-05-01 04:14:45

do i have the ability to for example adding --force to the pacman command

Xyne · 2017-05-01 17:02:22

SolarAquarion wrote:

do i have the ability to for example adding --force to the pacman command

Yes. All pacman commands should be passed through to pacman.

SolarAquarion · 2017-05-02 16:35:28

So do I just edit the bauerbill conf for that to work. What's the version of pacaur -S $AURPACKAGE --force for bauerbill since i would probably need to edit build.sh for each $AURPACKAGE or can i edit bauerbill.json

Last edited by SolarAquarion (2017-05-02 16:37:27)

Xyne · 2017-05-02 17:05:59

SolarAquarion wrote:

So do I just edit the bauerbill conf for that to work. What's the version of pacaur -S $AURPACKAGE --force for bauerbill since i would probably need to edit build.sh for each $AURPACKAGE or can i edit bauerbill.json

bauerbill -S --aur --force $AURPACKAGE

A better question is why are you using force in the first place?

edit
Hmm, nevermind, I thought that worked.

Edit bauerbill.json and add "--force" to the makepkg command. You can use grep patterns to only use force on select packages via the "custom" field.

Last edited by Xyne (2017-05-02 17:10:02)

SolarAquarion · 2017-05-03 01:20:23

I deleted the pacman database for stupid reasons and now i'm matching files with packages

Arch Linux

#151 2017-03-10 23:37:59

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#152 2017-03-11 18:44:11

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#153 2017-03-11 18:46:08

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#154 2017-03-19 19:59:58

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#155 2017-03-25 17:58:17

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#156 2017-03-27 04:59:03

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#157 2017-03-29 01:00:27

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#158 2017-03-29 06:43:58

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#159 2017-03-29 12:07:07

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#160 2017-03-29 20:11:15

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#161 2017-03-30 14:17:07

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#162 2017-03-30 17:49:42

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#163 2017-03-30 22:54:41

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#164 2017-03-31 13:38:46

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#165 2017-03-31 14:32:24

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#166 2017-04-01 00:07:54

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#167 2017-04-01 02:43:58

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#168 2017-04-01 10:19:18

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#169 2017-04-03 23:59:19

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#170 2017-04-04 20:08:49

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#171 2017-05-01 04:14:45

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#172 2017-05-01 17:02:22

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#173 2017-05-02 16:35:28

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#174 2017-05-02 17:05:59

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

#175 2017-05-03 01:20:23

Re: Bauerbill reborn: pacman/powerpill+AUR+ABS+configurability

Board footer