mandb-ondemand: speed-up pacman updates to the manpage database

graysky · 2016-07-23 15:44:50

Purpose
With the deployment of pacman hooks, maintenance of the manpage index database (/var/cache/man/index.db) has been integrated into pacman operations. Anytime a package writes or removes files to /usr/share/man, a hook triggers the rebuild of the index database which can be slow -- really slow. This is particularly noticeable on low powered devices such as Arch ARM PCs. The delay can be annoying as the terminal (more likely the ssh window) remains locked while the rebuild is in progress.

Solution
mandb-ondemand flags the index database for a rebuild via a pacman hook that triggers a systemd unit to do the dirty work while pacman happily finishes thus leaving the shell free of the dreaded pause!

Without mandb-ondemand:

root@box # pacman -S foo bar
...
:: Running post-transaction hooks...
(1/1) Updating manpage index...
 <<<WAIT>>>    <<<WAIT>>>    <<<WAIT>>>
root@box #

With mandb-ondemand:

root@box # pacman -S foo bar
...
:: Running post-transaction hooks...
(1/1) Asking for an update to mandb's database...
root@box #

Links
AUR package: https://aur.archlinux.org/packages/mandb-ondemand
Upsteam code: https://github.com/graysky2/mandb-ondemand

What it supplies
It's just a few symlinks, a hook, and 2 systemd units. Tiny and simple (less than 16 K):

% pacman -Ql mandb-ondemand
mandb-ondemand /etc/pacman.d/hooks/man-db.hook
mandb-ondemand /usr/lib/systemd/system/mandb-ondemand.path
mandb-ondemand /usr/lib/systemd/system/mandb-ondemand.service
mandb-ondemand /usr/lib/systemd/system/multi-user.target.wants/mandb-ondemand.path
mandb-ondemand /usr/share/libalpm/hooks/mandb-ondemand.hook
mandb-ondemand /usr/share/licenses/mandb-ondemand/LICENSE

Last edited by graysky (2016-07-24 10:25:10)

graysky · 2016-07-23 15:48:03

I should mention that I currently don't know of a way to keep a race condition between two mandb's if the user does multiple, rapid calls to pacman such that call #1 triggers the rebuild and call #2 also triggers a rebuild. I can do this with an intermediate script but that is not very elegant. Does anyone have a suggestion to prevent another mandb from running while one is active using the service units?

Detail: this provides two pairs of service/path units: one runs on an install/upgrade and the other runs on a remove.

The race condition only happens if you rapidly install then remove a package. On slow devices (like RPi for example), the first rebuild can take 20-30 seconds so it's possible that the user can:
pacman -S foo ... wait a sec and decide foo isn't needed and then pacman -Rs foo and thus have both running at the same time.

I guess one solution is to just have one pair running mandb --quiet on install/upgrade/remove rather than what I current have which is 1 running mandb --quiet --no-purge (on installs/upgrade) and another running mandb --quiet (on removes).

EDIT: <<headsmack>> I see how that I in fact only need one pair since Arch upstream is supplying a timer that runs the purge. I will make this modification in version 2.02 and release shortly.

Last edited by graysky (2016-07-23 16:25:32)

Spider.007 · 2016-07-23 15:55:35

Awesome; the pacman-triggers in general are a nice feature but the mandb trigger is a pain.

Shouldn't systemd already prevent the concurrency issue? I'm pretty sure you can't start the same units multiple times (while the first one is still running)

graysky · 2016-07-23 16:39:25

@Spider - Yes, I edited my #2 post above explaining things. In any case, no race condition and a more simplistic package in version 2.02 which is now live in the AUR.

R00KIE · 2016-07-24 01:31:29

Can you elaborate why you have gzip as an optional dependency? I'm probably missing something but I don't see anything that directly depends on gzip or might make use of it if installed.

graysky · 2016-07-24 10:12:27

R00KIE wrote:

Can you elaborate why you have gzip as an optional dependency? I'm probably missing something but I don't see anything that directly depends on gzip or might make use of it if installed.

My bad... I believe that initially I copied the PKGBUILD from man-db rather than /usr/share/pacman/PKGBUILD.proto and just forgot to delete that line. Fixed in 2.02-3.

maggie · 2016-08-22 07:17:12

Thank you for this upgrade. Why doesn't the official package do this?

Awebb · 2016-08-22 09:56:51

maggie wrote:

Thank you for this upgrade. Why doesn't the official package do this?

graysky still seems to be smoothing out the edges. When it's idiot proof, it's probably only a bug report and a short discussion away from the default.

R00KIE · 2016-08-22 10:47:05

The only thing I've noticed that _might_ go wrong is if you reboot before mandb has finished doing its business, but even then it wouldn't be something that would make the system unstable or unbootable.

graysky · 2016-08-22 11:46:07

Edges are pretty smooth now. I don't see why we or formal upstream wouldn't be interested. Frankly, on a fast desktop, it isn't needed since the rebuild is 5-10 sec... but for my arm devices, it is pretty necessary

R00KIE · 2016-08-22 13:29:26

At least here with an i7-6500U big rebuilds still take a good while (not as long as on arm obviously). Either way anything that can be done in the background is nice, specially with pacman since it frees the db lock for you to be able to do other things.

Awebb · 2016-08-22 19:29:56

graysky wrote:

Edges are pretty smooth now. I don't see why we or formal upstream wouldn't be interested. Frankly, on a fast desktop, it isn't needed since the rebuild is 5-10 sec... but for my arm devices, it is pretty necessary

I very much appreciate this tool on my Atom based home server :-)

Is there already a mailing list discussion or a bug report we can chime into?

graysky · 2016-08-22 19:49:54

I haven't opened a flyspray yet for our package nor have I upstream (if they would even have an interest)... please feel free to start with a flyspray with our system. If you wish to submit one.

Last edited by graysky (2016-08-22 19:52:31)

Awebb · 2016-08-22 19:52:26

graysky wrote:

I haven't opened a flyspray yet for our package nor have I upstream (if they would even have an interest).

I took a look at both, upstream and our package. It looks like the hook is added to the package downstream. I'd say such distribution specific changes are not very popular upstream.

graysky · 2016-08-22 22:22:58

Just realized upstream won't care since it's the pacman hook that sets-up the thing... probably best to send in to our package.

graysky · 2016-08-26 11:14:12

https://bugs.archlinux.org/task/50523

aude · 2016-09-10 22:45:32

This was really nagging me as well, the slow update post-install. Especially on my Raspberry Pi.

I made a different version, one that only updates the updated man pages: https://gist.github.com/aude/95b6402a01 … b4105c4e33

It is much faster. Care to test it?

graysky · 2016-09-11 01:14:48

There is no need now as Arch upstream has reverted to the timer.

aude · 2016-09-11 08:35:28

Yes, the hook has been removed. Though mandb could be run on demand using the --filename parameter as done in the gist I posted.
I will ask upstream if they want this way to do on-demand.

Awebb · 2016-09-11 08:36:26

aude wrote:

Yes, the hook has been removed. Though mandb could be run on demand using the --filename parameter as done in the gist I posted.
I will ask upstream if they want this way to do on-demand.

Upstream meaning mandb upstream or the Arch package?

aude · 2016-09-11 09:37:50

Awebb wrote:

Upstream meaning mandb upstream or the Arch package?

The Arch package. I'll stop using the word upstream now and start using Arch.

Last edited by aude (2016-09-11 10:03:33)

Arch Linux

#1 2016-07-23 15:44:50

mandb-ondemand: speed-up pacman updates to the manpage database

#2 2016-07-23 15:48:03

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#3 2016-07-23 15:55:35

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#4 2016-07-23 16:39:25

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#5 2016-07-24 01:31:29

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#6 2016-07-24 10:12:27

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#7 2016-08-22 07:17:12

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#8 2016-08-22 09:56:51

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#9 2016-08-22 10:47:05

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#10 2016-08-22 11:46:07

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#11 2016-08-22 13:29:26

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#12 2016-08-22 19:29:56

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#13 2016-08-22 19:49:54

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#14 2016-08-22 19:52:26

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#15 2016-08-22 22:22:58

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#16 2016-08-26 11:14:12

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#17 2016-09-10 22:45:32

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#18 2016-09-11 01:14:48

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#19 2016-09-11 08:35:28

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#20 2016-09-11 08:36:26

Re: mandb-ondemand: speed-up pacman updates to the manpage database

#21 2016-09-11 09:37:50

Re: mandb-ondemand: speed-up pacman updates to the manpage database

Board footer