You are not logged in.

#1 2016-07-23 15:44:50

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

mandb-ondemand: speed-up pacman updates to the manpage database

Purpose
With the deployment of pacman hooks, maintenance of the manpage index database (/var/cache/man/index.db) has been integrated into pacman operations. Anytime a package writes or removes files to /usr/share/man, a hook triggers the rebuild of the index database which can be slow -- really slow. This is particularly noticeable on low powered devices such as Arch ARM PCs. The delay can be annoying as the terminal (more likely the ssh window) remains locked while the rebuild is in progress.

Solution
mandb-ondemand flags the index database for a rebuild via a pacman hook that triggers a systemd unit to do the dirty work while pacman happily finishes thus leaving the shell free of the dreaded pause!

Without mandb-ondemand:

root@box # pacman -S foo bar
...
:: Running post-transaction hooks...
(1/1) Updating manpage index...
 <<<WAIT>>>    <<<WAIT>>>    <<<WAIT>>>
root@box #

With mandb-ondemand:

root@box # pacman -S foo bar
...
:: Running post-transaction hooks...
(1/1) Asking for an update to mandb's database...
root@box #

Links
AUR package: https://aur.archlinux.org/packages/mandb-ondemand
Upsteam code: https://github.com/graysky2/mandb-ondemand

What it supplies
It's just a few symlinks, a hook, and 2 systemd units.  Tiny and simple (less than 16 K):

% pacman -Ql mandb-ondemand
mandb-ondemand /etc/pacman.d/hooks/man-db.hook
mandb-ondemand /usr/lib/systemd/system/mandb-ondemand.path
mandb-ondemand /usr/lib/systemd/system/mandb-ondemand.service
mandb-ondemand /usr/lib/systemd/system/multi-user.target.wants/mandb-ondemand.path
mandb-ondemand /usr/share/libalpm/hooks/mandb-ondemand.hook
mandb-ondemand /usr/share/licenses/mandb-ondemand/LICENSE

Last edited by graysky (2016-07-24 10:25:10)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#2 2016-07-23 15:48:03

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database

I should mention that I currently don't know of a way to keep a race condition between two mandb's if the user does multiple, rapid calls to pacman such that call #1 triggers the rebuild and call #2 also triggers a rebuild.  I can do this with an intermediate script but that is not very elegant.  Does anyone have a suggestion to prevent another mandb from running while one is active using the service units?

Detail: this provides two pairs of service/path units: one runs on an install/upgrade and the other runs on a remove.

The race condition only happens if you rapidly install then remove a package.  On slow devices (like RPi for example), the first rebuild can take 20-30 seconds so it's possible that the user can:
pacman -S foo ... wait a sec and decide foo isn't needed and then pacman -Rs foo and thus have both running at the same time.

I guess one solution is to just have one pair running mandb --quiet on install/upgrade/remove rather than what I current have which is 1 running mandb --quiet --no-purge (on installs/upgrade) and another running mandb --quiet (on removes).

EDIT:  <<headsmack>> I see how that I in fact only need one pair since Arch upstream is supplying a timer that runs the purge.  I will make this modification in version 2.02 and release shortly.

Last edited by graysky (2016-07-23 16:25:32)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#3 2016-07-23 15:55:35

Spider.007
Member
Registered: 2004-06-20
Posts: 1,175

Re: mandb-ondemand: speed-up pacman updates to the manpage database

Awesome; the pacman-triggers in general are a nice feature but the mandb trigger is a pain.

Shouldn't systemd already prevent the concurrency issue? I'm pretty sure you can't start the same units multiple times (while the first one is still running)

Offline

#4 2016-07-23 16:39:25

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database

@Spider - Yes, I edited my #2 post above explaining things.  In any case, no race condition and a more simplistic package in version 2.02 which is now live in the AUR.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#5 2016-07-24 01:31:29

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: mandb-ondemand: speed-up pacman updates to the manpage database

Can you elaborate why you have gzip as an optional dependency? I'm probably missing something but I don't see anything that directly depends on gzip or might make use of it if installed.


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#6 2016-07-24 10:12:27

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database

R00KIE wrote:

Can you elaborate why you have gzip as an optional dependency? I'm probably missing something but I don't see anything that directly depends on gzip or might make use of it if installed.

My bad... I believe that initially I copied the PKGBUILD from man-db rather than /usr/share/pacman/PKGBUILD.proto and just forgot to delete that line.  Fixed in 2.02-3.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#7 2016-08-22 07:17:12

maggie
Member
Registered: 2011-02-12
Posts: 255

Re: mandb-ondemand: speed-up pacman updates to the manpage database

Thank you for this upgrade. Why doesn't the official package do this?

Offline

#8 2016-08-22 09:56:51

Awebb
Member
Registered: 2010-05-06
Posts: 6,272

Re: mandb-ondemand: speed-up pacman updates to the manpage database

maggie wrote:

Thank you for this upgrade. Why doesn't the official package do this?

graysky still seems to be smoothing out the edges. When it's idiot proof, it's probably only a bug report and a short discussion away from the default.

Offline

#9 2016-08-22 10:47:05

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: mandb-ondemand: speed-up pacman updates to the manpage database

The only thing I've noticed that _might_ go wrong is if you reboot before mandb has finished doing its business, but even then it wouldn't be something that would make the system unstable or unbootable.


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#10 2016-08-22 11:46:07

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database

Edges are pretty smooth now.  I don't see why we or formal upstream wouldn't be interested.  Frankly, on a fast desktop, it isn't needed since the rebuild is 5-10 sec... but for my arm devices, it is pretty necessary smile


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#11 2016-08-22 13:29:26

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: mandb-ondemand: speed-up pacman updates to the manpage database

At least here with an i7-6500U big rebuilds still take a good while (not as long as on arm obviously). Either way anything that can be done in the background is nice, specially with pacman since it frees the db lock for you to be able to do other things.


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#12 2016-08-22 19:29:56

Awebb
Member
Registered: 2010-05-06
Posts: 6,272

Re: mandb-ondemand: speed-up pacman updates to the manpage database

graysky wrote:

Edges are pretty smooth now.  I don't see why we or formal upstream wouldn't be interested.  Frankly, on a fast desktop, it isn't needed since the rebuild is 5-10 sec... but for my arm devices, it is pretty necessary smile

I very much appreciate this tool on my Atom based home server :-)

Is there already a mailing list discussion or a bug report we can chime into?

Offline

#13 2016-08-22 19:49:54

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database

I haven't opened a flyspray yet for our package nor have I upstream (if they would even have an interest)... please feel free to start with a flyspray with our system.  If you wish to submit one.

Last edited by graysky (2016-08-22 19:52:31)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#14 2016-08-22 19:52:26

Awebb
Member
Registered: 2010-05-06
Posts: 6,272

Re: mandb-ondemand: speed-up pacman updates to the manpage database

graysky wrote:

I haven't opened a flyspray yet for our package nor have I upstream (if they would even have an interest).

I took a look at both, upstream and our package. It looks like the hook is added to the package downstream. I'd say such distribution specific changes are not very popular upstream.

Offline

#15 2016-08-22 22:22:58

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database

Just realized upstream won't care since it's the pacman hook that sets-up the thing... probably best to send in to our package.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#16 2016-08-26 11:14:12

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#17 2016-09-10 22:45:32

aude
Member
Registered: 2016-09-10
Posts: 3
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database

This was really nagging me as well, the slow update post-install. Especially on my Raspberry Pi.

I made a different version, one that only updates the updated man pages: https://gist.github.com/aude/95b6402a01 … b4105c4e33

It is much faster. Care to test it?

Offline

#18 2016-09-11 01:14:48

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database

There is no need now as Arch upstream has reverted to the timer.


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#19 2016-09-11 08:35:28

aude
Member
Registered: 2016-09-10
Posts: 3
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database

Yes, the hook has been removed. Though mandb could be run on demand using the --filename parameter as done in the gist I posted.
I will ask upstream if they want this way to do on-demand.

Offline

#20 2016-09-11 08:36:26

Awebb
Member
Registered: 2010-05-06
Posts: 6,272

Re: mandb-ondemand: speed-up pacman updates to the manpage database

aude wrote:

Yes, the hook has been removed. Though mandb could be run on demand using the --filename parameter as done in the gist I posted.
I will ask upstream if they want this way to do on-demand.

Upstream meaning mandb upstream or the Arch package?

Offline

#21 2016-09-11 09:37:50

aude
Member
Registered: 2016-09-10
Posts: 3
Website

Re: mandb-ondemand: speed-up pacman updates to the manpage database

Awebb wrote:

Upstream meaning mandb upstream or the Arch package?

The Arch package. I'll stop using the word upstream now and start using Arch.

Last edited by aude (2016-09-11 10:03:33)

Offline

Board footer

Powered by FluxBB