[solved bug]man pages db sometimes zapped (if man-db.timer on battery?

cfr · 2022-03-12 02:11:49

I'm using the systemd timer to update manual pages daily.

/etc/systemd/system/timers.target.wants/man-db.timer

is just a symlink to the standard timer

lrwxrwxrwx 1 root root 36 Chw  10  2019 /etc/systemd/system/timers.target.wants/man-db.timer -> /usr/lib/systemd/system/man-db.timer

This had been working, as far as I know, perfectly. Twice in the last couple of weeks or so, however, something has gone wrong even though systemctl doesn't seem aware of it. That is, systemctl --failed yields no units. systemctl list-timers produces

NEXT                        LEFT           LAST                        PASSED     UNIT                         ACTIVATES                     
Sat 2022-03-12 01:02:15 GMT 5min left      Sat 2022-03-12 00:00:47 GMT 55min ago  fangfrisch.timer             fangfrisch.service
Sat 2022-03-12 05:48:30 GMT 4h 52min left  Fri 2022-03-11 13:02:55 GMT 11h ago    updatedb.timer               updatedb.service
Sat 2022-03-12 08:24:11 GMT 7h left        Fri 2022-03-11 13:02:55 GMT 11h ago    man-db.timer                 man-db.service
Sat 2022-03-12 15:01:19 GMT 14h left       Fri 2022-03-11 03:08:24 GMT 21h ago    etckeeper.timer              etckeeper.service
Sat 2022-03-12 15:01:19 GMT 14h left       Fri 2022-03-11 03:08:24 GMT 21h ago    systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sun 2022-03-13 00:00:00 GMT 23h left       Sat 2022-03-12 00:00:02 GMT 56min ago  logrotate.timer              logrotate.service
Sun 2022-03-13 00:00:00 GMT 23h left       Sat 2022-03-12 00:00:02 GMT 56min ago  pkgfile-update.timer         pkgfile-update.service
Sun 2022-03-13 00:00:00 GMT 23h left       Sat 2022-03-12 00:00:02 GMT 56min ago  shadow.timer                 shadow.service
Mon 2022-03-14 00:00:00 GMT 1 day 23h left Mon 2022-03-07 00:00:28 GMT 5 days ago paccache.timer               paccache.service
Mon 2022-03-14 01:38:53 GMT 2 days left    Mon 2022-03-07 00:12:44 GMT 5 days ago fstrim.timer                 fstrim.service

10 timers listed.
Pass --all to see loaded but inactive timers, too.

But apropos returns no results for anything I've tried including, for example, sysctl and bash.

The first time this happened, I think I triggered the timer immediately, though I'm not positive about that. Certainly, the index was easily restored.
However, now it has happened again. Since I was on battery this time, I didn't initially try triggering the timer as it is AC only.

Running /usr/bin/mandb manually restores the index of manual pages, as expected, but the output is not entirely expected (to me).

/usr/bin/mandb: can't update index cache /var/cache/man/172611: No such file or directory
Updating index cache for path `/usr/share/man/man8'. Wait.../usr/bin/mandb: warning: /usr/share/man/man8/swapoff.8.gz: whatis parse for swapoff(8) failed
/usr/bin/mandb: warning: /usr/share/man/man8/vigr.8.gz: whatis parse for vigr(8) failed
Updating index cache for path `/usr/share/man/man1'. Wait.../usr/bin/mandb: warning: /usr/share/man/man1/lastb.1.gz: whatis parse for lastb(1) failed
/usr/bin/mandb: warning: /usr/share/man/man1/escp2topbm.1.gz: whatis parse for escp2topbm(1) failed
/usr/bin/mandb: warning: /usr/share/man/man1/pamfixtrunc.1.gz: whatis parse for pamfixtrunc(1) failed
Updating index cache for path `/usr/share/man/man3'. Wait.../usr/bin/mandb: warning: /usr/share/man/man3/uuid_generate_random.3.gz: whatis parse for uuid_generate_random(3) failed
/usr/bin/mandb: warning: /usr/share/man/man3/uuid_generate_time.3.gz: whatis parse for uuid_generate_time(3) failed
/usr/bin/mandb: warning: /usr/share/man/man3/uuid_generate_time_safe.3.gz: whatis parse for uuid_generate_time_safe(3) failed
/usr/bin/mandb: warning: /usr/share/man/man3/TimeDate.3pm.gz: whatis parse for TimeDate(3pm) failed
...

Many similar warnings then follow concerning whatis parse failures for other manual pages.

...
Processing manual pages under /usr/local/man...
/usr/bin/mandb: can't update index cache /var/cache/man/oldlocal/172611: No such file or directory
Purging old database entries in /usr/local/texlive/current/texmf-dist/doc/man/...
Processing manual pages under /usr/local/texlive/current/texmf-dist/doc/man/...
99 man subdirectories contained newer manual pages.
25884 manual pages were added.
0 stray cats were added.
0 old database entries were purged.

The journal includes

Maw 10 00:13:03 XXX systemd[1]: Starting Daily man-db regeneration...
Maw 10 00:13:04 XXX systemd[1]: man-db.service: Deactivated successfully.
Maw 10 00:13:04 XXX systemd[1]: Finished Daily man-db regeneration.
...
Maw 11 13:02:55 XXX systemd[1]: Daily man-db regeneration was skipped because of a failed condition check (ConditionACPower=true).

The first 3 lines are for Thursday, when I was on AC.
The last line is for Friday, when I was on battery.
Is it possible that rather than doing nothing at all on battery, the unit is deleting the index (or some crucial thing accessing the index requires) but not regenerating it?

systemctl start man-db.service

As expected, the condition fails as I'm on battery

○ man-db.service - Daily man-db regeneration
     Loaded: loaded (/usr/lib/systemd/system/man-db.service; static)
     Active: inactive (dead) since Thu 2022-03-10 00:13:04 GMT; 2 days ago
TriggeredBy: ● man-db.timer
  Condition: start condition failed at Sat 2022-03-12 01:47:22 GMT; 1min 0s ago
             └─ ConditionACPower=true was not met
       Docs: man:mandb(8)
    Process: 144138 ExecStart=/usr/bin/install -d -o root -g root -m 0755 /var/cache/man (code=exited, status=0/SUCCESS)
    Process: 144139 ExecStart=/usr/bin/find /var/cache/man -type f -name *.gz -atime +6 -delete (code=exited, status=0/SUCCESS)
    Process: 144140 ExecStart=/usr/bin/mandb --quiet (code=exited, status=0/SUCCESS)
   Main PID: 144140 (code=exited, status=0/SUCCESS)
        CPU: 616ms

Maw 10 00:13:03 XXX systemd[1]: Starting Daily man-db regeneration...
Maw 10 00:13:04 XXX systemd[1]: man-db.service: Deactivated successfully.
Maw 10 00:13:04 XXX systemd[1]: Finished Daily man-db regeneration.
Maw 11 13:02:55 XXX systemd[1]: Daily man-db regeneration was skipped because of a failed condition check (ConditionACPower=true).
Maw 12 01:47:22 XXX systemd[1]: Daily man-db regeneration was skipped because of a failed condition check (ConditionACPower=true).

And the journal shows the same thing as before.

systemd[1]: Daily man-db regeneration was skipped because of a failed condition check (ConditionACPower=true).

But, this time, apropos bash shows no damage was done. The index wasn't updated, but the old index wasn't destroyed or rendered inaccessible either.

Is there something different about triggering a .service manually and using a timer which could explain the difference? Or should I be looking elsewhere entirely? Obviously something else could be periodically destroying or corrupting the index or whatever, but, right now, I've no idea what.

Last edited by cfr (2024-04-01 14:55:15)

cfr · 2022-10-29 01:48:38

I marked this as resolved, but have discovered it is not.

If man-db.service is triggered by the timer while on battery, the journal says the job is skipped, but the database is entirely or partly erased. In this case, the only man pages remaining were in /usr/local/. The journal has

systemd[1]: Daily man-db regeneration was skipped because of a failed condition check (ConditionACPower=true).

which is fine, but

apropos bash
apropos ls
apropos systemd

return no relevant results. (Unless there happens to be a match in TeX Live's manual pages, which are under /usr/local in my case.)

Starting the service on AC resolves the issue, of course, but why are the cached results even touched when the service is triggered by the timer on battery power? I'm perfectly happy for the service not to run, but I don't want it destroying the existing database.

Tblue · 2024-03-05 15:42:39

I have the same issue, and it's slightly annoying.

I think what's happening is:

As you discovered, "man-db.service" (triggered by "man-db.timer") has "ConditionACPower=true", so it only runs on AC power.
Presumably, this means that on our systems, the "man-db.service" unit only runs very rarely since the system usually happens to be on battery power when the timer triggers.
"mandb" stores its DB in /var/cache/man/, and files older than 1 week in that directory are periodically deleted by "systemd-tmpfiles" (see /usr/lib/tmpfiles.d/man-db.conf).

So, we're probably just unlucky, and "mandb" is run less than once a week on our systems. Thus, "systemd-tmpfiles" goes and cleans up the "old" files in /var/cache/man/, and there we go.

For now, I told "systemd-tmpfiles" to delete the files in /var/cache/man/ only after a month by creating the following file:

# /etc/tmpfiles.d/man-db.conf
#
# Default age is 1 week, but man-db.service runs only on AC power; thus, the DB
# is sometimes completely erased by systemd-tmpfiles because mandb happens to
# run less than once a week.
#
# Let's try 4 weeks instead:
d /var/cache/man 0755 root root 4w

That might help since mandb seems to at least run successfully once per month:

$ sudo journalctl -u man-db|grep Finished
Jan 10 02:52:16 gumshoe systemd[1]: Finished Daily man-db regeneration.
Jan 31 00:18:47 gumshoe systemd[1]: Finished Daily man-db regeneration.
Feb 08 02:08:51 gumshoe systemd[1]: Finished Daily man-db regeneration.
Feb 22 11:10:47 gumshoe systemd[1]: Finished Daily man-db regeneration.
Feb 23 20:44:14 gumshoe systemd[1]: Finished Daily man-db regeneration.
Feb 24 03:30:07 gumshoe systemd[1]: Finished Daily man-db regeneration.

If that doesn't help, I might try to just allow "man-db.service" to run on battery power.

cfr · 2024-03-29 06:59:39

Tblue wrote:

I have the same issue, and it's slightly annoying.
I think what's happening is:
As you discovered, "man-db.service" (triggered by "man-db.timer") has "ConditionACPower=true", so it only runs on AC power.
Presumably, this means that on our systems, the "man-db.service" unit only runs very rarely since the system usually happens to be on battery power when the timer triggers.
"mandb" stores its DB in /var/cache/man/, and files older than 1 week in that directory are periodically deleted by "systemd-tmpfiles" (see /usr/lib/tmpfiles.d/man-db.conf).
So, we're probably just unlucky, and "mandb" is run less than once a week on our systems. Thus, "systemd-tmpfiles" goes and cleans up the "old" files in /var/cache/man/, and there we go.

Ahhhh! That makes perfect sense.

But do we know **why** `systemd-tmpfiles` is configured to delete the cache if it is more than a week old? Under what circumstances is this in any way desirable? The update to the manual pages doesn't depend on prior deletion so I don't see what purpose the deletion serves.

seth · 2024-03-29 09:30:57

It makes even less sense as the service already has/d

# Expunge old catman pages which have not been read in a week.
ExecStart=/usr/bin/find /var/cache/man -type f -name *.gz -atime +6 -delete

and mandb defaults to purge and the service doesn't use "-p" leading to https://gitlab.com/man-db/man-db/-/comm … ab7c5e9b55

Something's Denmark in the state of mandb here…

Tblue · 2024-03-29 10:44:33

cfr wrote:

But do we know **why** `systemd-tmpfiles` is configured to delete the cache if it is more than a week old? Under what circumstances is this in any way desirable? The update to the manual pages doesn't depend on prior deletion so I don't see what purpose the deletion serves.

Yeah, I also asked myself that, especially since some of the mandb options suggest that a database can be purged of invalid entries etc. after initially being created.

I don't have an answer, but the systemd-tmpfiles configuration snippet comes from the upstream project, and is not an Arch-specific addition.

I created https://gitlab.com/man-db/man-db/-/issues/36 to try and shed some light on this behavior.

(To me it even looks like systemd-tmpfiles might corrupt the DB when it goes and deletes random files from it, but I'm no expert.)

Tblue · 2024-03-31 19:13:22

Good news, man-db 2.12.1 will fix this issue by reducing the scope of the cleanup to cat pages only (which apparently was the original intent). So, the manual database itself should stay intact in the future, if I understand correctly (and I can remove my workaround from my previous post).

Tblue · 2024-04-15 17:58:17

man-db 2.12.1 is in the repos now and should fix this issue.

Arch Linux

#1 2022-03-12 02:11:49

[solved bug]man pages db sometimes zapped (if man-db.timer on battery?

#2 2022-10-29 01:48:38

Re: [solved bug]man pages db sometimes zapped (if man-db.timer on battery?

#3 2024-03-05 15:42:39

Re: [solved bug]man pages db sometimes zapped (if man-db.timer on battery?

#4 2024-03-29 06:59:39

Re: [solved bug]man pages db sometimes zapped (if man-db.timer on battery?

#5 2024-03-29 09:30:57

Re: [solved bug]man pages db sometimes zapped (if man-db.timer on battery?

#6 2024-03-29 10:44:33

Re: [solved bug]man pages db sometimes zapped (if man-db.timer on battery?

#7 2024-03-31 19:13:22

Re: [solved bug]man pages db sometimes zapped (if man-db.timer on battery?

#8 2024-04-15 17:58:17

Re: [solved bug]man pages db sometimes zapped (if man-db.timer on battery?

Board footer