any standard utility to grep the rendered text of manpages?

s7hoang · 2022-01-24 04:43:04

question

Is there a good alternative to man -K that searches the final plain text?

I'm looking for something like man -K but for the output. Or even better,
recursively searching from a starting page and following all the links within.

example:

man groff_man:

...
DESCRIPTION
The man macro package ...
...

So I want to look for "man macro package"; not an unreasonable search query I
think.
Both apropos "man macro package" and man -K "man macro package" turn up empty.

man man has this to say about -K:

Note that this searches the sources of the manual pages, not the rendered text, and so
may include false positives due to things like comments in source files. Searching the
rendered text would be much slower.

so it seems like correct behaviour albeit a false negative in this case.

That's because the actual text looks like this:

_/usr/share/man/man7/groff_man.7_:

.\" ====================================================================
.SH DESCRIPTION
.\" ====================================================================
.
The
.I man
macro package for

so the line is split across two lines and so a quick grep doesn't quite do it either.
You have to either zcat | groff -man -Tutf8 | col -b | grep or man | grep
will do as well. The main alternative I'm seeing in general is to search the web version instead.

Given all the people shouting RTFM, is there really not a
standard utility to search the actual outputted plain text of manpages?

jasonwryan · 2022-01-24 04:45:53

man groff_man | less -p "man macro package"

seth · 2022-01-24 08:16:58

Errr… I might be missing the point, but the man pager has a search function - hit "/" enter the interesting token and hit enter.
Subsequent "/" searches will default to the previous one and there's readline support so you can use up/down to navigate the search history.

Raynman · 2022-01-24 10:36:16

I think the point (for this example) is to find "groff_man" given only "man macro package". I have never stumbled upon something like that (and not needed it often enough to find/build it). A web search (perhaps with site:man.archlinux.org) could work.

seth · 2022-01-24 13:38:56

Since the manpage content isn't indexed anywhere that would be the most efficient way, but required internet access.
Rendering and grepping them would be almost silly, but one could of course

for page in /usr/share/man/**/*; do man $page | grep $token && printf "$page\n--------------------\n"; done

Trilby · 2022-01-24 13:57:40

Piping man to grep will regularly fail. As will `less -p "$search-string"`. These both fail* for my for the original example provided in this thread as there are extra spaces between words in the groff formatted output.

The real hurdle (at least for anyone who is not a groff wizard) is to get each paragraph on a single line rather than wrapped. With "default" settings for man page, groff will full-justify the text which includes hyphens, line breaks, and extra spaces which would cause effectively 'random' failures in matching search terms.

The other alternative would be to preprocess the search string by tokenizing it and allowing for optional excess whitespace characters between tokens (while disabling hyphenation in groff which is easy enough).

If one were to actually untertake such a task, I suspect you'd not want to go through all of these steps for every search, but rather use xapian (or a similar search / indexer) to preprocess all the man pages in the above manner and creating a search index which would then be used for individual searches.

*note: if you test these approaches, they may appear to work, because there is a second incidence of the same phrase in the groff_man man page, but the first one is missed.

Last edited by Trilby (2022-01-24 14:06:46)

Slithery · 2022-01-24 14:11:49

I'm not exactly sure how https://man.archlinux.org/ parses the raw data but it seems to provide a match for the description...
https://man.archlinux.org/search?q=man+macro+package

Edit - It appears that they use mandoc...

archmanweb readme wrote:

We used a dynamic approach instead of building a website consisting of completely static pages. The main building blocks are the Django web framework, the PostgreSQL database server, the mandoc tool from the mandoc toolset for the conversion to HTML and the pyalpm library for data extraction from the Arch repositories. The code is available in the archmanweb repository at GitHub.
Overall, this approach allows us to provide the following features without rebuilding the whole website from scratch:
Listings with custom filters and orderings.
Links to other versions of the same manual provided by different packages.
Links to similar manuals available in other sections or languages.
Searching in the names and descriptions of packages and manuals, similarly to apropos(1).

Note the last sentence above.

https://gitlab.archlinux.org/archlinux/archmanweb

Trilby · 2022-01-24 14:58:59

That search for all the words in the search input, but not the phrase. E.g., searching for "man macro package" is the same as searching for "man package macro". Comparable results can be acheived with apropos:

$ apropos -ea man macro package
groff_man (7)        - GNU roff macro package for formatting man pages

And to illustrate the challenge resulting from formating characters between words (which is why matching all words is easy, but matching a phrase is hard):

$ apropos -ea man "macro package"
groff_man (7)        - GNU roff macro package for formatting man pages

$ apropos -ea "man macro package"
man macro package: nothing appropriate.

EDIT: OOPS, this is all a coincidence as these words are all in the title / summary. Niether the web interface nor the apropos use above does a full text search. The same groff_man man page includes the word "thematically", but if you put that in the web interface search, you do *not* get the groff_man page listed as a result.

Last edited by Trilby (2022-01-24 15:05:33)

Arch Linux

#1 2022-01-24 04:43:04