Making arch-wiki-docs useful

keenerd · 2011-02-28 13:57:58

While we can read the wiki offline, it is not terribly good. There is no search, no layout and no good way to use it from a terminal. See for yourself.
pacman -S arch-wiki-docs
and play with it for a bit. It is awkward if you don't already know what you are looking for.

"But I can just grep over the html to search!" you say.

Meh. Try it. You'll get a lot of hits. You'll miss many due to inline html formatting and silly unicode escapes. The hits need to be sorted by count. You'll want to filter by your language of choice. You really need a case intensive search. Case insensitive grep is about 15 times slower than normal grep. After all this, you still need a program to view html.

Here is where wiki-search comes in.
1) Case insensitive search that is 15x faster than grep -i
2) Convenient UI
3) Gives top ten results by count
4) Language filtering (and language summary)
5) Built in viewer with match highlighting
6) Depends only on core utilities

Install with pacman -S arch-wiki-lite

After some cleanup and testing, I would like to get this script bundled on the install CD. At only 10MB uncompressed, it seems a worthwhile addition.

I'll take general usability requests. If anything seems awkward or difficult, it is a bug and I'd like to fix it!

Also, my bash still sucks.

edit: Example use case

$ wiki-search suspend
0  Suspending_to_Disk_with_hibernate-script  00002479.html
1  Suspending_to_RAM_with_hibernate-script   00004319.html
2  Pm-utils                                  00003964.html
3  Tuxonice                                  00006750.html
[continues]

$ wiki-search 1
[page pops up in less or $PAGER]

$ wiki-search --lang
    210 简体中文
    182 Русский
    177 Italiano
     84 Česky
     80 Türkçe
     68 Français
[continues]

$ export wiki_lang="Italiano"
$ wiki-search xorg
0  Xorg_(Italiano)                    00003441.html
1  ATI_Catalyst_(Italiano)            00009340.html
2  Beginners'_Guide_(Italiano)        00003672.html
[continues]

Last edited by keenerd (2011-08-04 13:51:25)

milomouse · 2011-02-28 14:41:38

first off; thank you for this. i've always wanted this kind of functionality as i use the arch-wiki-docs quite often when i'm offline or prefer Arch advice over random google advice. although i do have a few questions/suggestions/opinions:

1) i noticed the built in viewer is my $PAGER (which is `most'). is there anyway to have a --browser option so i can use `w3m' or `luakit' instead or should i just pipe it those applications? i know i can use `w3m' as my pager but prefer `most' for those tasks. because as it stands my pager can't select links and jump to those sections. (e.g. 0000xxxx.html#sectionname)

2) also, you need to type in the `wiki-search 0000xxxx.html' or `wiki-search xxxx' to view that page, correct? not really a big deal but creates a slight delay on my part (as the user) when there's a lot of results and numbers above and below each other to distinguish from (maybe it's just me). perhaps incorporate an interactive feature where there's a prompt at the end of the results reading user input in the form of an integer, where you can view by result #. e.g. 12 results 1-12, type in 3 at the prompt to view result 3 or something. when i think about it it seems like adding code complexity for something trivial so feel free to discard this (or a derived form of this) as unnecessary. just brainstorming as some people might be dyslexic. might be useful to show desired example:

$ wiki-search --interactive/-i lisp
1. Lisp_Package_Guidelines               00004750.html
2. Stumpwm                               00004391.html
3. Emacs                                 00003611.html
4. Slime                                 00003390.html
5. Kiloseconds                           00006950.html
6. Sawfish                               00006887.html
7. Arch_is_the_best                      00004833.html
8. Org-mode_手册                         00009044.html
9. Arch_Packaging_Standards              00001210.html
10. Comparison_of_Tiling_Window_Managers  00006139.html
> Enter result number to view: 7

.. 7 brings up $PAGER/browser with 00004833.html (Arch_is_the_best)

again, if this will not be incorporated i'll just edit my own version instead of having it where other people may not want it, no big deal. it shouldn't add much.

thanks again :}

Last edited by milomouse (2011-02-28 14:56:46)

keenerd · 2011-02-28 17:04:30

Using the browser as a pager won't work. The output is stripped of all tags and escapes, there is no html left. I do like the menu, will see what can be done with that. In the mean time, you can get around both issues with

echo "<html><body>"
wiki-search $* | awk '{print "<br><a href=\"/usr/share/doc/arch-wiki/html/"$2"\">"$1"</a>"}'
echo "</body></html>"

and sending that into your browser.

Last edited by keenerd (2011-02-28 17:06:53)

sironitomas · 2011-03-01 00:10:09

Did you know there is a index.html in the folder containing all the docs?

karol · 2011-03-01 00:47:48

sironitomas wrote:

Did you know there is a index.html in the folder containing all the docs?

Sure, but the index contains (by definition) only the titles. This script searches the whole wiki, so it will find occurrences of the phrase even if it's not in the page title.
<'m sleepy so somebody correct me if I'm wrong. Nighty night, Archers.>

Edit: Oh, and OP wanted a menu with only the pages that came in the search, not all pages the are in the wiki.

Last edited by karol (2011-03-01 00:50:39)

keenerd · 2011-03-01 16:39:56

milomouse: Html browser support added. Export $wiki_browser beforehand. Still playing with the menu thing.

milomouse · 2011-03-01 17:40:25

works like a charm. much appreciated, keenerd.
as far as the menu goes; not that you might need the help, but i'll look at adding to the code on my end and see what comes up, as it may be different then your method. it will have to be later tonight when i do it, though. regardless, nice work so far. thanks again.

keenerd · 2011-03-02 01:01:25

I wanted to find a balance between keeping the user trapped in a looping menu and the freedom of sitting at a shell prompt. Here's what came out.

$ wiki-search mouse
0  Get_All_Mouse_Buttons_Working         00001539.html
1  Bluetooth_Mouse                       00002596.html
2  Bluetooth_mouse_manual_configuration  00009695.html
3  Console_Mouse_Support                 00003406.html
[continues]

$ wiki-search 3
[views page]

$ !!:0 3
[also views page]

The old methods of choosing a page are supported as well. Your thoughts?

karol · 2011-03-02 01:08:13

Both https://bbs.archlinux.org/viewtopic.php … 27#p897527 and 'wiki-search 3' are fine. Thanks :-)

keenerd · 2011-03-02 12:58:48

Updated with a bit of a speed tweak. Really large docs (russian beginners guide) took several minutes to render. Now just several seconds!

kgas · 2011-03-02 13:30:47

keenerd, thanks for the script. I did not go thro ' it fully but one point if the user did not install the wiki after the message it should exit with an error code or will fail in the next line.

Last edited by kgas (2011-03-02 13:32:09)

keenerd · 2011-03-02 13:47:33

Fixed.

kgas · 2011-03-02 14:14:54

keenerd, it is not return 1, it should be exit <some number>

milomouse · 2011-03-02 22:13:15

sorry, i got sidetracked due to my brother's laptop being zapped by lightning. still working on that issue.. but i did get a chance to look at your updates and from a glance it seems fine using a tmp file instead of keeping user in a loop. but i guess this is all(?) temporary if you're going to do it in C.

keenerd · 2011-03-05 15:36:41

Complete overhaul. Check out arch-wiki-lite in the AUR. It is a replacement for arch-wiki-docs.

The main selling point is the size, only 16MB to the 83MB of arch-wiki-docs. And the built in search and built in viewer, of course.

Last edited by keenerd (2011-03-05 15:53:08)

skodabenz · 2011-03-05 15:51:21

How about using a proper wiki mirroring software? http://openzim.org/Main_Page as mentioned at https://bbs.archlinux.org/viewtopic.php … 63#p881263.

Last edited by skodabenz (2011-03-05 15:53:26)

karol · 2011-03-05 16:11:08

keenerd wrote:

Complete overhaul. Check out arch-wiki-lite in the AUR. It is a replacement for arch-wiki-docs.
The main selling point is the size, only 16MB to the 83MB of arch-wiki-docs. And the built in search and built in viewer, of course.

So you can't click on e.g. 'related' topics because it's plaintext and not html, right? I have to perform extra searches.

milomouse · 2011-03-05 19:01:35

@karol: i'm guessing this lite version is geared more towards being integrated into Arch Boot/install images (trying to slim down, simplify, etc) rather than the other extrinsic wiki-search script which is possibly more suited for the desktop environment relying on html and the original larger arch-wiki-docs. i think that was his original intention, but i could be wrong. not sure if both are going to be maintained or just the lite version. either way, i'm going to continue to use the html version for my own purposes (e.g. clicking links, html formatting, etc). thanks again, keenerd

keenerd · 2011-03-05 19:43:55

skodabenz: Somehow I completely missed that thread. Thanks.

milomouse: Spot on about intentions. For now I'll maintain both. They share a lot of code in common. The only real difference is the light version pre renders (and compresses) all of the documents during packaging.

What do you think of bundling the original html based app in with arch-wiki-lite? Assuming desktop users don't mind the extra 16 MB of redundancy . I imagine you'd just delete it or something.

milomouse · 2011-03-05 20:30:23

@keenerd:
i personally wouldn't mind, that way it would be easier for me to follow if the html version never made it to the AUR on it's own.
maybe make arch-wiki-docs an optdepends too (as well as makedepends, which it already is) for html-version of script to work. optdepends=('arch-wiki-docs: for wiki-search-html to work') or something. because some programs may remove arch-wiki-docs are installing since it's just a makedepend. this way user would know to keep it if they wanted html version too. because i know if it shipped with Arch install it would already be compiled for the user without html so there's no problem there. i don't mind the extra gruff. i may need just the text versions sometimes so i'm all for it.

Last edited by milomouse (2011-03-05 20:31:38)

keenerd · 2011-03-05 21:15:51

Wiki-search-html is in, as is the optdep.

Install size is 40% smaller (down to 9MB) and searching is 40% faster too.

milomouse · 2011-03-05 23:09:47

fantastic. voted

keenerd · 2011-07-22 16:14:38

Update!

Bug fix:
report no matches found instead of Ada

New features:
pretty line wrapping to your terminal width
highlighted regex matches (sadly not case insensitive)

keenerd · 2011-08-02 20:52:55

More escape sequences added.

prebuilt package:

pacman -U http://pkgbuild.com/~kkeen/packages/arch-wiki-lite-20110802-1-any.pkg.tar.xz

Last edited by keenerd (2011-08-02 20:53:23)

keenerd · 2011-08-04 13:50:34

Fixed a bug where bash color escapes were taken too literally.

Links are now highlighted.

Also, now in [community].

Arch Linux

#1 2011-02-28 13:57:58

Making arch-wiki-docs useful

#2 2011-02-28 14:41:38

Re: Making arch-wiki-docs useful

#3 2011-02-28 17:04:30

Re: Making arch-wiki-docs useful

#4 2011-03-01 00:10:09

Re: Making arch-wiki-docs useful

#5 2011-03-01 00:47:48

Re: Making arch-wiki-docs useful

#6 2011-03-01 16:39:56

Re: Making arch-wiki-docs useful

#7 2011-03-01 17:40:25

Re: Making arch-wiki-docs useful

#8 2011-03-02 01:01:25

Re: Making arch-wiki-docs useful

#9 2011-03-02 01:08:13

Re: Making arch-wiki-docs useful

#10 2011-03-02 12:58:48

Re: Making arch-wiki-docs useful

#11 2011-03-02 13:30:47

Re: Making arch-wiki-docs useful

#12 2011-03-02 13:47:33

Re: Making arch-wiki-docs useful

#13 2011-03-02 14:14:54

Re: Making arch-wiki-docs useful

#14 2011-03-02 22:13:15

Re: Making arch-wiki-docs useful

#15 2011-03-05 15:36:41

Re: Making arch-wiki-docs useful

#16 2011-03-05 15:51:21

Re: Making arch-wiki-docs useful

#17 2011-03-05 16:11:08

Re: Making arch-wiki-docs useful

#18 2011-03-05 19:01:35

Re: Making arch-wiki-docs useful

#19 2011-03-05 19:43:55

Re: Making arch-wiki-docs useful

#20 2011-03-05 20:30:23

Re: Making arch-wiki-docs useful

#21 2011-03-05 21:15:51

Re: Making arch-wiki-docs useful

#22 2011-03-05 23:09:47

Re: Making arch-wiki-docs useful

#23 2011-07-22 16:14:38

Re: Making arch-wiki-docs useful

#24 2011-08-02 20:52:55

Re: Making arch-wiki-docs useful

#25 2011-08-04 13:50:34

Re: Making arch-wiki-docs useful

Board footer