Arch Linux

asg1448

Member

Registered: 2014-04-12

Posts: 74

[Solved] Read wikipedia offline

I am looking to setup an offline arch laptop with a full copy of wikipedia.

Now the obvious solution is to setup MediaWiki package (https://wiki.archlinux.org/index.php/MediaWiki), and import wikimedia XML dump (https://en.wikipedia.org/wiki/Wikipedia … e_download) using `importDump.php` tool. Except it doesn't work. English XML dump is ~15 million pages, and import works at 5 pages/sec or less, so importing will take months.

I wrote some code in attempt to speed this up (https://github.com/galets/FastImportDump), but it does not help much, import is still very slow.

There are two other tools - mwdumper and some other perl tool. Can't find a name of 2nd tool, but that's irrelevant, cuz both of them churned around, and it looks like they are working, but didn't really import anything. mwdumper doesn't work with latest version at all, but it works with mediawiki LTS (insofar that it doesn't crash right away), but it doesn't really import anything :@

There's tons of tools out there claiming they can be used to set up readable wikipedia from dump, and I'm yet to find one that works.

Any suggestions (please based on experience only)?

**Update:** I solved the problem by installing kiwix package from AUR. It will download and index wikipedia all by itself. It did take almost 2 days for it to index wikipedia, this thing is huge!!!

Last edited by asg1448 (2015-07-22 19:39:08)

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: [Solved] Read wikipedia offline

What exactly are you trying to do? Import the whole wikipedia to read a few (outdated?) pages now and then?

Xyne

Forum Fellow

Registered: 2008-08-03

Posts: 6,965

Website

Re: [Solved] Read wikipedia offline

If you make karol a good offer, he would probably come over and recite any page you like from memory.

---

My Arch Linux Stuff • Forum Etiquette • Community Ethos - Arch is not for everyone

asg1448

Member

Registered: 2014-04-12

Posts: 74

Re: [Solved] Read wikipedia offline

What exactly are you trying to do? Import the whole wikipedia to read a few (outdated?) pages now and then?

have a view-able copy if something happens to internet access

progandy

Member

Registered: 2012-05-17

Posts: 5,322

Re: [Solved] Read wikipedia offline

You should probably use the sql dump and import directly into your database.

---

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' | alias ENGLISH='LANG=C.UTF-8 ' |