Using MediaWiki to pull text from a Wikia page but it comes back in a big mess is there a better way I could do this to pull text from each section?

Question

I am developing an Android app that pulls information from a Wikia page and displays it in the app. I currently am pulling all Categories to navigate and have my app set up to display the page in a WebView but I would like to just pull the info and format myself instead of cheapening it by passing to WebView.

What I am using to get the text is: http://scottlandminecraft.wikia.com/api.php?format=xml&action=query&titles=ZackScott&prop=revisions&rvprop=content

My problem is the text comes back in a big clump, does anyone have any ideas as to how to get this more formatted so I could parse from tags or am I wasing my time trying to find that? If so would it be better to find a way to parse the text I need by going from identifiers in the text this pulls, or is there a better way?

Thank you for your input and time.

Answer 1

The easiest way, if you don't want to parse the wiki markup yourself, is to retrieve the parsed HTML version of the page and then process it using an HTML parser (like jsoup, as recommended by Hasham).

Besides just scraping the normal wiki user interface (which will give you the page HTML wrapped in the navigation skin), there are two ways of getting the HTML text of a MediaWiki page:

1. use the API with action=parse, which will return the page HTML wrapped in a MediaWiki API XML (or JSON / YAML / etc.) response, like this:

   • http://scottlandminecraft.wikia.com/api.php?format=xml&action=parse&page=ZackScott
     
     

2. or use the main index.php script with action=render, which will return just the page HTML:

   • http://scottlandminecraft.wikia.com/index.php?action=render&title=ZackScott

Ps. Since you mention sections in your question, let me note that the action=parse API module can return information about the sections on the page using prop=sections (or even prop=sections|text). For an example, see this API query:

• http://scottlandminecraft.wikia.com/api.php?format=xml&action=parse&page=ZackScott&prop=sections