Mining Groups of people from Wikipedia

Question

I am trying to get the list of people from the http://en.wikipedia.org/wiki/Category:People_by_occupation . I have to go through all the sections and get people from each section.

How should i go about it ? Should I use a crawler and get the pages and search through those using BeautifulSoup ?
Or is there any other alternative to get the same from Wikipedia ?

Answer 1

I would go with Pywikipediabot python project.

Have a look to category.py. You could use:

* tree        - show a tree of subcategories of a given category
* listify     - make a list of all of the articles that are in a category