Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Retrieve a list of all Wikipedia languages programmatically

I need to retrieve a list of all existing languages for a certain wiki project. For example, all Wikivoyage or all Wikipedia languages, just like on their landing pages.

I prefer to do this via MediaWiki API, if it's possible.

Thanks for your time.

like image 756
Damian Pavlica Avatar asked Nov 09 '15 12:11

Damian Pavlica


3 Answers

Approach 3: Using an API in the Wikimedia wiki farm and Extension:Sitematrix

https://commons.wikimedia.org/w/api.php?action=sitematrix&smtype=language

While this will return all wikis, the matrix knows about, it is easily filtered client side by code [as of now, one of: wiki (Wikipedia), wiktionary, wikibooks, wikinews, wikiquote, wikisource, wikiversity, wikivoyage] and by its closed state. One request with just some response body overhead but since it's easily cached and compresses well, not that serve.

like image 91
Rainer Rillke Avatar answered Nov 14 '22 04:11

Rainer Rillke


Approach 1: Using an API in the Wikimedia wiki farm

To get all interwiki prefixes that a wiki knows of, use the meta module of the MediaWiki API, and query any project for siprop=interwikimap:

https://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=interwikimap

You will get a large array of objects like this:

{
    "prefix": "aa",
    "local": "",
    "language": "Qaf\u00e1r af",
    "url": "https://aa.wikipedia.org/wiki/$1",
    "protorel": ""
}

protorel tells you if the url is protocol relative or not (i.e. starting with //. For the WikiMedia wikis, they will start with https. The $1 in the URL is, as you would have imagined, a placeholder for the title.

To get only the wikis in the same wikifarm (e.g. Wikimedia wikis), add sifilteriw=local to your query:

https://sv.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=interwikimap&sifilteriw=local

To fetch the names in you langue use siinlanguagecode, like this (all Wikimedia wikis, with their Swedish names, retrieved from arabic Wikipedia, but could have been any endpoint in the wiki farm):

https://ar.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=interwikimap&sifilteriw=local&siinlanguagecode=sv

From here you would have to filter out e.g. the Wikipedias yourself.

Approach 2: Using Wikistats at wmflabs

A list already filtered by type of project is available at http://wikistats.wmflabs.org (csv), where you can filter out Wikipedia, Wikiversity, etc. The csv file is updated on daily basis, but the tool is experimental, and might not be there forever.

In either approach, Wikimedia Incubator wikis will not show up.

like image 36
leo Avatar answered Nov 14 '22 06:11

leo


Subtract closed.dblist from wikipedia.dblist (other lists), then remove wiki from the end and replace _ with -.

like image 3
Tgr Avatar answered Nov 14 '22 05:11

Tgr