Python and command-line MediaWiki access for Humans.
- get an HTML or plain text "extract" (lead or summary)
- get a representative image, pageimage, thumbnail
- get an Infobox as a python dictionary
- get any/all Wikidata by title
- get info in any language
- get random info
This package is intended to make it as easy as possible to get data from MediaWiki instances, expose more Wikidata, and extend the MediaWiki API just for kicks. We say "(for Humans)" because that is a goal. Please see NOTES for details. Questions, feedback, and especially contributions are welcome!
$ pip install wptools
✨🦄✨
>>> import wptools
An instance can be initialized by:
wptools.page (self)
None
: <NoneType>lang
: <str> MediaWiki language code (default='en')title
: <unicode> a MediaWiki article titlewiki
: <str> any MediaWiki sitewikibase
: <str> Wikidata item entity ID
The simplest way to begin is with a title:
>>> f = wptools.page("Flannery O'Connor")
Flannery_O'Connor (en)
{
lang: en
title: Flannery_O'Connor
}
The default language is 'en' (English):
>>> t = wptools.page('穐吉敏子')
穐吉敏子 (en)
{
lang: en
title: 穐吉敏子
}
Leaving off arguments invokes a random lookup in English:
>>> r = wptools.page()
en.wikipedia.org (action=random) None
Sylvia_Rivera (en)
{
lang: en
pageid: 3296309
title: Sylvia_Rivera
}
If you give only lang, you get a random article in that language:
>>> zh = wptools.page(lang='zh')
zh.wikipedia.org (action=random) None
哈莉特·塔布曼 (zh)
{
lang: zh
pageid: 211070
title: 哈莉特·塔布曼
}
You can also start with a wikibase item:
>>> q = wptools.page(wikibase='Q43303')
Q43303 (en)
{
lang: en
wikibase: Q43303
}
Or, another wiki site:
>>> m = wptools.page(wiki='en.wikiquote.org')
en.wikiquote.org (action=random) None
Malala_Yousafzai (en)
{
lang: en
pageid: 146817
title: Malala_Yousafzai
}
Instance attributes echo automatically. You can turn that off with
silent=True
:
>>> r = wptools.page(silent=True)
Request details echo to stderr with verbose=True
:
>>> r = wptools.page(verbose=True)
Get a representative image:
>>> frida = wptools.page('Frida Kahlo').get_query()
en.wikipedia.org (action=query) Frida_Kahlo
>>> frida.pageimage
u'https://upload.wikimedia.org/wikipedia/commons/0/06/Frida_Kahlo,_by_Guillermo_Kahlo.jpg'
frida.thumbnail
Note: A page's image can come from an Infobox, query pageimage
or thumbnail
, a RESTBase request, or from Wikidata Property:P18.
See Methods for more details.
Get a text (or HTML) extract:
>>> ella = wptools.page('Ella Fitzgerald').get_query()
en.wikipedia.org (action=query) Ella_Fitzgerald
>>> print ella.extext
**Ella Jane Fitzgerald** (April 25, 1917 – June 15, 1996) was an
American jazz singer often referred to as the First Lady of Song,
Queen of Jazz and Lady Ella. She was noted for her purity of tone,
impeccable diction, phrasing and intonation, and a "horn-like"
improvisational ability, particularly in her scat singing.
...
>>> print ella.extract
<p><b>Ella Jane Fitzgerald</b> (April 25, 1917 – June 15, 1996) was an
American jazz singer often referred to as the First Lady of Song,
Queen of Jazz and Lady Ella. She was noted for her purity of tone,
impeccable diction, phrasing and intonation, and a "horn-like"
improvisational ability, particularly in her scat singing.</p>
...
Get an Infobox as a python object:
>>> fela = wptools.page('Fela Kuti').get_parse()
en.wikipedia.org (action=parse) Fela_Kuti
>>> fela.infobox['instrument']
'Saxophone, vocals, keyboards, trumpet, guitar, drums'
Note: Getting data from Infoboxen may be unavoidable, but getting
Wikidata (via get_wikidata()
) is preferred. Wikidata is
structured but (sometimes) data poor, while Infoboxen are
unstructured and (frequently) data rich. Please consider updating
Wikidata if the information you want is only available in a MediaWiki
instance so that others may benefit from open, linked data.
Get an (album, book, film, etc.) cover image:
>>> p = wptools.page('The Sympathizer')
>>> p.get_parse()
en.wikipedia.org (action=parse) The_Sympathizer
>>> p.image
'https://upload.wikimedia.org/wikipedia/en/e/e8/The_Sympathizer_-_book_cover.jpg'
Get wikidata by title:
>>> fry = wptools.page('Stephen Fry').get_wikidata()
en.wikipedia.org (action=wikidata) Stephen_Fry
en.wikipedia.org (action=wikidata) Q8817795|Q6581097|Q145|Q5
Stephen_Fry (en)
{
claims: <dict(4)> {Q145, Q5, Q6581097, Q8817795}
description: English comedian, actor, writer, presenter, and activist
g_claims: <dict(3)> {info, query, response}
g_wikidata: <dict(3)> {info, query, response}
image: https://upload.wikimedia.org/wikipedia/commons/1/15/Stephen_Fry_cropped.jpg
label: Stephen Fry
lang: en
props: <dict(8)> {P18, P21, P27, P31, P345, P569, P856, P910}
title: Stephen_Fry
wikibase: https://www.wikidata.org/wiki/Q192912
wikidata: <dict(8)> {IMDB, birth, category, citizenship, class, ge...
}
Note: Resolved properties and claims are stored in the wikidata
attribute. Wikidata claims are processed via WPTools._WIKIPROPS
.
Properties (e.g. P17: country) are stored in props
and those properties
that have Wikidata entities for values (e.g. Q142 instead of France) are
stored in claims
and resolved by another Wikidata API call (as
shown above).
Extend Wikidata claims to be resolved:
>>> simone = wptools.page('Simone de Beauvoir')
>>> simone._WIKIPROPS['P21'] = 'gender'
>>> simone.get_wikidata()
en.wikipedia.org (action=wikidata) Simone_de_Beauvoir
en.wikipedia.org (action=wikidata) Q142|Q8745624|Q1214721|Q6581072|Q5
>>> simone.wikidata['gender']
'female'
Get special lead section HTML:
>>> buddha = wptools.page('Buddha').get_rest()
en.wikipedia.org (action=/page/mobile-text/) Buddha
>>> buddha.lead
u'<p heading><a href="https://en.wikipedia.org/wiki/Buddha">Buddh...
<img pageimage src="https://upload.wikimedia.org/wikipedia/common...
<p snipped><span><b>Gautama Buddha</b>, also known as <b>Siddh...
Gautama taught a <a href="https://en.wikipedia.org/wiki/Middle_Wa...
Gautama is the primary figure in Buddhism. He is recognized by Bu...
<p metadata>Last modified: 2016-09-01T08:15:49Z</p>'
Note: The lead attribute contains the assembled stand-alone, encyclopedia-like HTML fragment:
<p heading>
wiki-linked title and description<img {type}>
{image, pageimage, or thumbnail}<p snipped>
lead paragraphs with (noprint, reference, &c.) snipped<p metadata>
available metadata (e.g. Last modified)
Get all the things:
>>> jill = wptools.page('Jill Lepore').get()
en.wikipedia.org (action=query) Jill_Lepore
en.wikipedia.org (action=parse) Jill_Lepore
en.wikipedia.org (action=wikidata) Q6192915
en.wikipedia.org (action=wikidata) Q30|Q5
Jill_Lepore (en)
{
claims: <dict(2)> {Q30, Q5}
description: American historian
extext: <str(1016)> **Jill Lepore** (born August 27, 1966) is an A...
extract: <str(1107)> <p><b>Jill Lepore</b> (born August 27, 1966) ...
g_claims: <dict(3)> {info, query, response}
g_parse: <dict(3)> {info, query, response}
g_query: <dict(3)> {info, query, response}
g_wikidata: <dict(3)> {info, query, response}
infobox: <dict(39)> {academic_advisors, alma_mater, alt, author_ab...
label: Jill Lepore
lang: en
pageid: 22469182
parsetree: <str(20662)> <root><template><title>Infobox scientist</...
props: <dict(3)> {P27, P31, P569}
random: Kunming Medical University
title: Jill_Lepore
url: https://en.wikipedia.org/wiki/Jill_Lepore
urlraw: https://en.wikipedia.org/wiki/Jill_Lepore?action=raw
wikibase: https://www.wikidata.org/wiki/Q6192915
wikidata: <dict(3)> {birth, citizenship, class}
wikitext: <str(13872)> {{Infobox scientist| name = Jill Lepore| na...
}
Query results are always cached:
>>> jill.get()
Request cached in g_query.
Request cached in g_parse.
Request cached in g_wikidata.
The wptools
user-agent will look like this:
wptools/version (https://github.com/siznax/wptools) pycurl libs
Get help on instance methods like this:
>>> help(wptools.core)
>>> help(<instance>)
get (self)
make all requests necessary to populate all the things
- get_query()
- get_parse()
- get_wikidata()
get_claims (self)
Wikidata:API action=wbgetentities for labels of claims
- e.g. {'Q298': 'country'} resolves to {'country': 'Chile'}
- use get_wikidata() to populate claims
get_parse (self)
MediaWiki:API action=parse request for:
- image: <str> Infobox image URL
- images: <dict> {pimage}
- infobox: <dict> Infobox data as python dictionary
- links: <list> interwiki links (iwlinks)
- pageid: <int> MediaWiki database ID
- parsetree: <unicode> XML parse tree
- wikibase: <unicode> Wikidata entity ID or wikidata URL
- wikitext: <unicode> raw wikitext URL
get_query (self)
MediaWiki:API action=query request for:
- extext: <unicode> plain text (Markdown) extract
- extract: <unicode> HTML extract via Extension:TextExtract
- images: <dict> {qimage, qthumb}
- pageid: <int> MediaWiki database ID
- pageimage: <unicode> pageimage URL via Extension:PageImages
- random: <unicode> a random article title with every request!
- thumbnail: <unicode> thumbnail URL via Extension:PageImages
- url: <unicode> the canonical wiki URL
- urlraw: <unicode> ostensible raw wikitext URL
get_random (self)
MediaWiki:API action=query request for:
- pageid: <int> MediaWiki database ID
- title: <unicode> article title
get_rest (self)
RESTBase /page/mobile-text/
request for:
- description: <unicode> apparently, Wikidata description
- images: <dict> {rimage, rthumb}
- lastmodified: <str> ISO8601 date and time
- lead: <str> encyclopedia-like lead section
- pageimage: <unicode> apparently,
action=query
pageimage - thumbnail: <unicode> larger
action=query
thumbnail - url: <unicode> the canonical wiki URL
- urlraw: <unicode> ostensible raw wikitext URL
get_wikidata (self)
Wikidata:API action=wbgetentities request for:
- claims: <dict> Wikidata claims (to be resolved)
- description: <unicode> Wikidata description
- image: <unicode> Wikidata Property:P18 image URL
- images: <dict> {wimage}
- label: <unicode> Wikidata label
- props: <dict> Wikidata properties
- wikibase: <str> Wikidata URL
- wikidata: <dict> resolved Wikidata properties and claims
show (self)
Pretty-print instance attributes.
Basic functionality on the command-line is provided by the wptool
command.
$ wptool -h
usage: wptool [-h] [-H] [-l L] [-n] [-q] [-s] [-t T] [-v] [-w W]
Get Wikipedia article info and Wikidata via MediaWiki APIs.
Gets a random English Wikipedia article by default, or in the
language -lang, or from the wikisite -wiki, or by specific
title -title. The output is a plain text extract unless -HTML.
optional arguments:
-h, --help show this help message and exit
-H, -HTML output HTML extract
-l L, -lang L language code
-n, -nowrap do not wrap text
-q, -query show query and exit
-s, -shh quiet output to stderr
-t T, -title T get a specific title
-v, -verbose HTTP status to stderr
-w W, -wiki W use alternative wikisite
Powered by https://github.com/siznax/wptools/
For example:
$ wptool -t "Jeanne d'Arc" -l fr -s
JEANNE_D'ARC—sainte et héroïne de l'histoire de France
![Jeanne d'Arc](https://upload.wikimedia.org/wikipedia/commons/3/39/...)
**Jeanne d'Arc**, née vers 1412 à Domrémy village du duché de Bar dont
une partie relevait du royaume de France pour le temporel et de
l'évêché de Toul pour le spirituel (actuellement dans le département
des Vosges en Lorraine), et morte sur le bûcher le 30 mai 1431 à
Rouen, capitale du duché de Normandie alors possession du royaume
d'Angleterre, est une héroïne de l'histoire de France, chef de guerre
et sainte de l'Église catholique, surnommée depuis le XVIe siècle «
_la Pucelle d'Orléans_ » et, depuis le XIXe siècle, « _mère de la
nation française_ ».
...
<https://fr.wikipedia.org/wiki/Jeanne_d%27Arc>
<https://www.wikidata.org/wiki/Q7226>
Please enjoy!
@siznax 👹