Name server list contains incorrect encodings
peterdesmet opened this issue · comments
In the interim and processed data of the name server species, some names are written with incorrect encoding characters, such as Agaricus xantholepis F.H. Møller, 1952
instead of Agaricus xantholepis F.H. Møller, 1952
(e.g. in this file).
The problem is related to the way the GBIF API is requested. With the requests
library (used in e.g. the gbif_name_matching utility), there are no issues:
import requests
results = requests.get('http://api.gbif.org/v1/species/match?verbose=False&strict=true&name=Agaricus+xantholepis')
results.json()
Returns:
{'canonicalName': 'Agaricus xantholepis',
'class': 'Agaricomycetes',
'classKey': 186,
'confidence': 99,
'family': 'Agaricaceae',
'familyKey': 4170,
'genus': 'Agaricus',
'genusKey': 7893259,
'kingdom': 'Fungi',
'kingdomKey': 5,
'matchType': 'EXACT',
'order': 'Agaricales',
'orderKey': 1499,
'phylum': 'Basidiomycota',
'phylumKey': 34,
'rank': 'SPECIES',
'scientificName': 'Agaricus xantholepis F.H. Møller, 1952',
'species': 'Agaricus xantholepis',
'speciesKey': 5243402,
'status': 'ACCEPTED',
'synonym': False,
'usageKey': 5243402}
But the name server list is so big, that an asynchronous request is necessary. This is implemented in this notebook. Apparently the asyncio
or other library that is used to request GBIF does not handle the result as UTF8?