World Factbook Corpus
The CIA World Factbook is a Public Domain data set comprising of geographical, economic and political data on every country in the world.
Data types include free text, currency, percentages, longitude & latitude, altitude, taxonomies, and as such it makes a viable test & demonstration corpus for search applications, on top of the intrinsic value of the data.
Since the Factbook is not available in an easily machine-readable format, we've created a crawler to extract the data in a way that should be easier to consume.
Implementation
The crawler was written using Node.js and outputs in both XML and JSON. Pre-generated output is provided.
Run the crawler
The command below will extract data from the dataset in ./factbook-crawler/data and export it to ./data
node factbook-crawler/index.js
Use the data
var fs = require('fs'),
path = require('path');
fs.readdirSync('./data/json').forEach(function(file){
var country = JSON.parse(fs.readFileSync('./data/json/'+file));
console.log( country.name )
});