niderhoff / big-data-datasets

Curated list of Publicly available Big Data datasets. Uncompressed size in brackets. No Blockchains.

Big Data Datasets

Curated list of Publicly available Big Data datasets. Uncompressed size in brackets. No Blockchains.

Structured

• CelebFaces Attributes - 1.2 GB
  • Over 200k images of celbrities with 40 binary attribute annotations

Text

• CommonCrawl (AWS) - A corpus of web crawl data composed of over 25 billion web pages.
  • Semi-Structured (includes Metadata): 250 TB
• DBpedia - curated wikipedia data
• Freebase
  • Freebase: 22 GB (250 GB)
  • Freebase Deleted Triples: 2 GB (8 GB)
  • Freebase/wikidata Mappings: 22 MB (243 MB)
• StackOverflow Data (BigQuery) - 182 GB

Image

• Landsat 8 (AWS)
• Uber Self Driving Car Challenge
  • 200 GB+ (compressed)

Audio

• Million Song Dataset (AWS)

Bonus: API / Streamdata / "Self-Service"

• Facebook Social Graph API

Bonus: Opendata / Census / Government data

• EU Open Data Portal
• data.gov
• US Census
• data.gov.uk
• CIA World Factbook
• healthdata.gov
• digital.nhs.uk
• Gapminder
• National Centers for Environmental Information

Meta / Lists / Sources

These pages might link to datastes which are already in the list.

• Data Science Central List of Big Data Sets
• AWS Open Data
  • Usage examples
• Kaggle Datastes
• Data Science Central: Big data sets available for free
• KDNuggets