vdmitriyev / datasets-links-collection

A collection of links for various datasets

💬 About

A collection of links for various datasets.

🔎 Search Engines

• Google Dataset Search
• Hugging Face Datasets
• PapersWithCode, ML Datasets
• UC Irvine Machine Learning Repository

🇩🇪 Datasets Links (Germany / in German)

• Das Datenportal für Deutschland

  • https://www.govdata.de/web/guest/daten

• Data Open Data Portal (Oldenburg)

  • https://opendata.oldenburg.de

• Open Data of city Zürich

  • A really impressive collection of datasets about the city
  • https://data.stadt-zuerich.ch/

• Deutsche Bahn (Open Data Portal)

  • https://data.deutschebahn.com/dataset?groups=datasets

• GENESIS-Online Database (Statistisches Bundesamt)

  • https://www-genesis.destatis.de/genesis/online

• Daten aus dem Handelsregister

  • https://offeneregister.de/

📑 Collections

• Awesome Public Datasets (GitHub)
  • https://github.com/caesar0301/awesome-public-datasets
• Dataset on Kaggle
  • https://www.kaggle.com/datasets
  • ML-friendly Public Datasets (article on Kaggle)
• Collections - high quality data and datasets organized by topic
  • https://datahub.io/collections
• RELATIONAL DATASET REPOSITORY
  • https://relational.fit.cvut.cz/search
• Links to 1 111 (and counting) interestingd datasets (shared via Google Spreadsheet)
  • https://docs.google.com/spreadsheets/d/1wZhPLMCHKJvwOkP4juclhjFgqIY8fQFMemwKL2c64vk/edit#gid=0
• Open Data on AWS
  • https://registry.opendata.aws/
• KDnuggets datasets
  • http://www.kdnuggets.com/datasets/index.html
• 70+ Machine Learning Datasets – Gain real-world experience with Data Science projects
  • https://data-flair.training/blogs/machine-learning-datasets/
• Descriptions are given in Russian, but the datasets are in various languages
  • Подборка датасетов для машинного обучения
• Data from Publications
  • http://figshare.com/

📋 Datasets Links

Dataset Name	Links
Airbnb	- Airbnb datasets by Inside Airbnb - Python scripts for Airbnb listings
EU Open Data Portal	- https://data.europa.eu/euodp/data/dataset
DataHub of ready-to-use NLP datasets	- https://github.com/huggingface/datasets
Music	- Million Playlist Dataset (RecSys Challenge 2018) - Million Song Dataset
WikiData	- http://www.wikidata.org/ - Wikipedia Revision History with ~ 314 millions of rows
DBpedia	- http://dbpedia.org/
Twitter corpus of moral sentiment (35k)	- https://osf.io/k5n7y/
Web data: Amazon reviews (~35 million reviews)	- https://snap.stanford.edu/data/web-Amazon.html
Datasets for machine learning in Python	- https://github.com/jaberg/skdata
Open Data by Socrata	- https://opendata.socrata.com/
Networks Datasets	- https://west.uni-koblenz.de/research/datasets
Open Data ( source: comments on ResearchGate)	- http://web.ist.utl.pt/acardoso/datasets/ - http://webspam.lip6.fr/wiki/pmwiki.php
SNAP - Stanford Network Analysis Project	- http://snap.stanford.edu/data/ - http://snap.stanford.edu/data/links.html - http://snap.stanford.edu/data/other.html
The daily news cycle [9*~2.0 GB]	- http://www.memetracker.org/data.html - https://pslcdatashop.web.cmu.edu/index.jsp
S5 - A Labeled Anomaly Detection Dataset [~16M]	- http://webscope.sandbox.yahoo.com/catalog.php?datatype=s&did=70
Data Sets on AWS	- http://aws.amazon.com/publicdatasets/#1
Linked Open Data by NY Times[RDF]	- http://data.nytimes.com/#
Global Database of Events, Language, and Tone (GDELT)	- https://www.gdeltproject.org/#downloading - Basic data import can be found here
PSLC DataShop in Pittsburgh	- https://pslcdatashop.web.cmu.edu/index.jsp?datasets=public
Labeled Faces in the Wild [images of faces]	- http://vis-www.cs.umass.edu/lfw/index.html
Mobile Data Challenge (MDC) Dataset	- https://www.idiap.ch/dataset/mdc
(RU) Хаб открытых данных на русском языке	- http://hubofdata.ru/
US Open Data Action Plan and Datasets	- http://www.kdnuggets.com/2014/05/us-open-data-action-plan-data-sets.html
StatLib - Datasets Archive	- http://lib.stat.cmu.edu/datasets/
100+ Interesting Data Sets for Statistics	- http://rs.io/100-interesting-data-sets-for-statistics/
Repositories of datasets	- http://www.trustlet.org/datasets/
Quantnet	- https://quantnet.hu-berlin.de/
Quora on open datasets	- http://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public?q=dataset
Enron Email Dataset (Famous Public E-mails Dataset)	- https://www.cs.cmu.edu/~./enron/
The 20 Newsgroups data set (~ 20,000 docs)	- http://qwone.com/~jason/20Newsgroups/
Stack Overflow Database (up to 320 Gb)	- https://www.brentozar.com/archive/2015/10/how-to-download-the-stack-overflow-database-via-bittorrent/
TimeDial and Disfl-QA (Conversational NLP)	https://ai.googleblog.com/2021/08/two-new-datasets-for-conversational-nlp.html
Miscellaneous	- https://archive-it.org/explore?show=Collections - https://delicious.com/pskomoroch/dataset - http://www.hpi.uni-potsdam.de/naumann/projekte/repeatability/datasets.html

Microsoft

• Microsoft Research Open Data
  • https://msropendata.com/
• GeoLife GPS Trajectories [298.66 MB]
  • http://research.microsoft.com/en-us/downloads/b16d359d-d164-469e-9fd4-daa38f2b2e13/
• T-Drive trajectory data sample [~140 MB]
  • http://research.microsoft.com/apps/pubs/?id=15288
• Microsoft Data Science Initiative
  • http://research.microsoft.com/apps/mobile/showpage.aspx?page=/en-us/projects/data-science-initiative/default.aspx#datasets
• LETOR: Learning to Rank for Information Retrieval
  • http://research.microsoft.com/en-us/um/beijing/projects/letor/

💡 Energy

• Solar Measurement Grid Oahu, Hawaii by NREL
  • First analysis - https://github.com/vdmitriyev/data-analysis-notebooks/tree/master/nrel-oahu-solar
• UK Domestic Appliance-Level Electricity (UK-DALE) dataset
  • Github repository - https://github.com/JackKelly/UK-DALE_metadata
• Open PV Dataset (Tracking the Sun, Berkeley)
• Energy efficiency Data Set from UCI
  • Analysis is here - https://github.com/vdmitriyev/data-analysis-notebooks/tree/master/energy-residential-buildings
• PLAID: the Plug Load Appliance Identification Dataset
  • https://github.com/jingkungao/PLAID/blob/master/ParseData.ipynb
• Indian Dataset for Ambient Water and Energy (IWAE)
• SMARD.DE Data
• Non-Intrusive Load Monitoring Toolkit (nilmtk)
• Global Power Plant Database
  • Some data analysis results can be found here - https://github.com/wri/global-power-plant-database
• Erneuerbare-Energien-Gesetz aggregated data
  • Figures, data and information concerning the EEG
  • EEG-Anlagenstammdaten
• BLOND, a building-level office environment dataset of typical electrical appliances
  • publication - https://www.nature.com/articles/sdata201848
  • dataset - https://mediatum.ub.tum.de/1375836

⚓ Water

• SmartH2O Project
  • https://zenodo.org/record/556152#.WgsNsWjWxhF
  • http://smarth2o.deib.polimi.it/results/datasets/

🤖 LLMs - Evaluation Datasets

• truthful_qa - Datasets at Hugging Face
• wino_bias - Datasets at Hugging Face
• bold - Datasets at Hugging Face

⚙️ Software To Work Data

• CKAN - the world’s leading open-source data portal platform
  • http://ckan.org/
• Datasette - A tool for exploring and publishing data
  • https://github.com/simonw/datasette/blob/master/README.md
• Amundsen - data discovery and metadata engine
  • https://github.com/amundsen-io/amundsen
• OpenRefine - a powerful tool for working with messy data
  • https://openrefine.org/