bytewax / awesome-public-real-time-datasets

A list of publicly available datasets with real-time data maintained by the team at bytewax.io

Home Page:https://bytewax.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Awesome Public Real-Time Datasets and Sources

Awesome

This list is inspired by awesome public datasets, but for real-time datasets and sources. Normally accessed via HTTP or Websockets.

The list is separated into Free and Paid and broken into subsections based on loose categories.

Free

Finance/Crypto

  • Coinbase Market Data - Coinbase websocket to market data including level 2 orderbook data.
  • Blockchain transactions - Provides real-time notifications about new transactions and blocks.
  • Yahoo Financewss://streamer.finance.yahoo.com/ - This is not advertised in developer documentation, but discoverable as the websocket is used to update their website.
  • Finnhub - Limited free usage with a premium data sources also available.
  • CoinCheck - a cryptocurrency API that has a WebSocket interface (in beta)
  • Alpaca Markets Real-Time and historical market data via HTTP and Websocket.

Transportation

  • Open Rail Data - A collection of APIs that provide data relating to the UK rail network, including reference data, train timetables, and live service updates. The live data is streamed using the STOMP protocol.
  • GBFS New York - GBFS is the standard for bike share data with many locations around the world. Find more information here
  • Open Sky Flight - Data from Open Sky Api via HTTP endpoint. Supports real-time, but not streaming. Need to continually poll.
  • Open Glider Network - The OGN provides real-time traffic for gliders and other light aircraft. You can use an OGN client like python-ogn-client to connect to OGN servers, parse the APRS messages and push them to a broker like Kafka for streaming processing.
  • MTA GTFS Feed - Transit data in GTFS format for transit systems like NYC subway and Caltrain.
  • NY 511 live camera data - This live camera data requires some scraping to use. From this list of cameras you can then source the individual camera id and then request the timestamped image or most recent image by building the url like - https://511ny.org/map/Cctv/<image-id-goes-here>
  • Transport for London (TfL) - live data about the tube, buses, and more
  • Norwegian Coastal Administration - AIS data from vessels within the Norwegian economic zone and the protection zones off Svalbard and Jan Mayen.

Information

Other

Paid

Finance/Crypto

  • IEX Trading - IEX was created in response to questionable trading practices that had become widely used across traditional exhcanges! Their API provides streaming Stock market data.
  • Twitter - Twitter provides a streaming interface for research or enterprise. It is unknown what support will be post Twitter purchase in 2022.

Transportation

  • AIS Data Maritime, Aviation and weather data available via Spire.

Information

About

A list of publicly available datasets with real-time data maintained by the team at bytewax.io

https://bytewax.io

License:Creative Commons Zero v1.0 Universal