Giters

blockchain-etl / blockchain-etl-streaming

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Home Page:https://medium.com/google-cloud/live-ethereum-and-bitcoin-data-in-google-bigquery-and-pub-sub-765b71cd57b5

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Implement streaming_service.py

medvedev1088 opened this issue · comments

commented

Streaming service

  1. Continuously polls for new blocks using Ethereum JSON RPC API (e.g. every 10 seconds).
  2. Outputs data to GooglePubSubItemExporter, which publishes data to PubSub.
  3. Saves the last synchronized block to last_sync_block.txt file on every period.

This script can be used as a basis https://github.com/blockchain-etl/ethereum-etl/blob/feature/streaming/stream.py

================================

PubSub publisher guide https://cloud.google.com/pubsub/docs/publisher

There is some code in this fork https://github.com/airswap/ethereum-etl/commit/cc7f86e139d317452c9cf0c975b99feb6ed8d7a0#diff-a8e6a25bc32464a59378bac1befac9eb

TODO: Consider using geth subscriptions to handle chain reorganizations https://github.com/ethereum/go-ethereum/wiki/RPC-PUB-SUB#newheads