Giters

blockchain-etl / blockchain-etl-streaming

Streaming Ethereum and Bitcoin blockchain data to Google Pub/Sub or Postgres in Kubernetes

Home Page:https://medium.com/google-cloud/live-ethereum-and-bitcoin-data-in-google-bigquery-and-pub-sub-765b71cd57b5

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Implement Kubernetes manifests for ethereum-etl-streaming

medvedev1088 opened this issue · comments

commented

A single Pod runs:

  1. download-last_synced_block - Job that downloads last_synced_block.txt from GCS bucket on node start up. See also init containers.
  2. streaming_service.py - streams blockchain data to PubSub
  3. health_checker.py - listens on HTTP port for health check requests
  4. gcs-syncd - uploads last_synced_block.txt periodically to GCS bucket and has a preStop hook that uploads the file to GCS bucket before Pod is terminated.
  5. fluentd daemon - uploads logs to GCS bucket

====================

Some code can be found here https://github.com/airswap/ethereum-etl/commit/cc7f86e139d317452c9cf0c975b99feb6ed8d7a0#diff-aa0ae5de65a67b108e5a26ddde9d3adf