Giters

bmoscon / cryptostore

A scalable storage service for cryptocurrency data

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

No data for COINBASE-l2_book-BTC-USD

krishvishal opened this issue · comments

commented

Describe the bug

To Reproduce

python 3.8
Redis server v=6.0.6
mongodb 4.4.5

Steps to reproduce the behavior:

pip install git+https://github.com/manahl/arctic.git
pip install git+https://github.com/bmoscon/cryptostore.git
cryptostore
  • Provide config (at least as much as is relevant)
cache: redis

redis:
    ip: '127.0.0.1'
    port: 6379
    socket: null
    del_after_read: true
    retention_time: null
    start_flush: true


exchanges:
    COINBASE:
        channel_timeouts:
            l2_book: -1
            trades: 120
            ticker: 120
        retries: -1
        trades: [BTC-USD]
        l2_book:
            symbols: [BTC-USD]
            max_depth: 20
            book_delta: true
            book_interval: 0

storage: [arctic]
storage_retry_wait: 30

arctic: mongodb://127.0.0.1

storage_interval: 5

  • redis or kafka?
    redis

  • What behavior did you see?
    For binance the above config worked fine but with coinbase L3 and L2 book its failing. I'm able to access coinbase L3 and L2 api from this machine.

  • Provide any tracebacks if applicable

2021-04-28 01:10:24,435 : INFO : Reading COINBASE-trades-BTC-USD
2021-04-28 01:10:24,436 : INFO : No data for COINBASE-trades-BTC-USD
2021-04-28 01:10:24,436 : INFO : Reading COINBASE-l2_book-BTC-USD
2021-04-28 01:10:24,436 : INFO : No data for COINBASE-l2_book-BTC-USD
.
.
.
.

Expected behavior
Is to cache L2 data to redis and write it to arctic.
Screenshots
If applicable, add screenshots to help explain your problem.

Operating System:

  • Ubuntu 18.04
    Cryptofeed Version
    cryptofeed: 1.9.0
    cryptostore: 0.4.0
commented

why is book interval set to 0?

commented

I was changing the book interval to different values for testing. The default book interval value also results in the same behavior.

commented

I have no problems with coinbase:

2021-04-27 19:43:51,073 : INFO : Reading COINBASE-l2_book-BTC-USD
2021-04-27 19:43:51,073 : INFO : No data for COINBASE-l2_book-BTC-USD
2021-04-27 19:43:51,073 : INFO : Reading COINBASE-trades-BTC-USD
2021-04-27 19:43:51,074 : INFO : No data for COINBASE-trades-BTC-USD
2021-04-27 19:43:51,362 : INFO : Collector for COINBASE running on PID 5309
2021-04-27 19:44:50,762 : INFO : Reading COINBASE-l2_book-BTC-USD
2021-04-27 19:44:50,840 : INFO : l2_book-COINBASE-BTC-USD: Read 14718 messages from Redis
2021-04-27 19:44:51,156 : INFO : l2_book-COINBASE-BTC-USD: Removed 14718 entries through id 1619567090752-0
2021-04-27 19:44:51,157 : INFO : Write Complete COINBASE-l2_book-BTC-USD
2021-04-27 19:44:51,157 : INFO : Reading COINBASE-trades-BTC-USD
2021-04-27 19:44:51,158 : INFO : trades-COINBASE-BTC-USD: Read 185 messages from Redis
2021-04-27 19:44:51,211 : INFO : trades-COINBASE-BTC-USD: Removed 185 entries through id 1619567091012-0
2021-04-27 19:44:51,211 : INFO : Write Complete COINBASE-trades-BTC-USD

Using this for the coinbase config:

exchanges:
    COINBASE:
        retries: -1
        l2_book:
            symbols: [BTC-USD]
            book_delta: true
            book_interval: 100000
        trades: [BTC-USD]
commented

@krishvishal were you able to resolve the issue? What was it?

commented

It worked after restarting the mongodb and also I made book_interval: 100000, it was not working with book_interval: 0. Can you please explain what book_interval does?

From what I understand it saves a lob snapshot every book_interval number of updates and when it does it also deletes the intermediate updates. Is my understanding correct? If yes, is there a way to save both LOB updates and the snapshots.

commented

no, it doesnt delete anything. it just controls how frequently complete snapshots are written, so 100000 means after 100k deltas you'll get the complete book written, then 100k deltas, then the snapshot, then 100k deltas, etc.