Giters

nautechsystems / nautilus_trader

A high-performance algorithmic trading platform and event-driven backtester

Home Page:https://nautilustrader.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Process `OrderBookDeltas` for backtest streaming

cjdsellers opened this issue · comments

commented

The way to do this will be to use the deltas flags field, and per the Databento implementation for these https://databento.com/docs/knowledge-base/new-users/standards-conventions/flags when F_LAST (128) is seen then this marks the end of an order book event "Marks the last record in a single event for a given instrument_id".

  • The data will need to have the flags field correctly populated
  • This is currently how the Nautilus Databento adatper buffers MBO (market-by-order / L3) messages into groups of deltas
  • This option should probably be configurable
commented

Wondering what is still missing to use this feature. It would be great to start using deltas. Right now, the backtesting is not realistic enough, because the flag is not set AFAIK. Most venues send multiple deltas at once, hence it is unclear during backtesting which delta is the last one.

commented

Happy to help further develop this. For example testing or even making adapter specific changes if needed.

commented

Hi @davidsblom

This could be handled at the DataEngine by buffering deltas until an F_LAST flag is seen, and then creating an OrderBookDeltas object which would then be published to subscribers (including the data engine itself maintaining a book). This could be an option on the DataEngineConfig such as buffer_deltas etc.

In this case we'd be relying on an accurately populated flags field, otherwise the deltas may be buffered "forever" or up to some configured bound.

The Databento adapter currently does this in Rust for live feeds, because we can rely on the flags field being correct there. It's also far more performant to do this in Rust and outside the Python event loops main thread.

I'm not sure how applicable this would be for some of the other adapters. For instance Binance when the depth feed is throttled anyway - there isn't really a concept of event boundaries. Other adapters may be different, do you have any thoughts here?

So one gray area to determine here is this DataEngine config is applicable for both backtest and live. In the case of Databento, individual deltas are never seen live - but we currently don't replicate this in a backtest (per this ticket).

I still think this is most applicable to Databento data, not sure how it may translate to other order book data?