confluentinc / parallel-consumer

Parallel Apache Kafka client wrapper with per message ACK, client side queueing, a simpler consumer/producer API with key concurrency and extendable non-blocking IO processing.

Home Page:https://confluent.io/confluent-accelerators/#parallel-consumer

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Encoding back pressure system so that offset payloads are prevented from being too large

astubbs opened this issue · comments

At the moment the system does't protect against the space required to encode the offset overrun data from becoming too large to fit in the space available.

An approximation of the continuous approach is to keep some history of record per byte performance, and use that rate to restrict new messages.

For an exact approach, see #53

Approximate approach merged for 0.3.0