thelabnyc / django-logpipe

Automated mirror of https://gitlab.com/thelabnyc/django-logpipe

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Infinite loop in case of ProvisionedThroughputExceededException from Kinesis

thiago-sylvain opened this issue · comments

Hi, thanks for your amazing package!

We use it as a consumer for Kinesis.
We use the suggested configuration with multiconsumer:

multi = MultiConsumer(people_consumer, places_consumer)
multi.run()

At some point, our Kinesis service raises a ProvisionedThroughputExceededException, multiple times a row, and it ends up with the _get_records method of the kinesis based consumer returning None.

This further leads to a StopIteration but coupled with the indefinite consumer, it ends up in an infinite loop.

For the moment we overrode the _get_records to generate a specific exception and crash for good, but there must be a cleaner solution to avoid entering this infinite loop.

Thank you very much for your advice!