RLP Gateway keeps sending data to dropped connections
pianohacker opened this issue · comments
Thanks for your submission!
Here are some guidance for submitting your issue.
Tell us why you are submitting?
- I found a bug - here are some steps to recreate it.
- I have an idea for a new feature - please document as "As a user, I would
like to..." - I am having trouble getting setup, understanding documentation, or using
Loggregator in some way - please highlight any resources you are using. - This is an architecture change that will result in cleaner more efficient
code - Tell us why you think this is a good idea.
We review issues and PR's on a weekly basis and try to schedule and prioritize
items. If you are wondering about status of an item feel free to contact us on
our slack channel.
Expected Behavior
When a consumer drops a connection to the RLP Gateway, all other consumers with the same shard ID should receive all of the envelopes.
Actual Behavior
The RLP Gateway will continue to give part of the envelopes to the disconnected consumer, and the remaining consumers with the same shard ID will only receive part of the envelopes for roughly 14 minutes.
This seems to be due to the connection within the RLP Gateway staying around after the client disconnects, and thus receiving its share of the logs until the stream timeout is hit.
We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.
The labels on this github issue will be updated when the story is started.
We can't reproduce this behavior. Every time we kill a rlp consumer, the load gets balanced within 3-5 seconds.
We suspect this might be due to firewalls or load balancers holding on to connections. That would be the next place to look.