implydata / plywood

A toolkit for querying and interacting with Big Data

Home Page:https://plywood.imply.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

concurrent limit requester getting stuck

cedrics opened this issue · comments

We recently started observing plywood seemingly getting stuck. After some investigation we realized the concurrent limit requester is getting stuck and from its perspective the number of requests didn't decrease anymore.

I am not very familiar with node's streaming API but we patched the requester to listen to the close event instead of end & error and haven't observed a similar issue since, however. I am not a 100% sure if this actually fixed the issue as it only popped up sporadically or if there any drawbacks of using close. The node documentation states that the end event will not be triggered unless all data is read from the stream and my assumption is that we are somehow running into that edge case