improve auto batching logic
aniketmaurya opened this issue · comments
RIght now automatic batching is done in the LoadBalancer which is sequential.
We can move it to Model server level for more concurrency.
@ethanwharris and @aniketmaurya are checking this