Why not set up a dedicated thread to generate tokens?
changweige opened this issue · comments
From what I understand about Token Bucket, if there is a dedicated thread to generate tokens continuously, it can handle a flow burst easily. But it seems that no such a generator exists in this crate.
The existing implementation can handle bursts. For example, if the capacity is set to 10, up to 10 tokens are available in the bucket. We don't need a separate thread to handle this.
The bookkeeping done during try_wait is likely lightweight enough to be used within a worker thread without much impact.
I'm seeing effective rates of > 20M/s for a single thread on a 2018 MacBook - which means the average case is about 50ns delay for try_wait. This seems pretty reasonable for ratelimiting typical client and server activities (eg, ratelimiting HTTP/API requests, connection establishment, etc). I'd be hesitant to switch to a model that requires an additional thread for token generation - as it may have more unpredictable delay waiting for the "token generator" to schedule onto a core or cause additional resource utilization. This cost could quickly add up if a client needs multiple separate ratelimits - eg, if a remote API has different limits for different types of requests - having a separate thread for each of those might get quite expensive.
https://crates.io/crates/ratelimit actually took the approach of using a separate thread for "generating" tokens - technically, it works in reverse, draining spent tokens from a MPSC queue. I've since moved to this strategy using atomics and amortizing the cost of token bucket bookkeeping across the consumer threads.
Can you share some details about your use-case? There may be some ways to extend this library to handle it without necessarily impacting some of the design decisions that were made in this particular implementation. It is also possible that other implementations may be more suited to your use-case.
Closing due to inactivity.
Thanks for your elaboration and it makes sense.
My use case is to limit filesystem backend stream bandwidth.'
After delving a little more into ratelimit library, it looks like the limiter will enter a busy loop consuming cpu cycles constantly. For my use case, it might be a problem when we set the limit very low.
Moreover, precise_time_ns use u64 to represent nano-seconds from epoch, it is not reliable I suppose.
And unwrap should come alongside a comment line telling its safety.