Question about limiting concurrency in pools
srabraham opened this issue · comments
Hi @camdencheek, thanks for your work on this!
I have questions around the use of limitations on concurrency in the pool package. My understanding is that Go is built for having huge numbers of concurrent goroutines, since it has its own scheduler that can assign many lightweight goroutines to each OS thread (I found this article very enlightening). By having a default MaxGoroutines of runtime.GOMAXPROCS(0), the package is generally going to cap the concurrency such that the Go scheduler can't be maximally fast or clever. Also, anytime one of those goroutines gets blocked on a channel, you'll have one fewer live thread.
I know this is how a pool is supposed to work, but I bet a lot of people will be wanting to use a conc pool because they want a more sophisticated errgroup. Concurrency capping is useful in some cases, but not always, and I'm wondering if people might shoot themselves in the foot. I actually can't think of a case when runtime.GOMAXPROCS(0) would be a good default. In a concurrent API call situation, I'd want to cap the QPS, not the concurrency, and in a CPU-bound situation, there's no harm in allowing many more concurrent goroutines and allowing the Go scheduler to choose how to allocate goroutines to OS threads.
Sorry for the huge message and thanks again for your work!
@srabraham You are totally right. I actually changed the default to unlimited yesterday, very shortly after I cut the v0.1.0 release and this repo got posted publicly 😄 That change will make it into the v0.2.0 release once it gets a little more testing.
In a concurrent API call situation, I'd want to cap the QPS, not the concurrency
This obviously doesn't apply to all situations, but a common pattern at Sourcegraph (whose use cases inspired most of this package) is to make an API call for each item in a list. We want to do this as fast as possible, and a QPS limiter would leave speed on the table. Limiting concurrency lets you still process things as fast as the server will respond, but without overloading it with a burst of requests, which can massively increase latency of the first response (which we also care about at Sourcegraph).
If you care, some more detail on why it was initially implemented this way
In the internal version of this package, we had a "pool" and a "group". A "group" spawned a goroutine per task. A "pool" spawned a fixed pool of goroutines and reused them to run tasks. The APIs for both were nearly identical, and it was often confusing deciding which to use since, quite frequently, it didn't matter at all. The big difference was that a group had no limit by default and a pool had a fixed limit.
When I unified those two things in this package, the default runtime.GOMAXPROCS(0) tidbit survived.
Awesome, thanks for the thorough (and speedy!) explanation. I must've been looking at the v0.1.0 release's code. I agree that "unlimited" is probably the right default behavior. Thanks again!