Query around pmap function
anshumandixit9 opened this issue · comments
Does pmap create a new object of ConcurrentLinkedQueue for every method call?
Yes, you can see here https://github.com/reborg/parallel/blob/master/src/parallel/core.clj#L527
Inputs are queued into the queue straight way and n (future) workers are created that immediately start de-queueing. The following loop collects available results at each round, essentially waiting for the slowest of the n workers to return their result. It then starts another round of n concurrent workers until the queue is empty.
Like core.pmap, this approach to parallelism works well when workers are roughly doing the same amount of work. If some item in the queue requires substantially more effort to produce, then it's not particularly efficient as every other worker in the queue needs to wait. But regardless, the existence of this parallel.pmap is more about independence from the size of sequential chunks (normally 32) and the number of cores, so you can control exactly how many threads are running in parallel independently from the platform and the input sequence type.