Best multithreading setup for large projects
kLy opened this issue · comments
Hi. I have a pretty large project with thousands of audio files, but most of them are quite short. When I look at the documentation, it seems like rhubarb only properly utilises multi-threading with longer audio files. So I'm wondering what would be a good way to do batch processing of large numbers of audio files to most effectively use the multi-threading? I'm thinking:
- Use the existing setup, and batch the files through one at a time. According to the documentation, this seems like it will make least use of the multithreading built into the tool.
- Force thread count to 1 on rhubarb, but launch a pool of worker threads that call rhubarb in parallel, one for each processor core I have.
- Have some kind of functionality for rhubarb to input a list of files rather than just process one at a time. This would make rhubarb handle the multithreading in the way that it deems the most efficient, though of course that would mean adding support for this feature.
Curious to hear your thoughts. My current plan is to go with 2. Let me know if that's a good / bad idea. Thanks! :)
My thoughts on that topic are exactly the same. For Rhubarb 2, I'm thinking of implementing a built-in batch mode that will take a glob pattern of files and process them all using smart multi-threading.
Until then, your best bet when it comes to processing speed is to go with your option 2.