Multi-threaded file IO for loading halo catalogs
lgarrison opened this issue · comments
Lehman Garrison commented
On network file systems, we can probably get better IO performance in CompaSOHaloCatalog
by reading multiple files in parallel. Hopefully the time-intensive parts of the IO release the GIL, so we could just spin up Python-level threads and use a queue
to pass the loaded arrays back to the main thread.
Note that each IO thread will additionally spin off 1 to 4 Blosc threads to do the decompression.