abacusorg / abacusutils

Python code to interface with halo catalogs and other Abacus N-body data products

Home Page:https://abacusutils.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Multi-threaded file IO for loading halo catalogs

lgarrison opened this issue · comments

On network file systems, we can probably get better IO performance in CompaSOHaloCatalog by reading multiple files in parallel. Hopefully the time-intensive parts of the IO release the GIL, so we could just spin up Python-level threads and use a queue to pass the loaded arrays back to the main thread.

Note that each IO thread will additionally spin off 1 to 4 Blosc threads to do the decompression.