abacusorg / abacusutils

Python code to interface with halo catalogs and other Abacus N-body data products

Home Page:https://abacusutils.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Unpack cleaned subsamples directly into subsample table

lgarrison opened this issue · comments

Currently, we build a concatenated table of all cleaned particles, reindex it, merge it with the original subsamples, then do the RVint unpacking on the whole table. We may be able to achieve better performance by never constructing the concatenated cleaned particle table and instead do the RVint unpacking directly into the final location in the combined particle table.

In #16, I noticed that the peak RSS usage is way higher than the final usage. I think the ASDF objects are holding onto internal buffers with the uncompressed data, so garbage collection requires deleting the ASDF handles. So we will want to pay attention to that in this refactor. But it may be moot if we get an ASDF "read_into" functionality working.