Sparse HEALPix file format
kadrlica opened this issue · comments
This is a great space saver, but uses a non-standard file format. We should investigate whether healpy
has evolved to have a sparse file format.
It appears that healpy
does have the ability to read and write sparse healpix files, which they called "partial" format. The benefit is that there is a "standard" definition for partial healpix map files. The down side is that the healpy.fitsfunc
interface relies on the creation of the full pixel array for writing/reading, which imposes significant overhead.
From a brief look at the healpy code, the "standard" requires two header keywords:
INDXSCHM = 'EXPLICIT'
OBJECT = 'PARTIAL'
The first column must contain the pixel indexes, while the second through nth column contain values. The healpy convention is to call the first column PIXEL
; however, this is not enforced in read_map
. (The IACT community choses to call the first column PIX
instead.)
I would suggest that we conform to the standard header keywords defined by fitsfunc.write_map
. However, we probably want to keep our own version of the read/write functions to avoid creating full arrays.