fil2h5 crashed during conversion of a 59 GiB filterbank file

[Richard Elkins]

It also caused a whole lot of pain in several processes in the same system!

[Richard Elkins]

See UCBerkeleySETI/turbo_seti#267

[Richard Elkins]

Note:

fil2h5 needs to do the following (same as __make_h5_file() in data_handler.py in turbo_seti):

• from blimpy.io.hdf_writer import __write_to_hdf5_heavy as write_to_h5
• Waterfall(....., load_data=False) # Do not load the entire data matrix at once.
• write_to_h5(.....)

This will facilitate the following:

• The filterbank file is loaded initially with only the header.
• The data matrix is read from the filterbank file and written to the HDF5 file, one "blob" at a time.