Issue reading the last `isub` at the end of the last file.
devanshkv opened this issue · comments
The file has 66 isub(s). I think this bit of the code needs to be checked.
2019-11-11 17:17:56,463 - to_dada -your.dada - DEBUG - Data read is 6291456, Data step is 32768
2019-11-11 17:17:56,463 - get_data -your - DEBUG - Reading from 32768 samples from sample 6291456
2019-11-11 17:17:56,463 - get_data -your.psrfits - DEBUG - Startsub 384, endsub 386
2019-11-11 17:17:56,463 - get_data -your.psrfits - DEBUG - isub is 384
2019-11-11 17:17:56,464 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 17:17:56,464 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-11 17:17:56,464 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f4691973b38>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f469196d390>]
2019-11-11 17:17:56,464 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - isub is 385
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-11 17:17:56,515 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f4691973b38>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f469196d390>]
2019-11-11 17:17:56,515 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - isub is 386
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-11 17:17:56,571 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f4691973b38>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f469196d390>]
Process p1:
Traceback (most recent call last):
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/dada.py", line 114, in to_dada
data_input = self.your_object.get_data(data_read, self.data_step)
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/__init__.py", line 65, in get_data
return PsrfitsFile.get_data(self, nstart, nsamp)
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/psrfits.py", line 254, in get_data
data.append(self.read_subint(int(isub % self.nsubints)))
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/psrfits.py", line 138, in read_subint
sdata = self.fits['SUBINT'].data[isub]['DATA']
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/astropy/io/fits/fitsrec.py", line 492, in __getitem__
out = self.view(np.recarray)[key]
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/numpy/core/records.py", line 506, in __getitem__
obj = super(recarray, self).__getitem__(indx)
IndexError: index 66 is out of bounds for axis 0 with size 66
I don't think it is reading files in the correct order. puppi_58763_J1935+2154_0296_0006.fits is supposed to be the 6th file, with fileId = 5, but clearly file id is still 1.
Also, I am not able to replicate this error. I tried setting filename to puppi_58763_J1935+2154_0296_0006.fits and fileid to 1, with isub=386, and tried both the functions: get_data and read_subint.
It seems to work fine at least for this simple test. It is reading the correct file corresponding to fileid=1. I suspect that the your.filelist is getting reordered somewhere in the pipeline.
l = glob.glob('/hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296*fits')
y = Your(l)
y.get_data(6291456, 32768)
2019-11-11 22:36:09,942 - get_data -your - DEBUG - Reading from 32768 samples from sample 6291456
2019-11-11 22:36:09,944 - get_data -your.psrfits - DEBUG - Updating fileid to 1
2019-11-11 22:36:09,945 - get_data -your.psrfits - DEBUG - Delted mmap'ed object
2019-11-11 22:36:09,946 - get_data -your.psrfits - DEBUG - Reading file ID: 1
2019-11-11 22:36:09,954 - get_data -your.psrfits - DEBUG - Startsub 384, endsub 386
2019-11-11 22:36:09,955 - get_data -your.psrfits - DEBUG - isub is 384
2019-11-11 22:36:09,956 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 22:36:09,957 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0002.fits
2019-11-11 22:36:09,962 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7ff7d3cd56a0>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7ff7d3c68978>]
2019-11-11 22:36:09,985 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I
2019-11-11 22:36:10,084 - get_data -your.psrfits - DEBUG - isub is 385
2019-11-11 22:36:10,085 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 22:36:10,086 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0002.fits
2019-11-11 22:36:10,087 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7ff7d3cd56a0>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7ff7d3c68978>]
2019-11-11 22:36:10,088 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I
2019-11-11 22:36:10,172 - get_data -your.psrfits - DEBUG - isub is 386
2019-11-11 22:36:10,173 - get_data -your.psrfits - DEBUG - file id is 1
2019-11-11 22:36:10,174 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0002.fits
2019-11-11 22:36:10,174 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7ff7d3cd56a0>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7ff7d3c68978>]
2019-11-11 22:36:10,176 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I```
For some reason you are reading /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0002.fits and not /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits?
I am not forcing it to read that file, I am just giving it a list of files.
In the normal (correct) behavior, it should read the file puppi_58763_J1935+2154_0296_0002.fits. Which is why I think that the ordering of filelist is being modified somewhere in the pipeline.
I ran your_heimdall.py on these files, and it is reading the files with correct indexes now, although I still get the same error. Not sure why it was reading the files incorrectly earlier.
2019-11-14 15:08:36,004 - get_data -your - DEBUG - Reading from 32768 samples from sample 27262976
2019-11-14 15:08:36,004 - get_data -your.psrfits - DEBUG - Startsub 1664, endsub 1666
2019-11-14 15:08:36,004 - get_data -your.psrfits - DEBUG - isub is 1664
2019-11-14 15:08:36,004 - get_data -your.psrfits - DEBUG - file id is 5
2019-11-14 15:08:36,004 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-14 15:08:36,004 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f8f01ae29e8>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f8f01af9f98>]
2019-11-14 15:08:36,005 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I
2019-11-14 15:08:36,065 - get_data -your.psrfits - DEBUG - isub is 1665
2019-11-14 15:08:36,065 - get_data -your.psrfits - DEBUG - file id is 5
2019-11-14 15:08:36,065 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-14 15:08:36,065 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f8f01ae29e8>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f8f01af9f98>]
2019-11-14 15:08:36,065 - read_subint -your.psrfits - WARNING - Polarization is IQUV, just using Stokes I
2019-11-14 15:08:36,167 - get_data -your.psrfits - DEBUG - isub is 1666
2019-11-14 15:08:36,167 - get_data -your.psrfits - DEBUG - file id is 5
2019-11-14 15:08:36,167 - get_data -your.psrfits - DEBUG - Reading file: /hyrule/data/grbfrb/P3231D/puppi_58763_J1935+2154_0296_0006.fits
2019-11-14 15:08:36,167 - get_data -your.psrfits - DEBUG - Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f8f01ae29e8>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f8f01af9f98>]
Process p1:
Traceback (most recent call last):
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/dada.py", line 114, in to_dada
data_input = self.your_object.get_data(data_read, self.data_step)
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/__init__.py", line 65, in get_data
return PsrfitsFile.get_data(self, nstart, nsamp)
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/psrfits.py", line 254, in get_data
data.append(self.read_subint(int(isub % self.nsubints)))
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.4.1-py3.6.egg/your/psrfits.py", line 138, in read_subint
sdata = self.fits['SUBINT'].data[isub]['DATA']
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/fitsrec.py", line 492, in __getitem__
out = self.view(np.recarray)[key]
File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/numpy/core/records.py", line 506, in __getitem__
obj = super(recarray, self).__getitem__(indx)
IndexError: index 66 is out of bounds for axis 0 with size 66```
Fixed with 7423f5b