IndexError: tuple index out of range
devanshkv opened this issue · comments
your_object = Your("/hyrule/data/grbfrb/P3231_ALFA/test/4bit-p3231.20191118.J2229+2643.b0g0_0001.fits")
your_object.get_data(0,1024)
IndexError Traceback (most recent call last)
<ipython-input-8-926a6d8290ce> in <module>
----> 1 your_object.get_data(0,1024)
~/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.2-py3.6.egg/your/__init__.py in get_data(self, nstart, nsamp)
64 return SigprocFile.get_data(self, nstart, nsamp)
65 else:
---> 66 return PsrfitsFile.get_data(self, nstart, nsamp)
67
68 def __repr__(self):
~/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.2-py3.6.egg/your/psrfits.py in get_data(self, nstart, nsamp)
265 logger.debug(f'Reading subint {fsub} in file {self.filename}')
266 try:
--> 267 data.append(self.read_subint(fsub))
268 except KeyError:
269 logger.warn(f"Encountered KeyError, maybe mmap'd object was delected")
~/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.2-py3.6.egg/your/psrfits.py in read_subint(self, isub, apply_weights, apply_scales, apply_offsets)
140 if self.nbits < 8: # Unpack the bytes data
141 if (shp[0] != self.nsamp_per_subint) and \
--> 142 (shp[1] != self.nchan * self.nbits / 8):
143 sdata = sdata.reshape(self.nsamp_per_subint,
144 int(self.nchan * self.nbits / 8))
IndexError: tuple index out of range
Can you run it in debug mode and give the corresponding logs for the same?
DEBUG:your:Reading the fits file: /hyrule/data/grbfrb/P3231_ALFA/test/4bit-p3231.20191118.J2229+2643.b0g0_0001.fits
WARNING: VerifyWarning: Invalid keyword for column 17: The repeat count of the column format 'DATA' for column '7329600B' is fewer than the number of elements per the TDIM argument '(1,960,1,15270)'. The invalid TDIMn value will be ignored for the purpose of formatting this column. [astropy.io.fits.column]
WARNING:astropy:VerifyWarning: Invalid keyword for column 17: The repeat count of the column format 'DATA' for column '7329600B' is fewer than the number of elements per the TDIM argument '(1,960,1,15270)'. The invalid TDIMn value will be ignored for the purpose of formatting this column.
WARNING:your.psrfits:Channel spacing changes in file 0!
WARNING: VerifyWarning: Invalid keyword for column 17: The repeat count of the column format 'DATA' for column '7329600B' is fewer than the number of elements per the TDIM argument '(1,960,1,15270)'. The invalid TDIMn value will be ignored for the purpose of formatting this column. [astropy.io.fits.column]
WARNING:astropy:VerifyWarning: Invalid keyword for column 17: The repeat count of the column format 'DATA' for column '7329600B' is fewer than the number of elements per the TDIM argument '(1,960,1,15270)'. The invalid TDIMn value will be ignored for the purpose of formatting this column.
DEBUG:your:Generating unified header for file /hyrule/data/grbfrb/P3231_ALFA/test/4bit-p3231.20191118.J2229+2643.b0g0_0001.fits
DEBUG:your:Successfully generated unified header for file /hyrule/data/grbfrb/P3231_ALFA/test/4bit-p3231.20191118.J2229+2643.b0g0_0001.fits
DEBUG:your:Reading from 1024 samples from sample 0
DEBUG:your.psrfits:Startsub 0, endsub 0
DEBUG:your.psrfits:isub is 0
DEBUG:your.psrfits:file id is 0
DEBUG:your.psrfits:Using: [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7f58fa5f6978>, <astropy.io.fits.hdu.table.BinTableHDU object at 0x7f58fa57f128>]
DEBUG:your.psrfits:Reading subint 0 in file /hyrule/data/grbfrb/P3231_ALFA/test/4bit-p3231.20191118.J2229+2643.b0g0_0001.fits
Traceback (most recent call last):
File "fail.py", line 9, in <module>
your_object.get_data(0,1024)
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.2-py3.6.egg/your/__init__.py", line 66, in get_data
return PsrfitsFile.get_data(self, nstart, nsamp)
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.2-py3.6.egg/your/psrfits.py", line 267, in get_data
data.append(self.read_subint(fsub))
File "/home/dagarwal/soft/conda/envs/numba_gpu/lib/python3.6/site-packages/your-0.4.2-py3.6.egg/your/psrfits.py", line 142, in read_subint
(shp[1] != self.nchan * self.nbits / 8):
IndexError: tuple index out of range
The shape of data being returned by self.fits['SUBINT'].data[isub]['DATA'] is incorrect. Instead of a multidimensional array, it is giving a 1D array, which is causing this error. Converting the original data to 4bit is causing this issue.
As the original file for this wasn't present in the directory, I tested it with p3231.20191011.B0301+19.b0s0g0.00000_4bit.00000.fits and its original p3231.20191011.B0301+19.b0s0g0.00000.fits. Again, as suspected for the 4bit case the array returned has incorrect shape, while for the original it works fine.
Most of the mock data which we have is 4bit and all of them have the same issue, we would need to figure out a way to fix this.
The fits weren't written properly.