Giters

Keck-DataReductionPipelines / KcwiDRP

KCWI data reduction pipeline. Developers: Don Neill, Matt Matuszewski, Chris Martin (Caltech)

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Calibration files not found

hebeling opened this issue · comments

commented

Kcwi_prep works once the file root is provided explicitly but everything beyond stage 1 of the pipeline runs into problems. Specifically, the DRP does not appear to create / find the necessary calibration files:

41/ 41 88 2 2 TUP 0 10 000 Med BM KBlu 5000.0 2458573.138 10.0 object 88 308.59116667 25.06380556 90.00 1.370 wolf1346
KCWI_STAGE2DARK [INFO 1 ] input reduced image: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_int.fits
KCWI_DO_DARK [INFO 1 ] DS Test (drk?, msig, tsig): NO 0.1864 0.2841
KCWI_STAGE2DARK [WARNING] cannot associate with any master dark: kb190330_00088_int.fits
KCWI_STAGE2DARK [INFO 1 ] scattered light subtracted
KCWI_WRITE_IMAGE [INFO 1 ] wrote image file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_mskd.fits
KCWI_WRITE_IMAGE [INFO 1 ] wrote image file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_vard.fits
KCWI_WRITE_IMAGE [INFO 1 ] wrote image file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00088_intd.fits

KCWI_STAGE2DARK [INFO 1 ] run time in seconds: 117.51911
KCWI_STAGE2DARK [INFO 1 ] finished on Fri Apr 12 11:57:07 2019
IDL> kcwi_stage3geom
% Compiled module: KCWI_STAGE3GEOM.
fatal: Not a git repository (or any of the parent directories): .git
KCWI_READ_PROC [INFO 1 ] Master proc file: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kcwi.proc
KCWI_READ_PROC [INFO 1 ] Fri Apr 12 11:57:15 2019
KCWI_STAGE3GEOM [INFO 1 ] Number of input images: 41

1/ 41 27 2 2 TUP 0 10 000 Med BM KBlu 5000.0 2458572.590 36.7 dflat 27 58.00000000 45.00000000 0.00 1.410
KCWI_STAGE3GEOM [INFO 1 ] input 2-D image: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00027_intd.fits
KCWI_STAGE3GEOM [WARNING] missing calibration file(s) for: /Volumes/T01/d1/keck/kcwi/raw/mar19/redux/kb190330_00027_intd.fits

The AAAREADME file provides no information that would help me solve these problems - or did I miss it? All my original FITS files are in the same directory, and I specified "." as the "calibration directory" during the kcwi_prep phase.

Thanks in advance!

Harald

commented

A few questions:
1- Are you using the original file names or the KOA generated file names?
2- After you run kcwi_prep, there are some diagnostic files that identify the file types. One of them is what.list. Could you upload it here?

commented
commented

In this case, can you send the result of running just the kcwi_prep? that produces a file called a proc table

commented
commented

Could you try to run the pipeline on a directory that only contains data from the 190330 data? Remove any reference to 190331. I am not sure how froot works and if it is implemented correctly.

I have your data on disk, and I have no problem running the pipeline, but I only have 190330.

The relevant line I get from running kcwi_prep is:

% Compiled module: JULDAY.
KCWI_PREP [INFO 1 ] Number of raw input images: 65
% Compiled module: KCWI_PRINT_CFGS.

While you get:
% Compiled module: JULDAY.
KCWI_PREP [INFO 1 ] Number of raw input images: 1
% Compiled module: KCWI_PRINT_CFGS.

commented
commented
commented
commented

I am on travel until the end of the week. I will be able to look into this further when I get back to Waimea after Easter.

We can reduce your data on your computers there, and find a way to transfer the results to you, next week.

commented
commented

Hi Harald,
I think I found the problem with your data.
We actually discussed it briefly when you were here, but I think we both lost track of it.
Some of your calibration files don't have the correct header information. Specifically, there is a keyword called CALTYPE, and I am sure that if you look at your files it is probably set to unknown.

For that reason, and to reduce the data online, we created a short script that fixes the header. The script is still in the reduced data directory on our computers.

Here it is:

from astropy.io import fits

contbars = [11,30,42]
arclamps = [12,13, 31,32, 43,44]
flatlamps = [14,15,16,17,18,19, 33,34,35,36,37,38, 45,46,47,48,49,50]
bias = [20,21,22,23,24,25,26]

prefix = "kb190330"

for frame in contbars:
    file_name = "%s_%05d.fits" % (prefix, frame)
    fits.setval(file_name, 'CALTYPE', value='cbars')

for frame in arclamps:
    file_name = "%s_%05d.fits" % (prefix, frame)
    fits.setval(file_name, 'CALTYPE', value='arcflat')

for frame in flatlamps:
    file_name = "%s_%05d.fits" % (prefix, frame)
    fits.setval(file_name, 'CALTYPE', value='cflat')

for frame in bias:
    file_name = "%s_%05d.fits" % (prefix, frame)
    fits.setval(file_name, 'CALTYPE', value='dark')

Make sure that the file numbers at the top correspond to the actual calibrations, but it seems correct to me.

Maybe make a copy of your data, and then run this script on the raw data before you do any pipeline step, including kcwi_prep.

commented
commented
commented

Let's verify that you are running the correct version of the code.
Did you install it using a tar file or using git?

If you used tar, I recommend that you switch to git.

git clone https://github.com/Keck-DataReductionPipelines/KcwiDRP.git

The installation instructions might be different, but this is my recommendation. This will allow us to verify that you are running the correct version by using the command:

git pull

If the result is that nothing needs to be downloaded, then you are on the latest version.

commented
commented
commented
commented
commented