3.3 IR Data Calibration Steps

The IR data reduction process begins with the raw IR image file. This data file contains all the non-destructive readouts from an exposure, stored in reverse time order i.e. the first extension corresponds to the last readout. Most of the calibration steps are applied independently to each readout. For example, the DQICORR, NLINCORR, and DARKCORR steps apply the bad pixel flags, non-linearity correction coefficients, and dark image, respectively, to each readout. On the other hand, the CRCORR step, which performs the up-the-ramp fit and removes the effects of cosmic ray hits, utilizes the values from all readouts of individual pixels simultaneously. Detailed descriptions of each step are provided in the following sections.

All steps up through UNITCORR are applied to an in-memory image stack that contains all the readouts. The CRCORR step produces an additional single image that gives the best-fit count rate for each pixel. The remaining steps in the process - FLATCORR and image statistics - are then applied to the full stack of readouts and to the single image produced by CRCORR.

Upon completion of the IR data calibration process, two output files are produced. The Intermediate MultiAccum (ima) file contains the full stack of calibrated readouts while the final calibrated image (flt) file is the single image produced by CRCORR with subsequent flat fielding applied.

Figure 3.3 shows a schematic representation of all the IR calibration steps, which are also briefly summarized below, in the order they are executed, with the corresponding calibration switch keyword in parenthesis:

• Initialize data quality, DQ, array (DQICORR)
• Estimate amount of signal in zeroth-read (ZSIGCOR)
• Subtract bias level from reference pixels (BLEVCORR)
• Subtract zeroth read image (ZOFFCORR)
• Initialize error array, ERR (NOISCORR)
• Correct for detector non-linear response (NLINCORR)
• Subtract dark current image (DARKCORR)
• Compute photometric keyword values for header (PHOTCORR)
• Convert to units of count rate (UNITCORR)
• Fit readout signal and identify CR hits (CRCORR)
• Divide by flat-field image(s) and apply gain conversion (FLATCORR)
• Compute image statistics (No switch)

3.3.1 Data Quality Initialization

• Header Switch: DQICORR
• Reference File: BPIXTAB

This step populates the data quality (DQ) array in all IR readouts by reading a table of known bad pixels for the detector, stored in the ‘Bad Pixel’ reference table BPIXTAB. The appropriate BPIXTAB is selected based on the value of the DETECTOR and USEAFTER keywords.

The types of bad pixels that can be flagged are listed in Table 3.3.

The DQ array is no longer updated to reflect any Take Data Flag (TDF) transition during the sample (see Section 5 of STAN 12, Jan 2013). Other DQ values will only be marked during further processing (such as cosmic-ray rejection).

If the user wishes to update the DQ array themselves before running further processing, they should first complete the DQ initialization step, and remember that the data in the DQ extension is always in units of UNSIGNED INTEGER.

Table 3.3: Data quality array flags for IR files

3.3.2 IR Zero-Read Signal Correction

• Header Switch: ZSIGCORR
• Reference Files: DARKFILE, NLINFILE

At the beginning of an IR observation, the detector pixels are reset and read out to record the bias level. An interval of approximately 2.9 seconds elapses between the time a given pixel is reset and then read. Because the IR channel does not have a shutter, signal from the field of view under observation, as well as persistent signal from previous observations, accumulates during that 2.9 second interval. When the initial (or ‘zeroth’) read is later subtracted from subsequent readouts, any signal in addition to the bias in the zeroth read will be subtracted. 

The linearity correction and saturation checking, executed in the NLINCORR step (see Section 3.3.13), both depend on the absolute signal level in a pixel at the time it was read, and thus, any pixels that were exposed to bright sources during zeroth read will have a large signal and will over-subtract that signal from subsequent reads. This can lead to inaccurate linearity corrections, as well as the failure to detect saturation pixels. The ZSIGCORR  step is used to estimate the amount of source signal in the field of view (plus any persistence signal) in the zeroth read and to supply this estimate to the  NLINCORR step. 

Such an estimate is given by the difference between the zeroth read in the linearity reference file (NLINFILE) and the science zero read exposure.  In brief the ZSIGCORR  module contains the following steps:

• Copy the zeroth signal image from the linearity reference file (super zero).
• Compute any subarray offsets.
• Subtract the super zero read reference image from the zero read science image (bias-subtracted zeroth read).
• Compute the noise in the subtracted zero image.
• A sigma clipping (zTHRESH*noise) is done for the bias subtracted zeroth read and any pixel having signal above the threshold noise is flagged with a DQ value of 2048; pixels below the threshold noise are ignored. This sigma clipped bias subtracted zeroth read is provided as one of the inputs to the NLINCORR step.
• The linearity correction file (NLINFILE)also has an extension with saturation values. Pixels that are saturated in the zeroth or first reads are flagged in the DQ array and the number of saturated pixels is reported.

The ZSIGCORR step estimates the source signal in the science zeroth read, as outlined above, by subtracting the super zeroth read from the science zeroth read. Prior to March 2011, this module calculated an estimated source signal in the zeroth read based on the signals in the first read and zeroth read + the exposure time between them. The current method not only avoids the issue due to difference in readout times for subarrays, but also, the dark current subtraction is no longer necessary for the source signal estimate in the zeroth read (the DARKFILE is no longer used by this step).

Note that this technique will not work well for pixels that are exposed to very bright targets where the signal is already beginning to saturate either the zeroth or first reads. The zero read correction, ZSIGCORR, thus checks for saturation in the zeroth and first read arrays and flags the saturated pixels with a value of 256 in the DQ extension of all the corresponding readouts.

Pixels that are determined to have detectable signal in the zeroth read are flagged in the DQ arrays of the first readout of the output ima file with a data quality value of 2048 and carried over into the rest of the processing up to the final flt file. The NLINFILE is chosen based on the values of the DETECTOR and USEAFTER keywords, while the DARKFILES are selected based on the values of the DETECTOR, CCDAMP, CCDGAIN, SAMP_SEQ, SUBTYPE and USEAFTER keywords.

3.3.3 Bias Correction

• Header Switch: BLEVCORR
• Reference Files: OSCNTAB

The BLEVCORR step uses the reference pixels located around the perimeter of the IR detector to track and subtract the bias level during an exposure. For each raw readout array, the average signal level of the reference pixels is computed (via a sigma-clipped mean), subtracted from the array, and recorded in the MEANBLEV keyword in the SCI header of each readout.

The reference pixels located at the ends of each image row are used in this computation. Reference pixels are also located along the bottom and top of the detector, at the ends of each column, but those have been found to be less reliable and thus are not used.  There are 5 reference pixels around the perimeter but the outermost pixel is ignored on each side, for a total of 8 reference pixels per row used in the BLEVCORR step.

As with the UVIS overscan correction, the boundaries of the reference pixel regions that are used in the computation are defined in the OSCNTAB reference table, in the BIASSECT* columns. The BIASSECTA[1,2] values indicate the starting and ending column numbers for the reference pixels on the left edge of the image, and the BIASSECTB[1,2] give the values for the right side of the image.

The reference pixel regions are retained in the fits file throughout the remainder of processing but are usually ignored or skipped over in the actual application of calibration algorithms. They are left in place in the calibrated data stored in the ima file at the end of processing but are trimmed from the flt image file.

The reference file for bias level correction, OSCNTAB, is selected based on the value of the DETECTOR keyword only.

3.3.4 IR Zero-read Image Subtraction

• Header Switch: ZOFFCORR
• Reference Files: None

The ZOFFCORR step subtracts the zeroth read from all readouts in the exposure, including the zeroth read itself, resulting in a zero-read image that is exactly zero in the remainder of processing. The zeroth-read image is propagated through the remaining processing steps and included in the output products, so that a complete history of error estimates and data quality (DQ) flags is preserved.

3.3.5 Error Array Initialization

• Header Switch: NOISCORR (not listed explicitly in image header, see text)
• Reference Files: CCDTAB

This step computes an estimate of the uncertainties associated with the raw science data based on a noise model for the detector. The NOISCORR keyword is not user-accessible; this step is always performed by calwf3. Currently, the noise model (in DN) is a simple combination of detector read noise (RN) and Poisson noise in the signal, such that:

where RN (readnoise), dark, and flux are in units of electrons, and exptime (exposure time) is in units of seconds. The detector read noise and gain are read from the CCDTAB reference file and are different for each of the quadrants that are read out by the four amplifiers.

The uncertainties are then propagated, in quadrature, through each step of the calibration process, as appropriate. The ERR array for the final calibrated flt image is populated by the CRCORR step, based on the calculated uncertainty of the count rate fit to the MultiAccum samples (see Section 3.3.13 for details).

The CCDTAB reference file used in this step is selected based on the value of the DETECTOR keyword only.

3.3.6 Detector Non-linearity Correction

• Header Switch: NLINCORR
• Reference Files: NLINFILE

In this step, the integrated counts in the IR science images are corrected for the non-linear response of the detector, flagging each pixel that is saturated (as defined in the saturation extension of the NLINFILE reference image.) The observed response of the detector can be represented by two regimes:

• At low and intermediate signal levels the detector response deviates from the incident flux in a way that is correctable using the following expression:

where c1, c2, c3, and c4 are the correction coefficients (calculated from a large number of flat images that are median combined and then fit with a polynominal), F is the uncorrected flux in DN and Fc is the corrected flux. The current form of the correction uses a third-order polynomial, but the algorithm can handle an arbitrary number of coefficients. The number of coefficients and their uncertainties are given by the values of the NCOEFF and NERR keywords in the header of the NLINFILE.

• At high signal levels, as saturation sets in, the response becomes highly non-linear and is not correctable to a scientifically useful degree.

The signal in the zero read is temporarily added back to the zeroth read image of the science data before the linearity correction is applied and before the saturation is evaluated. Once the correction has been applied the zero read signal is once again removed. As with other calibration steps, this step is executed if ZSIGCORR is set to PERFORM. Saturation values for each pixel are stored in the NODE extension of the NLINFILE. Each pixel up the ramp (readout) is checked for saturation and if is saturated then the DQ array, for that readout, is updated and all subsequent readouts up the ramp are set as saturated. This is necessary since, in most cases, the readouts after a very saturated pixel can have a signal values that are lower than the saturation limit, and if not flagged will give incorrect flux values for these pixel. The SAMP and TIME arrays are not modified during this step. The NLINFILE reference files are selected based on the value of the DETECTOR keyword only.

3.3.7 Dark Current Subtraction

• Header Switch: DARKCORR
• Reference Files: DARKFILE

The DARKCORR step subtracts the detector dark current from the science data. The reference file listed under the DARKFILE header keyword is used to subtract the dark current from each sample in the ramps. Appropriate DARKFILE reference file are chosen based on the DETECTOR, CCDAMP, CCDGAIN, SAMP_SEQ, and SUBTYPE keywords of the science image. Due to potential non-linearities in some of the signal components, such as reset-related effects in the first one or two reads of an exposure, the dark current subtraction is not applied by simply scaling a generic reference dark image by the exposure time and then subtracting it. Instead, a library of dark current images are maintained, and includes darks taken in each of the available predefined MULTIACCUM sample sequences for both full-frame, as well as sub-array readout modes. The dark reference file is subtracted read-by-read from the stack of science image readouts so that there is an exact match in the timings and other characteristics of the dark image and the science image. The subtraction does not include the reference pixels. The ERR and DQ arrays from the reference dark file are combined with the SCI and DQ arrays from the science image, but the SAMP and TIME arrays are unchanged. The mean of the dark image is saved to the MEANDARK keyword in the output science image header.

3.3.8 Photometry Keywords

• Header Switch: PHOTCORR
• Reference Files: IMPHTTAB

The PHOTCORR step updates the image header with keywords that allow the user to convert their data from counts rates to absolute fluxes, and perform calibrated photometry. The step is performed using tables of precomputed values stored in the IMPHTTAB file. The appropriate entry in IMPHTTAB is selected according to the observing mode, whose value is stored in the image header keyword, PHOTMODE. The updated keywords are:

• PHOTFLAM: the inverse sensitivity in units of erg cm^-2 Å^-1 electron^-1
• PHOTFNU: the inverse sensitivity in units of Jy sec electron^-1
• PHOTZPT: the STMAG zero point
• PHOTPLAM: the bandpass pivot wavelength in Å
• PHOTBW: the band width in Å

3.3.9 Conversion To Signal Rate

• Header Switch: UNITCORR
• Reference Files: None

This step converts the science data from a time-integrated signal to a signal rate by dividing the SCI and ERR arrays for reach readout by the TIME array. No reference file is needed when executing this step. The BUNIT keyword in the output data header is updated and reflects the appropriate data units. This step is skipped if the BUNIT value is already COUNTS/S. The flat fielding process (performed if FLATCORR is set to PERFORM), further changes the BUNIT by multiplying the image by the gain. Therefore, the final BUNIT value depends on the value of both UNITCORR and FLATCORR, as illustrated in Table 3.4.

Table 3.4: Possible values of BUNIT after calibrating IR data with calwf3.

UNITCORR

FLATCORR

BUNIT AFTER CALIBRATION

OMIT

OMIT

COUNTS

OMIT

COMPLETE

ELECTRONS

COMPLETE

OMIT

COUNTS/S

COMPLETE

COMPLETE

ELECTRONS/S

3.3.10 Up-the-ramp Fitting and Cosmic Ray Identification

• Header Switch: CRCORR
• Reference Files: CCREJTAB

CRCORR combines the data from all readouts into a single image and identifies and flags pixels suspected of containing cosmic-ray (CR) hits. The method is described in detail in Fixsen et al. (2000).

The data from all readouts up the ramp are analyzed pixel-by-pixel, iteratively computing a linear fit to the accumulating counts-versus-exposure time relation. Pixels flagged as bad in the DQ arrays, such as when saturation occurs midway through the exposure, are rejected from the fitting process. CR hits are identified by searching for outliers from the fit results. The rejection threshold is set by the value in the CRSIGMAS column of the CR rejection parameters reference table CRREJTAB, and has a default value of 4. When a CR hit is detected, a linear fit is then performed independently for the sets of readouts before and after the hit; if a CR hit is identified to have occurred during a sample, the value measured for that sample is included in the ‘after’ ramp segment. Those fitting results are then again checked for outliers. This process is iterated until no new CRs are detected.

Pixel samples identified as containing a CR hit are flagged in the DQ arrays of the intermediate MultiAccum (ima) file, with DATAREJECT DQ value of 8192.

The DATAREJECT DQ flag is also set for all samples following the CR in order to indicate that the absolute value of the pixel is wrong after the first hit. However, this flagging smears the location of any hits which might occur after the first CR.

In addition to CRs, "negative CR hits" in the accumulated counts vs. time relation have occasionally been observed in WFC3 data. They appear as sudden “drops” in the accumulated counts vs. time plots for individual pixels (e.g. Figure 1 in WFC3 ISR 2010-13). These “negative CR hits” are also identified in the CRCORR step, and flagged with the SPIKE DQ flag, value = 1024. Appendix B of WFC3 ISR 2009-40 gives a possible explanation for a sub-class of such events: normal cosmic rays that traverse the detector but instead of hitting the photo-sensitive HgCdTe pixel bulk, they strike other parts of the pixel (e.g. the electronics) that are sensitive to charged particles. These events are sometimes clearly associated with CRs by the physical trail visible in raw images e.g. as shown in Figure 3.4. The trail has a lower signal than in the neighboring pixels (when the CR goes through the electronics) to undetectable (when the CR travels through layers that are unaffected by its passage) to higher signal than in the neighboring pixels (when charge is released in the active HgCdTe region of the pixel), see Appendix B of WFC3 ISR 2009-40. Other negative spikes are sometimes observed in isolated pixels and are attributed to “burst noise”, also known as popcorn noise or random telegraph signal.

Once all outliers have been identified, the slopes of each segment of non-flagged samples are computed via a linear fit to the counts vs. time data. This fit includes optimal weighting with individual data point uncertainties as well as contributions from the read noise and the Poisson noise for the source and the dark current. The linear fit reports the best fitting slope and its uncertainty. The final count rate value for a pixel (and its uncertainty) is determined by computing the weighted mean (and its uncertainty) of the individual sample's slopes. The result of this operation is stored in the output flt file, where the SCI array contains the final slope computed for each pixel, the ERR array contains the estimated uncertainty in the slope, the SAMP array contains the total number of non-flagged samples used to compute the slope, and the TIME array contains the total exposure time of those samples.

Pixels for which there are no unflagged samples, e.g., permanently hot or cold pixels, still have a slope computed (recorded in the SCI array of the output flt file) but they are flagged in the DQ array of the flt file. Users should therefore be always check the flt file DQ arrays to help determine whether a given SCI image pixel value is trustworthy for subsequent analysis.

The basic rule of thumb is that in order for a DQ value to propagate into the flt, it must be present in all the reads of the ima. The 8192 flag is not propagated into the flt because calwf3 has already accounted for the effects of the CRs when performing the up-the-ramp fit. The DQ arrays in the ima files contain the complete record of when and where exactly each cosmic ray hit the detector.

A similar propagation scheme occurs for the saturation flag (DQ = 256). If, for example, a pixel is saturated in the last two reads of a ramp, then those two reads are flagged with 256 in the ima file, and calwf3 ignores them during line-fitting. The resulting DQ value in the flt file is 0 because calwf3 has dealt with the saturation and the effects are not present in the flt. If saturation occurs in the first read of a ramp, the SCI extension of the flt file for that pixel contains an estimate of the flux equal to the value in the input zeroth read image, but the DQ extension of the flt does not have a 256 value added to it. If the zeroth read is also saturated, the flt file still contains the same flux estimate as in the first-read saturation case, but in this case, the DQ flag 256 is propagated into the output flt DQ extension.

Pixels where calwf3 finds 4 or more CRs up the ramp are flagged as UNSTABLE (32). These pixels are masked in the flt since that many signal jumps for a given pixel in a single ramp are an indication that likely the pixel should not be trusted. DQ values from any sample are carried through to the flt file if a pixel has no good samples.

3.3.11 Flat-field Correction

• Header Switch: FLATCORR
• Reference Files: PFLTFILE, LFLTFILE, DFLTFILE

The FLATCORR step corrects for pixel-to-pixel and large-scale sensitivity variations across the detector by dividing the science images by one or more flat-field images. A combined flat is created within calwf3 using up to three flat-field reference files: the pixel-to-pixel flat (c), the low-order flat (LFLTFILE), and the delta flat (DFLTFILE). FLATCORR also multiplies the science data by the detector gain, using the mean gain from all the amplifiers. Therefore the calibrated data will be in units of electrons per second (or electrons if UNITCORR = OMIT).

The PFLTFILE is a pixel-to-pixel flat-field correction file containing the small-scale flat-field variations. The PFLTFILE is always used in the calibration pipeline, while the other two flats are optional. The LFLTFILE is a low-order flat that corrects for any large-scale sensitivity variations across the detector. This file can be stored as a binned image, which is then expanded when being applied by calwf3. Finally, the DFLTFILE is a delta-flat containing any needed changes to the small-scale PFLTFILE. If the LFLTFILE and DFLTFILE are not specified in the SCI header, only the PFLTFILE is used for the flat-field correction. If two or more reference files are specified, they are read in and multiplied together to form a combined flat-field correction image.

The flat-field correction is applied to all readouts of the calibrated IR MultiAccum stack, as well as the single image produced by the CRCORR function. Due to geometric distortion effects, the area of the sky seen by different pixels is not constant and therefore observations of a constant surface brightness object will have counts per pixel that vary over the detector. To produce images that appear uniform with uniform illumination, the same counts per pixel variation across the field is left in place in the flat-field images, so that when a science image is divided by the flat it makes an implicit correction for the distortion effects on photometry. A consequence of this procedure is that two point-source objects of equal brightness will not have the same total counts after the flat-fielding step, thus point source photometry requires the application of a pixel area map (PAM) correction.

All flat-field reference images are selected based on the DETECTOR, CCDAMP, and FILTER used for the observation. A sub-array science image uses the same reference file(s) as a full-size image; calwf3 extracts the appropriate region from the reference file(s) and applies it to the sub-array input image.

3.3.12 Image Statistics Calculation

• Header Switch: None
• Reference Files: None

This step computes image statistics using the “good pixels”, i.e. with DQ value equal to 0, and records them in image header keywords. The operation is performed for every readout in the calibrated MultiAccum stack (ima), as well as the final (CRCORR-produced) calibrated image (flt). The values computed and captured in keywords are: the minimum, mean, and maximum values (GOODMIN, GOODMEAN, GOODMAX, respectively), the number of good pixels (NGOODPIX), as well as the minimum, mean, and maximum signal-to-noise ratio (i.e. the ratio of the SCI and ERR pixel values) which are SNRMIN, SNRMEAN, SNRMAX, respectively. The minimum, mean, and maximum statistics are computed for the ERR arrays as well.

3.3.13 Cosmic-ray rejection

• Header Switch: RPTCORR
• Reference Files: CCREJTAB

Associations with more than one member, i.e. have been associated using REPEAT-OBS, are combined using wf3rej (see Section 3.4.5 for more details). CR-SPLIT is not available for the IR channel. The task uses the same statistical detection algorithm developed for ACS (acsrej), STIS (ocrrj) and WFPC2 (crrej), providing a well-tested and robust procedure. For all associations (including dithered observations), the DRZ products will be created by Astrodrizzle, which performs both cosmic ray detection (in addition to wf3rej, for REPEAT-OBS observations) and corrects for geometric distortion.