E.2 The STScI Reduction and Calibration Pipeline

In this section of the appendix, we summarize the basic reductions and calibrations that are performed in the STScI WFC3 calibration pipeline, calwf3. The material in this appendix is intended to provide only enough background to develop robust observing proposals. The WFC3 Data Handbook provides more detailed information needed for analyzing your data.

E.2.1 Selected Pipeline History

In February 2016, the pipeline began to process WFC3 data with calwf3 version 3.3, which incorporates two fundamental changes to the way UVIS exposures are calibrated and corrected. First, calwf3 applies pixel-based CTE (Charge Transfer Efficiency) corrections. (Note that calwf3 version 3.4, which added CTE corrections for subarray apertures, became operational in October 2016.) Second, photometric calibrations are determined and applied independently for each CCD chip. See WFC3 STAN issue 22 for a summary of the changes underlying and included in calwf3 version 3.3. WFC3 ISR 2016-01 and WFC3 ISR 2016-02 provide a reference guide and cookbook for calwf3 version 3.3, respectively. WFC3 ISR 2016-03 provides chip-dependent inverse sensitivities. WFC3 ISR 2016-04 describes the creation of chip-dependent flat fields, which included a correction for the effect of the crosshatch pattern in the UV filters on sensitivity calibrations, based on analysis presented in WFC3 ISR 2015-18 and described in Section 5.4.3. WFC3 ISR 2016-07 documents the changes in WFC3/UVIS component files used by synphot (Lim et al. 2016) and pysynphot (Lim, Diaz, & Laidler 2015) to simulate HST photometric measurements. Note that pysynphot was retired in 2022, and the currently-supported counterpart to synphot is stsynphot (see Lim et al. 2016 for observatory-specific functionality).

In late 2020, a new set of UVIS (as well as IR) inverse sensitivities, i.e. zeropoints, were derived to incorporate improvements to the HST CALSPEC models (Bohlin et al. 2020) as well as an increase in the Vega reference flux. As a consequence of the model adjustments, the standard white dwarf fluxes increase by ~2% for wavelengths in the range 0.15 - 0.4 micron and ~1.5% in the range 0.4 - 1.6 micron covered by both detectors. The updated UVIS calibration includes the new models as well as a correction for the time-dependent detector sensitivity (changes of ~ 0.1 - 0.2 %/year)  derived from over 10 years of monitoring data (WFC3 ISR 2021-04). The IR inverse sensitivity changes installed in late 2020 were primarily the result of the new models, although the new zeropoints also incorporate new IR flat fields in the calibration of the flux standards (WFC3 ISR 2020-10). For more information on the updated photometric calibration, see Calamida et al. (2022). The updated IR 'pixel to pixel' flats, computed by stacking deep exposures acquired over 10 years, correct for spatial sensitivity residuals up to 0.5% in the center of the detector and up to 2% at the edges (WFC3 ISR 2021-01). A new set of 'delta' flats available for six filters (F098M, F105W, F110W, F125W, F140W, and F160W) correct for low-sensitivity artifacts known as 'blobs', as new blobs appear over time (WFC3 ISR 2021-10).

In December 2023, calwf3 version 3.7.1 was released, which adopted a new method of UVIS full well saturation flagging that uses a two-dimensional array of pixel threshold values instead of a single scalar threshold per amplifier quadrant. While this change does not meaningfully affect science data (and thus users should not need to redownload/reprocess data accessed before the pipeline update), it does lay the foundation for the future delivery of a spatially-dependent saturation map that accounts for the variation in full well depth across the detectors (Section 5.4.5, specifically Figure 5.6). See the release notes for hstcal version 2.7.6 and WFC3 ISR 2023-08 for more information regarding this change to the pipeline.

E.2.2 Processing Data

Science data taken by WFC3 are received from the Space Telescope Data Capture Facility and sent to the STScI data processing pipeline, where the data are unpacked, keyword values are extracted from the telemetry stream, and the science data reformatted and repackaged into uncalibrated FITS files by the generic conversion process. All WFC3 science data products are two-dimensional images that are stored in FITS image-extension files. Like ACS and STIS images, WFC3 UVIS channel exposures are stored as triplets of FITS image extensions, consisting of science (SCI), error (ERR), and data quality (DQ) arrays. There is one triplet of image extensions for each CCD chip used in an exposure. Full-frame exposures, using both chips, therefore have two triplets of SCI, ERR, and DQ extensions in a single FITS file. UVIS subarray exposures, which use only one CCD chip, have a single triplet of extensions in their FITS files. After the new Enhanced Pipeline Products code has been used to reprocess data, there will be a number of extensions past 6 related to the WCS (world coordinate system) and the astrometric solutions. Description of those products and their use is at https://outerspace.stsci.edu/pages/viewpage.action?spaceKey=HAdP&title=Improvements+in+HST+Astrometry.

WFC3 IR channel exposures use the NICMOS file structure, which are quintuplets of FITS image extensions, consisting of science (SCI), error (ERR), data quality (DQ), number of samples (SAMP), and integration time (TIME) arrays. There is one quintuplet of extensions for each of the non-destructive detector readouts that make up an IR exposure. Using the maximum number of readouts (16: from NSAMP=15 plus the zeroth read) in an IR exposure therefore results in a single FITS file containing a total of 80 image extensions.

E.2.3 Calibrating Data

The uncalibrated ("RAW") FITS files are processed through calwf3, the software task that calibrates the data for individual exposures, producing calibrated FITS files. Exposures that are obtained as part of an associated set, such as dithered images, have calwf3 calibration applied to the individual exposures before being processed as a set for the purpose of image combination. All calibrated images will be processed further with the DrizzlePac software for the purpose of removing geometric distortions from individual exposures and for combining associated exposures.

The FITS file name suffixes given to WFC3 raw and calibrated data products are described in Table E.2 and closely mimic the suffixes used by ACS and NICMOS. The initial input files to calwf3 are the RAW files from generic conversion and the association (ASN) table, if applicable, for the complete observation set. 

Most WFC3/UVIS RAW images first go through pixel-based CTE correction, producing a temporary, CTE-corrected RAW file with the suffix “RAC_TMP”. The RAC_TMP and original RAW files have the same calibration steps applied, producing two sets of final calibrated products (one uncorrected and one corrected for CTE). For WFC3/UVIS images, a temporary file, with the suffix “BLV_TMP” (BLC_TMP for CTE products), is created by calwf3 once bias levels have been subtracted and the overscan regions trimmed. This file is renamed using the "FLT" ("FLC" for CTE-corrected products) suffix after the remaining standard calibrations (dark subtraction, flat fielding, etc.) have been completed. For exposures taken as part of a UVIS CR-SPLIT or REPEAT-OBS set, a parallel set of processing is performed, using the BLV_TMP/BLC_TMP files as input to an image combination and cosmic ray rejection routine. The resulting CR-combined image, with a temporary file name suffix of “CRJ_TMP” (“CRC_TMP” for CTE products), then receives the remaining standard calibrations, after which it is renamed using the “CRJ” (“CRC” for CTE products) suffix.

Table E.2: WFC3 File Name Suffixes.

Processing of WFC3/IR exposures results in an intermediate MULTIACCUM ("IMA") file, which is a file that has had all calibrations applied to all of the individual readouts of the IR exposure if the *CORR keywords were populated with PERFORM in the raw file (excluding DRIZCORR). This includes calibrations such as dark subtraction, linearity correction, and flat fielding (see below for list of calibration steps). A final step in calwf3 processing of WFC3/IR exposures produces a combined image from the individual readouts, which is stored in an FLT output product file. Note: we recommend observers inspect not only their FLT files but the IMA products as well.

calwf3 performs the following basic science data calibrations:

• Pixel-based CTE correction (UVIS only)
• Bad pixel flagging
  
• Bias level subtraction (UVIS); Reference pixel subtraction (IR)
  
• Bias image subtraction (UVIS); Zero-read subtraction (IR)
  
• Dark current subtraction
  
• Post-Flash subtraction
  
• Non-linearity correction
  
• Flat-field correction and gain calibration
  
• Shutter shading correction (UVIS only)
  
• Up-the-ramp fitting (IR only)
  
• Photometric calibration
  
• CR-SPLIT/REPEAT-OBS image combination
  

As noted in the list above, the details of some calibration steps differ for UVIS and IR exposures, while others do not apply at all. The process of bias subtraction, in particular, differs for UVIS and IR exposures. The UVIS channel CCDs include regions of overscan, which are used for measuring and subtracting the overall bias level from each CCD exposure. A bias reference image is also subtracted from each science exposure to remove spatial variations in the bias. For IR exposures, the reference pixels located around the perimeter of the detector are used to track and remove changes in the overall bias level between readouts, while the image from the initial (“zeroth”) readout of the exposure is subtracted from all subsequent readouts to remove spatial bias structure.

UVIS shutter shading correction is in principle only necessary for very short duration exposures. Note, however, that testing has shown that the shading amounts to only a 0.2-0.3% variation across the field and therefore this step is not applied.

Up-the-ramp fitting is applied to IR exposures to determine a final signal rate for each pixel in the image. This process not only determines the best-fit rate from the individual readouts of the exposure, but also detects and removes effects due to cosmic-ray hits. This process is also capable of recovering a useful signal for pixels that go into saturation during the exposure by using only the non-saturated readouts to compute the fit.

WFC3 grism observations are handled in a special way by the pipeline. Grism observations require a special flat-fielding procedure, where the flat-field value for each pixel is based on the wavelength of the detected signal. Processing of grism images in calwf3 therefore uses an “identity” flat-field reference image (an image filled with values of 1.0 at each pixel), which allows for the gain calibration part of the flat-fielding step to still be applied without actually flat-fielding the science image. Separate software packages are used to extract and calibrate one-dimensional spectra from WFC3 grism exposures (see Section 8.5). These packages are used to locate and extract spectra of individual sources from calibrated images and perform wavelength calibration, background subtraction, flat fielding, and absolute flux calibration for the extracted spectra.

Table E.3 shows the values assigned to pixels in the DQ arrays of calibrated images, which indicate anomalous conditions and are frequently used in downstream processes to reject a pixel value. If more than one data quality condition applies to a pixel, the sum of the values is used. Note that some flag values have different meanings for UVIS and IR images.

Table E.3: WFC3 Data Quality Flags.