E.2 The STScI Reduction and Calibration Pipeline
In this section of the appendix, we summarize the basic reductions and calibrations that are performed in the STScI WFC3 calibration pipeline, calwf3
. The material in this appendix is intended to provide only enough background to develop robust observing proposals. The WFC3 Data Handbook provides more detailed information needed for analyzing your data.
E.2.1 Selected Pipeline History
In February 2016, the pipeline began to process WFC3 data with calwf3
version 3.3, which incorporates two fundamental changes to the way UVIS exposures are calibrated and corrected. First, calwf3
applies pixel-based CTE (Charge Transfer Efficiency) corrections. (Note that calwf3
version 3.4, which added CTE corrections for subarray apertures, became operational in October 2016.) Second, photometric calibrations are determined and applied independently for each CCD chip. See WFC3 STAN issue 22 for a summary of the changes underlying and included in calwf3
version 3.3. WFC3 ISR 2016-01 and WFC3 ISR 2016-02 provide a reference guide and cookbook for calwf3
version 3.3, respectively. WFC3 ISR 2016-03 provides chip-dependent inverse sensitivities. WFC3 ISR 2016-04 describes the creation of chip-dependent flat fields, which included a correction for the effect of the crosshatch pattern in the UV filters on sensitivity calibrations, based on analysis presented in WFC3 ISR 2015-18 and described in Section 5.4.3. WFC3 ISR 2016-07 documents the changes in WFC3/UVIS component files used by synphot
(Lim et al. 2016) and pysynphot
(Lim, Diaz, & Laidler 2015) to simulate HST photometric measurements. Note that pysynphot
was retired in 2022, and the currently-supported counterpart to synphot
is stsynphot
(see Lim et al. 2016 for observatory-specific functionality).
In late 2020, a new set of UVIS (as well as IR) inverse sensitivities, i.e. zeropoints, were derived to incorporate improvements to the HST CALSPEC models (Bohlin et al. 2020) as well as an increase in the Vega reference flux. As a consequence of the model adjustments, the standard white dwarf fluxes increase by ~2% for wavelengths in the range 0.15 - 0.4 micron and ~1.5% in the range 0.4 - 1.6 micron covered by both detectors. The updated UVIS calibration includes the new models as well as a correction for the time-dependent detector sensitivity (changes of ~ 0.1 - 0.2 %/year) derived from over 10 years of monitoring data (WFC3 ISR 2021-04). The IR inverse sensitivity changes installed in late 2020 were primarily the result of the new models, although the new zeropoints also incorporate new IR flat fields in the calibration of the flux standards (WFC3 ISR 2020-10). For more information on the updated photometric calibration, see Calamida et al. (2022). The updated IR 'pixel to pixel' flats, computed by stacking deep exposures acquired over 10 years, correct for spatial sensitivity residuals up to 0.5% in the center of the detector and up to 2% at the edges (WFC3 ISR 2021-01). A new set of 'delta' flats available for six filters (F098M, F105W, F110W, F125W, F140W, and F160W) correct for low-sensitivity artifacts known as 'blobs', as new blobs appear over time (WFC3 ISR 2021-10).
In December 2023, calwf3
version 3.7.1 was released, which adopted a new method of UVIS full well saturation flagging that uses a two-dimensional array of pixel threshold values instead of a single scalar threshold per amplifier quadrant. While this change does not meaningfully affect science data (and thus users should not need to redownload/reprocess data accessed before the pipeline update), it does lay the foundation for the future delivery of a spatially-dependent saturation map that accounts for the variation in full well depth across the detectors (Section 5.4.5, specifically Figure 5.6). See the release notes for hstcal version 2.7.6 and WFC3 ISR 2023-08 for more information regarding this change to the pipeline.
E.2.2 Processing Data
Science data taken by WFC3 are received from the Space Telescope Data Capture Facility and sent to the STScI data processing pipeline, where the data are unpacked, keyword values are extracted from the telemetry stream, and the science data reformatted and repackaged into uncalibrated FITS files by the generic conversion process. All WFC3 science data products are two-dimensional images that are stored in FITS image-extension files. Like ACS and STIS images, WFC3 UVIS channel exposures are stored as triplets of FITS image extensions, consisting of science (SCI), error (ERR), and data quality (DQ) arrays. There is one triplet of image extensions for each CCD chip used in an exposure. Full-frame exposures, using both chips, therefore have two triplets of SCI, ERR, and DQ extensions in a single FITS file. UVIS subarray exposures, which use only one CCD chip, have a single triplet of extensions in their FITS files. After the new Enhanced Pipeline Products code has been used to reprocess data, there will be a number of extensions past 6 related to the WCS (world coordinate system) and the astrometric solutions. Description of those products and their use is at https://outerspace.stsci.edu/pages/viewpage.action?spaceKey=HAdP&title=Improvements+in+HST+Astrometry.
WFC3 IR channel exposures use the NICMOS file structure, which are quintuplets of FITS image extensions, consisting of science (SCI), error (ERR), data quality (DQ), number of samples (SAMP), and integration time (TIME) arrays. There is one quintuplet of extensions for each of the non-destructive detector readouts that make up an IR exposure. Using the maximum number of readouts (16: from NSAMP=15 plus the zeroth read) in an IR exposure therefore results in a single FITS file containing a total of 80 image extensions.
E.2.3 Calibrating Data
The uncalibrated ("RAW") FITS files are processed through calwf3
, the software task that calibrates the data for individual exposures, producing calibrated FITS files. Exposures that are obtained as part of an associated set, such as dithered images, have calwf3
calibration applied to the individual exposures before being processed as a set for the purpose of image combination. All calibrated images will be processed further with the DrizzlePac software for the purpose of removing geometric distortions from individual exposures and for combining associated exposures.
The FITS file name suffixes given to WFC3 raw and calibrated data products are described in Table E.2 and closely mimic the suffixes used by ACS and NICMOS. The initial input files to calwf3
are the RAW files from generic conversion and the association (ASN) table, if applicable, for the complete observation set.
Most WFC3/UVIS RAW images first go through pixel-based CTE correction, producing a temporary, CTE-corrected RAW file with the suffix “RAC_TMP”. The RAC_TMP and original RAW files have the same calibration steps applied, producing two sets of final calibrated products (one uncorrected and one corrected for CTE). For WFC3/UVIS images, a temporary file, with the suffix “BLV_TMP” (BLC_TMP for CTE products), is created by calwf3
once bias levels have been subtracted and the overscan regions trimmed. This file is renamed using the "FLT" ("FLC" for CTE-corrected products) suffix after the remaining standard calibrations (dark subtraction, flat fielding, etc.) have been completed. For exposures taken as part of a UVIS CR-SPLIT or REPEAT-OBS set, a parallel set of processing is performed, using the BLV_TMP/BLC_TMP files as input to an image combination and cosmic ray rejection routine. The resulting CR-combined image, with a temporary file name suffix of “CRJ_TMP” (“CRC_TMP” for CTE products), then receives the remaining standard calibrations, after which it is renamed using the “CRJ” (“CRC” for CTE products) suffix.
Table E.2: WFC3 File Name Suffixes.
File Suffix | Description | Units |
_RAW (_RAC_TMP) | Raw data (with CTE correction) | DN |
_ASN | Association file for observation set | |
_SPT | Telemetry and engineering data | |
_TRL | Trailer file with processing log | |
_BLV_TMP (_BLC_TMP) | Bias subtracted individual UVIS exposure (with CTE correction) | DN |
_CRJ_TMP (_CRC_TMP) | Uncalibrated, CR-rejected combined UVIS image (with CTE correction) | DN |
_IMA | Calibrated intermediate IR exposure | e–/s |
_FLT (_FLC) | Calibrated individual exposure (with CTE correction) | e– (UVIS) |
_CRJ (_CRC) | Calibrated, CR-rejected, combined UVIS image (with CTE correction) | e– |
_DRZ | Calibrated, geometrically-corrected, dither-combined image | e–/s |
Processing of WFC3/IR exposures results in an intermediate MULTIACCUM ("IMA") file, which is a file that has had all calibrations applied to all of the individual readouts of the IR exposure if the *CORR keywords were populated with PERFORM in the raw file (excluding DRIZCORR). This includes calibrations such as dark subtraction, linearity correction, and flat fielding (see below for list of calibration steps). A final step in calwf3
processing of WFC3/IR exposures produces a combined image from the individual readouts, which is stored in an FLT output product file. Note: we recommend observers inspect not only their FLT files but the IMA products as well.
calwf3
performs the following basic science data calibrations:
- Pixel-based CTE correction (UVIS only)
- Bad pixel flagging
- Bias level subtraction (UVIS); Reference pixel subtraction (IR)
- Bias image subtraction (UVIS); Zero-read subtraction (IR)
- Dark current subtraction
- Post-Flash subtraction
- Non-linearity correction
- Flat-field correction and gain calibration
- Shutter shading correction (UVIS only)
- Up-the-ramp fitting (IR only)
- Photometric calibration
- CR-SPLIT/REPEAT-OBS image combination
As noted in the list above, the details of some calibration steps differ for UVIS and IR exposures, while others do not apply at all. The process of bias subtraction, in particular, differs for UVIS and IR exposures. The UVIS channel CCDs include regions of overscan, which are used for measuring and subtracting the overall bias level from each CCD exposure. A bias reference image is also subtracted from each science exposure to remove spatial variations in the bias. For IR exposures, the reference pixels located around the perimeter of the detector are used to track and remove changes in the overall bias level between readouts, while the image from the initial (“zeroth”) readout of the exposure is subtracted from all subsequent readouts to remove spatial bias structure.
UVIS shutter shading correction is in principle only necessary for very short duration exposures. Note, however, that testing has shown that the shading amounts to only a 0.2-0.3% variation across the field and therefore this step is not applied.
Up-the-ramp fitting is applied to IR exposures to determine a final signal rate for each pixel in the image. This process not only determines the best-fit rate from the individual readouts of the exposure, but also detects and removes effects due to cosmic-ray hits. This process is also capable of recovering a useful signal for pixels that go into saturation during the exposure by using only the non-saturated readouts to compute the fit.
WFC3 grism observations are handled in a special way by the pipeline. Grism observations require a special flat-fielding procedure, where the flat-field value for each pixel is based on the wavelength of the detected signal. Processing of grism images in calwf3
therefore uses an “identity” flat-field reference image (an image filled with values of 1.0 at each pixel), which allows for the gain calibration part of the flat-fielding step to still be applied without actually flat-fielding the science image. A separate software package, hstaxe
(available from the Github repository), is used to extract and calibrate one-dimensional spectra from WFC3 grism exposures (see Section 8.5). The hstaxe
software is used to locate and extract spectra of individual sources from calibrated images and performs wavelength calibration, background subtraction, flat fielding, and absolute flux calibration for the extracted spectra. It is a descendant of the software package aXe
, which was developed at ST-ECF and previously used for processing NICMOS and ACS spectral observations.
Table E.3 shows the values assigned to pixels in the DQ arrays of calibrated images, which indicate anomalous conditions and are frequently used in downstream processes to reject a pixel value. If more than one data quality condition applies to a pixel, the sum of the values is used. Note that some flag values have different meanings for UVIS and IR images.
Table E.3: WFC3 Data Quality Flags.
FLAG Value | Data Quality Condition | |
UVIS | IR | |
0 | OK | OK |
1 | Reed-Solomon decoding error | Reed-Solomon decoding error |
2 | Data replaced by fill value | Data replaced by fill value |
4 | Bad detector pixel | Bad detector pixel |
8 | (unused) | Unstable in zero-read |
16 | Hot pixel | Hot pixel |
32 | Unstable pixel | Unstable pixel |
64 | Warm pixel | (Obsolete: Warm pixel) |
128 | Bad pixel in bias | Bad reference pixel |
256 | Full- well saturation | Full-well saturation |
512 | Bad or uncertain flat value | Bad or uncertain flat value |
1024 | Charge trap (sink pixel) | (unused) |
2048 | A-to-D saturation | Signal in zero-read |
4096 | Cosmic ray detected by AstroDrizzle | Cosmic ray detected by AstroDrizzle |
8192 | Cosmic ray detected during CR-SPLIT or REPEAT-OBS combination | Cosmic ray detected during up-the-ramp fitting |
16384 | Pixel affected by ghost or crosstalk (not used) | Pixel affected by crosstalk (not used) |
-
WFC3 Instrument Handbook
- • Acknowledgments
- Chapter 1: Introduction to WFC3
- Chapter 2: WFC3 Instrument Description
- Chapter 3: Choosing the Optimum HST Instrument
- Chapter 4: Designing a Phase I WFC3 Proposal
- Chapter 5: WFC3 Detector Characteristics and Performance
-
Chapter 6: UVIS Imaging with WFC3
- • 6.1 WFC3 UVIS Imaging
- • 6.2 Specifying a UVIS Observation
- • 6.3 UVIS Channel Characteristics
- • 6.4 UVIS Field Geometry
- • 6.5 UVIS Spectral Elements
- • 6.6 UVIS Optical Performance
- • 6.7 UVIS Exposure and Readout
- • 6.8 UVIS Sensitivity
- • 6.9 Charge Transfer Efficiency
- • 6.10 Other Considerations for UVIS Imaging
- • 6.11 UVIS Observing Strategies
- Chapter 7: IR Imaging with WFC3
- Chapter 8: Slitless Spectroscopy with WFC3
-
Chapter 9: WFC3 Exposure-Time Calculation
- • 9.1 Overview
- • 9.2 The WFC3 Exposure Time Calculator - ETC
- • 9.3 Calculating Sensitivities from Tabulated Data
- • 9.4 Count Rates: Imaging
- • 9.5 Count Rates: Slitless Spectroscopy
- • 9.6 Estimating Exposure Times
- • 9.7 Sky Background
- • 9.8 Interstellar Extinction
- • 9.9 Exposure-Time Calculation Examples
- Chapter 10: Overheads and Orbit Time Determinations
-
Appendix A: WFC3 Filter Throughputs
- • A.1 Introduction
-
A.2 Throughputs and Signal-to-Noise Ratio Data
- • UVIS F200LP
- • UVIS F218W
- • UVIS F225W
- • UVIS F275W
- • UVIS F280N
- • UVIS F300X
- • UVIS F336W
- • UVIS F343N
- • UVIS F350LP
- • UVIS F373N
- • UVIS F390M
- • UVIS F390W
- • UVIS F395N
- • UVIS F410M
- • UVIS F438W
- • UVIS F467M
- • UVIS F469N
- • UVIS F475W
- • UVIS F475X
- • UVIS F487N
- • UVIS F502N
- • UVIS F547M
- • UVIS F555W
- • UVIS F600LP
- • UVIS F606W
- • UVIS F621M
- • UVIS F625W
- • UVIS F631N
- • UVIS F645N
- • UVIS F656N
- • UVIS F657N
- • UVIS F658N
- • UVIS F665N
- • UVIS F673N
- • UVIS F680N
- • UVIS F689M
- • UVIS F763M
- • UVIS F775W
- • UVIS F814W
- • UVIS F845M
- • UVIS F850LP
- • UVIS F953N
- • UVIS FQ232N
- • UVIS FQ243N
- • UVIS FQ378N
- • UVIS FQ387N
- • UVIS FQ422M
- • UVIS FQ436N
- • UVIS FQ437N
- • UVIS FQ492N
- • UVIS FQ508N
- • UVIS FQ575N
- • UVIS FQ619N
- • UVIS FQ634N
- • UVIS FQ672N
- • UVIS FQ674N
- • UVIS FQ727N
- • UVIS FQ750N
- • UVIS FQ889N
- • UVIS FQ906N
- • UVIS FQ924N
- • UVIS FQ937N
- • IR F098M
- • IR F105W
- • IR F110W
- • IR F125W
- • IR F126N
- • IR F127M
- • IR F128N
- • IR F130N
- • IR F132N
- • IR F139M
- • IR F140W
- • IR F153M
- • IR F160W
- • IR F164N
- • IR F167N
- Appendix B: Geometric Distortion
- Appendix C: Dithering and Mosaicking
- Appendix D: Bright-Object Constraints and Image Persistence
-
Appendix E: Reduction and Calibration of WFC3 Data
- • E.1 Overview
- • E.2 The STScI Reduction and Calibration Pipeline
- • E.3 The SMOV Calibration Plan
- • E.4 The Cycle 17 Calibration Plan
- • E.5 The Cycle 18 Calibration Plan
- • E.6 The Cycle 19 Calibration Plan
- • E.7 The Cycle 20 Calibration Plan
- • E.8 The Cycle 21 Calibration Plan
- • E.9 The Cycle 22 Calibration Plan
- • E.10 The Cycle 23 Calibration Plan
- • E.11 The Cycle 24 Calibration Plan
- • E.12 The Cycle 25 Calibration Plan
- • E.13 The Cycle 26 Calibration Plan
- • E.14 The Cycle 27 Calibration Plan
- • E.15 The Cycle 28 Calibration Plan
- • E.16 The Cycle 29 Calibration Plan
- • E.17 The Cycle 30 Calibration Plan
- • E.18 The Cycle 31 Calibration Plan
- • Glossary