3.1 The calwf3 Data Processing Pipeline

«Previous Next»

The main executable which processes data taken with either the UVIS or IR detectors of the WFC3 instrument onboard the Hubble Space Telescope is called calwf3. The code is organized into subroutines that are called by the calwf3 executable. The subroutines may be called independently if users desire specialized processing for their dataset. The subroutines used for processing UVIS images are called w3cte, wf3ccd, and wf32d. The main subroutine used for processing IR images is wf3ir. The wf3rej subroutine is shared between the UVIS and IR pipelines and is used for combining CR-SPLIT or REPEAT-OBS image sets. Figure 3.1 shows the flow diagram for the UVIS pipeline as a whole, while Figure 3.3 contains the flow for the IR pipeline.

A detailed description of the improvements in calwf3 v3.3, which is more generally referred to as the UVIS2.0 update, can be found in the Feb 2016 STAN. Each WFC3 image is calibrated with reference files particular to the observing mode used. Currently, raw images and telemetry files are processed and calibrated, using the most up-to-date reference files, parameters, and software (see Section 1.1.1 of the Introduction to the HST Data Handbooks). Then the data products, as well as the raw data are stored by MAST (Mikulski Archive for Space Telescopes).  As of August 2021, the version of calwf3 is 3.6.2; information concerning the software releases are on the STScI GIT repository. 

During automatic pipeline processing by the STScI archive, Astrodrizzle follows calwf3. All calibrated images are corrected for geometric distortion and associated sets of dithered images are combined into a single product.

3.1.1 Where to Find calwf3

calwf3 is part of the HSTCAL package, which can be downloaded separately from its GIT repository in the Space Telescope Area. Its binaries are also installed along with the STScI distributed package Astroconda. 

3.1.2 Running calwf3

calwf3 can be run on a single input raw file or an asn table listing the members of an association. When processing an association, it retrieves calibration switch and reference file keyword settings from the first image listed in the association (asn) table (see Section 2.1.2). calwf3 does not accept a user-defined list of input images on the command line (e.g. *raw.fits to process all raw files in the current directory). The wf3ccd, wf32d, wf3cte, wf3ir tasks on the other hand, will accept such user-defined input file lists, but they will not accept an association table (asn) as input.

Below are some basic examples illustrating how to run calwf3 in python and directly from a terminal. Although pure python commands may still be used in a PyRAF session, please note that IRAF/PyRAF have been deprecated and support is no longer available. Users are strongly encouraged to switch to python; more advanced usage examples are available in Section 3.5.

Running calwf3 from a python terminal using wfc3tools

from wfc3tools import calwf3
calwf3(filename)

Running many files at the same time with python

The recommended method for running calwf3 in batch mode is to use Python and the wfc3tools package in the Astroconda distribution.
For example:

from wfc3tools import calwf3
from glob import glob

for fits in glob('i*_raw.fits'):
    calwf3(fits)

Displaying output from calwf3 in a Jupyter Notebook

When calling calwf3 from a Jupyter notebook using the calwf3() function from wfc3tools, informational text output from the underlying calwf3.e program will be passed through the print function as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to calwf3. As output is read from the underlying program, the calwf3 Python wrapper will call log_func with the contents of each line. (print is an obvious choice for a log function, but this also provides a way to connect calwf3 to the Python logging system by passing the logging.debug function or similar). If log_func=None is passed, informational text output from the underlying program will be ignored, but the program’s exit code will still be checked for successful completion.

Command line options for the calwf3 executable

calwf3 can also be called directly from the OS command line by supplying the executable calwf3.e with an input file and a list of options. This is the same executable that the wfc3tools package calls.

calwf3.e -vts iaa012wdq_raw.fits
input: str
   Name of input files
       a single filename (``iaa012wdq_raw.fits``)
       the filename of an ASN table (``*_asn.fits``)
   -t : print a detailed time stamp
   -s : save temporary files
   -d : print optional debugging statements
   -1 : suppress the OpemMP parallel processing for the
UVIS CTE correction
   -v : Print verbose time stamps and information
   -q : Print messages only to trailer file

Running many files at the same time from the command line

The command line executable only accepts one file at a time, but you can use operating system tools like awk or xargs to process everything in a directory:

ls *raw.fits | awk '{print "calwf3.e",$1}' | csh

or alternatively:

ls *raw.fits | xargs -I % calwf3.e %

3.1.3 Keyword Usage

calwf3 processing is controlled by the values of keywords in the input image headers. Certain keywords, referred to as calibration switches, are used to control which calibration steps are performed. Reference file keywords indicate which reference files to use in the various calibration steps. Users who wish to perform custom reprocessing of their data may change the values of these keywords in the _raw FITS file primary headers and then rerun the modified file through calwf3.

Other keyword values record instrument and detector parameters that are used in the calibration and some record information that is computed or derived during calibration. Table 3.1 provides a summary of these keywords used by calwf3, specifying whether they function as input or output to the task(s) listed in column 2. For a definition of each keyword see Tables 2.8, 2.9, and 2.10. 

Table 3.1: WFC3 keywords used with calwf3  

KeywordTask(s)I/OHeaderSample of Possible Values
APERTURE

wf3ccd,wf32d, wf3ir

Input

Primary

UVIS1, UVIS2, UVIS1-FIX, UVIS2-FIX, UVIS, IR, ...

ATODGNA,ATODGNB, ATODGNC,ATODGND

wf3ccd,wf32d, wf3ir, wf3rej

Output, Input

Primary

1.56, 2.26

BIASLVA,BIASLEVB, BIASLEVC,BIASLEVD

wf3ccd

Output

Primary

2502.23, 2605.48

BINAXIS1,BINAXIS2

wf3ccd,wf32d, wf3ir

Input

SCI

1, 2, 3

CAL_VER

wf3ccd,wf32d, wf3ir

Output

Primary

2.1 (15-May-2010)

CCDAMP

wf3ccd,wf32d, wf3rej

Input

Primary

ABCD, AC, BD, A, B, C, D

CCDCHIP

wf3ccd,wf32d, wf3rej

Input

SCI

1, 2

CCDGAIN

wf3ccd,wf32d, wf3ir, wf3rej

Input

Primary

1.5, 2.5

CCDOFSTA,CCDOFSTB
CCDOFSTC,CCDOFSTD

wf3ccd,wf32d, wf3rej

Input

Primary

3

DETECTOR

wf3ccd,wf32d, wf3ir, wf3rej

Input

Primary

UVIS, IR

EXPSTART, EXPEND, EXPTIME

wf3ccd,wf32d, wf3ir, wf3rej

Input

Primary

51475.159

FILTER

wf3ccd,wf32d, wf3ir, wf3rej

Input

Primary

F606W, F160W, G102

FLASHDUR,FLASHSTA

wf3ccd,wf32d, wf3rej

Input

Primary

0.2, SUCCESSFUL

LTM1_1,LTM2_2

wf3ccd,wf3ir

Input

SCI, ERR, DQ

1.0, 0.5, 0.333

LTV1,LTV2

wf3ccd,wf32d, wf3ir

Input

SCI, ERR, DQ

0.0, 25.0, 19.0, 5.0

MEANBLEV

wf3ccd,wf3ir

Output

SCI

2554.763, 14201.36

MEANDARK

wf32d, wf3ir

Output

SCI

3.20642E-01

MEANFLSH

wf3ccd

Output

SCI

N/A

NEXTEND

wf3ccd,wf32d, wf3ir, wf3rej

Input

Primary

3, 6, 80

NSAMP

wf3ir

Input

Primary

2-16

OBSTYPE

wf3ccd,wf32d, wf3ir, wf3rej

Input

Primary

Imaging, Spectroscopic

PHOTMODE

wf32d, wf3ir

Output

SCI, Primary

“WFC3 UVIS1 F606W”

PHOTFLAM

wf3ir

Output

SCI, Primary

Inverse sensitivity (erg cm-2 A-1 e-1)

PHOTFNU

wf3ir

Output

Primary

Inverse sensitivity (Jy sec e-1)

PHOTZPT

wf32d, wf3ir

Output

SCI, Primary

ST magnitude zero point

PHOTPLAM

wf32d, wf3ir

Output

SCI, Primary

Pivot wavelength

PHOTBW

wf32d, wf3ir

Output

SCI, Primary

rms bandwidth of filter plus detector

PHTFLAM1

wf32d

Output

SCI, Primary

Chip1 Inverse Sensitivity for infinite aperture erg cm-2 A-1 e-1

PHTFLAM2

wf32d

Output

SCI, Primary

Chip1 Inverse Sensitivity for infinite aperture erg cm-2 A-1 e-1

PHTRATIO

wf32d

Output

SCI, Primary

PHTFLAM2/PHTFLAM1 ratio

READNSEA, READNSEB,
READNSEC, READNSED

wf3ccd,wf32d, wf3ir, wf3rej

Output, Input

Primary

calibrated read noise for amplifier A, B, C, and D (electrons)

ROOTNAME

wf3ccd,wf32d, wf3ir, wf3rej

Input

Primary

rootname of the observation set

SAMP_SEQ

wf3ir

Input

Primary

RAPID, SPARS25,...

SAMPTIME

wf3ir

Input

SCI

Total integration time (sec)

SAMPZERO

wf3ir

Input

Primary

Sample time of MULTIACCUM zeroth read (sec)

SDQFLAGS

wf3ccd,wf32d, wf3ir

Input

SCI

Serious data quality flags considered “bad” by calwf3

SUBARRAY

wf3ccd,wf32d, wf3ir

Input

Primary

T, F

SUBTYPE

wf3ir

Input

Primary

FULLIMAG, SQ64SUB

TDFTRANS

wf3ir

Input

SCI

0, 1

IMAGE STATISTICS





NGOODPIX

wf32d, wf3ir

Output

SCI, ERR

number of good pixels

GOODMIN, GOODMAX, GOODMEAN

wf32d, wf3ir

Output

SCI, ERR

min, max and mean values of good pixels (electrons)

SNRMIN, SNRMAX, SNRMEAN

wf32d, wf3ir

Output

SCI

min, max, and mean signal to noise of good pixels

CR-REJ PARAMETERS





BADINPDQ

wf3rej

Output

Primary

data quality flag used for rejection

CRMASK

wf3rej

Output

Primary

T, F

CRRADIUS

wf3rej

Output

Primary

3.0

CRSIGMAS

wf3rej

Output

Primary

6.5, 5.5, 4.5

CRTHRESH

wf3rej

Output

Primary

rejection propagation threshold

EXPSTART, EXPEND, EXPTIME, TEXPTIME

wf3rej

Output

Primary

exposure start, and end times (modified Julian date)

EXPTIME, TEXPTIME

wf3rej

Output

Primary

total exposure duration (seconds)--calculated

INITGUES

wf3rej

Output

Primary

minimum, mode

MEANEXP

wf3rej

Output

Primary

Average exposure time (sec) for each image

NCOMBINE

wf3rej

Output

SCI

number of image sets combined during CR rejection

REJ_RATE

wf3rej

Output

Primary

rate at which pixels are affected by cosmic rays

SCALENSE

wf3rej

Output

Primary

Multiplicative term (in percent) for the noise model

SKYSUB

wf3rej

Output

Primary

Sky value subtracted (mode, none)

SKYSUM

wf3rej

Output

Primary

Sky level from the sum of all constituent images

3.1.4 Using CRDS To Update Reference Files

The HST Calibration Reference Data System (CRDS) is the reference file management software used by STScI for organizing and assigning reference files to datasets. Users can query CRDS to get the best references files for their data at hand; the manual recalibration Jupyter notebook referenced in Section 3.5.2 provides an example of how to perform such a 'bestref' query. The WFC3 team recommends that the most up-to-date reference files be used for re-processing. 

Users should ensure that the same version of all reference files and the pipeline software is used to analyze their entire dataset. This is particularly important for data taken at multiple epochs. Retrieving data from MAST will provide the most up-to-date calibration (software and reference files) available at the time of retrieval. 

For example, if the data are retrieved from MAST at different times (e.g., after the execution of each visit), it may be possible to observe systematic differences, due to changes in the reference files, or more generally, to the whole data processing flow. Thus it is recommended that the whole dataset pertaining to the given scientific investigation is retrieved simultaneously. Alternatively, the data need to be reprocessed off-line (starting from the unprocessed raw files) using a self-consistent configuration of pipeline and reference files. Rarely there may be an instrument mode for which there is no calibration data. In these cases there will be a placeholder reference file in CRDS that is filled with zeros or ones, as appropriate. Such reference files are identified by having their header keyword PEDIGREE set to DUMMY. When calwf3 encounters one of these reference files it will automatically skip the calibration step for which the file is used (e.g., DARKCORR will be set to the value SKIPPED if the DARKFILE is dummy).

«Previous Next»