3.1 The calwf3 Data Processing Pipeline

«Previous Next»

The main calibration pipeline which processes data taken with either the UVIS or IR detectors of the WFC3 instrument onboard the Hubble Space Telescope (HST) is called calwf3. It is a part of the software package (HSTCAL) that STScI uses for calibration of all HST science data. calwf3 is an executable written in C that can be called via the command line interface, or run by the wfc3tools package in Python. The pipeline is organized into subroutines that are called by the calwf3 executable, but can be invoked as standalone executables if users desire specialized processing of their dataset. The subroutines used for processing UVIS images are called w3cte, wf3ccd, and wf32d, while the main subroutine used for processing IR images is wf3ir. A subroutine, wf3rej, is shared between the UVIS and IR pipelines and is used for combining datasets obtained using CR-SPLIT or REPEAT-OBS modes. Figure 3.1 shows the flow diagram for the UVIS pipeline as a whole, while Figure 3.3 shows the flow diagram for the IR pipeline.

The calwf3 pipeline processes every WFC3 image and the data are calibrated with reference files that are appropriate to the observing mode used. Currently, raw images and telemetry files are processed and calibrated, using the most up-to-date reference files, parameters, and software (see Section 1.1.1 of the Introduction to the HST Data Handbooks). Then the data products, as well as the raw data are stored on MAST (Mikulski Archive for Space Telescopes).  As of June 2024, the most recent version of calwf3 is v.3.7.1, which was first released with HSTCAL v2.7.6. The HSTCAL software releases can be found on the "Releases" pages of the STScI GIT repository. 

During automatic pipeline processing on MAST,  the calibrated output products generated by calwf3 are run through Astrodrizzle, which is a part of the DrizzlePac software package. The main functions of Astrodrizzle are to correct calibrated images for geometric distortion, and combine dithered images into a single product. 

3.1.1 Where to Find calwf3

As mentioned above, calwf3 is part of the HSTCAL software, which is located in the STScI GIT repository.  There are three ways to install the HSTCAL software, two of which require using Conda. HSTCAL and its binaries are a part of the STScI software distribution stenv. It also independently resides on conda-forge. Alternatively, users can choose to install HSTCAL using the source code (requires compilation of C codes). The instructions for all three methods can be found under the README section of the HSTCAL repository.

3.1.2 Running calwf3

Before running the calwf3 pipeline, the raw data need to be prepared for calibration by updating the calibration reference files section of the fits data header (see Section 3.1.4 and Section 3.5 for more details). calwf3 can be run on a single input RAW file or an ASN table listing the members of an association.

When processing an association of CR-SPLIT exposures, calwf3 only retrieves the calibration switch and reference files from the first image listed in the ASN table, and applies them to the entire dataset.

calwf3 does not accept a user-defined list of input images on the command line (e.g. *raw.fits to process all raw files in the current directory). The wf3ccd, wf32d, wf3cte, wf3ir tasks on the other hand, will accept such user-defined input file lists, but they will not accept an association table (_asn.fits) as input.

Below are some basic examples illustrating how to run calwf3 in Python and directly from a terminal. Although pure Python commands may still be used in a PyRAF session, please note that IRAF/PyRAF have been deprecated and support is no longer available. Users are strongly encouraged to switch to Python; more advanced usage examples are available in Section 3.5.

Running calwf3 from a Python session using wfc3tools

from wfc3tools import calwf3
calwf3(filename)

Running multiple input files at the same time with Python

The recommended method for running calwf3 in batch mode is to use Python and the wfc3tools package in the conda-forge distribution.
For example:

from wfc3tools import calwf3
from glob import glob

for fits in glob('i*_raw.fits'):
    calwf3(fits)

Displaying output from calwf3 in a Jupyter Notebook

When running the pipeline from a Jupyter notebook using the calwf3 module from wfc3tools, informational text output from the underlying calwf3.e program is passed to the print function and is displayed in the user’s cell output. This behavior can be customized by passing your own function as the log_func keyword argument to calwf3. As output is read from the underlying program, the calwf3 Python wrapper will call log_func with the contents of each line (this provides a way to connect calwf3 to the Python logging system by passing the logging.debug function or similar).

If log_func=None is passed, informational text output from the underlying program will be ignored, but the program’s exit code will still be checked for successful completion.

Command line options for the calwf3 executable

calwf3 can also be called directly from a shell command line by supplying the executable calwf3.e with an input file and a list of options. This is the same executable that the wfc3tools package calls.

   calwf3.e [options] input 

   calwf3.e -vts iaa012wdq_raw.fits 

   input : str 

       Name of input file 

       - single filename (_raw.fits or _crj.fits) 

       - filename of an ASN table (_asn.fits) 

   options 

       -d : print optional debugging statements 

       -q : print messages only to the trailer file 

       -r : print version number and date of software (e.g., Current version: 3.6.2 (May-27-2021)) 

       -s : save temporary files 

       -t : print a detailed time stamp 

       -v : print verbose time stamps and information

       -1 : suppress the OpemMP parallel processing for the UVIS CTE correction 

   --help : print the syntax for executing this command 

--version : print version number of software (e.g., 3.6.2) 

--gitinfo : print git information (if it can be obtained) 

Running many files at the same time from the command line

The command line executable only accepts one file at a time, but you can use operating system tools like awk to process many _raw.fits files in a directory:

ls *raw.fits | awk '{print "calwf3.e",$1}' | csh

3.1.3 Keyword Usage

calwf3 processing is controlled by the values of keywords in the input image headers. Certain keywords, referred to as calibration switches, are used to control which calibration steps are performed. Reference file keywords indicate which reference files to use in the various calibration steps. Users who wish to perform custom reprocessing of their data may change the values of these keywords in the _raw.fits file primary headers and then rerun the modified file through calwf3.

Other keyword values record instrument and detector parameters that are used in the calibration and some record information that is computed or derived during calibration. Table 3.1 provides a summary of these keywords used by calwf3, specifying whether they function as input or output to the task(s) listed in column 2. For a definition of each keyword see Tables 2.8, 2.9, and 2.10. 

Table 3.1: WFC3 keywords used with calwf3  

KeywordTask(s)I/OHeaderSample of Possible Values
APERTURE

wf3ccd, wf32d, wf3ir

Input

Primary

UVIS1, UVIS2, UVIS1-FIX, UVIS2-FIX, UVIS, IR, ...

ATODGNA,ATODGNB,ATODGNC,ATODGND

wf3ccd, wf32d, wf3ir, wf3rej

Output, Input

Primary

1.56, 2.26

BIASLEVA,BIASLEVB,BIASLEVC,BIASLEVD

wf3ccd

Output

Primary

2502.23, 2605.48

BINAXIS1,BINAXIS2

wf3ccd, wf32d, wf3ir

Input

SCI

1, 2, 3

CAL_VER

wf3ccd, wf32d, wf3rej, wf3ir

Output

Primary

2.1 (15-May-2010)

CCDAMP

wf3ccd, wf32d, wf3rej, wf3ir

Input

Primary

ABCD, AC, BD, A, B, C, D

CCDCHIP

wf3cte, wf3ccd, wf32d, wf3rej

Input

SCI

1, 2

CCDGAIN

wf3ccd, wf32d, wf3ir, wf3rej

Input

Primary

1.5, 2.5

CCDOFSTA,CCDOFSTB
CCDOFSTC,CCDOFSTD

wf3ccd, wf32d, wf3rej

Input

Primary

3

DETECTOR

wf3ccd, wf32d, wf3ir, wf3rej

Input

Primary

UVIS, IR

EXPSTART,EXPEND,EXPTIME

wf3ccd, wf32d, wf3ir, wf3rej

Input

Primary

51475.159

FILTER

wf3ccd, wf32d, wf3ir, wf3rej

Input

Primary

F606W, F160W, G102

FLASHDUR,FLASHSTA

wf3ccd, wf32d, wf3rej

Input

Primary

0.2, SUCCESSFUL

LTM1_1,LTM2_2

wf3ccd, wf3ir

Input

SCI, ERR, DQ

1.0, 0.5, 0.333

LTV1,LTV2

wf3ccd, wf32d, wf3ir

Input

SCI, ERR, DQ

0.0, 25.0, 19.0, 5.0

MEANBLEV

wf3ccd, wf3ir

Output

SCI

2554.763, 14201.36

MEANDARK

wf32d, wf3ir

Output

SCI

3.20642E-01

MEANFLSH

wf3ccd

Output

SCI

N/A

NEXTEND

wf3ccd,wf32d, wf3ir, wf3rej

Input

Primary

3, 6, 80

NSAMP

wf3ir

Input

Primary

2-16

OBSTYPE

wf3ccd,wf32d, wf3ir, wf3rej

Input

Primary

Imaging, Spectroscopic

PHOTMODE

wf32d, wf3ir

Output

SCI, Primary

“WFC3 UVIS1 F606W”

PHOTFLAM

wf32d, wf3ir

Output

SCI, Primary

Inverse sensitivity (erg cm-2 A-1 e-1)

PHOTFNU

wf32d, wf3ir

Output

Primary

Inverse sensitivity (Jy sec e-1)

PHOTZPT

wf32d, wf3ir

Output

SCI, Primary

ST magnitude zero point

PHOTPLAM

wf32d, wf3ir

Output

SCI, Primary

Pivot wavelength

PHOTBW

wf32d, wf3ir

Output

SCI, Primary

rms bandwidth of filter plus detector

PHTFLAM1

wf32d

Output

SCI, Primary

Chip1 Inverse Sensitivity for infinite aperture erg cm-2 A-1 e-1

PHTFLAM2

wf32d

Output

SCI, Primary

Chip1 Inverse Sensitivity for infinite aperture erg cm-2 A-1 e-1

PHTRATIO

wf32d

Output

SCI, Primary

PHTFLAM2/PHTFLAM1 ratio

READNSEA,READNSEB,
READNSEC,READNSED

wf3ccd, wf32d, wf3ir, wf3rej

Output, Input

Primary

calibrated read noise for amplifier A, B, C, and D (electrons)

ROOTNAME

wf3ccd, wf32d, wf3ir, wf3rej

Input

Primary

rootname of the observation set

SAMP_SEQ

wf3ir

Input

Primary

RAPID, SPARS25,...

SAMPTIME

wf3ir

Input

SCI

Total integration time (sec)

SAMPZERO

wf3ir

Input

Primary

Sample time of MULTIACCUM zeroth read (sec)

SDQFLAGS

wf3ccd, wf32d, wf3ir

Input

SCI

Serious data quality flags considered “bad” by calwf3

SUBARRAY

wf3ccd, wf32d, wf3ir

Input

Primary

T, F

SUBTYPE

wf3ir

Input

Primary

FULLIMAG, SQ64SUB

TDFTRANS

wf3ir

Input

SCI

0, 1

IMAGE STATISTICS





NGOODPIX

wf32d, wf3ir

Output

SCI, ERR

number of good pixels

GOODMIN,GOODMAX,GOODMEAN

wf32d, wf3ir

Output

SCI, ERR

min, max and mean values of good pixels (electrons)

SNRMIN,SNRMAX,SNRMEAN

wf32d, wf3ir

Output

SCI

min, max, and mean signal to noise of good pixels

CR-REJ PARAMETERS





BADINPDQ

wf3rej

Output

Primary

data quality flag used for rejection

CRMASK

wf3rej

Output

Primary

T, F

CRRADIUS

wf3rej

Output

Primary

3.0

CRSIGMAS

wf3rej

Output

Primary

6.5, 5.5, 4.5

CRTHRESH

wf3rej

Output

Primary

rejection propagation threshold

EXPSTART,EXPEND,EXPTIME,TEXPTIME

wf3rej

Output

Primary

exposure start, and end times (modified Julian date)

EXPTIME,TEXPTIME

wf3rej

Output

Primary

total exposure duration (seconds)--calculated

INITGUES

wf3rej

Output

Primary

minimum, mode

MEANEXP

wf3rej

Output

Primary

Average exposure time (sec) for each image

NCOMBINE

wf3rej

Output

SCI

number of image sets combined during CR rejection

REJ_RATE

wf3rej

Output

Primary

rate at which pixels are affected by cosmic rays

SCALENSE

wf3rej

Output

Primary

Multiplicative term (in percent) for the noise model

SKYSUB

wf3rej

Output

Primary

Sky value subtracted (mode, none)

SKYSUM

wf3rej

Output

Primary

Sky level from the sum of all constituent images

3.1.4 Using CRDS To Update Reference Files

The HST Calibration Reference Data System (CRDS) is a Python library, set of command line tools and family of web servers used to manage and assign the HST and JWST calibration reference files. Users can query CRDS to identify and retrieve the best references files for their data. The following link explains how you can use this facility via a web interface, Using CRDS to find the best reference files for your data, or from the command line.

Users should ensure that the same version of all reference files and the pipeline software is used to analyze their entire dataset. This is particularly important for data taken at multiple epochs. Retrieving data from MAST will provide the most up-to-date calibration (software and reference files) available at the time of retrieval. 

For example, if the data are retrieved from MAST at different times (e.g., after the execution of each visit), it may be possible to observe systematic differences due to changes in the reference files, or more generally, to the whole data processing flow. Thus, it is recommended that the whole dataset pertaining to the given scientific investigation is retrieved simultaneously. Rarely there may be an instrument mode for which there is no calibration data. In these cases there will be a placeholder reference file in CRDS that is filled with zeros or ones, as appropriate. Such reference files are identified by having their header keyword PEDIGREE set to DUMMY. When calwf3 encounters one of these reference files it will automatically skip the calibration step for which the file is used (e.g., DARKCORR will be set to the value SKIPPED if the DRKFILE is a dummy reference file).

«Previous Next»