The following section contains information regarding the 5 high-level tasks that are called by the main program calwf3, namely wf3cte, wf3ccd, wf32d, wf3ir, wf3rej. These tasks can be called individually by the users from either the command line or within a python interpreter. They generally are “umbrella” tasks that can perform multiple data processing steps. However, the user can control the processing flow (and limit it to individual steps) by properly setting the header keyword switches, as detailed in Sections 3.2 (UVIS) and 3.3 (IR).

3.4.1 wf3cte

This routine is used to correct for Charge Transfer Efficiency (CTE) losses that cause electrons that are generated on the UVIS channel to be trapped by detector defects during readout. More details on the CTE phenomenon are given in Chapter 6, see also the WFC3 CTE webpage. Note that while the CTE model has recently been updated (WFC3 ISR 2021-09), the interface to the code as described below remains the same.

The PCTETAB reference file contains extensions of calibration images and tables of parameters used during the CTE correction stage. The header of this file also contains parameters for the CTE correction algorithm. These parameters, as well as others defaults used by wf3cte to correct the data, are stored in the output image headers. Although not recommended, users who wish to use other settings for the CTE correction algorithm can adjust the pertinent keywords in their dataset header and calwf3 will give them preference (WFC3 ISR 2016-02); as noted above, a new CTE model is currently in place in the pipeline.

wf3cte performs the CTE correction on raw data files. If the calibration step keyword PCTECORR is set to PERFORM then the CTE correction will be applied to the dataset. Some caveats for its use:

• CTE corrections can ONLY be performed on raw data which has not been calibrated in any way.
• Data which have BLEVCORR, BIASCORR or DARKCORR set to COMPLETE will be rejected.
• As of calwf3 v3.4, both full frame and most subarray modes (Table 3.5) are corrected for CTE. The calwf3 version used to process any data is stored in the header keyword "CAL_VER". The CTE correction is unavailable for subarray images acquired with aperture UVIS2-M1K1C-SUB or UVIS2-M512C-SUB as these images, near the middle of the chip, lack bias overscan pixels. The CTE correction code requires bias overscan to calculate a secondary bias subtraction for the image before the CTE is measured. 

Table 3.5: Sub-array modes for which CTE correction is enabled in calwf3 version 3.4. For sub-arrays not listed here, a workaround is available

UVIS1-2K2A-SUB
UVIS1-2K2B-SUB
UVIS2-2K2C-SUB
UVIS2-2K2D-SUB
UVIS2-C1K1C-SUB
UVIS2-C512C-SUB
UVIS2-C512D-SUB
UVIS1-C512A-SUB
UVIS1-C512B-SUB
UVIS1-2K4-SUB
UVIS-QUAD-SUB
UVIS2-2K4-SUB

Calling the wf3cte executable produces a FITS file with the extension "_rac" appended. This contains only the CTE-corrected raw data, no other calibrations have been performed.

Running wf3cte from a python terminal

from wfc3tools import wf3cte
wf3cte(filename)

Displaying output from wf3cte in a Jupyter Notebook

When calling wf3cte from a Jupyter notebook, informational text output from the underlying wf3cte.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3cte. As output is read from the underlying program, the wf3cte Python wrapper will call log_func with the contents of each line (print is an obvious choice for a log function, but this also provides a way to connect wf3cte to the Python logging system by passing the logging.debug function or similar).

If log_func=None is passed, informational text output from the underlying program will be ignored, but the program’s exit code will still be checked for successful completion.

Input Parameters for the Python interface

input: str
Name of input files
	· a single filename (iaa012wdq_raw.fits)
	· a Python list of filenames
	· a partial filename with wildcards (*raw.fits)
	· filename of an ASN table (*asn.fits)
	· an at-file (@input)
-1: value, optional
	as in minus one, this will make sure only 1
	processor/thread is used during processing,
	otherwise all available are used.
-v: verbose: bool, optional
	Print verbose time stamps?

Command Line Options for the wf3cte executable

wf3cte.e input [-options]

The options include:

-v: verbose
-1: turn off multiprocessing

Basic Steps In The CTE Correction

• The reference bias image named in the BIACFILE header keyword is subtracted from the data
• Parameters from the CTE parameter table, referenced in the PCTETAB header keyword, are read and stored
• The data are reformatted: each quadrant is rotated such that the readout amp is located at the lower left of the array. The reoriented four quadrants are then arranged into a single 8412 × 2070 image (including the overscan pixels) with amps CDAB in that order. In this format, the pixels are all parallel-shifted down, then serial-shifted to the left
• An additional bias correction is performed using the residual bias level measured for each amplifier from the steadiest pixels in the horizontal overscan, this value is then subtracted from all the pixels in each respective amp
• The image is multiplied by the gain to convert to electrons
• CTE correction v1.0 (calwf3 v3.5.2 and earlier). The bias-subtracted image contained contribution from read-noise. Those read-noise “electrons”, however, are generated at the readout and are not affected by CTE thus the bias-subtracted image was separated at this stage into two components: one that was consistent with being pure read-noise and another that was a smooth version of the astronomical scene. The CTE reconstruction algorithm then operated on the "smooth" version of the scene, determining which pre-read-out charge distribution could be CTE-trailed to result in the "smooth" scene. The difference between these two images is an estimate of how CTE likely redistributed charge in the exposure. This difference was added to the original raw, uncorrected, uncalibrated image. 
• CTE correction v2.0 (calwf3 v3.6.0 and later). While v1.0 worked well for a number of years, once the CTE correction became larger than a perturbation, read-noise was significantly amplified and required a new approach. In v2.0, pixel-to-pixel variations smaller than the background fluctuations (or 10 electrons, whichever is smaller) are essentially untouched by the reconstruction algorithm. The penalty is that the faintest sources will not be CTE-corrected and users will need to apply other strategies to correct the faintest measurements. Observers can override the default performance to mitigate the background noise by setting the PCTERNOI keyword in the raw science image header with a values between 3.75 and 9.75 electrons and calwf3 will use that value.
• The corrected image is now ready to continue through the rest of the pipeline. When the DARKCORR header keyword is set to PERFORM, the CTE corrected image will be dark-subtracted using the dark reference file referred to in the DRKCFILE header keyword.
• In the case of a subarray image, the same steps are performed as above after the image has been placed into the correct full-frame reference position since the correction is dependent on the distance of the pixels away from the read-out amplifier.

The PCTETAB and Algorithm Parameters

The following are new primary header keywords that will be updated in the data headers during the wf3cte step. They are also specified in the PCTETAB reference file.

The PCTETAB reference file has 4 extensions, two tables, and two images:

Filename: 54l1347ei_cte.fits
No. Name 		Type 		Cards	Dimensions	Format
0 	PRIMARY 	PrimaryHDU 	80		()
1 	QPROF 		BinTableHDU	18 		999R x 4C 	['I', 'J', 'E', '20A']
2 	SCLBYCOL	BinTableHDU	22 		8412R x 6C 	['I', 'E', 'E', 'E', 'E', '20A']
3 	RPROF 		ImageHDU 	33 		(999, 100) 	float32
4 	CPROF 		ImageHDU 	33 		(999, 100) 	float32

The first extension lists the charge-trap levels. The columns are respectively the trap number, the charge-packet size it applies to (in electrons), and the size of the trap (also in electrons).

The second extension contains the CTE scalings as a function of column number. There are 5 columns, each with 8412 elements. The first column contains the integer column number in the amp readout-aligned large array (rac). The other columns contain the CTE scaling appropriate for that column at the 512th, 1024th, 1536th and 2048th rows respectively.

The third extension contains the differential CTE trail profile as a function of charge level in the form of an image.

The fourth extension contains the cumulative CTE trail profile as a function of charge level, also in the form of an image.

Note: the PCTETAB and pixel-based CTE-correction algorithm was updated in April 2021 in calwf3 v3.6.0 to mitigate background noise A summary of the change is in STScI Newsletter Volume 38, Issue 1 while more details are available in WFC3 ISR 2021-09.  

Output Files

If the user is running the separate wf3cte.e step a _rac.fits file will be produced. This is the same as a _raw.fits file except the CTE correction has been applied to the data.

If the PCTECORR step is set to PERFORM:

• When the _raw.fits file enters calwf3, no intermediate _rac.fits file will be saved unless you specify the -s flag, which instructs calwf3.e to save all intermediate files.
• The calwf3 pipeline will produce both CTE calibrated products and non-CTE calibrated products. The CTE products have a ‘c’ at the end of their extension names, such as _blc, _rac, _crc, _flc, and the non-CTE calibrated products contain the familiar: _blv, _crj, _flt.

Note: The CTE correction step uses all available CPUs (while the rest of the calibration steps do not).

3.4.2 wf3ccd

This routine performs the initial processing steps for all the WFC3 UVIS channel data. These steps are:

• DQICORR - initializing the data quality array
• ATODCORR - perform the a-to-d conversion correction
• BLEVCORR - subtract the bias level from the overscan region
• BIASCORR - subtract the bias image

• FLSHCORR - subtract the post-flash image

wf3ccd first subtracts the bias and trims the overscan regions from the image. If an associated set of UVIS CR-SPLIT or REPEAT-OBS images is being processed, all of the overscan-trimmed images are sent through wf3rej to be combined and receive cosmic-ray rejection. The resulting combined image then receives final calibration with wf32d, which includes dark subtraction and flat fielding. If there are multiple sets of CR-SPLIT or REPEAT-OBS images in an association, each set goes through the cycle of wf3ccd, wf3rej and wf32d processing.

Only those steps with a switch value of PERFORM in the input files will be executed, after which the switch will be set to COMPLETE in the corresponding output files.

Running wf3ccd from a python terminal

from wfc3tools import wf3ccd
wf3ccd(filename)

Displaying output from wf3ccd in a Jupyter Notebook

When calling wf3ccd from a Jupyter notebook, informational text output from the underlying wf3ccd.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3ccd. As output is read from the underlying program, the wf3ccd Python wrapper will call log_func with the contents of each line (print is an obvious choice for a log function, but this also provides a way to connect wf3ccd to the Python logging system by passing the logging.debug function or similar).

If log_func=None is passed, informational text output from the underlying program will be ignored, but the program’s exit code will still be checked for successful completion.

Input parameters for the Python interface

input: str
Name of input files
	· a single filename (iaa012wdq_raw.fits)
	· a Python list of filenames
	· a partial filename with wildcards(*raw.fits)
	· filename of an ASN table (*asn.fits)
	· an at-file (@input)
output: str
	Name of the output FITS file.
dqicorr: str, “PERFORM/OMIT”, optional
	Update the dq array from bad pixel table
atodcorr: str, “PERFORM/OMIT”, optional
	Analog to digital correction
blevcorr: str, “PERFORM/OMIT”, optional
	Subtract bias from overscan regions
biascorr: str, “PERFORM/OMIT”, optional
	Subtract bias image
flashcorr: str, “PERFORM/OMIT”, optional
	Subtract post-flash image
verbose: bool, optional
	Print verbose time stamps?
quiet: bool, optional
	Print messages only to trailer file?

Command Line Options for the wf3ccd executable

wf32ccd.e input output [-options]

Input may be a single filename.
The options include:

-v: verbose
-f: print time stamps
-dqi: udpate the DQ array
-atod: perform gain correction
-blev: subtract bias from overscan
-bias: perform bias correction
-flash: remove post-flash image

3.4.3 wf32d

This routine performs the remaining series of tasks in the UVIS pipeline. The wf32d primary functions include:

• DARKCORR: dark current subtraction
• FLATCORR: flat-fielding

• PHOTCORR: photometric keyword calculations

• FLUXCORR: photometric normalization of the UVIS1 and UVIS2 chips

Only those steps with a switch value of PERFORM in the input files will be executed, after which the switch will be set to COMPLETE in the corresponding output files.

Running wf32d from a python terminal

from wfc3tools import wf32d
wf32d(filename)

Displaying output from wf32d in a Jupyter Notebook

When calling wf32d from a Jupyter notebook, informational text output from the underlying wf32d.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf32d. As output is read from the underlying program, the wf32d Python wrapper will call log_func with the contents of each line (print is an obvious choice for a log function, but this also provides a way to connect wf32d to the Python logging system by passing the logging.debug function or similar).

Parameters

input: str
Name of input files
	· a single filename (iaa012wdq_raw.fits)
	· a Python list of filenames
	· a partial filename with wildcards (*raw.fits)
	· filename of an ASN table (*asn.fits)
	· an at-file (@input)
output: str
	Name of the output FITS file.
dqicorr: str, “PERFORM/OMIT”, optional
	Update the dq array from bad pixel table
darkcorr: str, “PERFORM/OMIT”, optional
	Subtract the dark image
flatcorr: str, “PERFORM/OMIT”, optional
	Multiply by the flatfield image
shadcorr: str, “PERFORM/OMIT”, optional
	Correct for shutter shading (CCD)
photcorr: str, “PERFORM/OMIT”, optional
	Update photometry keywords in the header
fluxcorr; str, “PERFORM/OMIT”, optional
	Perform chip photometry normalization
verbose: bool, optional
	Print verbose time stamps?
quiet: bool, optional
	Print messages only to trailer file?

Command Line Options for the wf32d executable

wf32d.e input output [-options]

Input may be a single filename.
The options include:

3.4.4 wf3ir

This routine contains all the instrumental calibration steps for WFC3 IR channel images. The steps are:

• DQICORR - initialize the data quality array
• ZSIGCORR - estimate the amount of signal in the zeroth-read
• BLEVCORR - subtract the bias level from the reference pixels
• ZOFFCORR - subtract the zeroth read image
• NLINCORR - correct for detector non-linear response
• DARKCORR - subtract the dark current image
• PHOTCORR - compute the photometric keyword values
• UNITCORR - convert to units of count rate
• CRCORR - fit accumulating signal and identify the cr hits
• FLATCORR - divide by the flat-field images and apply gain conversion

The output images include the calibrated image (flt file) and the intermediate multi-accumulated ramp image (ima file).

Only those steps with a switch value of PERFORM in the input files will be executed, after which the switch will be set to COMPLETE in the corresponding output files.

Running wf3ir from a python terminal

from wfc3tools import wf3ir
wf3ir(filename)

Displaying output from wf3ir in a Jupyter Notebook

When calling wf3ir from a Jupyter notebook, informational text output from the underlying wf3ir.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3ir. As output is read from the underlying program, the wf3ir Python wrapper will call log_func with the contents of each line (print is an obvious choice for a log function, but this also provides a way to connect wf3ir to the Python logging system by passing the logging.debug function or similar).

Input Parameters for the Python interface

input: str
		Name of input files
	· a single filename (iaa012wdq_raw.fits)
	· a Python list of filenames
	· a partial filename with wildcards (*raw.fits)
	· filename of an ASN table (*asn.fits)
	· an at-file (@input)
output: str
		Name of the output FITS file.
verbose: bool, optional
		Print verbose time stamps?
quiet: bool, optional
Print messages only to trailer file?

Command Line Options for the wf3ir executable

wf32ir.e input output [-options]

Input may be a single filename. The options include:

-v: verbose
-f: print time stamps

3.4.5 wf3rej

wf3rej, the cosmic-ray rejection and image combination task in calwf3, combines CR-SPLIT or REPEAT-OBS exposures into a single image, first detecting and then replacing flagged pixels. The task uses the same statistical detection algorithm developed for ACS (acsrej), STIS (ocrrej), and WFPC2 data (crrej), providing a well-tested and robust procedure.

First, wf3rej temporarily removes the sky background from each input image (if requested via the SKYSUB parameter in the CRREJTAB), usually computed using the mode of each image. Sky subtraction is performed before any statistical checks are made for cosmic rays. Next, wf3rej constructs an initial comparison image from each sky-subtracted exposure. This comparison image can either be a median- or minimum-value sky-subtracted image constructed from all the input images, and it represents the ‘initial guess’ of a cosmic-ray free image. The comparison image serves as the basis for determining the statistical deviation of each pixel within the input images.

A detection threshold is then calculated for each pixel based on the comparison image:

τ_n = σ² × (noise+value/gain+(scale×value)²)/T_n²

where:

• σ is the sigma value used as the detection limit (CRSIGMAS),
• noise is the readnoise in DN squared and gain is the e-/DN of the amplifier used to read the pixel,
• scale is the scale factor for the noise model,
• T_n is the exposure time for the input image, and
•  value is the pixel value (in DN) from the median or minimum combined comparison image. 

The actual detection criterion for a cosmic ray is determined as:

Δ = ((pix_n –sky_n)/T_n–median)²

where:

• pixel_n is the pixel value from input image n,
• sky_n is the sky background of image n, and
• median is the median or minimum pixel value from the comparison image.

If Δ >τ_n the pixel is flagged as a cosmic ray in the input image’s DQ array and is ignored when images are summed together. Surrounding pixels within some expansion radius (CRRADIUS) are marked as ‘SPILL’ pixels and are given less stringent detection thresholds.
When all input images have been processed, the values of the non-rejected pixels are summed over all input images. Each pixel in the summed output array is then scaled by the total exposure time:

where:

• Τ_n is the exposure time for image n,
• m_n(x,y) is the mask value (0 for rejected pixels, 1 for good data) for the n-th image at pixel (x, y),
• Τ is the total exposure time (regardless of whether all input images were used for that particular pixel). This corresponds to the value recorded in the header keywords TEXPTIME and EXPTIME.

The following keywords are also derived from the variables in this equation:

• TEXPTIME = EXPTIME = T
• SKYSUM = ∑_nsky_n
• REJ_RATE = ₍∑_n T_nm_n(x,y)/Τ) averaged over all pixels
• NCOMBINE = n

The remaining keywords EXPSTART, EXPEND are updated based on the values corresponding to the first and last input images, respectively. In summary, the cosmic ray rejection task sums all non-rejected pixel values, computes the true exposure time for that pixel, and scales the sum to correspond to the total exposure time. The final scaled, cleaned pixel is written to the comparison image to be used for the next iteration. This process is then repeated with increasingly stringent detection thresholds, as specified by CRSIGMAS.

Cosmic Ray Rejection Table

wf3rej uses the Cosmic Ray Rejection parameter table (CRREJTAB) to determine the number of iterations for cosmic-ray rejection, the sigma levels to use for each iteration, and the spill radius to use during detection. This allows the rejection process to be tuned to each detector and observation, with suitable defaults being applied during pipeline processing. Observers may fine-tune the cosmic-ray rejection parameters when manually reprocessing data with wf3rej by editing the CRREJTAB.

The CRREJTAB reference file contains the basic parameters necessary for performing cosmic-ray rejection. The column names and default values for the CRREJTAB are given in Table 3.6. The appropriate row is selected based on the chip being processed (CCDCHIP), the number of images into which the exposure was split (CR-SPLIT), and the exposure time of each CR-SPLIT image (MEANEXP). If an exact match is not found for the exposure time, the table row with the closest value is used. If the CR-SPLIT value of the input images exceeds the values in the table, the table row with the largest CR-SPLIT value will be used. The sky fitting algorithm is controlled by the parameter SKYSUB, which can have values of ‘mode’, ‘mean’ or ‘none’. The ‘initial guess’ image is created using the median or minimum value of the input exposures, as specified by the value of INITGUES.

Cosmic-ray detection requires the specification of a threshold above which a pixel value is considered a cosmic ray. This threshold was defined above and uses the sigma rejection thresholds. These sigmas correspond to the CRSIGMAS column values in the CRREJTAB file. SCALENSE is a multiplicative term (in percent) for the noise model and is given as scale in the threshold equation above. This term can be useful when the pointing of the telescope has changed by a small fraction of a pixel between images. Under such circumstances, the undersampling of the image by the detector will cause stars to be mistakenly rejected as cosmic rays if a scale noise term is not included. This is a crude but effective step taken to satisfy the maxim of ‘do no harm’. However, for cases in which there have been no image-to-image offsets or the image is locally well-sampled, this will unduly bias against rejecting cosmic rays.

Pixels within a given radius, CRRADIUS, of a cosmic ray will also be treated as cosmic rays. A less stringent rejection threshold, CRTHRESH, can be used for detecting pixels adjacent to a cosmic ray. As for CRSIGMAS, CRTHRESH is also given as a sigma value. If CRTHRESH is exceeded, pixels within the defined radius of the cosmic ray will also be flagged. All pixels determined to be affected by a cosmic ray will have their DQ values set to 8192, as described in Table 3.6.

Table 3.6: Columns in the cosmic-ray rejection parameter table.

Running wf3rej from a python terminal

from wfc3tools import wf3rej
wf3rej(filename)

Displaying output from wf3rej in a Jupyter Notebook

When calling wf3rej from a Jupyter notebook, informational text output from the underlying wf3rej program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3rej. As output is read from the underlying program, the wf3rej Python wrapper will call log_func with the contents of each line (print is an obvious choice for a log function, but this also provides a way to connect wf3rej to the Python logging system by passing the logging.debug function or similar).

Input Parameters for the Python interface

input : str, Name of input files
	· a single filename (iaa012wdq_raw.fits)
	· a Python list of filenames
	· a partial filename with wildcards (\*raw.fits)
	· filename of an ASN table (\*asn.fits)
	· an at-file (@input)
output: string
	Name of the output FITS file.
crrejtab: string
	reference file name
scalense: string
	scale factor applied to noise
initgues: string
	intial value estimate scheme (min|med)
skysub: string
	how to compute the sky (none|mode|mean)
crsigmas: string
	rejection levels in each iteration
crradius: float
	cosmic ray expansion radius in pixels
crthresh: float
	rejection propagation threshold
badinpdq: int
	data quality flag bits to reject
crmask: bool
	flag CR in input DQ imageS?
shadcorr: bool
	perform shading shutter correction?
verbose: bool, optional
	Print verbose time stamps?

Command Line Options for the wf3rej executable

wf3rej.e input output [-options]

Input can be a single file.
The options include:

	t: print the timestamps
	v: verbose
	shadcorr: perform shading shutter correction?
	crmask: flag CR in input DQ images?
	table <filename>: the crrejtab filename
	scale <number>: scale factor for noise
	init <med|min>: initial value estimate scheme
	sky <none|median|mode>: how to compute sky
	sigmas: rejection leves for each iteration
	radius <number>: CR expansion radius
	thresh <number> : rejection propagation threshold
	pdq <number>: data quality flag bits to reject