3.4 Pipeline Tasks

The following section contains information regarding the 5 high-level tasks that are called by the main program calwf3, namely wf3cte, wf3ccd, wf32d, wf3ir, wf3rej. These tasks can be called individually by the users from either the command line or within a python interpreter. They generally are “umbrella” tasks that can perform multiple data processing steps. However the user can control the processing flow (and limit it to individual steps) by properly setting the header keyword switches, as detailed in Sections 3.2 (UVIS) and 3.3 (IR).

3.4.1 wf3cte

This routine is used to correct for Charge Transfer Efficiency (CTE) losses that cause electrons that are generated on the UVIS channel to be trapped by detector defects during readout. More details on the CTE phenomenon are given in Chapter 6, see also the WFC3 CTE webpage.

The PCTETAB reference file contains extensions of calibration images and tables of parameters used during the CTE correction stage. The header of this file also contains parameters for the CTE correction algorithm. These parameters, as well as others defaults used by wf3cte to correct the data are stored in the output image headers. Users who wish to use other settings for the CTE correction algorithm can adjust the pertinent keywords in their dataset header and calwf3 will give them preference, see WFC ISR 2016-02 for more information.

wf3cte performs the CTE correction on raw data files. If the calibration step keyword PCTECORR is set to PERFORM then the CTE correction will be applied to the dataset. Some caveats for its use:

  • CTE corrections can ONLY be performed on raw data which has not been calibrated in any way.
  • Data which have BLEVCORR, BIASCORR or DARKCORR set to COMPLETE will be rejected.
  • The CTE correction step in is implemented only for FULL FRAME images in calwf3 v3.3, but v3.4 also corrects for CTE for the subarray apertures indicated in Table 3.5 (the calwf3 version used to process any data is stored in the header keyword "CAL_VER"). The primary distinction is that these apertures have physical overscan pixels included which are used to calculate a secondary bias subtraction for the image before the CTE is measured; a future version of calwf3 may enable CTE corrections for the remaining subarrays which don’t have physical overscan pixels, but is still being validated by the WFC3 science team.

Table 3.5: Sub-array modes for which CTE correction is enabled in calwf3 version 3.4. For sub-arrays not listed here, a workaround is available

UVIS1-2K2A-SUB
UVIS1-2K2B-SUB
UVIS2-2K2C-SUB
UVIS2-2K2D-SUB
UVIS2-C1K1C-SUB
UVIS2-C512C-SUB
UVIS2-C512D-SUB
UVIS1-C512A-SUB
UVIS1-C512B-SUB
UVIS1-2K4-SUB
UVIS-QUAD-SUB
UVIS2-2K4-SUB

Calling the wf3cte executable produces a FITS file with the extension "_rac" appended. This contains only the CTE corrected data, no other calibrations have been performed.

Running wf3cte from a python terminal

In Python without TEAL:

from wfc3tools import wf3cte
wf3cte(filename)


In Python with TEAL:

from stsci.tools import teal
from wfc3tools import wf3cte
teal.teal('wf3cte')

Displaying output from wf3cte in a Jupyter Notebook

When calling wf3cte from a Jupyter notebook, informational text output from the underlying wf3cte.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3cte. As output is read from the underlying program, the wf3cte Python wrapper will call log_func with the contents of each line (print is an obvious choice for a log function, but this also provides a way to connect wf3cte to the Python logging system by passing the logging.debug function or similar).

If log_func=None is passed, informational text output from the underlying program will be ignored, but the program’s exit code will still be checked for successful completion.

Input Parameters for the Python interface

input: str
Name of input files
	· a single filename (iaa012wdq_raw.fits)
	· a Python list of filenames
	· a partial filename with wildcards (*raw.fits)
	· filename of an ASN table (*asn.fits)
	· an at-file (@input)
-1: value, optional
	as in minus one, this will make sure only 1
	processor/thread is used during processing,
	otherwise all available are used.
-v: verbose: bool, optional
	Print verbose time stamps?

Command Line Options for the wf3cte executable

wf3cte.e input [-options]


Where the options include:

-v: verbose
-1: turn off multiprocessing

Basic Steps In The CTE Correction

  • The reference bias image named in the BIACFILE header keyword is subtracted from the data
  • Parameters from the CTE parameter table, referenced in the PCTETAB header keyword, are read and stored
  • The data are reformatted: each quadrant is rotated such that the readout amp is located at the lower left of the array. The reoriented four quadrants are then arranged into a single 8412 × 2070 image (including the overscan pixels) with amps CDAB in that order. In this format, the pixels are all parallel-shifted down, then serial-shifted to the left
  • An additional bias correction is performed using the residual bias level measured for each amplifier from the steadiest pixels in the horizontal overscan, this value is then subtracted from all the pixels in each respective amp
  • The image is multiplied by the gain
  • The bias subtracted image still contains contribution from the read-noise. The read-noise “electrons”, however are generated at readout and are not affected by CTE. Therefore, the bias subtracted image is separated at this stage into two components: one that is consistent with being pure read-noise and another that is a smooth version of the astronomical scene.
  • The CTE reconstruction algorithm then operates on the "smooth" version of the scene, determining which pre-read-out charge distribution could be CTE-trailed to result in the "smooth" scene. The difference between these two images is an estimate of how CTE likely redistributed charge in the exposure. We add this difference to the original raw, uncorrected, uncalibrated image. 
  • The corrected image is now ready to continue through the rest of the pipeline. When the DARKCORR header keyword is set to PERFORM, the CTE corrected image will be dark-subtracted using the dark reference file referred to in the DRKCFILE header keyword.
  • In the case of subarray image, the same steps are performed as above after the image has been placed into the correct full-frame reference position since the correction is dependent on the distance of the pixels away from the read-out amplifier.

The PCTETAB and Algorithm Parameters

The following are new primary header keywords which will be updated in the data headers during the wf3cte step. They are also specified in the PCTETAB reference file.

KEYWORD

DESCRIPTION

CTE_NAME

name of cte algorithm [string]

CTE_VER

version number of cte algorithm [string]

CTEDATE0

date of wfc3/uvis installation in HST, in modified Julian days (MJD)

CTEDATE1

reference date of CTE model pinning, in modified Julian days (MJD)

PCTETLEN

max length of CTE trail

PCTERNOI

read-noise amplitude for clipping

PCTENFOR

number of iterations used in CTE forward modeling

PCTENPAR

number of iterations used in the parallel transfer

PCTENSMD

read-noise mitigation algorithm

PCTETRSH

over-subtraction threshold

PCTEFRAC

CTE scaling frac calculated from expstart and used in the algorithm

PCTERNOI

the read-noise clipping level to use

FIXROCR

make allowance for readout cosmic rays


The PCTETAB reference file has 4 extensions, two tables and two images:

Filename: zcv2057mi_cte.fits
No. Name 		Type 		Cards	Dimensions	Format
0 	PRIMARY 	PrimaryHDU 	21		()
1 	QPROF 		BinTableHDU	16 		999R x 4C 	['i', 'j', 'e', '20A']
2 	SCLBYCOL	BinTableHDU	20 		8412R x 6C 	['i', 'e', 'e', 'e', 'e', '20A']
3 	RPROF 		ImageHDU 	12 		(999, 100) 	float32
4 	CPROF 		ImageHDU 	12 		(999, 100) 	float32

The first extension lists the charge-trap levels, the columns are respectively the trap number, the charge-packet size it applies to (in electrons), and the size of the trap (also in electrons).

The second extension contains the CTE scalings as a function of column number. There are 5 columns, each with 8412 elements. The first column contains the integer column number in the amp readout-aligned large array. The other columns contain the CTE scaling appropriate for that column at the 512th, 1024th, 1536th and 2048th rows respectively.

The third extension contains the differential CTE trail profile as a function of charge level in the form of an image

The fourth extension contains the cumulative CTE trail profile as a function of charge level, also in the form of an image.

Output Files

If the user is running the separate wf3cte.e step a _rac.fits file will be produced. This is the same as a _raw.fits file except the CTE correction has been applied to the data.

If the PCTECORR step is set to PERFORM:

  • when the _raw.fits file enters calwf3, then no intermediate _rac.fits file will be saved, unless you specify the -s flag, which instructs calwf3.e to save all intermediate files.
  • the calwf3 pipeline will produce both CTE calibrated product and non-CTE calibrated products. The CTE products have a ‘c’ at the end of their extension name, such as _blc, _rac, _crc, _flc, and the non-CTE calibrated products contain the familiar : _blv, _crj, _flt.

Note: The CTE correction step uses all available CPUs (while the rest of the calibration steps do not).

3.4.2 wf3ccd

This routine performs the initial processing steps for all the WFC3 UVIS channel data. These steps are:

  • DQICORR - initializing the data quality array
  • ATODCORR - perform the a-to-d conversion correction
  • BLEVCORR - subtract the bias level from the overscan region
  • BIASCORR - subtract the bias image
  • FLSHCORR - subtract the post-flash image

wf3ccd first subtracts the bias and trims the overscan regions from the image. If an associated set of UVIS CR-SPLIT or REPEAT-OBS images is being processed, all of the overscan-trimmed images are sent through wf3rej to be combined and receive cosmic-ray rejection. The resulting combined image then receives final calibration with wf32d, which includes dark subtraction and flat fielding. If there are multiple sets of CR-SPLIT or REPEAT-OBS images in an association, each set goes through the cycle of wf3ccd, wf3rej and wf32d processing.

Only those steps with a switch value of PERFORM in the input files will be executed, after which the switch will be set to COMPLETE in the corresponding output files.

Running wf3ccd from a python terminal

In Python without TEAL:

from wfc3tools import wf3ccd
wf3ccd(filename)

In Python with TEAL:

from stsci.tools import teal
from wfc3tools import wf3ccd
teal.teal('wf3ccd')

Displaying output from wf3ccd in a Jupyter Notebook

When calling wf3ccd from a Jupyter notebook, informational text output from the underlying wf3ccd.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3ccd. As output is read from the underlying program, the wf3ccd Python wrapper will call log_func with the contents of each line. (print is an obvious choice for a log function, but this also provides a way to connect wf3ccd to the Python logging system by passing the logging.debug function or similar.)

If log_func=None is passed, informational text output from the underlying program will be ignored, but the program’s exit code will still be checked for successful completion.

Input Parameters for the Python interface

input: str
Name of input files
	· a single filename (iaa012wdq_raw.fits)
	· a Python list of filenames
	· a partial filename with wildcards(*raw.fits)
	· filename of an ASN table (*asn.fits)
	· an at-file (@input)
output: str
	Name of the output FITS file.
dqicorr: str, “PERFORM/OMIT”, optional
	Update the dq array from bad pixel table
atodcorr: str, “PERFORM/OMIT”, optional
	Analog to digital correction
blevcorr: str, “PERFORM/OMIT”, optional
	Subtract bias from overscan regions
biascorr: str, “PERFORM/OMIT”, optional
	Subtract bias image
flashcorr: str, “PERFORM/OMIT”, optional
	Subtract post-flash image
verbose: bool, optional
	Print verbose time stamps?
quiet: bool, optional
	Print messages only to trailer file?

Command Line Options for the wf3ccd executable

wf32ccd.e input output [-options]

input may be a single filename
Where the options include:

-v: verbose
-f: print time stamps
-dqi: udpate the DQ array
-atod: perform gain correction
-blev: subtract bias from overscan
-bias: perform bias correction
-flash: remove post-flash image

3.4.3 wf32d

This routine performs the remaining series of tasks in the UVIS pipeline
The wf32d primary functions include:

  • DARKCORR: dark current subtraction
  • FLATCORR: flat-fielding
  • PHOTCORR: photometric keyword calculations

  • FLUXCORR: photometric normalization of the UVIS1 and UVIS2 chips

Only those steps with a switch value of PERFORM in the input files will be executed, after which the switch will be set to COMPLETE in the corresponding output files.

Examples

In Python without TEAL:

from wfc3tools import wf32d
wf32d(filename)


In Python with TEAL:

from stsci.tools import teal
from wfc3tools import wf32d
teal.teal('wf32d')

Displaying output from wf32d in a Jupyter Notebook

When calling wf32d from a Jupyter notebook, informational text output from the underlying wf32d.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf32d. As output is read from the underlying program, the wf32d Python wrapper will call log_func with the contents of each line. (print is an obvious choice for a log function, but this also provides a way to connect wf32d to the Python logging system by passing the logging.debug function or similar.)

Parameters

input: str
Name of input files
	· a single filename (iaa012wdq_raw.fits)
	· a Python list of filenames
	· a partial filename with wildcards (*raw.fits)
	· filename of an ASN table (*asn.fits)
	· an at-file (@input)
output: str
	Name of the output FITS file.
dqicorr: str, “PERFORM/OMIT”, optional
	Update the dq array from bad pixel table
darkcorr: str, “PERFORM/OMIT”, optional
	Subtract the dark image
flatcorr: str, “PERFORM/OMIT”, optional
	Multiply by the flatfield image
shadcorr: str, “PERFORM/OMIT”, optional
	Correct for shutter shading (CCD)
photcorr: str, “PERFORM/OMIT”, optional
	Update photometry keywords in the header
fluxcorr; str, “PERFORM/OMIT”, optional
	Perform chip photometry normalization
verbose: bool, optional
	Print verbose time stamps?
quiet: bool, optional
	Print messages only to trailer file?

Command Line Options for the wf32d executable

wf32d.e input output [-options]

input may be a single filename
Where the options include:

-v: verbose
-f: print time stamps
-d: debug
-dark: perform dark subtraction
-dqi: update the DQ array
-flat: perform flat correction
-shad: perform shading correction
-phot: perform phot correction

3.4.4 wf3ir

This routine contains all the instrumental calibration steps for WFC3 IR channel images. The steps are:

  • DQICORR - initialize the data quality array
  • ZSIGCORR - estimate the amount of signal in the zeroth-read
  • BLEVCORR - subtract the bias level from the reference pixels
  • ZOFFCORR - subtract the zeroth read image
  • NLINCORR - correct for detector non-linear response
  • DARKCORR - subtract the dark current image
  • PHOTCORR - compute the photometric keyword values
  • UNITCORR - convert to units of count rate
  • CRCORR - fit accumulating signal and identify the cr hits
  • FLATCORR - divide by the flat-field images and apply gain conversion

The output images include the calibrated image (flt file) and the intermediate multi-accumulated ramp image (ima file).

Only those steps with a switch value of PERFORM in the input files will be executed, after which the switch will be set to COMPLETE in the corresponding output files.

Running wf3ir from a python terminal

In Python without TEAL:

from wfc3tools import wf3ir
wf3ir(filename)


In Python with TEAL:

from stsci.tools import teal
from wfc3tools import wf3ir
teal.teal('wf3ir')

Displaying output from wf3ir in a Jupyter Notebook

When calling wf3ir from a Jupyter notebook, informational text output from the underlying wf3ir.e program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3ir. As output is read from the underlying program, the wf3ir Python wrapper will call log_func with the contents of each line. (print is an obvious choice for a log function, but this also provides a way to connect wf3ir to the Python logging system by passing the logging.debug function or similar.)

Input Parameters for the Python interface

input: str
		Name of input files
	· a single filename (iaa012wdq_raw.fits)
	· a Python list of filenames
	· a partial filename with wildcards (\*raw.fits)
	· filename of an ASN table (\*asn.fits)
	· an at-file (@input)
output: str
		Name of the output FITS file.
verbose: bool, optional
		Print verbose time stamps?
quiet: bool, optional
Print messages only to trailer file?

Command Line Options for the wf3ir executable

wf32ir.e input output [-options]

input may be a single filename
Where the options include:

-v: verbose
-f: print time stamps

3.4.5 wf3rej

wf3rej, the cosmic-ray rejection and image combination task in calwf3, combines CR-SPLIT or REPEAT-OBS exposures into a single image, first detecting and then replacing flagged pixels. The task uses the same statistical detection algorithm developed for ACS (acsrej), STIS (ocrrej), and WFPC2 data (crrej), providing a well-tested and robust procedure.

First, wf3rej temporarily removes the sky background from each input image (if requested via the SKYSUB parameter in the CRREJTAB), usually computed using the mode of each image. Sky subtraction is performed before any statistical checks are made for cosmic rays. Next, wf3rej constructs an initial comparison image from each sky-subtracted exposure. This comparison image can either be a median- or minimum-value sky-subtracted image constructed from all the input images, and it represents the ‘initial guess’ of a cosmic-ray free image. The comparison image serves as the basis for determining the statistical deviation of each pixel within the input images.

A detection threshold is then calculated for each pixel based on the comparison image:

τn = σ× (noise+value/gain+(scale×value)2)/Tn2

where:

  • σ is the sigma value used as the detection limit (CRSIGMAS),
  • noise is the readnoise in DN squared and gain is the e-/DN of the amplifier used to read the pixel,
  • scale is the scale factor for the noise model,
  • Tis the exposure time for the input image, and
  •  value is the pixel value (in DN) from the median or minimum combined comparison image. 

The actual detection criterion for a cosmic ray is determined as:

Δ = ((pixn –skyn)/Tn–median)2

where:

  • pixeln is the pixel value from input image n,
  • skyn is the sky background of image n, and
  • median is the median or minimum pixel value from the comparison image.

If Δ >τthe pixel is flagged as a cosmic ray in the input image’s DQ array and is ignored when images are summed together. Surrounding pixels within some expansion radius (CRRADIUS) are marked as ‘SPILL’ pixels and are given less stringent detection thresholds.
When all input images have been processed, the values of the non-rejected pixels are summed over all input images. Each pixel in the summed output array is then scaled by the total exposure time:

where:

  • Τis the exposure time for image n,
  • mn(x,y) is the mask value (0 for rejected pixels, 1 for good data) for then-th image at pixel (x, y),
  • Τ is the total exposure time (regardless of whether all input images were used for that particular pixel). This corresponds to the value recorded in the header keywords TEXPTIME and EXPTIME.

The following keywords are also derived from the variables in this equation:

  • TEXPTIME = EXPTIME = T
  • SKYSUM = nskyn
  • REJ_RATE = (Tnmn(x,y)/Τ) averaged over all pixels
  • NCOMBINE = n

The remaining keywords EXPSTART, EXPEND are updated based on the values corresponding to the first and last input images, respectively.
In summary, the cosmic ray rejection task sums all non-rejected pixel values, computes the true exposure time for that pixel, and scales the sum to correspond to the total exposure time. The final scaled, cleaned pixel is written to the comparison image to be used for the next iteration. This process is then repeated with increasingly stringent detection thresholds, as specified by CRSIGMAS.

Cosmic Ray Rejection Table

wf3rej uses the Cosmic Ray Rejection parameter table (CRREJTAB) to determine the number of iterations for cosmic-ray rejection, the sigma levels to use for each iteration, and the spill radius to use during detection. This allows the rejection process to be tuned to each detector and observation, with suitable defaults being applied during pipeline processing. Observers may fine-tune the cosmic-ray rejection parameters when manually reprocessing data with wf3rej by editing the CRREJTAB.

The CRREJTAB reference file contains the basic parameters necessary for performing cosmic-ray rejection. The column names and default values for the CRREJTAB are given in Table 3.6. The appropriate row is selected based on the chip being processed (CCDCHIP), the number of images into which the exposure was split (CR-SPLIT), and the exposure time of each CR-SPLIT image (MEANEXP). If an exact match is not found for the exposure time, the table row with the closest value is used. If the CR-SPLIT value of the input images exceeds the values in the table, the table row with the largest CR-SPLIT value will be used. The sky fitting algorithm is controlled by the parameter SKYSUB, which can have values of ‘mode’, ‘mean’ or ‘none’. The ‘initial guess’ image is created using the median or minimum value of the input exposures, as specified by the value of INITGUES.

Cosmic-ray detection requires the specification of a threshold above which a pixel value is considered a cosmic ray. This threshold was defined above and uses the sigma rejection thresholds. These sigmas correspond to the CRSIGMAS column values in the CRREJTAB file. SCALENSE is a multiplicative term (in percent) for the noise model and is given as scale in the threshold equation above. This term can be useful when the pointing of the telescope has changed by a small fraction of a pixel between images. Under such circumstances, the undersampling of the image by the detector will cause stars to be mistakenly rejected as cosmic rays if a scale noise term is not included. This is a crude but effective step taken to satisfy the maxim of ‘do no harm’. However, for cases in which there have been no image-to-image offsets or the image is locally well-sampled, this will unduly bias against rejecting cosmic rays.

Pixels within a given radius, CRRADIUS, of a cosmic ray will also be treated as cosmic rays. A less stringent rejection threshold, CRTHRESH, can be used for detecting pixels adjacent to a cosmic ray. As for CRSIGMAS, CRTHRESH is also given as a sigma value. If CRTHRESH is exceeded, pixels within the defined radius of the cosmic ray will also be flagged. All pixels determined to be affected by a cosmic ray will have their DQ values set to 8192, as described in Table 3.6.

Table 3.6: Columns in cosmic-ray rejection parameter table.

Column Name

Default Value

Contents

CRSPLIT

-

Number of exposures into which observation was split

CCDCHIP

-

Chip to which this conversion applies

MEANEXP

-

Average exposure time (sec) for each image

SCALENSE

30.0

Multiplicative term (in percent) for the noise model

INITGUES

minimum

Method for computing initial-guess image (minimum, median)

SKYSUB

mode

Sky fitting algorithm (mode, none)

CRSIGMAS

6.5, 5.5, 4.5

Rejection thresholds (sigma)

CRRADIUS

2.1

Radius (in pixels) for propagating cosmic ray

CRTHRESH

0.5555

Propagation factor

BADINPDQ

39

Data quality file bits to reject

CRMASK

yes

Flag CR-rejected pixels in input files?

Running wf3rej from a python terminal

In Python without TEAL:

from wfc3tools import wf3rej
wf3rej(filename)


In Python with TEAL:

from stsci.tools import teal
from wfc3tools import wf3rej
teal.teal('wf3rej')

Displaying output from wf3rej in a Jupyter Notebook

When calling wf3rej from a Jupyter notebook, informational text output from the underlying wf3rej program will be passed through print as the calibration runs by default, and show up in the user’s cell. This behavior can be customized by passing your own function as the log_func keyword argument to wf3rej. As output is read from the underlying program, the wf3rej Python wrapper will call log_func with the contents of each line. (print is an obvious choice for a log function, but this also provides a way to connect wf3rej to the Python logging system by passing the logging.debug function or similar.)

Input Parameters for the Python interface

input : str, Name of input files
	· a single filename (iaa012wdq_raw.fits)
	· a Python list of filenames
	· a partial filename with wildcards (\*raw.fits)
	· filename of an ASN table (\*asn.fits)
	· an at-file (@input)
output: string
	Name of the output FITS file.
crrejtab: string
	reference file name
scalense: string
	scale factor applied to noise
initgues: string
	intial value estimate scheme (min|med)
skysub: string
	how to compute the sky (none|mode|mean)
crsigmas: string
	rejection levels in each iteration
crradius: float
	cosmic ray expansion radius in pixels
crthresh: float
	rejection propagation threshold
badinpdq: int
	data quality flag bits to reject
crmask: bool
	flag CR in input DQ imageS?
shadcorr: bool
	perform shading shutter correction?
verbose: bool, optional
	Print verbose time stamps?

Command Line Options for the wf3rej executable

wf3rej.e input output [-options]

Input can be a single file
Where the options include:


	t: print the timestamps
	v: verbose
	shadcorr: perform shading shutter correction?
	crmask: flag CR in input DQ images?
	table <filename>: the crrejtab filename
	scale <number>: scale factor for noise
	init <med|min>: initial value estimate scheme
	sky <none|median|mode>: how to compute sky
	sigmas: rejection leves for each iteration
	radius <number>: CR expansion radius
	thresh <number> : rejection propagation threshold
	pdq <number>: data quality flag bits to reject