2.1 Types of WFC3 Files

Science data taken in orbit by WFC3 are received from the Space Telescope Data Capture Facility and sent to the STScI Operational Pipeline Unified System (OPUS) pipeline, where the data are unpacked, keyword values extracted from the telemetry stream, and the science data reformatted and repackaged into raw (uncalibrated) FITS files by the Generic Conversion process (see Section 1.1.1 of the Introduction to the HST Data Handbooks). All WFC3 science data products are two-dimensional images stored in Multi-Extension FITS format files. For each exposure taken with WFC3, there is one FITS file with a unique 9-character rootname followed by a 3-character suffix: rootname_xxx.fits. The rootname identifies the observation and the suffix denotes what type of file it is (see Chapter 5 of the Introduction to the HST Data Handbooks for more details on HST file names).
WFC3 data files are given the following definitions:

  • An exposure is a single multi-extension FITS file, the atomic unit of HST data.

  • A dataset is a collection of files having a common rootname.
  • An association is a set of exposures that can be meaningfully combined into a single product (e.g. a set of dithered exposures, or a series of CR-SPLIT exposures)
  • A sub-product is a dataset created by combining a subset of the exposures in an association.
  • A product is a dataset created by combining sub-products of an association.

2.1.1 Data Files and Suffixes

The suffixes used for WFC3 raw and calibrated data products are described in Table 2.1 and closely mimic those used by ACS and NICMOS.

Table 2.1: WFC3 data file suffixes.

File Suffix

Description

Units

Uncalibrated


 

 

_raw

raw data

DN

Distortion Corrected


 

 

_drz

UVIS and IR calibrated exposure, corrected for geometric distortion

e¯/s

_drc

UVIS calibrated exposure, corrected for geometric distortion and CTE

e¯/s

Intermediate


 

 

_rac_tmp

UVIS CTE corrected raw data, no other calibration

DN

_blv_tmp

overscan-trimmed UVIS exposure

DN

_blc_tmp

overscan-trimmed UVIS exposure, CTE corrected exposure

DN

_crj_tmp

uncalibrated, cosmic-ray-rejected combined

DN

_crc_tmp

uncalibrated, cosmic-ray-rejected, CTE cleaned

DN

_ima

calibrated intermediate IR multiaccum image

 e¯/s

Calibrated


 

 

_flt

UVIS calibrated exposure

_flc

UVIS calibrated exposure including CTE correction

_flt

IR calibrated exposure

e¯/s

_flt

IR calibrated exposure (SCAN mode)

_crj

UVIS calibrated, cosmic-ray-rejected image

_crj

IR calibrated, cosmic-ray-rejected image

 e¯/s

_crc

UVIS calibrated, cr rejected, CTE cleaned image

Auxiliary


 

 

_asn

association file for observation set


 

_spt

telescope and WFC3 telemetry and engineering data.


_trl

trailer file contains processing messages


 


The initial input files to the calibration program calwf3 are the raw files (raw) from Generic Conversion and the association (asn) table, if applicable, for the complete observation set.

For UVIS images, temporary files are created once bias levels are subtracted and the overscan regions are trimmed. These temporary files have suffixes which also relate whether they have been corrected for Charge Transfer Efficiency (CTE) trailing. Files which have been corrected for CTE effects have a "c" as the third letter in their suffix, so it follows that "_blv_tmp" files have completed the BIASCORR stage but were not corrected for CTE like the "_blc_tmp" files. The CTE correction is the very first step in the correction process for UVIS datasets. If the save option is specified when calwf3 is called on a "_raw" image, then an intermediate "_rac" file will also be saved. This contains the original data with just the CTE correction applied.

The final output files will be renamed with the "_flt" or "_flc" suffixes after the standard calibrations (flat fielding, dark subtraction, etc.) are complete. The "_blv_tmp" files serve as input for cosmic ray rejection, if required. For UVIS CR-SPLIT and REPEAT-OBS exposures, a temporary CR-combined image is created and then renamed with either the "_crj" or "_crc" suffix once basic calibrations of that image are complete. With the addition of the CTE correction, which is the very first step, the pipeline now follows two distinct lines of processing, one with the CTE correction applied and one without. Users running raw data through calwf3 which has PCTECORR set to PERFORM should expect to see both sets of calibrated files when the tasks are complete.

For the IR detector, an intermediate MultiAccum (ima) file is the result after all calibrations are applied (dark subtraction, linearity correction, flat fielding, etc.) to all of the individual readouts of the IR exposure. A final step in calwf3 processing of IR exposures produces a combined, CR-cleaned, image from the individual readouts, which is stored in an "_flt" product file.

The final products of the calwf3 pipeline are not corrected for geometric distortion. AstroDrizzle is used to correct all WFC3 images for geometric distortion, whether they are taken as single exposures or as part of an association. AstroDrizzle is distributed as part of the Drizzlepac package, used for aligning and combining all HST images. For CR-SPLIT and REPEAT-OBS, AstroDrizzle supersedes the calwf3 cosmic-ray rejection processing and uses the individual "_flt" or "_flc" files directly as input, performing cosmic-ray rejection in the process of producing the final drizzled image from multiple exposures (see Table 2.2). This has significant advantages in cases where small numbers of CR-SPLIT images were obtained at a small number of different dither positions, because AstroDrizzle will use all the information from all the input files to produce the best cosmic-ray rejection. The resulting drizzled images should generally be useful for science, although subsequent reprocessing off-line may be desirable in some cases to optimize the data for specific scientific applications.

Table 2.2: The calwf3 and AstroDizzle input and output products.

UVIS
CALWF3AstroDrizzle




Output

CTE Corrected?

CR Rejected?

Distortion Corrected?

Input

Output

CTE Corrected?

CR Corrected?

Distortion Corrected?

Single

raw

flt

No

No

No

flt

drz

No

No

Yes

flc

Yes

No

No

flc

drc

Yes

No

Yes

CR-SPLIT

raw, asn

crj

No

Yes

No

flt, asn

drz

No

Yes

Yes

crc

Yes

Yes

No

flc, asn

drc

Yes

Yes

Yes

RPT-OBS

raw, asn

crj

No

Yes

No

flt, asn

drz

No

Yes

Yes

crc

Yes

Yes

No

flc, asn

drc

Yes

Yes

Yes

Dither Pattern

raw, asn

flt

No

No

No

flt, asn

drz

No

Yes

Yes

flc

Yes

No

No

flc, asn

drc

Yes

Yes

Yes

IR

CALWF3

AstroDrizzle

Image Type

Input

Output

CTE Corrected?

CR Rejected?

Distortion Corrected?

Input

Output

CTE Corrected?

CR Corrected?

Distortion Corrected?

Single

raw

ima

N/A

No

No

flt

drz

N/A

Yes

Yes

flt

N/A

Yes

No

RPT-OBS

raw, asn

crj

N/A

Yes

No

flt, asn

drz

N/A

Yes

Yes

Dither Pattern

raw, asn

flt

N/A

Yes

No

flt, asn

drz

N/A

Yes

Yes

For the UVIS channel, when there are multiple exposures, AstroDrizzle supersedes the calwf3 cosmic-ray rejection processing. It uses the "_flt" and "_flc" files produced by calwf3 as input, and performs cosmic-ray rejection in the process of producing the final distortion-corrected drizzled image.
For further information on drizzle please refer to the DrizzlePac Handbook for http://www.stsci.edu/files/live/sites/www/files/home/scientific-community/software/drizzlepac/_documents/drizzlepac-handbook.pdf.

2.1.2 Auxiliary Data Files

Association Tables (asn)

Association tables are useful for keeping track of the complex set of relationships that can exist between exposures taken with both WFC3 channels, especially with REPEAT-OBS, CR-SPLIT, and dithered exposures. Images taken at a given dither position may be additionally CR-SPLIT into multiple exposures (e.g., UVIS observations). In these cases, associations are built to describe how each exposure relates to the desired final product. As a result, WFC3 association tables can be used to create one or more science products from the input exposures. The relationships defined in the association tables determine how far through the calibration pipeline the exposures are processed and when the calibrated exposures get combined into sub-products for further calibration.
An association file has a single extension that is a binary FITS table. That table has three columns:

  • MEMNAME, the member name
  • MEMTYPE, the role which that member plays in the association
  • MEMPRSNT, a boolean value for whether the member is present or not.


The different MEMTYPE values are summarized in Table 2.3.

Table 2.3: Exposure types in WFC3 associations. The suffix "n" is appended to the MEMTYPE to denote multiple sets are present within a single association.

MEMTYPE

DESCRIPTION

EXP-CRJ

Input CR-SPLIT exposure (single set)

EXP-CRn

Input CR-SPLIT exposure for CR-combined image n (multiple sets)

PROD-CRJ

CR-combined output product (single set)

PROD-CRn

CR-combined output product n (multiple sets)

EXP-RPT

Input REPEAT-OBS exposure (single set)

EXP-RPn

Input REPEAT-OBS exposure for repeated image n (multiple sets)

PROD-RPT

REPEAT-OBS combined output product (single set)

PROD-RPn

REPEAT-OBS combined output product n (multiple sets)

EXP-DTH

Input dither exposure

PROD-DTH

Dither-combined output product


A sample association table for a two-position dithered observation with CR-SPLIT=2 is presented in Table 2.4 This example shows how both MEMNAME and MEMTYPE are used to associate input and output products. The MEMTYPE for each component of the first CR-SPLIT exposure, IxxxxxECQ and IxxxxxEGQ, are given the type EXP-CR1. The sub-product Ixxxxx011 is designated in the table with a MEMTYPE of PROD-CR1. The last digit of the product filename corresponds to the output product number in the MEMTYPE. A designation of zero for the last digit in the filename is reserved for the dither-combined product.

The column MEMPRSNT indicates whether a given file already exists. For example, if cosmic ray rejection has not yet been performed by calwf3, the PROD-CRn files will have a MEMPRSNT value of "no". The sample association table in Table 2.4 shows the values of MEMPRSNT prior to calwf3 processing.

Table 2.4: Sample association table ixxxxx010_asn.

MEMNAME

MEMTYPE

MEMPRSNT

IxxxxxECQ

EXP-CR1

Yes

IxxxxxEGQ

EXP-CR1

Yes

Ixxxxx011

PROD-CR1

No

IxxxxxEMQ

EXP-CR2

Yes

IxxxxxEOQ

EXP-CR2

Yes

Ixxxxx012

PROD-CR2

No

Ixxxxx010

PROD-DTH

No


Table 2.5: Extensions in UVIS _flt or _flc files after standard processing with calwf3 and AstroDrizzle.

Extension Number

Extension Name

Extension Description

Extension Type

Extension Dimensions

Data Format

Imset/Chip/Axis

0


 

Global Header


 

 

 

 

1

[SCI,1]

Science Image

Image

4096 × 2051

Float32

Imset 1 =

UVIS2

(CHIP2)

2

[ERR,1]

Error Array

Image

4096 × 2051

Float32

3

[DQ,1]

Data Quality Array

Image

4096 × 2051

Int16

4

[SCI,2]

Science Image

Image

4096 × 2051

Float32

Imset 2 =

UVIS1

(CHIP1)

5

[ERR,2]

Error Array

Image

4096 × 2051

Float32

6

[DQ,2]

Data Quality Array

Image

4096 × 2051

Int16

7

[D2IMARR,1]

Filter-independent

distortion

correction

(lithography)

Image

64 × 32

Float32

CHIP2, X axis

8

[D2IMARR,2]

Image

64 × 32

Float32

CHIP2, Y axis

9

[D2IMARR,3]

Image

64 × 32

Float32

CHIP1, X axis

10

[D2IMARR,4]

Image

64 × 32

Float32

CHIP1, Y axis

11

WCSDVARR

Filter-dependent

distortion

correction


 

Image

64 × 32

Float32

CHIP2, X axis

12

WCSDVARR

Image

64 × 32

Float32

CHIP2, Y axis

13

WCSDVARR

Image

64 × 32

Float32

CHIP1, X axis

14

WCSDVARR

Image

64 × 32

Float32

CHIP1, Y axis

15

WCSCORR

WCS change log

BINTABLE

14R × 24C


 

 

Note: At the time of publishing this document, only a subset of UVIS filters have calibrated filter-dependent distortion correction, and associated NPOLFILE. These filters are F225W, F275W, F336W, F350LP, F390M, F390W, F438W, F475W, F555W, F606W, F621M, F775W, F814W, F850LP, F953N. Images produced by the calwf3 pipeline for these filters have 15 extensions, images for the remaining filters have only 11 extensions (they have no WCSDVARR). The characterization of the filter-dependent distortion correction for the remaining UVIS filters is in progress.
 
In order to create a geometrically correct, drizzle-combined product, PROD-DTH exposures are combined only with AstroDrizzle, which executes after calwf3 has finished processing all members. PROD-RPT and PROD-CRJ products are combined using wf3rej and all output files have the “cr” extension.

Trailer Files (trl)

Each task used by calwf3 creates messages during processing that describe the progress of the calibration and are sent to STDOUT. In calibration pipelines written for other HST instruments, trailer files were created by simply redirecting the STDOUT to a file. Because multiple output files can be produced in a single run of calwf3, creating trailer files presents a unique challenge. Each task within calwf3 must decide which trailer file should be appended with comments and automatically open, populate, and close each trailer file.

calwf3 will always overwrite information in trailer files from previous runs of calwf3 while preserving any comments generated by Generic Conversion. This ensures that the trailer files accurately reflect the most recent processing performed. The string "CALWF3BEG" will mark the first comment added to the trailer file. If a trailer file already exists, calwf3 will search for this string to determine where to append processing comments. If it is not found, the string will be written at the end of the file and all comments will follow. Thus any comments from previous processing are overwritten and only the most current calibrations are recorded.

As each image is processed, an accompanying trailer file with the "*_trl.fits" suffix will be created. Further processing with calwf3 will concatenate all trailer files associated with an output product into a single file. Additional messages will then be appended to this concatenated file. This duplicates some information across multiple trailer files but ensures that for any product processed within the pipeline, the trailer file will contain processing comments from all the input files.

Linking trailer files together can result in multiple occurrences of the "CALWF3BEG" string. Only the first, however, determines where calwf3 will begin overwriting comments if an observation is reprocessed.

Support Files (spt)

The support files contain information about the observation and engineering data from the instrument and spacecraft that was recorded at the end of the observation. A support file can have multiple FITS image extensions within the same file. Each extension holds an integer (16-bit) image containing the data that populates the *_spt.fits header keyword values.