3.2 Pipeline Overview

Pipeline processing is carried out by two separate image processing packages: calacs corrects for instrumental effects to produce calibrated products. AstroDrizzle corrects for geometric distortion, performs cosmic ray rejection based on the individual images of the same scene, and attempts to correct for hot pixels using dithered MAST images.

3.2.1 calacs: Image Calibration

calacs controls the image calibration steps based on the type of images and/or associations:

  • For CCD images, bias level removal, as well as charge transfer efficiency (CTE) corrections for WFC images, are performed on each image using the task acsccd, followed by the task acscte.
  • If the association has CCD images created from "CR-SPLIT" observations, or from repeated non-dithered exposures (several sub-exposures per observation), the task acsrej is used to combine the images and reject cosmic rays.
  • The task acs2d continues with routine image reductions; MAMA images are dark-subtracted (omitted by default) and flat-fielded. CCD images—single images and images combined with acsrej—are, as appropriate, dark-subtracted, post-flash-subtracted, and flat-fielded.
  • SBC MAMA images in the association, created from repeated non-dithered exposures (several sub-exposures per observation), are summed using the task acssum.

Calibrated data products from calacs (with suffixes flt.fits/flc.fits, crj.fits/crc.fits, and sfl.fits) are in units of electrons.

calacs standard calibration final products have suffixes flt.fits and crj.fits. When run manually, if desired, calacs also creates temporary intermediate data products, such as those with the suffix blv_tmp.fits

For WFC images, calacs produces counterpart data files that have undergone pixel-based CTE corrections. The CTE-corrected final data products have suffixes flc.fits and crc.fits, to complement flt.fits and crj.fits files, respectively. When calacs is run manually for WFC data, temporary intermediate data products have the letter "" c"  in the suffix to indicate that it has also been corrected for CTE, such as blc_tmp.fits as the counterpart for blv_tmp.fits

Beyond calacs, the pipeline also produces two sets of drizzled data for WFC, with suffixes drz.fits and drc.fits. In this document, unless the context is specifically for one or the other, standard and CTE-corrected files mentioned by suffix will appear separated by a " /" for instance, flt.fits/flc.fits.

While intermediate steps in calacs make use of sky subtraction values to perform certain steps, such as in identifying cosmic rays, all data products created by the pipeline will not be sky subtracted.

Calibrated products from the pipeline may still contain some artifacts such as hot pixels, cosmic rays, and, in the case of post-SM4 WFC subarray images, bias striping. To correct for post-SM4 subarray bias striping, after bias subtraction but before the rest of the calibration steps, it is currently necessary to use a standalone Python routine (acs_destripe_plus) outside the calacs package (see Section 4.2.1 and Example 5 in Section 3.5.2). Residual hot pixels and cosmic rays may be rejected from dithered images using AstroDrizzle to process associations created from observations taken with the "POS TARG" or dither "PATTERN" special requirements in Phase II proposals.

calacs and Single Exposures

Each single-exposure raw image undergoes standard detector calibrations in calacs, such as bias subtraction, dark subtraction, and flat-fielding (see Section 3.3) to create a flt.fits image.

For full frame WFC images,1 by default, a CTE-corrected image with the suffix flc.fits is also created. This is done regardless of whether those single images will be combined in later calacs steps. Data in the "SCI" (science image) and "ERR" (error image) extensions of a calibrated flt.fits/flc.fits image are in units of electrons, whereas the raw ACS images are in units of counts.

calacs and Combining of Sub-Exposures

Depending on how multiple sub-exposures were executed, calacs has two different ways to combine them.

  1. If CCD images are flagged in an association table as belonging to a "CR-SPLIT" or repeated observations set,2 the following steps are performed by calacs:
    - Bias subtraction, dark subtraction, and flat-fielding are performed on each raw image. For WFC images with PCTECORR set to PERFORM, CTE correction is performed prior to dark subtraction.
    - Images are combined with cosmic ray rejection.

    - The combined image is flat-fielded to create a calibrated image file with suffix crj.fits. For images where CTE-correction is applicable, a CTE-corrected combined image with the suffix crc.fits is also created.A single fully-calibrated MAMA image is given the suffix flt.fits. (MAMA images do not have an overscan region, and they are not affected by cosmic rays and CTE.)
  2. If SBC MAMA images are flagged in an association table as belonging to a set of repeated sub-exposures, calacs takes the following actions:

- Each image is fully calibrated and flat-fielded to produce flt.fits files.

- The flt.fits images are summed to create an image with the suffix sfl.fits.

Note that each single exposure image from a "CR-SPLIT" or repeated sub-exposures set will also be calibrated individually to produce a flt.fits/flc.fits image for later use in AstroDrizzle if the header value EXPSCORR="PERFORM" (which is currently the default).

calacs and Dithered Exposures

calacs produces a calibrated flt.fits/flc.fits file for each single-exposure image in an association, including those created from using dither "PATTERN" and "POS TARG" special requirements in the Phase II proposal.

If there were two or more repeated sub-exposures at a pointing, calacs produces a cosmic ray-rejected combined image, crj.fits/crc.fits, for CCD data. For SBC MAMA data, a summed image is created with the suffix sfl.fits.

However, calacs will not combine images from multiple positions within an association (like those from a dither pattern). Later in the pipeline, after calacs processing is completed, flt.fits/flc.fits images will be corrected for geometric distortion and combined, with cosmic ray and hot pixel removal, by AstroDrizzle (crj.fits/crc.fits and sfl.fits files are not used in AstroDrizzle).

Table 3.1: Input and Output Image Suffixes from calacs and AstroDrizzle for Various Observing Modes

Image Type

Image Suffixes (suffix.fits)

calacs
Input

calacs Output

calacs
Cosmic Ray Rejected?

AstroDrizzle Input

AstroDrizzle Output

AstroDrizzle
Cosmic Ray Rejected?

Single

raw

flt/flca

No

flt/flc

drz/drc

No

Repeated
Observations

CCD

asn or raw

flt & crj
flc & crc

Yes
Yes

asn & flt, or flt
asn & flc, or flc

drz/drc

Yes

MAMA

asn or raw

flt & sfl

n/ab

asn & flt, or flt

drz

n/ab

CR-SPLIT


 

asn or raw

flt & crj
flc & crc

Yes

asn & flt, or flt
asn & flc, or flc

drz/drc

Yes

Dither PATTERN or POS TARG


 

asn or raw

flt & crj,
flc & crc,
flt & sfl (SBC only)

Maybec

asn & flt, or flt
asn & flc, or flc

drz/drc

Yes

CTE-corrected products from MAST only apply to full frame WFC images. Users wishing to correct old format (pre-Cycle 24, October 2016) 2K subarray images and any new format subarray images for CTE effects may do so using acs_destripe_plus in the acstools Python package. Other old format subarray sizes are not compatible with the pixel-based CTE correction.
SBC MAMA detectors are not sensitive to cosmic rays.
cDepends on the image type. For "CR-SPLIT" exposures, calacs creates crj.fits/crc.fits combined images. For repeated MAMA exposures, calacs creates a summed sfl.fits file. However, combined images are not used as input to AstroDrizzle. Only flt.fits/flc.fits files are the primary input to AstroDrizzle; they can also be represented by an association table, if one is available.

3.2.2 AstroDrizzle Processing in the Pipeline

During pipeline processing, calibrated ACS data that belong to an association are corrected for geometric distortion and drizzle-combined with cosmic ray rejection by AstroDrizzle. If the associated images are dithered, they are aligned using the WCS information in their headers before being drizzle-combined. If there is no association table, each single-exposure ACS image is drizzled to correct for geometric distortion.

The resulting drizzled image, in units3 of electrons/second, is written to a file with the suffix drz.fits/drc.fits. (For WFC, data from the two chips are mosaicked together as one image.)

In the pipeline, AstroDrizzle and its related software rely on these reference files:

  • IDCTAB reference table for a description of the distortion model.
  • D2IMFILE reference file for filter independent detector pixel grid defects or irregularities in X,Y in each WFC CCD chip, only for WFC images.
  • NPOLFILE reference file is for the non-polynomial filter dependent part of distortion, for residual distortions not accounted for by the IDCTAB distortion solution coefficients (nor corrected by the D2IMFILE, in the case of WFC images).

Information about geometric distortion from these reference files are stored as SIP header keywords and as FITS extensions in the flt.fits/flc.fits images. Please see ACS ISR 15-06 for more information on the distortion and Section 2.2 for details on the file structure.

The resulting drizzled images from the pipeline may be useful for science as-is, although subsequent manual reprocessing with AstroDrizzle is recommended, and sometimes required, for optimizing the data. For more information, please refer to the DrizzlePac website.

3.2.3 When is MAST Processing not Appropriate?

The goal of the ACS pipeline is to provide data calibrated to a level suitable for initial evaluation and analysis for all users. Observers require a detailed understanding of the calibrations applied to their data and the ability to repeat, often with improved products, the calibration process at their home institutions. There are several occasions when data processed via MAST from the Archive are not ideal, requiring off-line interactive processing:

  • Running calacs with different reference files than those specified in the image header.
  • Running calacs with non-default calibration switch values.
  • When images must be cleaned of artifacts such as hot pixels, cosmic rays, and residual artifacts such as bias striping.

Images combined by AstroDrizzle in the pipeline were produced using parameters that are suitable for the widest range of scientific applications (see ACS ISR 17-02 for more details). Some datasets, however, could benefit significantly from manual reprocessing, for instance, by using a different pixel scale or by modifying cosmic ray rejection parameters. The same AstroDrizzle task used in the pipeline is also available to users in AstroConda for off-line processing of flt.fits/flc.fits images retrieved from the Archive. For more information, please refer to DrizzlePac website.



1 Since 2014, users can apply CTE corrections to old format (pre-Cycle 24, October 2016) WFC 2K subarray images and all new format subarrays using the acs_destripe_plus tool in acstools. Only full-frame images are CTE-corrected in the archive. 

2 The Phase II proposal's exposure log sheet line parameter "Number_of_Interations" has an integer value greater than 1.

3 The final drizzled image's unit type is set in the AstroDrizzle task parameter final_units; the choices are cps (counts per second, the default value) or counts. The unit for counts is specified in the image header keyword BUNIT. For ACS images, BUNIT is set to ELECTRONS. Therefore, ACS drizzled images are, by default, in units of electrons/second.