2.1.1 Data Files and Suffixes

At various stages in the pipeline calibration process, a given data file can contain not only the raw data sent by HST down to the ground along with various pertinent information, but also data that has been processed through various subsequent stages in the calibration pipeline and the associated information propagated into in the various headers related to those pipeline processing steps. The calibration pipeline, called calacs, will be described in Chapter 3. In this Chapter we provide an overview of the data structures and where they stand in the calibration stages.

In this Section we describe the ACS data files and the meaning of the suffixes attached to a given file. We describe the files and suffixes in the order of calibration sequences, thus from the lower level to the higher level of processing. File suffixes for ACS data products are given in Table 2.1, and are described below.

• Initial input files to the calibration pipeline:
  • Raw (raw.fits) files from Generic Conversion.
  • If applicable, an association table (asn.fits) for a complete observation set.

• A single fully-calibrated MAMA image is given the suffix flt.fits. MAMA images do not have an overscan region, and they are not affected by cosmic rays and CTE.
• Calibration of a single CCD image:
  • After the bias image and bias level are subtracted, and the overscan regions trimmed, a temporary file with the suffix blv_tmp.fits is created. By default, CTE-corrected images are also created for WFC data so calacs creates a temporary file with suffix blc_tmp.fits.
  • Upon completion of additional calibration steps (dark subtraction, flat-fielding, etc.), the temporary file is renamed with the suffix flt.fits. For WFC data, CTE-corrected images, with suffix flc.fits, are also created. The flt.fits and flc.fits files will later serve as input for AstroDrizzle.

• A "CR-SPLIT" CCD observation:
  • Raw images from a "CR-SPLIT" observation undergo bias image and bias level subtraction. These images are then combined at the cosmic ray rejection step in calacs to create a temporary image with suffix crj_tmp.fits. For WFC images with CTE corrections, a file with suffix is crc_tmp.fits is also created.
  • Other basic calibrations (dark subtraction, flat-fielding, etc..) are performed on the temporary combined image. It is then renamed with the crj.fits suffix, and crc.fits suffix for WFC CTE-corrected images.
  • Individual calibrated images (flt.fits/flc.fits) are also created for each exposure in the "CR-SPLIT" observation.

• Multiple CCD exposures:
  • If the multiple exposures are all dithered, only subsets that are CR-SPLIT together are stacked into crj.fits and crc.fits images. The rest would be independently processed to flt.fits and flc.fits images.

• A MAMA observation consisting of several repeated sub-exposures:
  • Calibrated flt.fits images are created for each sub-exposure.
  • A summed flat-fielded image is created, with suffix sfl.fits.

Table 2.1: ACS File Suffixes

2.1.2 Association Tables

Association tables describe and track the relationship or "associations" between data products for a set of observations. Such relationships include repeated exposures in observations, "CR-SPLIT" observations, and dithered observations. ACS association tables can be used to instruct calacs to create different levels of calibration products. These tables are particularly useful for keeping track of complex observations like an observation at a specific dither position that may be additionally split into multiple exposures. Please note that standard association tables only include exposures from a single visit. Edited association tables can also be used with calacs to create non-default calibration products (see Example 3 in Section 3.5).

ACS data files are given the following definitions:

• A single image from an exposure or sub-exposure is the "atomic unit" of HST data.
• A dataset is a collection of files having a common rootname (first nine characters of the image name).
• A sub-product is created by combining a subset of the exposures in an association.
• A product is created by combining sub-products, or in some cases, individual exposures (before they were incorporated into a sub-product), of an association.

An ACS association table has three primary columns: MEMNAME, MEMTYPE, and MEMPRSNT. An example of an association table is shown in Table 2.3.

• The column MEMNAME lists the name of each exposure in the association and names of calacs output products.
• The column MEMTYPE describes the role of a file in the association. A unique set of MEMTYPES, specific to ACS, were adopted to provide descriptions for multiple products. These types are summarized in Table 2.2.
• The MEMPRSNT column indicates the calibration status of each product.

Table 2.2: Exposure Types, or MEMTYPEs, in ACS Associations

An example of an association table is shown in Table 2.3. But first, to trace back its origins, begin by looking at the Phase II proposal commands that created the data. This example came from proposal GO-10605, visit 1, exposure log sheet number 1. The observations, using ACS/WFC, were taken as a two-point dither with a "CR-SPLIT=2" at each dither point. (NOTE: Although shown for our purposes in this example, we no longer recommend CR-SPLIT exposures at the same dither position, but suggest individually dithered single exposures instead.) The proposal's dither pattern specification looks like this:

Pattern_Number: 1
                    Primary_Pattern               Secondary_Pattern
Pattern_Type        ACS-WFC-DITHER-LINE
Pattern_Purpose     DITHER
Number_Of_Points    2
Point_Spacing       3.011
Line_Spacing        <none>
Coordinate_Frame    POS-TARG
Pattern_Orient      85.28
Angle_Between_Sides <none>
Center_Pattern      NO

The exposure log sheet commands used to execute these observations were as follows:

-------------------------------------------------------------------------------------------------------------
Exp | Target| Instr| Oper.| Aper | Spectral| Central| Optional|Num| Time| Special
Num | Name | Config| Mode| or FOV| Element| Waveln.| Parameters|Exp|| Requirements
-------------------------------------------------------------------------------------------------------------
1 MESSIER-081 ACS/WFC ACCUM WFC F555W CR-SPLIT=2 1 2900 S PATTERN 1 1-2
 -DWARF-A
-------------------------------------------------------------------------------------------------------------
(line 2 is not shown)

There are two pairs of four single calibrated images, one pair with suffix flt.fits, and the other, corrected for CTE, with suffix flc.fits.

• From the "CR-SPLIT" sub-exposures at dither point 1:
  
  j9cm01jvq_flt.fits and j9cm01jvq_flc.fits
j9cm01jwq_flt.fits and j9cm01jwq_flc.fits

• From the "CR-SPLIT" sub-exposures at dither point 2:
  
  j9cm01k2q_flt.fits and j9cm01k2q_flc.fits
  j9cm01k4q_flt.fits and j9cm01k4q_flc.fits

• The combined images created by AstroDrizzle, in the pipeline, from four flt.fits and four flc.fits images, respectively, are: 
  
j9cm01010_drz.fits and j9cm01010_drc.fits

Table 2.3: Contents of Association Table, j9cm01010_asn.fits

Column       1              2           3    
Label  ___MEMNAME____ ___MEMTYPE____ MEMPRSNT
     1 J9CM01JVQ      EXP-CR1        yes     
     2 J9CM01JWQ      EXP-CR1        yes     
     3 J9CM01K2Q      EXP-CR2        yes     
     4 J9CM01K4Q      EXP-CR2        yes     
     5 J9CM01010      PROD-DTH       yes     
     6 J9CM01011      PROD-CR1       yes     
     7 J9CM01012      PROD-CR2       yes    

• Rows 1 and 2 describe the association's first "CR-SPLIT" observation at dither point 1.
  • "MEMNAME" shows the image rootnames for two sub-exposures in the first "CR-SPLIT" observation.
  • "MEMTYPE" of "EXP-CR1" means that the images came from the association's first "CR-SPLIT" observation.
  • "MEMPRSNT" set to "yes" indicates that those images underwent standard calibrations in calacs.

• Rows 3 and 4 describe the association's second "CR-SPLIT" observation at dither point 2. The two images have rootnames "J9CM01K2Q" and "J9CM01K4Q." "EXP-CR2" means they came from the second CR-SPLIT observation, and "yes" indicates that those images underwent standard calibrations in calacs.
• In row 5, "PROD-DTH" and "yes" indicate that the single exposure images in the association were drizzle-combined in the pipeline by AstroDrizzle, and named with the rootname specified in the MEMNAME column. In this example, four flt.fits images were drizzled-combined to produce j9cm01010_drz.fits, and four flc.fits images were drizzle-combined to create j9cm01010_drc.fits. If "MEMPRSNT" had been set to "no," the AstroDrizzle step would have been omitted in the pipeline.
• In row 6, "PROD-CR1" and "yes" indicate that the two images from the first "CR-SPLIT" (rows 1 and 2) were combined with cosmic ray rejection by calacs to create an image with the rootname specified in the MEMNAME column. This was done for flt.fits and flc.fits images to create the files J9CM01011_crj.fits and J9CM01011_crc.fits, respectively.
• In row 7, "PROD-CR2" and "yes" indicate that the two images from the second "CR-SPLIT" (rows 3 and 4) were combined with cosmic ray rejection by calacs to create an image with the rootname specified in the MEMNAME column. This was done for flt.fits and flc.fits images to create the files j9cm01012_crj.fits and j9cm01012_crc.fits, respectively.

2.1.3 Trailer Files

Each task in the calacs package creates messages that describe the progress of the calibration; these messages are directed to STDOUT (STanDard OUTput), which simply means that processing messages appear on the screen during the calacs run.

In pipeline processing for first and second generation HST instruments, where data files were calibrated one at a time, trailer files were created by simply redirecting the STDOUT contents to a file. However, the ACS pipeline was designed to calibrate several images at a time (like those described in Section 2.1.2), and create different types of output files. Therefore, each task within the calacs package must decide how to populate the trailer files associated with each product.

calacs will always overwrite information in existing trailer files from previous runs of calacs while preserving any comments generated by Generic Conversion. This ensures that the trailer files accurately reflect the most recent processing performed by calacs. After the Generic Conversion entries, the string "CALACSBEG" marks the first comment added to a trailer file by calacs. If the trailer file already exists, calacs searches for this string, then begins to write new processing comments from that point onwards, over-writing previous calacs comments. If "CALACSBEG" is not found in an existing trailer file, calacs will write that string at the end of the trailer file, then continue populating the trailer file with calibration processing comments.

As each image is reprocessed, an accompanying trailer text file with the suffix "tra" (without the .fits extension) is created. The trl.fits file from the Archive has the same content, in FITS table format.

Following the processing hierarchy specified in the association table, information in trailer files belonging to images used for creating a higher level product will be included in the trailer file of that higher level product. In other words, the trailer file for any product processed by the pipeline will contain processing comments from trailers belonging to each input file.

Linking trailer files together can result in multiple occurrences of the "CALACSBEG" string. Only the first, however, determines where calacs will begin overwriting comments if an observation is reprocessed.

2.1.4 Hubble Advanced Products (HAP)

(The following was reproduced, with minor details changed, from Section 2.1.3 of the WFC3 Data Handbook.) In late-2020, the HST data calibration and archive pipelines began producing new Hubble Advanced Products (HAP) to be distributed through MAST. These include two new types (or levels) of products for a given dataset, referred to as 'Single Visit Mosaics' (SVMs) and 'Multi Visit Mosaics' (MVMs), available for download from either the MAST Discovery Portal or the MAST HST Search Interface.  An example of the file naming convention for standard (HST) and advanced (HAP) products is shown in Table 2.4.

SVMs comprise the data from a single HST visit which are aligned to a common astrometric reference frame and then drizzled onto the same north-up pixel grid. Compared to standard HST data products, these include improved relative alignment across different exposures (and/or filters) acquired within the same visit, enabling easy comparison of the drizzled products. When possible, sources in the HST images have been aligned directly to an external reference catalog (e.g. Gaia eDR3, GSC v2.4.2, or 2MASS) and include improved absolute astrometry in the image World Coordinate System (WCS), carried as HDRLET extensions in the FITS files.  SVM data products with both relative alignment (by filter) and absolute alignment to Gaia will contain the string 'FIT_SVM_GAIAeDR3' in the 'WCSNAME' keyword in the science extension of the image header.  The SVM drizzled images are used to generate point source and segment catalogs during pipeline processing. These catalogs supersede those produced by the Hubble Legacy Archive and will be the basis of the next version of the Hubble Source Catalog.  Availability of these SVM data products was announced in the December 2020 MAST Newsletter article, and more details on the software used to compute these data products is available in the DrizzlePac documentation for Single Visit Mosaic Processing.

In mid-2022, MAST began distributing a new type of Hubble Advanced Product (HAP) referred to as Multi-Visit Mosaics (MVM). These are cross-visit, cross-proposal mosaics, which combine public observations of fields observed multiple times by ACS/WFC, WFC3/UVIS, or WFC3/IR.  MVM data products combine all exposures falling within a pre-defined 0.2° x 0.2° ‘sky cell’ for each detector+filter drizzled onto a common, pre-defined pixel grid.  Note that the MVMs are large files (up to 1TB, depending on the number of input exposures in each sky cell). Because they combine observations acquired over a range of dates, the MVMs may have photometric errors of several percent or systematic alignment errors when combining visits with different catalog solutions. These products are therefore recommend to be used as ‘discovery images’ for comparing observations in different detectors and passbands and not for precise scientific analysis.  Availability of the MVM products was announced in the May 2022 MAST Newsletter article. More detail on the software used to compute these data products is available in the DrizzlePac documentation for Multi-Visit Mosaic Processing.  

For statistics on the alignment for each HST detector, uncertainties in the WCS solutions when aligning to different reference catalogs, and examples of SVM and MVM data products, see 'Improved Absolute Astrometry for ACS and WFC3 Data Products' (ACS ISR 2022-03).

Table 2.4: MAST standard data products (HST) and Hubble Advanced Products (HAP) available for a sample dataset 'jfcp25010' taken in the F814W filter.