2.1.1 Data Files and Suffixes

At various stages in the pipeline  calibration process, a given data file can contain not only the raw data sent by HST down to the ground along with various pertinent information, but also data that has been processed through various subsequent stages in the calibration pipeline and the associated information propagated into in the various headers related to those pipeline processing steps. The calibration pipeline, called calacs, will be described in Chapter 3. In this Chapter we provide an overview of the data structures and where they stand in the calibration stages.

In this Section we describe the ACS data files and the meaning of the suffixes attached to a given file. We describe the files and suffixes in the order of calibration sequences, thus from the lower level to the higher level of processing. File suffixes for ACS data products are given in Table 2.1, and are described below.

• Initial input files to the calibration pipeline:
  • Raw (raw.fits) files from Generic Conversion.
  • If applicable, an association table (asn.fits) for a complete observation set.

• A single fully-calibrated MAMA image is given the suffix flt.fits. MAMA images do not have an overscan region, and they are not affected by cosmic rays and CTE.
• Calibration of a single CCD image:
  • After the bias image and bias level are subtracted, and the overscan regions trimmed, a temporary file with the suffix blv_tmp.fits is created. By default, CTE-corrected images are also created for WFC data so calacs creates a temporary file with suffix blc_tmp.fits.
  • Upon completion of additional calibration steps (dark subtraction, flat-fielding, etc.), the temporary file is renamed with the suffix flt.fits. For WFC data, CTE-corrected images, with suffix flc.fits, are also created. The flt.fits and flc.fits files will later serve as input for AstroDrizzle.

• A "CR-SPLIT" CCD observation:
  • Raw images from a "CR-SPLIT" observation undergo bias image and bias level subtraction. These images are then combined at the cosmic ray rejection step in calacs to create a temporary image with suffix crj_tmp.fits. For WFC images with CTE corrections, a file with suffix is crc_tmp.fits is also created.
  • Other basic calibrations (dark subtraction, flat-fielding, etc..) are performed on the temporary combined image. It is then renamed with the crj.fits suffix, and crc.fits suffix for WFC CTE-corrected images.
  • Individual calibrated images (flt.fits/flc.fits) are also created for each exposure in the "CR-SPLIT" observation.

• Multiple CCD exposures (e.g., CR-SPLIT):
  • If the multiple exposures are all dithered, only subsets that are CR-SPLIT together are stacked into crj.fits and crc.fits images. The rest would be independently processed to flt.fits and flc.fits images.

• A MAMA observation consisting of several repeated sub-exposures:
  • Calibrated flt.fits images are created for each sub-exposure.
  • A summed flat-fielded image is created, with suffix sfl.fits.

Table 2.1: ACS File Suffixes

2.1.2 Association Tables

Association tables describe and track the relationship or "associations" between data products for a set of observations. Such relationships include repeated exposures in observations, "CR-SPLIT" observations, and dithered observations. ACS association tables can be used to instruct calacs to create different levels of calibration products. These tables are particularly useful for keeping track of complex observations like an observation at a specific dither position that may be additionally split into multiple exposures. Please note that standard association tables only include exposures from a single visit. Edited association tables can also be used with calacs to create non-default calibration products (see Example 3 in Section 3.5).

ACS data files are given the following definitions:

• A single image from an exposure or sub-exposure is the "atomic unit" of HST data.
• A dataset is a collection of files having a common rootname (first nine characters of the image name).
• A sub-product is created by combining a subset of the exposures in an association.
• A product is created by combining sub-products, or in some cases, individual exposures (before they were incorporated into a sub-product), of an association.

An ACS association table has three primary columns: MEMNAME, MEMTYPE, and MEMPRSNT. An example of an association table is shown in Table 2.3.

• The column MEMNAME lists the name of each exposure in the association and names of calacs output products.
• The column MEMTYPE describes the role of a file in the association. A unique set of MEMTYPES, specific to ACS, were adopted to provide descriptions for multiple products. These types are summarized in Table 2.2.
• The MEMPRSNT column indicates the calibration status of each product.

Table 2.2: Exposure Types, or MEMTYPEs, in ACS Associations

An example of an association table is shown in Table 2.3. But first, to trace back its origins, begin by looking at the Phase II proposal commands that created the data. This example came from proposal GO-10605, visit 1, exposure log sheet number 1. The observations, using ACS/WFC, were taken as a two-point dither with a "CR-SPLIT=2" at each dither point. (NOTE: Although shown for our purposes in this example, we no longer recommend CR-SPLIT exposures at the same dither position, but suggest individually dithered single exposures instead.) The proposal's dither pattern specification looks like this:

Pattern_Number: 1
                    Primary_Pattern               Secondary_Pattern
Pattern_Type        ACS-WFC-DITHER-LINE
Pattern_Purpose     DITHER
Number_Of_Points    2
Point_Spacing       3.011
Line_Spacing        <none>
Coordinate_Frame    POS-TARG
Pattern_Orient      85.28
Angle_Between_Sides <none>
Center_Pattern      NO

The exposure log sheet commands used to execute these observations were as follows:

-------------------------------------------------------------------------------------------------------------
Exp | Target| Instr| Oper.| Aper | Spectral| Central| Optional|Num| Time| Special
Num | Name | Config| Mode| or FOV| Element| Waveln.| Parameters|Exp|| Requirements
-------------------------------------------------------------------------------------------------------------
1 MESSIER-081 ACS/WFC ACCUM WFC F555W CR-SPLIT=2 1 2900 S PATTERN 1 1-2
 -DWARF-A
-------------------------------------------------------------------------------------------------------------
(line 2 is not shown)

A standard calibrated data retrieval from the Archive for images taken by Visit 1, exposure 1, are listed below. (Intermediate calacs products from the Archive have to be specifically requested, therefore, the crj.fits and crc.fits files are not included in the delivery.)

j9cm01010_asn.fits  j9cm01jvq_flt.fits  j9cm01jwq_flt.fits  j9cm01k2q_flt.fits  j9cm01k4q_flt.fits
j9cm01010_drz.fits  j9cm01jvq_flc.fits  j9cm01jwq_flc.fits  j9cm01k2q_flc.fits  j9cm01k4q_flc.fits
j9cm01010_drc.fits  j9cm01jvq_spt.fits  j9cm01jwq_spt.fits  j9cm01k2q_spt.fits  j9cm01k4q_spt.fits
j9cm01010_spt.fits  j9cm01jvq_trl.fits  j9cm01jwq_trl.fits  j9cm01k2q_trl.fits  j9cm01k4q_trl.fits
j9cm01010_trl.fits

There are two pairs of four single calibrated images, one pair with suffix flt.fits, and the other, corrected for CTE, with suffix flc.fits.

• From the "CR-SPLIT" sub-exposures at dither point 1:
  
  j9cm01jvq_flt.fits and j9cm01jvq_flc.fits
j9cm01jwq_flt.fits and j9cm01jwq_flc.fits

• From the "CR-SPLIT" sub-exposures at dither point 2:
  
  j9cm01k2q_flt.fits and j9cm01k2q_flc.fits
  j9cm01k4q_flt.fits and j9cm01k4q_flc.fits

• The combined images created by AstroDrizzle, in the pipeline, from four flt.fits and four flc.fits images, respectively, are: 
  
j9cm01010_drz.fits and j9cm01010_drc.fits

Table 2.3: Contents of Association Table, j9cm01010_asn.fits

Column       1              2           3    
Label  ___MEMNAME____ ___MEMTYPE____ MEMPRSNT
     1 J9CM01JVQ      EXP-CR1        yes     
     2 J9CM01JWQ      EXP-CR1        yes     
     3 J9CM01K2Q      EXP-CR2        yes     
     4 J9CM01K4Q      EXP-CR2        yes     
     5 J9CM01010      PROD-DTH       yes     
     6 J9CM01011      PROD-CR1       yes     
     7 J9CM01012      PROD-CR2       yes    

• Rows 1 and 2 describe the association's first "CR-SPLIT" observation at dither point 1.
  • "MEMNAME" shows the image rootnames for two sub-exposures in the first "CR-SPLIT" observation.
  • "MEMTYPE" of "EXP-CR1" means that the images came from the association's first "CR-SPLIT" observation.
  • "MEMPRSNT" set to "yes" indicates that those images underwent standard calibrations in calacs.

• Rows 3 and 4 describe the association's second "CR-SPLIT" observation at dither point 2. The two images have rootnames "J9CM01K2Q" and "J9CM01K4Q." "EXP-CR2" means they came from the second CR-SPLIT observation, and "yes" indicates that those images underwent standard calibrations in calacs.
• In row 5, "PROD-DTH" and "yes" indicate that the single exposure images in the association were drizzle-combined in the pipeline by AstroDrizzle, and named with the rootname specified in the MEMNAME column. In this example, four flt.fits images were drizzled-combined to produce j9cm01010_drz.fits, and four flc.fits images were drizzle-combined to create j9cm01010_drc.fits. If "MEMPRSNT" had been set to "no," the AstroDrizzle step would have been omitted in the pipeline.
• In row 6, "PROD-CR1" and "yes" indicate that the two images from the first "CR-SPLIT" (rows 1 and 2) were combined with cosmic ray rejection by calacs to create an image with the rootname specified in the MEMNAME column. This was done for flt.fits and flc.fits images to create the files J9CM01011_crj.fits and J9CM01011_crc.fits, respectively.
• In row 7, "PROD-CR2" and "yes" indicate that the two images from the second "CR-SPLIT" (rows 3 and 4) were combined with cosmic ray rejection by calacs to create an image with the rootname specified in the MEMNAME column. This was done for flt.fits and flc.fits images to create the files j9cm01012_crj.fits and j9cm01012_crc.fits, respectively.

2.1.3 Trailer Files

Each task in the calacs package creates messages that describe the progress of the calibration; these messages are directed to STDOUT (STanDard OUTput), which simply means that processing messages appear on the screen during the calacs run.

In pipeline processing for first and second generation HST instruments, where data files were calibrated one at a time, trailer files were created by simply redirecting the STDOUT contents to a file. However, the ACS pipeline was designed to calibrate several images at a time (like those described in Section 2.1.2), and create different types of output files. Therefore, each task within the calacs package must decide how to populate the trailer files associated with each product.

calacs will always overwrite information in existing trailer files from previous runs of calacs while preserving any comments generated by Generic Conversion. This ensures that the trailer files accurately reflect the most recent processing performed by calacs. After the Generic Conversion entries, the string "CALACSBEG" marks the first comment added to a trailer file by calacs. If the trailer file already exists, calacs searches for this string, then begins to write new processing comments from that point onwards, over-writing previous calacs comments. If "CALACSBEG" is not found in an existing trailer file, calacs will write that string at the end of the trailer file, then continue populating the trailer file with calibration processing comments.

As each image is reprocessed, an accompanying trailer text file with the suffix "tra" (without the .fits extension) is created. The trl.fits file from the Archive has the same content, in FITS table format.

Following the processing hierarchy specified in the association table, information in trailer files belonging to images used for creating a higher level product will be included in the trailer file of that higher level product. In other words, the trailer file for any product processed by the pipeline will contain processing comments from trailers belonging to each input file.

Linking trailer files together can result in multiple occurrences of the "CALACSBEG" string. Only the first, however, determines where calacs will begin overwriting comments if an observation is reprocessed.