3.5.1 Requirements for Manual Recalibration

About this section:

This section contains information and examples about how to manually calibrate ACS data using calacs outside of the usual MAST pipeline. We also provide a Jupyter notebook for a full ACS/WFC image reduction example with the relevant Python code, which is another excellent resource for anyone learning how to calibrate ACS/WFC observations.

Software Requirements

An overview of HST image data analysis software is available in the Introduction to the HST Data Handbooks. STScI no longer supports IRAF/PyRAF for data analysis, and instead recommends the use of Python. Please note that Astroconda has been phased out as of February 1st, 2023, and is no longer supported by STScI. Stenv incorporates many of the Python tools used to calibrate and analyze HST data including calacs and AstroDrizzle. Please create a stenv environment before running the following examples

Data Retrieval

The Introduction to the HST Data Handbooks contains an overview of data retrieval from the Archive.

Setting up "jref"

Before any recalibration can be done, the directory location for calibration reference files must be defined. For ACS, this directory is referred to as "jref", and is used as a prefix in the reference file names in the image header (i.e., jref$qb12257gj_pfl.fits). In a Bash shell, export is used to set jref to a directory location. For example:
 
Bash INPUT:

Verify your jref location using echo $jref prior to using these examples.

Using Non-default Reference files and Calibration Switches

By default, the Archive provides calibrated images processed with the latest available reference files at processing time. In order to use non-default reference files and calibration switch settings, manual recalibration is required. These non-default settings have to be manually updated in the uncalibrated data (the raw FITS files) before running calacs. The table below gives a list of all of the calibration switches in the primary header.

Table 3.7: Calibration Switch Selection Criteria
The first column shows calibration switch header keywords. The second column is a description of the keyword, and the third column shows the default values.

Since calibration switches are tied to specific reference files, it is imperative to re-populate the reference files in the headers after updating calibration keywords in raw files. This can be accomplished by running crds bestrefs on the command line, in the directory where the updated, raw files are stored.

$ crds bestrefs --files *raw.fits --update-bestrefs

For more information on the crds tool, please refer to the documentation.

Post-SM4 WFC Image Artifact Correction for WFC Subarray Images

Certain artifacts present in post-SM4 WFC subarray images, including bias striping, bias shift, and CTE trailing, are not fully handled by calacs. The amplitude of the bias shift and the CTE trailing in the 512- and 1024-pixel subarrays of HST Cycles 17–23 are not well characterized. However, the 2K subarray CTE trailing is near-identical to full-frame readout, and is corrected in calacs with the full-frame algorithm. All the new subarray modes introduced for use in Cycle 24 onwards will have calacs correction of bias shift and CTE trailing. Note that CTE correction for subarray images is not performed automatically, and the image must first be de-striped (see below for more information).

Bias striping in post-SM4 subarray images is not corrected within calacs. Subarray stripe removal requires fitting across the entire image region, as discussed in Section 3.4.1, using the stand-alone task acs_destripe_plus in the acstools suite. (See also Example 5 in Section 3.5.2)

Bypassing the doPhot Step

During the doPhot step, done when PHOTCORR=PERFORM, pixel values and units are not changed. This step only calculates the values of the calibrated image's photometric header keywords, such as the inverse sensitivity conversion factor (PHOTFLAM). Please refer to Section 3.4.4 "doPhot - Photometry Keyword Calculator" for more information.

When populating the photometric keywords during the doPhot step, calacs uses the CRDS reference file IMPHTTAB. Some users find it cumbersome to keep up with the updates, and prefer to simply copy the photometric keyword values from the original calibrated data into the raw image's primary header, then run calacs with the PHOTCORR switch set to OMIT.

3.5.2 calacs Examples

In these examples, Python is used to run calacs. These Python functions are simply wrappers around the C code that comprises calacs, and as such they may also be run outside of the Python environment.

Example 1: Reprocessing a Single Exposure Using a Different Bias File

The following example uses WFC data from the Cycle 23 CAL/ACS program 14398 (PI: Chiaberge), which monitors the ACS/WFC CTE using observations of the globular cluster NGC 104 (47 Tucanae) in the F502N filter. The dataset names are JD1Y04RLQ and JD1Y04RNQ, and are part of the association JD1Y04010. Download the association (ASN), RAW, and FLT FITS files by using the MAST Search Form. In the "Dataset" field, use the association name JD1Y04010 to search. Once the results are populated, select "Download Data" then "Choose Files" to select the asn.fits, raw.fits, and flt.fits files to download. In our examples, we have stored the FITS files in a subdirectory called cal_fits/ so that we can copy the files to the current working directory while preserving the original files for other examples. Because this is an association that was made from the two parts of a CR-SPLIT exposure, the association name here ends with a "0", while the raw FITS files end with a "1".

Python INPUT:

from astropy.io import fits
from astropy.table import Table
import shutil
 
shutil.copy('cal_fits/jd1y04010_asn.fits', '.')
 
#Adding the Table() function from astropy.table makes some
#tasks more convenient and makes the formatting of the table
#more readable in the notebook
 
asn_hdu = fits.open('jd1y04010_asn.fits')
asn_table = Table(asn_hdu[1].data)
asn_hdu.close()
 
asn_table

Python OUTPUT:

For the purposes of this first example, assume that the observations are not part of an association. This example will illustrate the steps required to reprocess a single exposure (JD1Y04RLQ) after changing the bias reference file from the default value to a file specified by the user. Note that we will also CTE correct this image, but we will omit this for future examples. The CTE correction is computationally intensive and thus may take a long time to run on your machine. By default, it will use all of your machine's processors.

1. Make sure that the jref keyword is set to the ACS reference files as described earlier in this section.
2. Copy the original file into the current working directory as shown above for the association file.
3. To determine which bias reference file name was specified in the image header, open the raw FITS file and access the value for the header keyword BIASFILE 

   Python INPUT: 

   shutil.copy('cal_fits/jd1y04rlq_raw.fits', '.')
   hdu_raw = fits.open('jd1y04rlq_raw.fits', mode = 'update')
   hdr0 = hdu_raw[0].header
    
   print(hdr0['BIASFILE'])

   Python OUTPUT:

   jref$6641828tj_bia.fits

4. Edit the primary header of the raw image to enter the name of the new bias file called mybias.fits (for this example, mybias.fits is a copy of the default bias reference file listed in the header).

   Python INPUT:

   hdr0['BIASFILE'] = '<mybias.fits>'

5. Set the PHOTCORR processing step to OMIT and copy the values of two useful photometric keywords from the calibrated image (retrieved from the Archive) to the raw image (see "Bypassing the PHOTCORR" for more information.) The PHOTFLAM keyword will be useful for photometric calibration during image analysis, and PHOTMODE is useful as a concise description of the observation mode. Note: For WFC images, the keywords need to be edited for both SCI extensions, however the values in the extensions are identical.
6. The pixel-based CTE correction algorithm was updated in July 2017. So any files you may have downloaded from the Archive prior to August 3, 2017 will not have the new PCTETAB file in the header. In these cases, calacs will not run on files with PCTETAB set to an old reference file unless either PCTECORR is set to OMIT or PCTETAB is set to the latest reference file (alternatively, one can force calacs to run with the old method as well). In this example, we edit PCTETAB to point to the most recent *_cte.fits file in the jref directory. 

   Python INPUT:

   hdr0['PHOTCORR'] = 'OMIT'
    
   #Grab the values of the PHOTFLAM and PHOTMODE 
   #keywords from the FLT file in cal_fits/
   #Note that we have mixed the nomenclature to use the [1]
   #and ['sci',1] notations for the image extensions. Please
   #see the ACS Data Handbook Section 2.2 "ACS File Structure"
   #for more information.
   hdu_flt = fits.open('cal_fits/jd1y04rlq_flt.fits')
   hdr_flt = hdu_flt[1].header
   hdr1, hdr4 = hdu_raw['sci',1].header, hdu_raw['sci',2].header
   hdu_flt.close()
   photflam, photmode = hdr_flt['PHOTFLAM'], hdr_flt['PHOTMODE']
    
   #Update the header keywords
   hdr1['PHOTFLAM'] = photflam
   hdr1['PHOTMODE'] = photmode
   hdr4['PHOTFLAM'] = photflam
   hdr4['PHOTMODE'] = photmode
   hdr0['PCTETAB'] = 'jref$16k1747tj_cte.fits'
    
   #Close the raw file to push the changes to the header
   hdu_raw.close()

7. We can now run calacs on the raw FITS file, which will be processed with the user-specified bias reference file mybias.fits. The result will be two calibrated images: one with the flt.fits extension; and a CTE-corrected one with the flc.fits extension. A trailer file with the .tra extension describing the steps taken by calacs is also produced.

   NOTE: Importing calacs will print a message (see the code block below). This is done on import only and not every time calacs is run.
   Python INPUT: 

   from acstools.calacs import calacs
    
   calacs('jd1y04rlq_raw.fits')

   The following tasks in the acstools package can be run with TEAL:
   acs2d       acs_destripe        acs_destripe_plus
   acsccd      acscte                 acsrej
   acssum     calacs  
   PixCteCorr is no longer supported. Please use acscte.

    
   The cell below cleans up the current working directory by removing the files created by this example. Only run this when you are finished with this example.
   Python INPUT: 

   import os, glob
   files = glob.glob('jd1y04*') + ['<mybias.fits>']
   for x in files:
       if os.path.exists(x):
           os.remove(x)

Example 2: Reprocessing Multiple Exposures Taken with "CR-SPLIT" Within an Association

This example uses the same data from Example 1 and illustrates the steps required to reprocess an ACS association after changing the bias reference file from the default value to a file specified by the user. The steps required are similar to the previous example, with a few modifications.

1. The association table shows the images from two exposures. The MEMTYPE value "EXP-CRJ" indicate that those two images were created from a "CR-SPLIT" exposure. The cosmic ray-rejected product created by calacs has the rootname J1DY04010

   Python INPUT: 

   from astropy.io import fits
   from astropy.table import Table
   from acstools.calacs import calacs
   import shutil, glob
    
   shutil.copy('cal_fits/jd1y04010_asn.fits', '.')
   shutil.copy('cal_fits/<mybias.fits>', '.')
    
   #Adding the Table() function from astropy.table makes some
   #tasks more convenient and makes the formatting of the table
   #more readable in the notebook
   asn_hdu = fits.open('jd1y04010_asn.fits')
   asn_table = Table(asn_hdu[1].data)
   asn_hdu.close()
    
   asn_table

   Python OUTPUT:

   MEMNAME	MEMTYPE	MEMPRSNT
   str14	str14	bool
   JD1Y04RLQ	EXP-CRJ	True
   JD1Y04RNQ	EXP-CRJ	True
   JD1Y04011	PROD-CRJ	True

2. Copy the images from the cal_fits/ directory to the current working directory and edit the headers to point BIASFILE to mybias.fits and update the PHOTCORR, PHOTFLAM, and PHOTMODE keywords appropriately as in Example 1. As mentioned previously, in this and future examples, we will set the PCTECORR keyword to OMIT for the sake of expediency. Then run calacs.
   
   Python INPUT:

   #Here we have hardcoded the file names, but we could also 
   #access the rootnames from asn_table as asn_table['MEMNAME'][0]
   #and asn_table['MEMNAME'][1].
   shutil.copy('cal_fits/jd1y04rlq_raw.fits', '.')
   shutil.copy('cal_fits/jd1y04rnq_raw.fits', '.')
    
   #Use the glob function to create a list of the raw FITS files
   #that we can loop over
   raw_files = glob.glob('*raw.fits')
   for x in raw_files:
       
       #Open the primary header of the raw image and get the
       #rootname for accessing the calibrated files
       hdu_raw = fits.open(x, mode = 'update')
       hdr0, hdr1, hdr4 = hdu_raw[0].header, hdu_raw['sci',1].header, hdu_raw['sci',2].header
       rootname = hdr0['ROOTNAME'].lower()
       cal_image = 'cal_fits/' + rootname + '_flt.fits'
       
       #Get the PHOTFLAM and PHOTMODE keywords from the
       #calibrated image science header
       hdu_flt = fits.open(cal_image)
       hdr_flt = hdu_flt['sci',1].header
       hdu_flt.close()
       photflam, photmode = hdr_flt['PHOTFLAM'], hdr_flt['PHOTMODE']
       
       #Update the raw image headers
       hdr0['BIASFILE'] = 'mybias.fits'
       hdr0['PCTETAB']  = 'jref$16k1747tj_cte.fits'
       hdr0['PHOTCORR'] = 'OMIT'
       hdr0['PCTECORR'] = 'OMIT'
       hdr1['PHOTMODE'] = photmode
       hdr1['PHOTFLAM'] = photflam
       hdr4['PHOTMODE'] = photmode
       hdr4['PHOTFLAM'] = photflam
       
       #Close the raw file to push the changes to the header
       hdu_raw.close()
       
   #Run calacs on the association file
   calacs('jd1y04010_asn.fits')

   
   The products are three calibrated images: 1) two flt.fits files, one for each of the EXP-CRJ images in the association; and 2) one cosmic-ray cleaned image that is the combination of the two input images using the acsrej algorithm with no CTE correction applied.

 
The cell below cleans up the current working directory by removing the files created by this example. Only run this when you are finished with this example.
Python INPUT: 

import os, glob
files = glob.glob('jd1y04*') + ['<mybias.fits>']
for x in files:
    if os.path.exists(x):
        os.remove(x)

Example 3: Combining Exposures From Multiple Associations

This example illustrates the steps required to combine two sets of repeated observations to create a cosmic ray-rejected combined image. The data for this exercise comes from the ACS calibration program, CAL/ACS program 9662, that observed NGC 104 (47 Tucanae) using the HRC with a clear filter. The associations' names are J8IS01020 and J8IS01040, and they again, as in the examples immediately above, may be more difficult to find unless accessing them via the Program Information page and going to the Visit Status link and clicking on the Archive information links there. Each association comprises of two 1-second exposures, and share the same target pointing.

1. Copy the files from the cal_fits/directory to the current working directory, open the association FITS files and merge the two tables into one. Only use the first two rows of each file, as the third row of each table contains a combined product that we are going to exclude.

   from astropy.io import fits
   from astropy.table import Table, vstack
   from numpy import rec
   from acstools.calacs import calacs
   import shutil
    
   shutil.copy('cal_fits/j8is01020_asn.fits', '.')
   shutil.copy('cal_fits/j8is01040_asn.fits', '.')
    
   asn_hdu1 = fits.open('j8is01020_asn.fits')
   asn_hdu2 = fits.open('j8is01040_asn.fits')
    
   asn_tab1 = Table(asn_hdu1[1].data)
   asn_tab2 = Table(asn_hdu2[1].data)
   asn_hdu1.close()
   asn_hdu2.close()
    
   #Copy one of the association files to a new file called merged_asn.fits
   #for which we will overwrite the data
   shutil.copy('j8is01020_asn.fits', 'merged_asn.fits')
   merged_asn = fits.open('merged_asn.fits', mode = 'update')
    
   merged_table = vstack([asn_tab1[0:2], asn_tab2[0:2]])
   merged_table

   Python OUTPUT:

   MEMNAME	MEMTYPE	MEMPRSNT
   str56	str56	bool
   J8IS01J2Q	EXP-RPT	True
   J8IS01J3Q	EXP-RPT	True
   J8IS01J8Q	EXP-RPT	True
   J8IS01J9Q	EXP-RPT	True

2. Now that we have a table with each of the individual exposures in it, we need to add one more row to contain the cosmic-ray rejected product name (J8IS0xx1).

   Python INPUT: 

   merged_table.add_row(['J8IS01xx1', 'PROD_CRJ', 'yes'])
   merged_table

   Python OUTPUT:

   MEMNAME	MEMTYPE	MEMPRSNT
   str56	str56	bool
   J8IS01J2Q	EXP-RPT	True
   J8IS01J3Q	EXP-RPT	True
   J8IS01J8Q	EXP-RPT	True
   J8IS01J9Q	EXP-RPT	True
   J8IS01XX1	PROD_CRJ	True

   Python INPUT: 

   #Replace the data in the merged_asn.fits file with the new table.
   #The Astropy Table format is not the same as the FITS file table
   #format, so we must convert the Astropy format to a binary FITS
   #table HDU first, then replace the data array in the association
   #file with the data array of the newly created table HDU.
   #Then close the file to push the changes.
   fits_table = fits.table_to_hdu(merged_table)
   merged_asn[1].data = fits_table.data
   merged_asn.close()

3. As in the other examples, set PHOTCORR to OMIT and copy the PHOTFLAM and PHOTMODE keywords from the calibrated files. As this is HRC data, there is only one science extension, and thus we do not need to update the headers of multiple extensions.
   
   Python INPUT: 
   

   #Here we have used a more advanced syntax to compress our FOR
   #statement into one line called a list comprehension. However,
   #we are simply creating a list of the root names from the merged
   #association if they contain the letter "Q" (i.e., they are exposures
   #and not products)
   rootnames = [x.rstrip().lower() for x in merged_table['MEMNAME'] if 'Q' in x]
   for x in rootnames:
       shutil.copy('cal_fits/' + x + '_raw.fits', '.')
       raw_hdu = fits.open(x + '_raw.fits', mode = 'update')
       flt_hdu = fits.open('cal_fits/' + x + '_flt.fits')
       hdr0 = raw_hdu[0].header
       hdr1 = raw_hdu['sci',1].header
       flt_hdr = flt_hdu['sci',1].header
       flt_hdu.close()
       hdr0['PHOTCORR'] = 'OMIT'
       hdr1['PHOTFLAM'] = flt_hdr['PHOTFLAM']
       hdr1['PHOTMODE'] = flt_hdr['PHOTMODE']
   #Run calacs on the association
   calacs('merged_asn.fits')

The products are five images: four flt.fits calibrated images; and one crj.fits (cosmic-ray cleaned stack of the four flt.fits files).

The cell below cleans up the current working directory by removing the files created by this example. Only run this when you are finished with this example.
Python INPUT: 

import os, glob
files = glob.glob('j8is01*') + ['merged_asn.fits', 'merged.tra']
for x in files:
    if os.path.exists(x):
        os.remove(x)

Example 4: Reprocessing Images Taken as part of a Dither Pattern

The following example uses WFC data from the GOODS program GO program 9425. These observations are from visit 54, exposure 219; the target name was "CDF-South," observed with the F606W filter. The images were part of a 2-point line dither pattern with an exposure time of 480 seconds each, with rootnames J8E654C0Q and J8E654C4Q.

This example illustrates the steps needed to reprocess data that are part of a dither pattern using a non-default dark reference file.

1. Copy the association file and images to the current working directory, then view the contents of the association file. 

   from astropy.io import fits
   from astropy.table import Table
   from acstools.calacs import calacs
   import shutil
   shutil.copy('cal_fits/j8e654010_asn.fits', '.')
   shutil.copy('cal_fits/mydark.fits', '.')
   asn_hdu = fits.open('j8e654010_asn.fits')
   asn_table = Table(asn_hdu[1].data)
   asn_hdu.close()
   asn_table

   Python OUTPUT:

   MEMNAME	MEMTYPE	MEMPRSNT
   str14	str14	bool
   J8E654C0Q	EXP-RPT	True
   J8E654C4Q	EXP-RPT	True
   J8E654010	PROD-DTH	True

2. Edit the global image header for all the raw images to insert the name of the new dark reference file, mydark.fits (as in Example 1, we have copied the original dark file from the jref directory to the current working directory and renamed it mydark.fits).
3. Edit the PHOTCORR, PHOTFLAM, and PHOTMODE keywords as in previous examples.

4. Since mydark.fits does not have a counterpart CTE-corrected dark reference file, set PCTECORR to OMIT so that CTE-corrected images are not generated. 

   #Get the rootnames of the exposures directly from the association
   #table and copy the raw files from the cal_fits/ directory to the
   #current working directory
   raw_files = [asn_table['MEMNAME'][0].rstrip().lower(),
   			 asn_table['MEMNAME'][1].rstrip().lower()]
   
   for x in raw_files:
   	shutil.copy('cal_fits/' + x + '_raw.fits', '.')
   	#Get the raw and FLT image headers
   	hdu_raw = fits.open(x + '_raw.fits', mode = 'update')
   	hdu_flt = fits.open('cal_fits/' + x + '_flt.fits')
   	hdr0, hdr1, hdr4 = hdu_raw[0].header, hdu_raw['sci',1].header, hdu_raw['sci',2].header
   	hdr_flt = hdu_flt['sci',1].header
   	hdu_flt.close()
   	#Grab the photometry keywords from the FLT header
   	photmode, photflam = hdr_flt['PHOTMODE'], hdr_flt['PHOTFLAM']
   	#Set all of the necessary keywords in the raw FITS file
   	hdr0['DARKFILE'] = 'mydark.fits'
   	hdr0['PHOTCORR'] = 'OMIT'
   	hdr0['PCTECORR'] = 'OMIT'
   	hdr1['PHOTMODE'] = photmode
   	hdr1['PHOTFLAM'] = photflam
   	hdr4['PHOTMODE'] = photmode
   	hdr4['PHOTFLAM'] = photflam
   	#Close the raw FITS file to push the changes to the header
   	hdu_raw.close()
   #Run calacs on the association file
   calacs('j8e654010_asn.fits')

The result of this calacs run will be two calibrated images, one for each exposure in the association. This time, there will only be flt.fits files, not CTE-corrected flc.fits files, as we have set PCTECORR to OMIT. These files can then be used in AstroDrizzle to create a drizzled image.

The cell below cleans up the current working directory by removing the files created by this example. Only run this when you are finished with this example.
Python INPUT:

import os, glob
files = glob.glob('j8e654*') + ['mydark.fits']
for x in files:
	if os.path.exists(x):
		os.remove(x)	

Example 5: CTE Correction On Sub-array Images

The ACS Instrument Team implemented new flight software in Cycle 24 that changed the way CCD sub-array images are read out. This change makes it possible to apply the pixel based CTE correction to sub-array images. Here we illustrate the steps necessary to implement the correction in post processing.

The correction is implemented by using a tool within the acstools python package called acs_destripe_plus. This tool is useful for when the built-in calacs de-striping algorithm using overscans is insufficient or unavailable. The use of this tool to apply the CTE correction is limited to the following cases:

1. All 2K sub-arrays taken after SM4.
2. Smaller sub-arrays taken after October 2016.

For this example we will use the raw observation of the 47 Tuc calibration field with rootname JD5702JPQ, which we have copied to a new file called filename.fits for the following example. These data were taken using the WFC1B-1K aperture. This is a 1024 x 2048 pix sub-array.

Update the image header to turn on the PCTECORR switch and update reference file information as necessary. Subarray images do not have the PCTETAB and DRKFILE reference file names populated automatically, so these header keywords are updated with the code below.

PCTECORR = "PERFORM"
PCTETAB = "jref$19i16323j_cte.fits"
DRKCFILE = "jref$name_of_appropriate_dark_file_dkc.fits"
 
Python INPUT : 

from astropy.io import fits

with fits.open('filename.fits',mode='update') as hdu:
	hdu[0].header['PCTECORR'] = 'PERFORM'
	hdu[0].header['PCTETAB'] = 'jref$19i16323j_cte.fits'
	hdu[0].header['DRKCFILE'] = 'jref$19j14351j_dkc.fits'

Run acs_destripe_plus

Finally, to run the correction run the following commands in python:
Python INPUT: 

from acstools import acs_destripe_plus
 
acs_destripe_plus.destripe_plus('filename.fits',cte_correct=True)

The tool will take some time to run. In an eight-core machine it takes three minutes for the CTE correction to complete.

This tool will give you and FLT and an FLC file. The FLC file is the one that has been CTE corrected.

Correct acs_destripe_plus WCS

The subarray products produced by this process do not have the proper WCS information in the header. This is normally done by the pipeline via an additional call to AstroDrizzle. You can do this yourself with two python commands:
Python INPUT: 

from stwcs import updatewcs
 
updatewcs.updatewcs('*flc.fits')

With that, you should have calibrated files ready for scientific analysis or to process using Astrodrizzle.