5.2 AstroDrizzle The New Drizzle Workhorse

AstroDrizzle uses flat-field calibrated images (flc.flt/flt.fits, c0m.fits for WFPC2) as input files. There are two main components to the process: (1) create mask files for bad pixels and cosmic rays; (2) drizzle-combine the input images using the mask files, while applying geometric distortion corrections, to create a “clean” distortion-free combined image.

There are three components to mask files used by AstroDrizzle. Information about bad pixels flagged in the input images data quality array are taken into account. A “static pixel mask” is created by identifying pixels with abnormally low values in each image. The third component is a cosmic ray masks, one for each input flc.flt/flt.fits image. Creating cosmic ray masks is a major part of AstroDrizzle, so it is briefly described below (with more detailed information available later in this section):

• Each input flc.flt/flt.fits image is drizzled to create an undistorted copy in the output frame, where the WCS information in the single-drizzled image header is used to align it with respect to a reference image.
• The aligned single-drizzled images are combined to create a “clean” median image that approximates the appearance of a clean distortion-free combined image.
• The “clean” median image is “blotted” (or reverse-drizzled) back to the distorted image frames of each input flc.flt/flt.fits image.
• By comparing each flc.flt/flt.fits file with its counterpart median image, cosmic rays and other bad pixels can be identified and stored in a mask file.

These mask files are used in the final step where each input image isdrizzle-combined to create a clean distortion-free output image. Depending on the number of images and type of observations, users can elect to adjust parameters to improve the quality of the final drizzle-combined image.

Regarding WFC3/IR Image

For IR images, the steps "Drizzle separate images," “Create Median Image,” “Blot back the Median Image,” and “Remove Cosmic Rays with Deriv, Driz-CR” in AstroDrizzle may be turned off since cosmic rays are flagged in calwfc3 as part of the “up-the-ramp fitting.” However, it may be useful to run those steps (using a different bit flag, like 8192, for “cosmic rays” found during AstroDrizzle processing) to flag additional detector artifacts not present in the data quality arrays of the calibrated images. Note that it is very important to subtract the sky prior to drizzling the final image, or the science array will be compromised by increased noise.

Detailed Description of the AstroDrizzle algorithm

Each subsection below describes each step of the AstroDrizzle task in detail. 

Set-up and Initialize astrodrizzle Parameters

Several initialization steps are performed by AstroDrizzle to set some preferences on how the code will execute and to ensure the input data files have the correct format. These include specifying the input files, providing a name and format for the output image, and resetting cosmic ray flags in the data quality arrays of input images.

Create a Static Mask Containing Permanent Bad Pixels

A detector-based static mask is created to flag bad pixels in the input images. First, severely negative or low value pixels are identified in each image. These low values could be caused by over-subtraction of bad pixel values when applying the dark image correction in routine calibrations, or from post-pipeline over-subtraction of high sky levels. For each detector in the list of input images¹, the individual masks are combined to create a master-mask file for use in the final drizzle step. The clipping level for identifying bad pixels is provided by the static_sig parameter and is an integer multiple of the RMS below the image’s mode.

How the Static Mask is Used in Subsequent astrodrizzle Steps

The bad pixel mask used in the single-drizzle step in AstroDrizzle contains the static mask created as described in the previous paragraph, combined with a mask of bad pixels described in the input image’s data quality arrays, minus any bits marked as good in the driz_sep_bits parameter.

The bad pixel mask used in the final drizzle-combining step in AstroDrizzle contains all bad pixels from all input images as used in the single-drizzle step, as well as the mask of pixels identified as cosmic rays as determined from running the driz_cr step.

Users should consider the details of their science image and decide if creating this mask is appropriate for the resulting science. For instance, if the image field is very crowded, or if it contains mostly nebulous or extended objects, the statistics used to create the static mask could be heavily skewed resulting in a mask that flags valid pixels as bad.

Perform Sky Subtraction

In the pipeline, AstroDrizzle sky computation is based on the statistical distribution of pixel values in an input image. It is performed by iterative sigma-clipping, starting with the full range of pixel intensity values, to calculate the standard deviation. By default, pixel values deviating from the median value (specified by parameter skystat) by over four sigma are rejected, and this operation is repeated for a total of five iterations. The median value of the final distribution is used as the sky value, stored in the header keyword MDRIZSKY in the original flc.flt/flt.fits image (the original input images themselves are NOT sky-subtracted at this stage).

The default method provides good results for a wide range of datasets. However, the sky is occasionally slightly overestimated, and such cases can be improved in post-pipeline AstroDrizzle processing.

If the sky in a field is strongly affected by bright sources, the primary purpose of a background measurement is to ensure that there are no offsets in the background levels between exposures prior to combination. The variations in background levels may be due to any number of external sources (e.g., bright earth limb), but in a field of view dominated by bright source, there would not be enough sky pixels beyond the source to accurately determine the true sky. This leads to the possibility that sky variations from one exposure to the next could ultimately make the true background difficult or potentially impossible to determine.

Flat-field calibrated images (flc.flt/flt.fits, c0m.fits for WFPC2) delivered to users from the Archive are not sky-subtracted. However, the sky values calculated by AstroDrizzle in the pipeline are held in the science extension header keyword MDRIZSKY. The computed sky value is subtracted from a copy of the input image; these sky-subtracted image copies are later used in the final image combination step.

For cameras with multiple detectors (four in WFPC2, two each in ACS/WFC and WFC3/UVIS), sky values in an image are measured separately for each detector. The lowest measured detector sky value is adopted for all detectors for that exposure. This is based on the premise that the pixel intensity distribution will be higher in one or more detectors with extended or bright objects, thereby overestimating their sky value. In other words, the sky value in the detector least affected by bright or extended objects would provide a more realistic sky determination.

In the pipeline, sky subtraction is performed for broad-band data, and turned off for narrow-band data and UV observations that are “dark” (i.e., those that have no geocoronal emission in the filter bandpass). This is done for the following reasons:

• The sky background through such filters is much lower than through optical broad-band filters
• These observations are often extended diffuse emission-line targets with flux that is much higher than the background. In such situations, the automated sky calculations by AstroDrizzle will lead to the introduction of errors larger that the background value. However, if the user is able to determine an accurate background level for such images, from parts of the image with measurable sky, then a user-specified sky value may be used to propagate those values directly to AstroDrizzle.

Sky subtraction is generally recommended for optimal flagging and removal of cosmic rays, if the sky background is more than a few electrons. However, for some science applications where the sky should not be removed, the final drizzle step can be performed with no sky subtraction (skysub set to False in astrodrizzle). If sky subtraction is turned off, then the final_pixfrac parameter should also be set to 1; otherwise, variations in sky between images will add noise to the data.

Methodology

Steps for determining sky value for each image:

1. For each chip in the input image, obtain an estimate of the sky background by computing the clipped statistics (set by the parameters skyclip, skystat, skylsigma, and skyusigma).
2. The lowest sky value relative to area on the sky (in units of electrons/square-arcsecond) among all the chips is then adopted for all chips in the image. That value is then rescaled to the plate scale of each chip to become the sky value for that chip.
3. The science header of each input image is updated with this value.
4. Sky subtraction is not applied to the flc.fits/flt.fits images. Instead, the sky value is subtracted from the chip, on-the-fly, during the process of drizzling the image.

The default estimate of the sky background relies on computing the clipped median for each input chip. Offline processing by the user, however, could use a number of statistical operations to independently estimate the sky background, including the mode, mean and median.

A user-determined sky value, already subtracted from the input image, can be entered as a new keyword in the input file header–this keyword name is given by the astrodrizzle skyuser parameter. This user-supplied value will then get used, instead of AstroDrizzle’s computed sky value, when the sky-subtraction step is run.

However, if the sky subtraction step is turned off, AstroDrizzle will still use the sky value recorded in the MDRIZSKY keyword when performing single-image drizzling and cosmic ray identification, as it provides the only indication of the background sky level needed for the statistical computations used to identify cosmic rays.

Create Separately Drizzled and Registered Images

Each input image is drizzled to create an undistorted copy in the output frame². These images are registered with respect to a reference image (by default, the first image in the input list) using the WCS information in their headers, and will be used in the next three steps to create cosmic ray masks for each input image.

These single-drizzled images are generated using the full WCS information provided by each of the input images. Any WCS offsets will show up as misaligned sources in the single-drizzled images. In post-pipeline processing, these images can be used for refining the image registration for each of the input images, if the user decides to use image registration software other than the tweakreg task.

The default values in this step will define the output frame that will ideally include all input pixels from all the WCS-registered single-drizzled input images. But like the final drizzle step, the output frame may be redefined at this step using different parameter settings

Create a Median Image

Single-drizzled WCS-aligned images created from the previous step are combined to create a median image with statistical rejection of bad pixels from the image stack. It serves as an approximation of a combined distortion-free “clean” image with most cosmic rays and hot pixels removed. This median combination gets performed section-by-section from the input single-drizzled images. Each section corresponds to a contiguous set of lines from each image taking up no more than 1 Mb in memory, so that combining 10 input images would only require 10 Mb of memory for this step. This median image will then be used in the next two steps for creating cosmic ray and bad pixel masks.

Blot Median Image

The blot software takes a distortion-corrected image and applies (not removes) the full distortion model to recreate the original distorted input image. In other words, a drizzled image is “reverse-drizzled” to recreate the original distorted image.

In the previous step, a median image was created by combining the distortion-corrected single-drizzled images to generate an initial guess for the cosmic ray-cleaned combined output image. In this step, the median image is “blotted” to create clean versions of each input image at each of their respective dither locations.

This is done so that in the next step, these blotted images will be directly compared to their counterpart original distorted input images for detection of bad pixels, hot pixels, and cosmic rays to create bad pixel masks.

Create a Cosmic Ray Mask for Each Image

In this step, the software compares each blotted image with its counterpart original flc.fits/flt.fits image to detect spurious pixels such as cosmic rays and hot pixels. The spurious pixels are flagged in cosmic ray masks that will be used in the final drizzle-combine step. Spurious pixels are identified by comparing the original input flc.fits/flt.fits image and its corresponding cleaned blot image, and blot derivative image. (A derivative image provides a measure of how sharp the edges of sources are in the image.)

The process of identifying cosmic rays and other bad pixels requires the following operations:

• Take the spatial derivative of each blotted image from the previous step. This derivative image is used to estimate the degree to which the value of the blotted estimate has been distorted by errors in the image offset (computed from the WCS of each input image), and the blurring effect of taking the median.
• Compare each original image with the corresponding blotted image. Where the difference is larger than what would be expected from noise statistics or an error in the shift, the suspect pixel is masked. The statistical limit gets set by the user through the first term of the driz_cr_snr parameter.
• Repeat the previous step on pixels adjacent to pixels already masked, using a more stringent comparison criterion specified by the second term of the driz_cr_snr parameter.

The deriv algorithm uses the blotted median image to compute the absolute value of the difference between each pixel and its four surrounding neighbors; for each pixel, the largest of these four values is then used by the driz_cr algorithm to flag cosmic rays and other blemishes, such as satellite trails. Where the difference is larger than can be explained by noise statistics, the flattening effect of taking the median, or an error in the shift (the latter two effects are estimated using the image derivative), the suspect pixel is masked. Cosmic rays are flagged using the following rule:

where scale is defined as the multiplicative factor applied to the derivative, deriv.

This expression is used to determine if the difference between the data image and the blotted image is large enough to require masking. noise is calculated using a combination of the detector read noise and the poisson noise of the blotted median image, plus the sky background.

The user must specify two cut-off signal-to-noise (SNR) values for determining whether a pixel should be masked: the first for detecting the primary cosmic ray, and the second for masking lower-level bad pixels adjacent to those found in the first pass.

Since cosmic rays often extend across several pixels, the adjacent pixels make use of a slightly lower SNR threshold. If desired, a third-pass cosmic ray rejection can be carried out by “growing” the cosmic rays via the driz_cr_grow parameter.

When driz_cr_corr is set to yes, the task will create both a cosmic ray mask image (suffix sci?_crmask.fits where “?” is the extension number) and a clean version of the original input images (suffix crclean.fits), where flagged pixels are replaced by pixels from the blotted median. The cosmic ray masks are multiplied by the static pixel mask from the first step and masks created from pixels flagged as “bad” in the flc.fits/flt.fits DQ array, to create a final mask for each image. The optional parameter crbit allows the user to assign an alternate flag value to cosmic rays, and this flag will be written to the DQ array of each input image.

Create a Final Distortion-Free Combined Image

In the final step, the original input images are drizzle-combined, using the DQ, static, and cosmic ray masks to exclude bad pixels, hot pixels, and cosmic rays from the final image computation. The resulting drizzle-combined image is a registered, cosmic ray-cleaned, distortion-free, photometrically flat science image with associated weight and context images. By default, the output image will be written out as a single multi-extension FITS file, but the user could have them written out as separate simple FITS images.

The output frame, just like the single drizzle step, can be redefined using specific parameter settings; otherwise, default values will be used to include all the input pixels from all the WCS-registered input images.

Overriding Instrument-Specific Parameters

It is possible to override information in the image headers by setting these parameters directly so AstroDrizzle can work with data generated or modified by the user rather than working with data that came directly from the HST Archive.

A Note about Photometry and Weights in AstroDrizzle

AstroDrizzle combines data using weights, a practice that may be new to some users who routinely add images together. Summing images works if the sky is stable, or if the objects’ Poisson noise (not read noise) is the primary source of noise in the image; this is essentially equivalent to setting final_wht_type to EXP (exposure time weighting) in astrodrizzle.

However, for much of the work done with HST, sources of interest are faint compared to the sky and the budget of the noise is dominated by read noise and Poisson noise of the background. In this case, the inverse variance map (IVM) is a good weighting option because it weights each pixel by the sky, dark noise, and read noise, adjusting appropriately for the value of the flat field.

But then, why not do a full calculation of the noise in each pixel in each image instead? This can be done by setting final_wht_type to ERR in astrodrizzle. This weight is what statisticians call the “minimum variance estimator.” It has the smallest statistical error of all choices of weights. However, it is not an “unbiased minimum variance estimator.” If a pixel in an image has, by chance, a few lesser counts than one would expect on average in a particular pixel, then calculating the noise for that pixel based on the number of counts will underestimate the noise and thus overestimate the significance of that pixel. The inverse problem occurs if the pixel happens to be a bit brighter than average. As a result, using this weight produces a small bias (usually no more than one to two percent in HST images). Sources in the resulting final image will be slightly fainter than they really are.

In general, photometric software available to most users does not take advantage of the final weight map. Some users may wish to write their own code to use the information in the weights. In this case, there are several choices:

• If exposure time weighting (EXP) was used, the final image could be multiplied by the weight image to get the number of source counts in each pixel to estimate photon noise.
• For IVM weighting, and when there’s no concern for very bright sources in the field, the final IVM output weight map could be used.
• If the ERR array was used, this already provides a very good estimate of the noise (but not the bias).

For users who want an accurate weight map for use in photometry, a good approach may be to first run astrodrizzle using final_wht_type set to either EXP (when bright sources are most important) or IVM (if the faint sources are the primary objects of interest). This creates the final science image but not the final weight map. To create a final weight map, astrodrizzle needs to be run again, using the same weighting scheme as in the first drizzle; however, the original weight maps (drz_wht.fits images) should be drizzled rather than the images. This will give an output “image” that fully estimates the error at each pixel. At present, there’s no easy way for the user to do this. Perhaps this could be done by swapping in the flc.fits/flt.fits ERR array for the science (SCI) array, then running astrodrizzle on the modified images. If there is demand for this approach, it may be implemented as a feature in a future version of AstroDrizzle.

AstroDrizzle Memory Usage

The AstroDrizzle software will estimate the memory requirements for processing a particular set of data during the “Initialization Step” and print a summary to the screen.

The reported requirements include the size of the final output image in pixels, the number of cores to be used, and an estimate of the amount of memory required for processing. AstroDrizzle will estimate the maximum amount of memory required during processing based on the size of the input files, the number of cores to be used, and the size of the final output products. This information can be used by the user to determine whether or not their processing run will require more memory or disk space than installed on their system, allowing them to interrupt the processing as soon as possible using Ctrl-C so that input parameters can be reset as needed.

The astrodrizzle parameter called num_cores will enable parallel processing using multiple cores, greatly reducing processing time. This has been enabled for the driz_separate and driz_cr steps. Memory usage will be a prime concern for those who rely on this parallel processing feature as each core will require a separate copy of the output array to be stored in memory along with having each input image in memory. For example, processing four WFC3/UVIS images to generate a 5000 × 5000 mosaic on a quad-core system will require over 1.3Gb of memory during the single drizzle step. Users with little RAM are, therefore, strongly advised to run with num_cores=1.

An example of the processing time is shown in Figure 5.1 for three WFC3/UVIS images that were combined using astrodrizzle with default parameters. The total processing time using 6 cores was only 3.6 minutes. When num_cores was set to 1, the actual memory usage was lower, but the total processing time increased to 6.3 minutes.

1 AstroDrizzle input images can be from more than one instrument, such as cases where ACS/WFC and WFC3/UVIS images are being combined.

2 Two output files are created: a single-drizzled science image and a single-drizzled weight image, both in a simple FITS format.