3.5 Recalibration of STIS Data

Sometimes the default pipeline calibration, performed shortly after the data were obtained from the telescope, is not the best possible calibration for a specific science program. There are a number of reasons why it may be desirable to recalibrate STIS data. The most likely reasons include:

• More appropriate reference files have become available since the data were taken. CCD darks and biases are examples of reference files that are updated frequently, but they require some time to be installed in the pipeline. Further refinement of the dark files can be achieved by using daily dark exposures.
• Contemporaneous CCD flat fields were obtained with the science data for G750L or G750M NIR observations to remove fringing.
• Some steps need to be repeated with different input parameters. For example, one may wish to re-perform the cosmic ray rejection or the 1-D spectral extraction after adjusting the input parameters. The best target and background extraction regions for extracting 1-D spectra can depend on the science goals of the program.
• Integer pixel dithers were performed along the slit to move hot pixels in a series of CCD spectroscopic exposures. These exposures are not associated and are thus not combined in pipeline processing. The _flt images must be aligned before they can be combined with ocrreject.
• Spectral extractions will be made with heights less than the default size to isolate targets with small separations along the slit, or individual rows in a rectified spectral image of an extended target will be analyzed. More accurate spectral traces are needed to improve the extraction or rectification.

The simplest way to re-calibrate data with the most appropriate reference files is to request the data from the archive again. However, to tailor the calibration to individual preferences, it may be beneficial to run calstis on the user's local machine, or to use tasks that improve the reference files or allow customized treatment of the data. Calstis, its constituent programs, and other STIS-specific tasks are available in the stistools package. The syntax used for running stistools tasks is described in their documentation.

The STIS calibration pipeline was designed to accommodate the need for full or partial recalibration. As mentioned at the beginning of this chapter, calstis is re-entrant, so that certain calibration steps can be performed outside of the pipeline, and others can be executed multiple times, depending upon the science goals.

Generally, the calibration switches in the header control the operations that calstis performs on the data. There are two basic ways to select which operations are performed during calibration:

• Edit the calibration switches and run the calstis task.
• Use one or more of the pipeline subset tasks or other STIS-specific tasks described below, managing the calibration through task parameters.

This section describes the first two methods. In the end, the calibration switches in the headers of the calibrated data files will reflect the operations performed on the calibrated data and the reference files used.

3.5.1 Mechanics of Full Recalibration

If you choose to fully recalibrate STIS data, there is a certain amount of set up required for calstis to run properly. The operations mentioned in the checklist below will be described in detail in the following subsections:

• Set up a directory with the required reference files.
• Determine which reference files are needed and retrieve them from the Archive.
• Set the environment variable oref to point to your reference file directory.
• Set the calibration switches to perform the needed steps.
• Run calstis or a subset of its constituent tasks.

The stistools package requires the use of a Bash or Bash-compatible shell. Users are referred to: https://stenv.readthedocs.io/ for tips on getting started with the stenv environment.

Before running calstis, you will need to define an environment variable to indicate the location of the directory containing the required calibration reference files. The names of the calibration files are preceded with the logical path name “oref$” in the STIS science headers. In a Bash shell, you may set the path with this command:

$ export oref = "/data/calibration/oref/"

Note the inclusion of the trailing slash (/) above. If you plan to run calstis inside a Python, iPython, or Jupyter Notebook session, you may instead set the environment by doing:

>>> import os
>>> os.environ['oref'] = "/data/calibration/oref/"

An alternative to using the oref$ variable is specifying the full pathnames to the reference files in the science headers.

Retrieve Reference Files

To recalibrate STIS data, you will need to retrieve the reference files used by the different calibration steps to be performed. The names of the reference files to be used during calibration must be specified in the primary header of the input files, under the section “CALIBRATION REFERENCE FILES.” Note that the data headers will be populated already with the names of the reference files used during pipeline calibration at STScI.

To retrieve the best reference files for a given exposure via the MAST, check “Best Reference Files” in the “Reference Files” section of the Retrieval Options form. 

To retrieve the best reference files for a given exposure via the MAST HST Search Tool select a dataset and then select "Choose which files to download". In the file menu select the option for all reference files or choose a subset of reference files according to your needs. Alternatively, the reference files may be retrieved with a tool developed by the Calibration Reference Data System (CRDS) team that offers the advantage of retrieving the most recent reference files for observations in a working directory as well as updating the names of the reference files in the primary data headers of the files. In a BASH shell execute the following commands:

$ export CRDS_PATH="$HOME/crds_cache"
$ export CRDS_SERVER_URL=https://hst-crds.stsci.edu
$ export oref="${CRDS_PATH}/references/hst/oref/"

The above syntax define where your personal copies of CRDS reference files will be stored and the CRDS server that is used. Then the following command may be used to assign and obtain the best references files:

$ crds bestrefs --update-bestrefs --sync-references=1 --files *.fits

The STIS reference files are all in FITS format, and can be in either IMAGE or BINTABLE extensions. The names of these files along with their corresponding primary header keywords, extensions, and format (image or table), are listed in Chapter 2. The (somewhat obscure) rootname of a reference file is based on the time that the file was delivered to the CRDS. 

Edit the Calibration Header Keywords

To edit file headers in preparation for recalibration, use the astropy.io.fits package. The following syntax illustrates how to turn on the bias correction switch and update the name of the bias image reference file for a given STIS _raw image in the current directory:

>>> from astropy.io import fits
>>> hdu = fits.open('<name_raw.fits>', mode='update')
>>> header = hdu[0].header 
>>> header['BIASCORR'] = "PERFORM"
>>> header['BIASFILE'] = "oref$new_bias.fits"
>>> hdu.close()

The flags listed in Table 3.4 as controlling the basic calibration will usually be set to PERFORM, whether or not the data are is fully calibrated. For data not fully calibrated by the pipeline, some or all of the flags in the last column of Table 3.4 will be set to OMIT. To fully calibrate such data, the user can reset these flags in the primary extension header of the _raw file to PERFORM, and then run the calstis task with this _raw file as input. It is also possible to reset the necessary flags in the _flt or _crj file and then run calstis with one of these files as input to complete the steps that were omitted by the original calibration.

Table 3.4: STIS calibration switches commonly set to PERFORM

If WAVECORR is set to PERFORM, an appropriate STIS line lamp image should be named in the WAVECAL header keyword. This can be either the _wav file supplied with the dataset, or if a GO specified wavecal is used, the file name of the raw wavelength calibration lamp image should be put into this keyword.

The default calibration assumes that the target was centered in the aperture along the dispersion direction. If this is not correct, the wavelength scale will be incorrect, and this offset will apply the wrong sensitivity calibration at each wavelength. In such a case it may be better to perform only the basic 2-D calibrations using the calstis task, and to then perform the spectral extraction or rectification by running the x1d or x2d tasks independently, and taking the offset in the dispersion direction into account when running those tasks (see Section 5.5.2).

It is also possible to impose a wavelength offset by setting the WAVECORR keyword in the primary extension header to COMPLETE, and then setting the desired values for the SHIFTA1 and SHIFTA2 keywords in all science extension headers of the input image before running calstis. See the description of the WAVECORR task (Section 3.4.23) for an explanation of the SHIFTA1 and SHIFTA2 keywords. However, in the absence of a valid wavecal exposure, there is no straightforward way to determine what these values should be. In general they will be non-zero even in the absence of target offsets.

When running the calstis task itself, the calibration steps are controlled by setting the values of the calibration step keywords discussed in Section 3.4. These are set in the input data file’s primary extension header. However, when running the individual standalone tasks (e.g., basic2d or x1d), the calibration steps need to be turned on or off using the input parameters of each of the standalone tasks. The values of the calibration keywords in the data file’s header are usually ignored, unless they are set to COMPLETE. Most standalone tasks will not apply a correction if the header marks it in this way as already done.

However, regardless of whether calstis or one of the stand alone tasks is being used, all reference file names must be appropriately specified in the corresponding header keywords in the input data file’s primary extension header.

3.5.2 Rerunning Subsets of the Calibration Pipeline

Selected portions of the pipeline can be executed with special tasks in the stistools package. The tasks that can be simply used in this fashion are listed in Table 3.5 below. See also Table 3.1 for the association between basic2d, ocrreject, wavecal, x1d, and x2d and the components of the calstis pipeline. See Chapter 5 for further information about some of these tasks. When you run these tasks individually, the calibration parameters usually read from the reference file must be entered as command line arguments, however most corrections will not be applied if the corresponding header keyword is already set to COMPLETE (see the Appendix of STIS ISR 98-26 and Section 6 of STIS ISR 98-10 for details).

The inttag task for TIME-TAG data will accumulate selected events from the raw event table, writing the results as one or more image sets (imsets) in a single, output FITS file. You can optionally specify an explicit starting time, time interval, and number of intervals over which to integrate, and the collection of imsets will be written to the output file, simulating a REPEATOBS ACCUM observation. Breaking the data into multiple, short exposures can be useful not only for time series analysis but also to improve the flat fielding when the Doppler shift is significant. Once the new raw images have been created, it is straightforward to process them with calstis and to analyze the output image or spectra, as appropriate.

Table 3.5: stistools Calibration Tasks

3.5.3 Improving the Treatment of Hot Pixels

Radiation damage creates hot pixels in the STIS CCD Detector. Some of these hot pixels recover spontaneously. Many more have been repaired monthly by warming the CCD from its normal operating temperature to the ambient instrument temperature for several hours. Despite this annealing procedure, the number of permanently hot pixels has been increasing with time. The development of hot pixels is documented at: https://www.stsci.edu/~STIS/monitors/anneals/pre_post.pdf.

Because the set of hot pixels changes continually, it is important to use the dark reference file prepared for the week of your observation. You should use the corresponding bias reference file, which corrects hot columns. If you requested your data close to the time of the observation, you should check whether new reference files applicable to the data have been made available since. If so, you may want to request data from the archive again for automatic recalibration with the current reference files, since dark and bias correction occur so early in the calibration sequence.

The standard dark file subtracted from STIS data is based on a combination of several long dark files taken over an entire week. They usually don’t do a perfect job of subtracting hot pixels, both because the hot pixels can change on short time scales, and because some of the hotter pixels may have been saturated in the long dark images. It will therefore often be useful to construct a customized dark image for that particular day using short darks taken on the same day as your science image. Procedures for doing this are described at: https://refstis.readthedocs.io/en/latest/api/weekdark.html.

After you create your new daydark file, you should set the DARKFILE keyword in the primary header of the _raw files to reference this new dark file and then reprocess the data through calstis.

Even using a daydark will not give perfect removal of all bad pixels. There is no routine available that will automatically find and fix all the bad pixels. If a more customized removal of bad pixels is required, it will be necessary to identify them by hand and interpolate over them using neighboring good pixels. The NumPy module numpy.interp may be useful for the latter task.

3.5.4 Improving Cosmic Ray Rejection

For CCD datasets, calstis first runs the equivalent of basic2d with DQICORR, BLEVCORR, and BIASCORR set to PERFORM to initialize the ERR and DQ arrays, to trim the overscan regions and subtract the bias level measured there, and to subtract the bias image. It then performs cosmic ray rejection by running the equivalent of ocrreject. Next, the equivalent of basic2d is run with DARKCORR and FLATCORR set to PERFORM to perform dark subtraction and flat fielding. You can instead run basic2d just once on the raw data to perform all of its relevant tasks, then run ocrreject on the resulting _flt file, but this ordering of the tasks does not provide the optimum rejection of cosmic rays. The cosmic ray rejection is based on a "noise model." Part of that model is that the detected number of photons will follow a Poisson distribution. In order to compute the expected variation in detected photons at a pixel, i.e. to evaluate the noise model, you need a reasonable value for the number of electrons at that pixel. That estimate could be based on the median (for example) of the counts at that pixel in all the images that are to be combined. Because that value should reflect the number of detected photons, the bias level and bias image must first be subtracted from the images, because the electronic bias is an arbitrary offset. The data should not be corrected for dark counts or flat field, however. Dark counts do contribute to Poisson noise. The number of electrons to use for computing the Poisson distribution is the number that were actually detected, not the number corrected for variations in sensitivity. See STIS ISR 98-22 for further information on cosmic ray rejection. (Calstis was changed to perform the tasks in the order described here in July 1998, after the ISR was published.)

In cosmic ray rejection, one wants to strike a balance between failing to identify cosmic rays and clipping real flux from the target. The former lets spurious flux into the _crj image; the latter depresses the flux by systematically excluding values that are relatively high for that pixel because of noise fluctuations or small shifts of the target on the detector. Default values of cosmic ray rejection parameters to be used by calstis, or its component task ocrreject, are taken from the CRREJTAB reference file. If your _crj images still have many pixels affected by unrejected cosmic rays, you may want to try running ocrreject on the _flt files with different values for the rejection parameters, which will override the values taken from the CRREJTAB. Use crmask=yes to flag CRs in the input DQ images.

The optimum values of the cosmic ray rejection parameters depend on many factors, including the number of exposures to be combined, exposure time, signal-to-noise level, source structure, and the magnitude of the misalignment of the exposures due to drift of the target on the detector. (See STIS ISR 98-22.) For example, a small spatial misalignment of two exposures made with very high signal-to-noise may require one to increase the value of the parameter crsigmas (for a higher rejection threshold in sigmas) to avoid the systematic rejection of real flux. For a direct image or spectral image with only gradual spatial or spectral changes in flux from pixel to pixel, it may be possible to lower the rejection threshold to catch more cosmic rays without rejecting real flux. One may also need to change the value of crthresh, which sets a more stringent rejection level for the pixels surrounding a pixel that failed the crsigmas test. (e.g., for crthresh=0.5 and crsigmas=7, rejection occurs for surrounding pixels at a level of 0.5 * 7 sigma = 3.5 sigma out to a radius of crradius.)

To see if real flux is being rejected by ocrreject, one can compare the _crj image to a summed image and look for systematically lower fluxes in the _crj image. Problems will tend to occur where the pixel-to-pixel flux changes are the greatest. For a direct image, examine the pixels near the centers of point sources. For a spectral image, examine the peak rows in the spectral image. One can also examine the DQ extensions of the _flt files after they have been flagged by ocrreject (crmask = yes). Systematic problems due to spatial misalignment are especially conspicuous in the DQ extensions of the _flt files of spectral images. Improperly flagged spectral images will have strings of DQ=8192 running along a row near the spatial peak of the spectrum, since that row will be less well centered on the target in one exposure than in another. Remember to examine the DQ extension of the _flt files instead of the _crj file, since pixels in the latter will not be flagged for cosmic rays if their flux was computed from cosmic-ray-free pixels in any of the input exposures.

3.5.5 Removing Fringes from Near-IR Spectral Data

STIS CCD spectra taken at wavelengths >7000 (the G750M and G750L modes) are impacted by “fringing”, a phenomenon caused by interference of multiple reflections between the two surfaces of the CCD in cases where the wavelength of the incident light is a small integer multiple of the distance between the two surfaces of the CCD as presented in STIS ISR 98-29. The stistools package contains four tools used for correcting the fringing effect, normspflat, prepspec, mkfringeflat, and defringe, which are located in the stistools.defringe package. These tools are Python ports of the original IRAF/PyRAF stis defringing tools that are no longer supported. The Python version of the IR-fringe removal package is available and presents several practical examples for common use cases as well as detailed descriptions of each STIS task.

3.5.6 Using GO Wavecals

To use an exposure other than the default wavecal, the user can either (1) put the name of the raw exposure file of the lamp image to be used into the WAVECAL keyword in the primary header of the _raw science image, make sure that the WAVECORR keyword is set to PERFORM, and then run calstis as normal, or (2) use the stistools task wavecal, to measure the offset of the wavelength calibration image and populate the SHIFTA1 and SHIFTA2 keywords in the science file’s extension headers. For example, if o8mi02010 is the science observation and o8mi02ovq a corresponding wavelength lamp calibration image, the command would be

>>> stistools.wavecal.wavecal('o8mi02010_flt.fits','o8mi02ovq_raw.fits')

This will update the SHIFTA1 and SHIFTA2 keywords and set WAVECORR to complete. The modified science image file can then be processed with calstis and/or one or more of the stand alone STIS tasks, e.g.,

>>> stistools.x1d.x1d('o8mi02010_flt.fits')

Note that the wavecal task is meant to update the headers of spectral images, prior to running the x1d or x2d tasks. It should not be used on images or tables (e.g., _sx1, _sx2, _x1d, or _x2d files that have already had wavelengths assigned to individual pixels).

3.5.7 Correcting the Orientation of Spectral Traces

The orientation of spectral traces on the STIS detectors has been found to rotate slowly over time. Additionally, there is a small random offset in the orientation each time the MSM is positioned. The time-dependent component of the orientation has been calibrated for the most commonly used grating/central-wavelength combinations: G140L/1425, G230L/2376, G230LB/2375, G430L/4300, G750L/7751, G750M/6581, G750M/6768, and G750M/8561. Calstis was modified to apply time-dependent rotation to spectral traces, and the evolutionary parameters for these modes are included in SPTRCTAB reference files. For other L and M modes with uncalibrated trace rotation, the mktrace task can be used to generate a SPTRCTAB with accurately rotated traces for a specific science image. The task can also be applied to data taken with the calibrated modes to correct for the random offset in orientation due to the positioning of the MSM.

The mktrace task computes the trace of the science target using the default aperture location or accepting an input location. A linear fit is made to this trace. The user can choose to exclude portions of the trace from the fit because of velocity structure, low signal-to-noise, etc. A calibration trace is interpolated from the traces at the nearest rows in the SPTRCTAB, and a linear fit is made using the same ranges of columns selected for the fitting of the science trace. Image files of the target trace and calibration trace and of the linear fits are generated to assist the user in selecting the portions of the spectrum to include in the fits. The difference in the angle of the two linear fits is the correction that must be applied to the SPTRCTAB traces. A new SPTRCTAB is generated, applying this correction to all of the traces for that grating/cenwave combination, thus covering the entire detector. The keyword SPTRCTAB in the primary header of the science image is changed to the name of the new trace file, so that subsequent processing by calstis, x1d, or x2d will use the new file.