3.1 Pipeline Processing Overview

During operation, telemetry containing STIS science data is downlinked through a TDRSS satellite to a ground station in White Sands, NM. From there it is sent to Goddard Space Flight Center where the PACOR data capture facility collects the downlinked science data into telemetry “pod files”. These pod files are then transmitted to STScI where they are saved to a permanent storage medium. The STScI ingest pipeline then unpacks the data, extracts keywords from the telemetry stream, reformats the data, and repackages them into raw, uncalibrated, but scientifically interpretable data files. These raw files are then processed by the calstis software to produce a variety of calibrated data files. The results of these procedures are then used to populate the databases that form the searchable archive catalog describing the individual instrument exposures.

The STIS calibration pipeline, calstis, performs the calibration of STIS science data and is available to the community in the stenv package stistools. Most calstis modules were written in the C programming language, and may be called with Python wrapper interfaces, which are documented at https://stistools.readthedocs.io/, and are explained further in Chapter 5. Previously, STIS data taken prior to the 2004 Side-2 electronics failure were statically archived, and calibrated data products did not benefit from updates to calstis routines. However, those pre-SM4 data have now been recalibrated with the most recent version of calstis and will now be automatically reprocessed whenever relevant updates to the pipeline are implemented.

Conceptually, calstis is several pipelines in one, reflecting the complexity and diversity of STIS observing modes.  STIS data are calibrated to different levels, depending on their nature:

• ACQs and ACQ/PEAKs are not calibrated by calstis; only the raw data are available from observations taken in these modes.
• All other science data are processed through basic two-dimensional image reduction (available as basic2d in stistools), which includes such functions as bias subtraction, dark subtraction, flat fielding, and linearity correction. In the case of CCD CR-SPLIT or REPEATOBS data, the data are also passed through a cosmic ray rejection procedure (available as ocrreject in stistools).
• Data taken in TIME-TAG mode are available from the archive both as event streams (rootname_tag.fits binary tables) and as raw images equivalent to those produced for ACCUM mode observations. For TIME-TAG data, the accumulation into an image is done by the ground system, rather than onboard the spacecraft (as is the case for ACCUM mode data). The calstis software as run in the pipeline uses those raw image files as input in either case, and does not distinguish between TIME-TAG and ACCUM mode data. The calstis pipeline software does not operate on the _tag event files; see Section 5.6 for a discussion of how to analyze these files.
• For MAMA data, the input raw data format is 2048 × 2048 (so called high-res pixels), while the calibrated data are binned by the pipeline to 1024 × 1024 native format pixels (see Section 3.4.17).
• Spectral data that were taken using a sufficiently small aperture, and which were also taken together with a wavelength calibration spectrum, are then passed through spectroscopic reduction to produce flux and wavelength calibrated science data. For first order spectral modes, a two-dimensional rectified spectral image is produced, and for both echelle and first order modes, a one-dimensional, background subtracted spectrum is also produced. For first order spectral observations where the target was located behind one of the fiducial bars of one of the long slits, only two-dimensional rectified spectral images are produced.
• Spectral data taken with very large apertures are treated as slitless observations. For such data, as well as for data taken without contemporaneous wavecal observations, the presumption is that the target location along the dispersion direction is too uncertain to assign a reliable wavelength scale. No flux or wavelength calibrated spectra are produced by the pipeline for such observations. The definition of which apertures are treated as slitless varies depending on the grating in use.

See Chapter 2 for the naming conventions of the various input, intermediate, and output calibrated files.

As with the calibration pipelines for the other HST instruments, the specific operations that are performed during calibrations are controlled by calibration switches, which are stored in the image headers as KEYWORD=VALUE pairs. Any given step in the calibration process may require the application of zero, one, or more calibration reference files, the names of which are also found in the image header. The names of the keywords containing the switches and reference file names were introduced in the previous chapter.  Section 3.3 outlines the role those keywords play in the data reduction, and Section 3.4 describes the various steps in the calibration. The path taken by a specific STIS data file through the pipeline is determined by the calibration switches set in the primary header of the _raw data, which in turn depend directly on the characteristics of the data.

A few other general comments are in order. It is important to note that some of the STIS calibration reference data are obtained contemporaneously with the science observations. Those data may be used to refine the calibration process (as with the automatic wavecals), or may require replacement of a default calibration reference file with the contemporaneously obtained one, as in the case of a CCD near infrared (NIR) fringe flat. The details of how the contemporaneous calibration files are used in calstis can be found in Section 3.4. The STIS pipeline is also unusual in that the various modules are re-entrant. That is, a user running calstis off-line may choose to reprocess STIS data partially, performing one or more of the intermediate steps without re-running the complete calstis pipeline, for instance to perform cosmic ray rejection or one dimensional spectral extraction.  Section 3.5 explains the mechanics (and restrictions) of this kind of processing. Finally, as with other HST pipelines, calstis propagates statistical errors and tracks data quality flags throughout the calibration process.