2.1 Overview

Raw COS telescope data are processed through the STScI data processing (DP) pipeline. The DP pipeline first processes the data through Generic Conversion, where the data bits from individual exposures are unpacked and combined into files containing raw, uncalibrated data. Next, the data are processed through the COS calibration pipeline, calcos, which performs image and spectroscopic reduction to produce output files that can be used directly for scientific analysis (see Chapter 3 for a detailed description of calcos). Finally, the data are ingested into the Hubble Data Archive (HDA) through the Data Archive and Distribution System (DADS). This system populates a database containing header keywords which is accessible to users via the Mikulski Archive at STScI (MAST). The data (both calibrated and uncalibrated) are then available for distribution by MAST to the user.

Since COS reference files are frequently updated, the HST data archive may use different versions depending on the date the datasets are retrieved. In the event of an updated reference file or calibration software, users may re-calibrate their data in one of two ways. The preferred method is for the user to re-retrieve the data from the HDA using the MAST interface. This will provide the most recent cached reductions produced by the standard pipeline with the default settings. Alternatively, users can reprocess the data themselves through calcos using the most recent reference files and software (see "Run calcos" in Section 3.6.1). The second option will not include any changes in the data due to Generic Conversion updates, but will allow a customized calibration through the use of modified reference files or keyword switches. Also, the user will need to manually edit the header keywords stating which reference files should be used by calcos (Table 2.18 and Section 3.6.1). The most recent reference files can be obtained either from the CRDS website, or via MAST.

Once you have retrieved your data, you will need to understand:

• The naming conventions and file suffixes of the individual files (Section 2.2).
• The basic format in which the COS data are stored (Section 2.3).
• The structure and content of the individual files (Section 2.4).
• The size of the COS data files (Section 2.5).
• How to use the header keywords to identify the principal parameters of an observation and how to determine the calibration processing steps that were performed on a dataset (Section 2.6).
• The meanings of the error and data quality arrays, which are propagated through the pipeline for each COS science observation (Section 2.7).