2.4 COS Data Products

The following sections discuss the COS raw science data files, intermediate calibration products, final calibration products, and auxiliary data files. Uncalibrated science data include all raw science data generated during Generic Conversion that have not been processed through the calibration pipeline. These raw files are the input files to the calcos pipeline, usually as part of an association file (see "Association Tables (ASN)" in Section 2.4.4). The pipeline produces both individual calibrated exposure files and, when appropriate, a final combined product file.  Note that the final combined product files (x1dsumN, x1dsum) are only produced if calcos is run on the ASN file.

2.4.1 Uncalibrated Science Data Files

Raw ACCUM Images (rawaccum)

For ACCUM data, the raw files contain a set of images, as shown in Figure 2.1, and have filenames with the suffix rawaccum for NUV data, or rawaccum_a and rawaccum_b for the two segments of the FUV detector. The SCI extension contains an image of the total accumulated counts during an exposure. For NUV data the ERR and DQ extensions have only a header with no data. For FUV data the ERR extension has only a header with no data, and the DQ extension is populated with data quality information only for pixels that are outside the subarray boundaries (defined below). The DQ extensions will be populated in the flt files, after calibration pipeline processing. Even though FUV rawaccum_a[b] data are 16384 × 1024 images, only portions of them contain actual data. These portions are called subarrays. Typically, three subarrays are used for each segment of an FUV ACCUM image. Two are centered on the stim pulse positions and the third is a stripe 128 pixels high which is centered on the wavecal spectrum of the object. Figure 2.2 shows these spectral region subarrays superimposed on two FUV rawtag images. As Figure 2.2 shows, the wavecal spectrum falls outside of the subarray. Consequently, wavecals must be taken separately for ACCUM data.

Raw TIME-TAG Events Lists (rawtag)

Raw events tables contain the locations and arrival times of individual photon events collected in TIME-TAG mode. These files have the suffix rawtag for NUV or rawtag_a[b] for the two FUV segments. Figure 2.3 shows the format of a rawtag table. The first extension contains the events list, in which each row of the table corresponds to a single event in the data stream and the columns of the table contain scalar quantities that describe the event. The second extension contains the good time intervals (GTI) table, where an uninterrupted period of time is considered as one good time interval. Interruptions in the data taking due to memory overflow could result in more than one GTI. Table 2.2 shows the columns of a rawtag table.

 ¹ The PHA column is present in the NUV data only for symmetry with the FUV data columns. For NUV data the values in this column are set to 0, since no pulse height amplitudes are available.

For more information on working with TIME-TAG data see Section 5.4.

Pulse Height Amplitude Files (pha)

For FUV ACCUM data only, a 7-bit pulse height amplitude histogram is accumulated in the onboard detector electronics. This information is placed in a file with the suffix pha. The pulse-height histogram files contain a primary header with no data and a single FITS image SCI extension containing a histogram of the pulse-height distribution during the exposure. The pulse height amplitude files do not contain an ERR or DQ extension, as shown in Figure 2.4. The pulse height distribution is an image array of length 128, corresponding to the number of photons with pulse height values from 0 to 127, corresponding to the pulse heights of 0–31 available in TIME-TAG data.

2.4.2 Intermediate Science Data Files

Corrected Events Lists (corrtag)

The COS pipeline produces corrected TIME-TAG events lists and stores them in binary tables with suffix corrtag. These files have a main header and three extensions: a corrected events list extension, a good time interval extension, and a timeline table extension, with a format similar to the one shown in Figure 2.3. The first extension of the corrtag file is the events table (see Table 2.3) which includes X and Y event locations that have been corrected for thermal and geometric distortions and for walk (see Section 3.4), Doppler shift (from HST's orbital motion), and offsets due to OSM motions in both the dispersion and cross-dispersion directions. It also includes wavelengths associated with events that occur within the active area of the detectors and a data quality (DQ) flag for each event (see Table 2.19). The second extension gives the start and stop times of the good time intervals (as in the rawtag file), and the third extension is the timeline table. The timeline table includes second by second values for spacecraft position, solar and target altitude above the horizon, and count rates for the most prominent airglow lines and the background. These observed rates might include counts from other external sources in addition to the ones from the airglow line itself. The data in this extension can be useful for reprocessing TIME-TAG data to exclude, for example, daytime data using the Python tool costools.timefilter, described in Section 5.4.2.

For ACCUM data, the corrtag files are somewhat different. All of the time stamps in the first extension are set to the median value of the observation. Each count in the rawaccum file becomes an event so, for example, a pixel in the rawaccum that had 100 counts would have 100 entries in the corrtag file. The RAWX, XCORR and XDOPP entries are all the same for NUV data, but can be different for FUV. In addition, RAWY and YCORR entries will have the same values. However, XFULL and YFULL can be different. In the timeline extension, the SHIFT1, airglow and DARKRATE entries are fixed.

Table 2.3: Columns of a COS corrtag Table.

¹ The XCORR and YCORR columns are present in the NUV data only for symmetry with FUV data. Currently no distortion correction is applied to NUV data, so for NUV data the XCORR and YCORR columns are identical to the RAWX and RAWY columns.
² For FUV data extracted with the TWOZONE method, YFULL is now also corrected for the spectrum trace and offset from the template profile (see Section 3.4.14).
³ The PHA column is present in the NUV data only for symmetry with the FUV data columns. For NUV data this column is set to a default value of 0, since no pulse height amplitudes are available for NUV.

Lampflash Files (lampflash)

For TAGFLASH data, calcos produces an events list with suffix lampflash, that contains the extracted wavecal lamp flashes. Each row in the events list corresponds to a different segment or stripe and flash number (the first flash is number 1, the second is number 2, etc.). The lampflash files have the format shown in Figure 2.5. The contents of the columns in a lampflash events list are listed in Table 2.4. Columns TIME, LAMP_ON, and LAMP_OFF have the same temporal zero point as the TIME column of the rawtag and corrtag tables and the same unit (seconds). The shifts contained in the SHIFT_DISP and SHIFT_XDISP columns of the lampflash table are applied to the XDOPP and YCORR columns of the corrtag file to produce the X[Y]FULL entries. When multiple TAGFLASHES are present, the shifts are interpolated in time for events occurring between each set of flashes. Events occurring before the first flash are shifted by a value extrapolated using the slope defined by the first two flashes; events beyond the last flash are given the shift determined by the last flash. As a result, the difference between the X[Y]FULL and X[Y]CORR entries in the corrtag file can be a function of time.

As noted in the table below, the time column provides a median value.  Lamps are flashed for a fixed length, let's say 12s. As an example, if you have flash 1 that starts at 0 seconds and lasts 12 seconds, the median time is 5.5. The next flash in the exposure then executes at 600 seconds and lasts 12 seconds, so the median time of the flash is 605.5. So your time array would be (5.5, 605.5, 5.5, 605.5), repeated because one is for FUVA and one is for FUVB.

Counts Files (counts)

The counts images are an intermediate calibrated output product for both imaging and spectroscopic data with suffix counts. These files contain three extensions (SCI, ERR, and DQ) as shown in Figure 2.1. These files are constructed by summing up the events from each pixel using the XFULL and YFULL coordinates. The data are in units of counts per pixel. For FUV data the images are 16384 columns in the x (dispersion) direction by 1024 rows in the y (cross-dispersion) direction. The NUV images are 1274 columns in the x direction by 1024 rows in the cross-dispersion direction for spectroscopic data, and 1024 × 1024 for data obtained in imaging mode. The NUV spectroscopic files have more pixels in the dispersion direction than the actual NUV detector. This is because the counts files (and flt files) have been corrected for Doppler shift and OSM shift (including FP-POS offset), so the width was increased to accommodate those shifts. The FUV images are not extended since the active area is less than the size of the detector, so these effects can be incorporated into the images without the need to extend them. The FUV data are also corrected for walk and geometric distortions.

Flat-Fielded Image Files (flt)

For spectroscopic data a flat-fielded image is an intermediate calibrated data file. These files have a suffix, flt, and contain three extensions (SCI, ERR, and DQ) as shown in Figure 2.1. These files are constructed by summing up the values in the EPSILON column for each pixel using the XFULL and YFULL coordinates. The data are in units of the count rate. For FUV data the images are 16384 × 1024, and, like the counts images, the NUV images are 1274 × 1024 for spectroscopic data and 1024 ×1024 for data obtained in imaging mode. The flt images are corrected for deadtime effects. The NUV images are corrected for all flat-field effects and the FUV data are currently corrected for only the largest fixed-pattern features; the XDL grid-wire shadows, low-order flat-field variations (L-flats), and large geometric distortion artifacts.

2.4.3 Final Science Data Files (and Product Files)

The initial input files to calcos are the association tables with suffix asn. These files provide the calibration pipeline with information about how the data files are associated. In general, only exposures taken in sequence with the same spectral element, central wavelength (if applicable), and aperture at any FP-POS will be associated. For more information on COS association files see the "Association Tables (ASN)" portion of Section 2.4.4.

Processing of each individual exposure in the association produces a final calibrated result named with exposure rootname and suffix x1d (spectroscopy) or flt (imaging).

Next, for each FP-POS position <n> (where <n>=1, 2, 3, or 4), if there are multiple spectroscopic exposures in the association that use the same FP-POS position, calcos will combine their respective x1d into a file named with the association rootname and suffix x1dsum<n>, where <n> is the integer FP-POS value and the DQ_WGT is applied to the final, combined spectrum.  If there is a single exposure with a given FP-POS value in the association, the x1dsum<n> file contains the x1d spectrum to which the DQ_WGT is applied (see Section 3.4.22).

 Lastly, a final association product file is produced with association rootname and suffix x1dsum (spectroscopy) or fltsum (imaging) by combining all science exposures in the association.

One-Dimensional Extracted Spectra (x1d, x1dsum)

The COS pipeline produces extracted one-dimensional spectra and stores them in binary tables with suffix x1d, x1dsum<n>, or x1dsum. Figure 2.6 shows the format of the 1-D extracted spectra table.

Table 2.5: Columns of a COS Extracted Spectrum Table.

¹ Note that in the x1dsum & x1dsumN files, the DQ_WGT column is applied, while in the x1d files it is not.

Flat-Fielded Image Files (flt, fltsum)

For NUV imaging observations, the flt and fltsum images are the final data products, with the latter being a simple sum of the individuals when several exposures are processed together. They are fully linearized and flat-field corrected images. Unlike the flt files produced for the spectroscopic data (which are intermediate data products with a format of 1274 × 1024, see Section 2.4.2), the formats of the flt and fltsum files for imaging data are 1024 × 1024, since Doppler and OSM motions are not applied.

2.4.4 Auxiliary Data Files

Association Tables (ASN)

An association file is created for all COS observation sets, and has the suffix asn (e.g., lcwj01010_asn.fits). This file holds a single binary table extension, which can be displayed with the astropy.table.Table module.

Calcos calibrates raw data from multiple science exposures and any contemporaneously obtained line lamp calibration exposures through the pipeline as an associated unit. Each individual science exposure in an association is fully calibrated in the process. The information within an association table shows how a set of exposures are related, and informs the COS calibration pipeline how to process the data.

An example association table is shown below. Note that all related COS exposures will be listed in an association table, with the exception of acquisitions, darks, and flats. It is possible to have an association which contains only one exposure. The association file lists the rootnames of the associated exposures as well as their membership role in the association. The exposures listed in an association table directly correspond to individual raw FITS files. For example, the association table can describe how wavecal exposures are linked to science exposures. Table 2.6 summarizes the different exposure membership types (MEMTYPES) used for COS association tables.

Table 2.6: Member Types in COS Associations.

The table below illustrates the contents of the association table for a sequence of spectroscopic exposures for four FP-POS positions.

Sample Association Table l9v221010_asn. To display the association table for ldel05050_asn.fits:

> from astropy.io import fits
> fits.info('ldel05050_asn.fits')
Filename: ldel05050_asn.fits
No. Name  Ver Type     Cards Dimensions Format
  0 PRIMARY 1 PrimaryHDU  43 ()
  1 ASN     1 BinTableHDU 25 5R × 3C    [14A, 14A, L]
> from astropy.table import Table
> t = Table.read('ldel05050_asn.fits', hdu=1)
> print(t)

MEMNAME   MEMTYPE MEMPRSNT
--------- ------- --------
LDEL05JYQ EXP-FP  True
LDEL05K0Q EXP-FP  True
LDEL05K2Q EXP-FP  True
LDEL05K4Q EXP-FP  True
LDEL05050 PROD-FP True

In the above example, MEMTYPE describes the exposure membership type or role in the association. The column MEMPRSNT lists whether the member is present or not. The association file can be modified to not include a member during processing by changing the MEMPRSNT to ‘false.’

The association table above lists the names of the four associated exposures that are calibrated and combined to create the various association products which will have a rootname of ldel05050. This particular association is created from a single TIME-TAG spectroscopic APT specification with FP-POS=ALL specified in the Phase II file, which leads to a science exposure taken at each FP-POS location. For example, the first entry in the table, ldel05jyq, is the rootname of a single external science exposure taken with FP-POS=1. This exposure corresponds to the following rawtag files: ldel05jyq_rawtag_a.fits, ldel05jyq_rawtag_b.fits. The memtype of this exposure is EXP-FP which shows that it is an external exposure. Similar files correspond to the remaining three entries in the association file for data taken with the remaining three FP-POS positions. The pipeline will calibrate the members of an association as a unit, producing the calibrated data products for each individual exposure as well as the final combined association data product. For this particular association, the pipeline will produce a final combined association product, ldel05050_x1dsum.fits, which contains the final FP-POS combined, calibrated spectrum.

Trailer Files (TRL)

When COS data are processed in the HDA, the output messages from generic conversion and the different calibration steps are stored in a FITS ASCII table known as the trailer file, with suffix trl. Files ending with _log.txt are identical to files ending in trl, but are instead in TXT format.Each time the archive processes data, the old trailer file is erased and a new one created using the results of the most recent processing performed. The archive will produce a trailer file for each individual exposure and association product. Association product trailer files contain the appended information from all the exposures in the association, in order of processing. The order of processing is the same as the order of exposures in the association table, with the exception of AUTO or GO wavecals which are always processed first.

In the trailer files from the HDA, the output messages from generic conversion appear first in the file. This section contains information relevant to the selection of the best reference files and the population of some of the header keywords. The second part of this file contains information from calcos processing. Each task in the calcos pipeline creates messages during processing which describe the progress of the calibration, and appear in the order in which each step was performed. These messages are relevant to understand how the data were calibrated, and in some cases, to determine the accuracy of the products.

In the last section of the _trl file, the calcos steps are indicated by their module name. The calcos messages provide information on the input and output files for each step, the corrections performed, information regarding the reference files used, and in the case of FUV data, messages about the location of the stim pulses, or shift correction applied to the data. Calcos also gives warnings when the appropriate correction to the data could not be applied. For more detailed information on the calibration steps and structure of calcos, please refer to Chapter 3.

Calcos Trailer Files (TRA)

When calcos is run on a personal machine, calcos redirects the output of its steps to the STDOUT and an ASCII file with name rootname.tra. Note, the level of detail included in the output messages can be modified when running calcos (see about "verbosity" in "Run calcos"). So, when run on a personal machine, calcos will not overwrite the trl file but rather will direct the output to STDOUT and an ASCII tra file. The tra file is formatted like the trl file but with two exceptions: the tra file will not contain the output messages from generic conversion, and the tra file is not converted to FITS format. Each time calcos is run on a file, the STDOUT messages will be appended to the tra file if it already exists. Also, when running calcos on a personal machine there will be no tra created for the association products (ASN files). Instead, the calcos messages for association products will be sent only to STDOUT.

Support Files (SPT)

The support files contain information about the observation and engineering data from the instrument and spacecraft that were recorded at the time of the observation. A COS support file contains a primary header and a variable number of image extensions. Depending on the length of the exposure, the support file will contain one or more "imsets," each of which includes a support extension (EXTNAME = 'SUPPORT") and two snap extensions (EXTNAME = 'SNAP1' and 'SNAP2'):

The SUPPORT extension contains a header with the proposal information and an (16-bit) image array containing the data which populate the SPT header keyword values. The image array element values are used to populate the header keywords.

Following the support extension in each imset, there are two engineering snapshot extensions. These extensions contain samples of many instrument and telescope parameters from telemetry data during the course of the exposure. The SNAP1 extension in the first imset will always contain telemetry information collected immediately before the exposure begins. Subsequent SNAP1 extensions in all imsets will repeat the information contained in the first one. The SNAP2 extension in each imset will be populated with telemetry data taken during the course of the exposure. The final SNAP2 extension will be populated with data taken immediately after the completion of the science exposure.

Figure 2.7 depicts the structure of an N extension COS support file. With several snapshots of COS telemetry values, one may track the instrument status periodically throughout an exposure. For a schematic listing of the spt headers with detailed information about the spt header keywords, see:
http://stdatu.stsci.edu/keyword/cgi-bin/kdct-header.cgi?i=COS&s=20.1&db=Operational.

Acquisition Files (RAWACQ)

All COS acquisition exposures will produce a single raw data file with suffix rawacq. Almost all COS spectroscopic science exposures are preceded by an acquisition sequence or exposure to center the target in the aperture. Keywords in the header of COS science data identify the exposure names of relevant acquisition exposures in each visit. In addition, there are several other useful keywords in the COS acquisition exposures that describe the acquisition parameters used, as well as the calculated centroid positions and slew offsets. Table 2.7 lists all the relevant acquisition keywords.

Table 2.7: ACQ/IMAGE Header Keywords.

¹ These keywords are also found in the COS science headers in addition to being in the acquisition headers.

Table 2.8: ACQ/SEARCH Header Keywords.

¹ These keywords are also found in the COS science headers in addition to being in the acquisition headers.
² FUV only.
³ NUV only.

Table 2.9: ACQ/PEAKXD Header Keywords.

¹ These keywords are also found in the COS science headers in addition to being in the acquisition headers.
² NUV only.
³ FUV only.

 Table 2.10: ACQ/PEAKD Header Keywords.

¹ These keywords are also found in the COS science headers in addition to being in the acquisition headers.
² FUV only.
³ NUV only.

PEAKD and SEARCH Acquisitions

Acquisition peakups in the dispersion direction (ACQ/PEAKD) and acquisition spiral searches (ACQ/SEARCH) both use the flux from exposures taken at different dwell points to center the target. For more information on these types of COS acquisitions see Sections 8.6 and 8.3 respectively of the COS Instrument Handbook. Data for these acquisitions contain one binary table extension which describes the acquisition search pattern dwell point locations and counts as shown in Table 2.11 and Figure 2.8.

Table 2.11: Columns of an ACQ/SEARCH or ACQ/PEAKD Table.

¹ DISP_OFFSET is only present in tables for ACQ/SEARCH and PEAKD.

² DISP_OFFSET is not present in tables for ACQ/SEARCH and PEAKD.

PEAKXD Acquisition

Acquisition peakups in the cross-dispersion direction (ACQ/PEAKXD) use a TIME-TAG spectrum to center the target in the cross-dispersion direction. For more information on the ACQ/PEAKXD algorithm see Section 8.5 of the COS Instrument Handbook. An ACQ/PEAKXD exposure includes only a primary header and extension header.  Note that in almost all cases a PEAKXD is done first, then a PEAKD.

 IMAGE Acquisition

Acquisition images (ACQ/IMAGE) use an NUV image to center the target in the aperture. For more information on the ACQ/IMAGE algorithm see Section 8.4 of the COS Instrument Handbook. An ACQ/IMAGE exposure produces a raw data file containing two science image extensions corresponding to the initial and final pointing:

• [SCI,1] is an image of the initial target pointing.
• [SCI,2] is a confirmation image after the acquisition procedure has been performed.

See Figure 2.9 for the FITS file format for ACQ/IMAGE data.

Jitter Files (jit)

 The COS jitter files include engineering data that describe the performance of the Pointing Control System (PCS) including the Fine Guidance Sensors that are used to control the vehicle pointing. The jitter files report on PCS engineering data during the duration of the observation. The support files contain information about the observation and engineering data from the instrument and spacecraft that were recorded at the time of the observation. COS jitter files utilize the file format shown in Figure 2.10 for all science observations, excluding acquisitions.

Table 2.12: Columns of a jitter Table.

2-D Spacecraft Pointing Histogram (jif)

The COS jif files are a 2-D histogram of the corresponding jit file (see Jitter Files) and have the file format shown in Figure 2.11 for all science observations excluding acquisitions.