C.1 Hubble Advanced Spectral Products (HASP)

The Hubble Advanced Spectral Products (HASP) program is designed to robustly coadd Cosmic Origins Spectrograph (COS) and Space Telescope Imaging Spectrograph (STIS) spectra within the Mikulski Archive for Space Telescopes (MAST) in an automated fashion such that coadds are available for new data or archival data with updated calibrations. For each target within a visit or program, HASP employs a meticulous multi-stage filtering process to ensure data quality and creates coadded products for all central wavelengths (CENWAVEs) within specific gratings, as well as combined products using different gratings and instruments. As calibrations improve and new data are added to the archive, HASP products are re-created automatically so that they represent the best reduction of a given visit or program. Automated coadditions like those achieved by HASP can significantly enhance the combination of different CENWAVES, increase signal-to-noise ratios, and increase wavelength coverage. More details are available at the HASP website and in COS ISR 2024-01.

The default pipeline products of COS and STIS typically do not include combinations of different gratings, central wavelengths (CENWAVEs), or apertures for individual programs. This has historically meant that users were responsible for coadding different spectra for final scientific analyses. The HASP products are designed to work best for astrophysical sources that have compact angular sizes and are not variable, which ensures flux continuity across different gratings and apertures. In those cases, the coadditions should match each instrument's requirements for flux and wavelength repeatability and accuracy. However, HASP does not fit all science cases or requirements, so an integral part of the project is to provide user-friendly access to custom coaddition for situations when more individualized approaches might need to be taken.

C.1.1 Filtering Input Spectra

Default HASP coadded spectra are created through a three-step process in the MAST archive: automated collection of datasets, dataset filtering and validation, and coaddition via a publicly available coadd code. The HASP coadd code procedure employs a multi-stage filtering process. First, it compiles a list of all exposures associated with AlertObs reports and excludes those exposures. COS and STIS special calibration programs containing engineering commands, for example focus changes, or non-standard detector positions or modes are removed, as are datasets which are statically archived and do not undergo routine re-processing. It then removes any input spectra with header keywords that indicate data in a given program may have observing quality issues (often as a result of a failed or delayed guide star acquisition). See COS ISR 2024-01 for a detailed list of those conditions.

Despite several different checks for input spectra that may have been part of a failed observation, more subtle issues with target acquisition, spectroscopic extraction, or jitter may produce a default archival product that passes the HASP pre-filtering steps, but may still be inappropriate for coaddition. To handle such cases, the HASP wrapper code implements a flux checking step that compares input spectra from a single mode against that mode's final coadded spectrum. If an input spectrum is significantly fainter, the coadd wrapper code will reject the spectrum and recalculate the coadded spectrum until no more input spectra are rejected.

Only visit-level coadded spectra are produced for moving targets (i.e., Solar system objects). Targets with the MTFLAG set to true do not have coadded products generated at the program level due to their expected variability from visit to visit. If users wish to include any of the above types of data they may use the publicly available HASP coadd code with keyword-based filtering disabled, except for STIS PRISM data, which is always filtered.

C.1.2 Coadding Spectra

After filtering, the HASP coadd wrapper code organizes COS and STIS spectroscopic data in order to be processed. It works on all unfiltered x1d and sx1 files in the selected directory within the HASP workflow. Both COS and STIS have x1d files for UV observations, while STIS has sx1 files, derived from CCD observations subjected to CR-SPLITs. The coadd wrapper code works on all spectroscopic files for a single program.

The basic coaddition procedure for a single grating descends from the Ultraviolet Legacy Library of Young Stars as Essential Standards (ULLYSES) program's coadd code and the coaddition methodology detailed in COS ISR 2017-04. The HASP coadd wrapper code implements some differences in methodology with ULLYSES while maintaining backwards capability 

First, maximum and minimum values from the wavelength arrays of each input exposure are calculated, and these are rounded down (minimum wavelength) and up (maximum wavelength) to the nearest integer Angstrom. The largest wavelength bin-size between neighboring input wavelength arrays is used as the wavelength spacing between samples of the product spectrum.

In the coadd procedure, each input pixel's measurement is treated as an estimate of the monochromatic flux at its assigned wavelength. The output flux is obtained by calculating a weighted average of all the flux measurements falling within the output wavelength bin's bounds.  Throughput, which is the net count rate divided by flux and multiplied by the exposure duration, serves as the weighting factor for each input pixel.  This ensures that measurements with more counts have higher weights. Only input pixels with Data Quality (DQ) flags for COS (COS DHB Section 2.7 Error and Data Quality Arrays) and STIS (STIS DHB Section 2.5 Error and Data Quality Array) considered non-serious contribute to the output flux.  One exception is the DQ flag of 16 (which relates to flagging of pixels with dark rates >5σ times the median dark level) in STIS data products, which was found to be significantly over-flagged in current STIS data products. Errors are calculated from the square root of the total detected counts in each bin for data from all input photon counting modes (COS, STIS MAMA), and from the weighted sum of the squared errors for STIS CCD data.  The conversion from error counts to error flux in the former case uses the conversion calculated for the flux measurement at each wavelength. This procedure ignores any systematic uncertainties (such as those due to CCD fringing beyond 7000 Å and pixel-to-pixel variations in COS FUV spectra).

C.1.3 HASP Products

The coadd code implemented in the HASP project generates two distinct data products: coadded spectra and abutments, created at both the visit and program levels. Coadded products are the result of combining spectra from a common grating, while abutments involve the combination of spectra from different gratings and/or instruments. The process of abutment is an ”all-or-nothing” approach whereby all the data from one grating are used until a certain transition wavelength, and thereafter data from a different grating are used until the next grating transition wavelength.

The naming convention for data products is similar at both the visit-level and program-level, roughly following the naming convention of the ULLYSES high level science products (HLSPs).

Visit-level:

hst_<PID>_<instrument>_<target>_<opt_elem>_<ipppss>_cspec.fits

Program-level:

hst_<PID>_<instrument>_<target>_<opt_elem>_<ippp>_cspec.fits

In the above naming scheme, <PID> is the program ID, <instrument> is either COS or STIS or both, <target> refers to the Phase II target name, and <opt_elem> is the grating name for the relevant mode(s). If a grating name is shared by COS and STIS a "c" or "s" will denote its association with the corresponding instrument. Finally, <ipppss> refers to the instrument, program, and dataset identifier for visit-level coadds, while <ippp> refers to the instrument and program identifier for program-level coadds. These identifiers correspond to the MAST archive dataset naming conventions. As an example, the HASP coadd shown in Figure C.1 is hst_14910_cos_wd0308-565_g160m_ldel_cspec.fits, representing that this coadd is a visit-level coadd of just the G160M grating from Program 14910.

The HASP FITS structure for these coadded products consists of two BINTABLE extensions: A science extension containing the coadded spectra and accompanying noise and SNR estimates and a metadata extension recording attributes of each spectrum contributing to the combination, or so-called provenance table. The provenance table includes information such as the filename, telescope, instrument, detector, observation start and stop time in modified Julian dates, exposure time, aperture used, and the wavelength coverage of the observation.

The coadd code and the HASP wrapper are available in the HASP repository for public use. Both codes are scripted in Python (compatible with versions 3.9-3.11) and are installable using pip. More detailed installation and use instructions are given in our tutorial Jupyter notebooks, located at the HASP website.

Users can access HASP data products in a variety of ways. First, the MAST Portal contains all HASP products under the “HASP” Provenance, which can be accessed in an advanced search. The HST Mission Search form, under its default search results, will show whether a dataset has an associated HASP data product that can either be downloaded as part of the default download, or can be specifically selected by clicking on the “File Category” dropdown menu and selecting HASP products. Finally, since HASP data products are in MAST, users can download data directly through the astroquery API, which is detailed in the HASP Tutorial.

C.1.4 Custom Coadditions

In recognition that users may wish to customize how they coadd spectra, the HASP team has created several user-friendly Jupyter notebooks that demonstrate how to use the HASP scripts beyond the default process. These notebooks guide users through several custom processing examples, such as how to correct for slight flux variations between datasets by flux scaling, how to change the flux filtering threshold or turn off the filtering completely, and how to coadd datasets that were rejected by the pre-filter.