5.4 Working with TIME-TAG Data
COS detectors can be used in ACCUM or TIME-TAG modes, as described in Chapter 5 of the COS Instrument Handbook. In TIME-TAG mode, the position and detection time of every photon is recorded in an events list. Detection times are recorded with 32 millisecond precision, although events may be buffered for as long as 32 milliseconds prior to assignment of a detection time. This 32 millisecond rate may lead to a spurious four-second periodicity in plots of count rate vs. time.
For TIME-TAG datasets, the HST archive returns a raw events list in a file with a rawtag suffix. The rawtag file is a FITS file with two binary table extensions. The first extension contains the events list and the last extension a list of good time intervals, indicating time intervals when events are valid. More details are given in Section 3.2.
An events list in a rawtag file is a FITS binary table extension named EVENTS, containing four columns named TIME, RAWX, RAWY, and PHA. Note only FUV data will include the PHA columns in the _rawtag files.
The TIME in the events extension contains the time when each event was recorded, relative to the start time (MJD) of the exposure given in the EXPSTART keyword of the primary FITS header.
In TIME-TAG the RAWX column contains the pixel coordinate along the spectral axis where each event was recorded. Corrections to remove Doppler shifts introduced by the orbital motion of HST are applied by calcos and placed in the corrtag file. The correction depends on optical element and the projected orbital velocity of HST, which varies over the course of an observation. In ACCUM mode, this Doppler compensation is applied on orbit during an observation and is included in the RAWX column, but in TIME-TAG mode the uncorrected positions are downlinked and Doppler compensation is applied during ground processing. The RAWY column contains the pixel coordinate along the spatial, or cross-dispersion, axis. No Doppler compensation is applied. The PHA column (for FUV data only) contains the pulse height amplitude for each event as an integer on a 5-bit scale.
After all EVENTS extensions in a rawtag file, there will be one final binary table extension named GTI, containing columns named START and STOP. There will be associated start and stop times for every uninterrupted observing interval during a planned exposure. For most datasets, there will be only one START and one STOP time encompassing all buffer dumps in an exposure. Multiple good time intervals are possible, however—or example, if guide star lock is lost. Times in START and STOP are expressed in seconds since the start time (MJD) of the exposure given in the EXPSTART keyword of the primary FITS header is also expressed in seconds. The exposure start time (JD) is also provided in the EXPSTARTJ keyword of the primary FITS header. In Python, good time intervals can be examined using the fits module in the astropy.io package:
> from astropy.io import fits
> gti = fits.getdata('rootname_rawtag_a.fits', ext=2)
> print(gti)
[( 0., 289.024)]
where rootname must be replaced by the rootname of the rawtag file being examined. Note, the 2nd extension of the rawtag files includes all the GTI. In this example, there is just a single GTI.
5.4.1 Displaying TIME-TAG Data in DS9
To view a TIME-TAG file (rawtag_(a,b) in the FUV, rawtag in the NUV), open ds9, then choose ‘open’ from the menu bar at the top. The image will load but, save for a few pixels registering a value of 1, the remaining pixels will be zero.
Once the image is loaded, go to the menu item ‘bin’ and open the pull-down menu from that. From that pulldown menu you can choose the size of the image to view—generally you should make it as big as possible: 8192 × 8192 pixels for FUV data, 1024 × 1024 pixels for NUV.
NOTE: for the instructions below, the changes will not take effect until you click on the 'Apply' button.
Now choose 'Binning Parameters' from the 'bin' pulldown menu. This will open a new window with the binning parameters listed. You will notice right away that the bin columns are listed as TIME and RAWX. These are what is currently being displayed by ds9 (which is why the image looks so strange when initially loaded). However, what you really want is RAWX vs. RAWY, so change that in the pulldown menu under bin columns.
You can also set the blocking size of the image in the 'Binning Parameters’ window—just type in '2' in the Block field next to RAWX. By blocking this way along the dispersion direction, you can now see virtually all of the 16,384 pixels along the dispersion direction. If you are looking at NUV data, then no additional blocking is needed—just leave the blocksize as 1, but choose the image size as 1024 pixels from the 'bin' pulldown menu.
Spectroscopic Data
Next, from the 'Binning Parameters' window choose the part of the spectrum to be centered on the middle of the dispersion direction by clicking on the button marked 'or center of data'. Now press 'Apply' on the binning parameters window to update the ds9 display.
The spectrum should now be displayed, with the dispersion direction running from left to right. To better see the data, choose 'scale' under the main ds9 menu bar, and from that pulldown menu choose a square root stretch and min/max range. You can now pan your cursor over the image, while holding the right button down on your cursor, until the contrast looks just right. If you would like to smooth the data a bit (this can be useful for bringing out fainter features and increasing signal to noise along the display), choose the 'Analysis' menu item under the main ds9 menu bar and select 'smooth parameters.' A dialogue box will open, and from there you can set the number of pixels to smooth. Finally, you can also click on the 'Color' item on the ds9 menu bar and choose 'invert color map' to get an inverted color map.
You can also load a corrtag_(a,b) table in ds9, but in this case the appropriate columns to display are XFULL and YFULL. Otherwise, the same ds9 commands apply as for rawtag files. For both TIME-TAG and ACCUM spectroscopic data the flt and counts spectral images will load as simple 2-D images in ds9.
Imaging Data
For both TIME-TAG and ACCUM imaging data the flt and counts images will load as simple 2-D images in ds9.
TIME-TAG Animation
You can assign events registered during each time interval to a separate image in ds9, thereby creating a sequence of images which can be played as an animation. This can be useful in verifying the occurrence of lamp flashes in TAGFLASH data, in searching for the appearance of bursts in raw data (although bursts have not yet been seen with COS), and so on. To bin the images in time, set up the image as described above—with RAWX and RAWY chosen in the 'Binning Parameters' dialogue box. At the bottom of the 'Binning Parameters' box is a parameter called 'Bin 3rd Column,' Set the value of this parameter to TIME. Next, choose the number of bins you would like to divide the event file into under the 'Depth' parameter. Setting this value to 10, for example, will create 10 separate images, with the first one showing all events registered during the first (EXPTIME/10) seconds, the next one showing all events registered between (EXPTIME/10) and (2*EXPTIME/10) seconds, the next showing all events registered between (2*EXPTIME/10) and (3*EXPTIME/10), and so on up to EXPTIME. The 'Min' and 'Max' parameters let you choose the range of values in time to display—usually this is pre-set to 0 and EXPTIME, and can be left unchanged to bin the entire image as above. Select 'Apply' to do the binning.
Note that some time will be required to create the sequence of images, and that binning the events in time in ds9 is very memory intensive, and that it is easy to make ds9 crash if EXPTIME is large (for example >1000 seconds) and the number of bins in 'Depth' is set to a large value (for example 30). It is best to start with a small value for 'Depth' that works, then increase the value if needed.
After the binning is done, a new dialogue box will appear called 'Data Cube.' Numbered from left to right will be the enumeration of the bins (in the example above from 1 to 10), along with a slider underneath. Click on 'Play' in that window to start the animation—it will play each of the binned images sequentially in the ds9 window. Again, the spacing between each of the bins will be (EXPTIME/10) in seconds, or (EXPTIME/Nbin), where Nbin is the number of bins.
In the animation, it should be possible to see the TAGFLASH spectrum appear and disappear as the sequence progresses. Obviously the sequence will show the flashes only if the keyword TAGFLASH=AUTO or TAGFLASH="UNIFORMLY SPACED" is in the header of the event file.
To exit from the animation, close the 'Data Cube' window, and then set the 'Depth' parameter in the 'Binning Parameters' dialogue box to zero, and click 'Apply.' That will reset the image in ds9 to show all of the data again.
It is possible to bin in other parameters as well, such as PHA. The logic is the same as above.
5.4.2 Filtering Time-Tag Data
Filtering Events in the Timeline Extension
All corrtag files processed with calcos version 2.14 or later contain a timeline extension. The timeline extension can be operated on by the timefilter module to exclude photon events that match user-specified patterns. The timefilter module is available as part of costools, and requires calcos version 2.14 or later to work. One common use of timefilter is to exclude daytime events in order to minimize the contribution of geocoronal Lyman alpha or O I emission lines to your data. Timefilter will filter events according to a filter string passed to it.
The filter string consists of one or more filter conditions, separated by "and," "or," or "xor" (parentheses are currently unsupported). Each filter condition consists of a column name, a relation, and a cutoff value. Valid column names are "time," "longitude," "latitude," "sun_alt," "target_alt," "radial_vel," "shift1," "ly_alpha," "OI_1304," "OI_1356," and "darkrate" (see Table 2.3 for a description of the columns). Valid relations are '>', '>=', '<', '<=', '==', and '!='. Cutoff values are numerical values. In addition, it is possible to flag events based on one of the 32 SAA model contours with the filter condition "SAA #" where # is a number from 1 to 32. Events which match the filter string will be marked with the DQ flag 2048 (bad time interval), and will be excluded in the creation of flt and x1d files.
Timefilter can either modify an existing corrtag file in place, or create a new one, and it can be run in conjunction with splittag (although in that case, it is possible that some output files will contain no valid events at all). The produced corrtag file may be extracted with the calcos package as usual. It is possible to remove any events filtered with timefilter by setting the filter to "reset" followed by setting the filter to ''clear."
The following examples show common uses of Timefilter:
Take "test_corrtag_a.fits," flag all data taken during orbital day (sun_alt > 0), and save in the file "output_corrtag_a.fits"
> from costools import timefilter > timefilter.TimelineFilter('test_corrtag_a.fits', \ 'output_corrtag_a.fits', 'sun_alt > 0.')Filter "xyz_corrtag_b.fits" in place to remove data with (sun_alt > –10 AND ly_alpha > 2.5) OR taken in the SAA 31 profile
>from costools import timefilter >timefilter.TimelineFilter('xyz_corrtag_b.fits', '', \ 'sun_alt > -10 and ly_alpha > 2.5 or saa 31.')Remove filters from "xyz_corrtag_b.fits"
>from costools import timefilter >timefilter.TimelineFilter('xyz_corrtag_b.fits', '', 'reset') >timefilter.TimelineFilter('xyz_corrtag_b.fits', '', 'clear')
Manipulating TIME-TAG Data for Variability
Users may wish to process only sub-intervals of TIME-TAG events, to look for variability in the data. One way to do this would be to divide an exposure up into several sub-exposures before re-processing by using the splittag module. The splittag module is available as part the costools package within stenv.
Splittag is a useful tool for dividing a COS time-tag exposure (FUV or NUV) into a series of sub-exposures with time intervals specified by the user. The task operates on the calcos corrtag files, copying rows from a corrtag file into one or more output files. The number of files depends on the number of time intervals specified by the user. The resulting corrtag sub-exposures can then be run separately through calcos to extract one-dimensional, flux-calibrated spectra (*_x1d.fits files) for each file.
The following keywords are modified when splittag copies the time columns to the new corrtag files: EXPTIME, EXPEND and EXPENDJ. The keywords EXPSTART and EXPSTRTJ, on the other hand, are not changed. The EXPTIME keyword in each of the new corrtag files will be set to the duration of the time interval being extracted, while the modified Julian date and Julian date in EXPEND and EXPENDJ will be set to the following:
EXPEND = EXPSTART + t_end(i)
EXPENDJ = EXPSTARTJ + t_end(i) ,
where t_end(i) is the ending time of the i-th desired sub-exposure. In addition to the updated keywords, splittag also produces updated GTI (good time interval) tables for each of the output corrtag files. The GTI intervals are specified relative to the times of the original corrtag file, such that the split corrtag files will not include events outside the GTI values. The EXPTIME keyword written to the corrtag files affected by the GTI intervals is shortened accordingly.
There are two ways to run the splittag task: (1) specify a starting time, an increment, and an ending time, or (2) provide an explicit list of times (not necessarily adjacent to one another). In either case, the output corrtag files will have a root name specified by the user. If no root name is specified, the root name of the input corrtag will be used, appended with numbers 1,...N for N exposures.
The parameters input by the user for splittag include the following: an input corrtag file name, a root name for the output files, the starting time for the first event to be extracted, the time increment to be used in extracting the following intervals, and the ending time of the extraction. If option (1) from above is used, then the starting time and increment are specified, with the remaining parameters left at their indefinite values. This will extract however many corrtag files are needed until the ending time of the original exposure is reached. If option (2) is used, then the user can specify explicitly, in the form of start/stop pairs, which intervals are desired. For example, specifying time_list="0,20,100" will extract events in the range 0 < t < 20 seconds and output that to a corrtag file, then extract events in the range 20 < t < 100 seconds and write that to another file, and so on. Splittag can also read in a text file with the start/stop pairs entered (using the format in the example above). In that case, all the start/stop pairs would be listed in one line in the text file, separated by either commas or spaces. If this option is used (i.e., the time_list parameter is set to point to the text file), then the starttime and increment parameters are ignored.
For example, to split the exposure, l61h9002r_corrtag.fits into two sub-exposures, with an output of 'split':
> from costools import splittag
> splittag.splittag('rootname_corrtag_a.fits', 'split', \
starttime=None, increment=None, \
endtime=None, time_list='0, 60, 120')
Next, the two sub-exposures should be extracted with x1dcorr either as a separate task or by running calcos from the corrtag file. (Note, however, that the x1dcorr task is now broken; for more information, see Running x1dcorr task in costools and custom COS spectral extraction, in the HST Knowledge Base.) To instead split the exposure into 20-second increments, the following command would be used instead:
> from costools import splittag
> splittag.splittag('rootname_corrtag_a.fits', 'split', \
starttime=None, increment=20., \
endtime=None, time_list=None)