8.3 Mitigating the Effects of Persistence
Two possible ways to mitigate persistence are to 1) exclude the affected pixels from the analysis, or 2) subtract the persistence signal from the data and use the modified data in the analysis.
Simple procedures can be used to mark compromised pixels and modify the data quality extensions of the flt files. Typically one would use the persist.fits products to define an acceptable level of persistence, say 0.01 e¯/sec, based upon a visual inspection of the data and the corresponding persistence images. This level can be used to flag all pixels in the persist.fits image that have larger predicted persistence. The choice of level would be based on the science objective and the fraction of pixels impacted. For example, if the DQ extension of the flt.fits image is defined as im1 and the persist.fits product as im2, the following python code could be used:
The 1024 value is a currently unused data quality flag for the IR detector.
This method is typically very good at flagging persistence from previous observations in staring mode. However, the “A-gamma” model predictions show some discrepancy from the observed amount of persistence for images taken shortly before the image of interest (short-term persistence). While the model is still being improved, the user should be aware that the estimated internal persistence may differ from what is observed. In a similar fashion, when persistence is from prior scanned observations, the model significantly underestimates the level of persistence observed in the data. The “A-gamma” model relies on fluence estimates from the flt images of prior observations. Therefore, it implicitly assumes that the estimated flux in the flt reflects the actual impinging flux on the pixels. The CRCORR step of calwf3 works by fitting a line to the accumulating charge between reads, but for scan-mode observations, the flux is only accumulating in a very short interval, thus the pipeline is unable to report the true flux (See also Section 10.2 on IR scanned observations). This is true not only for the scans but also for other moving targets that end up in WFC3/IR observations, like asteroid trails in external observations ( WFC3 ISR 2017-18) or bright cities or even ships that are imaged during the observations of the dark-earth IR flats taken by the WFC3 team to calibrate the IR blobs, see Section 7.5. Persistence in these cases is very poorly estimated.
However, for both internal persistence and for persistence from moving targets, the “A-gamma” model does still predict the location of persistence on the detector very well. The actual predicted values could be off, however, by orders of magnitude. Therefore, the user is advised to tune the tolerance level to account for these possible mismatches.
Once the pixels are flagged, down-stream analysis proceeds as it normally would, assuming the tools that are used take data quality flags into account. Persistence is a property of the pixel: as long as the observer who planned the original observations created a set of dithered images, using this procedure to flag bad-persistence pixels should not significantly impact the science.
As an example, using AstroDrizzle to exclude those flagged pixels would result in a combined image that is largely cleansed of the effects of persistence. In some cases, the threshold defined from inspection of the flt.fits may not be low enough to detect lower-level persistence, so the deeper, combined drz.fits images may be used to verify whether persistence was fully removed.
An alternative is to use the persistence-subtracted files in down-stream analysis. If one adopts this approach, one should carefully inspect the _flt_cor.fits corrected data products (produced by simply subtracting the persist.fits model directly from the flt.fits image) to evaluate how well persistence has been subtracted from the image. It is important to inspect all of the images in a particular science observation as the persistence signal will decline with time. Typically in situations in which there is persistence, about 90% of the persistence signature is removed by the model. Because there are variations in persistence which are still being studied, including spatial variations across the detector, the corrected data products can be either under- or over-subtracted, particularly in the case of internal persistence where the power law model is not optimal for estimating the short term decay.
The persistence-subtracted image of our example is shown in Figure 8.5. The image is much cleaner, especially at the center of the image where the γ-ray burst had occurred. However, a careful inspection shows some residual signatures of the persistence, particularly due to the set of early images that produced the horizontally-stepped faint features in the original image. Sometimes one can do better by scaling the persistence model and producing one's own persistence-subtracted image. It is also, of course, possible to combine these approaches as well, flagging the worst pixels in the persistence-corrected images, and then using AstroDrizzle to produce a combined image.
In all cases in dealing with persistence, as is true of most problems associated with data analysis, first assess the severity of the problem and then choose a method of handling the issue that is consistent with the science being carried out. Users concerned about the effects of persistence on their data should check for updates on the WFC3 web pages, WFC3 Persistence pages in the periodically released STANs, as well as the Instrument Science Reports.
WFC3 Data Handbook
- • Acknowledgments
- Chapter 1: WFC3 Instruments
- Chapter 2: WFC3 Data Structure
- Chapter 3: WFC3 Data Calibration
- Chapter 4: WFC3 Images: Distortion Correction and AstroDrizzle
- Chapter 5: WFC3-UVIS Sources of Error
- Chapter 6: WFC3 UVIS Charge Transfer Efficiency - CTE
Chapter 7: WFC3 IR Sources of Error
- • 7.1 WFC3 IR Error Source Overview
- • 7.2 Gain
- • 7.3 WFC3 IR Bias Correction
- • 7.4 WFC3 Dark Current and Banding
- • 7.5 Blobs
- • 7.6 Detector Nonlinearity Issues
- • 7.7 Count Rate Non-Linearity
- • 7.8 IR Flat Fields
- • 7.9 Pixel Defects and Bad Imaging Regions
- • 7.10 Time Variable Background Contamination
- • 7.11 IR Photometry Errors
- • 7.12 References
- Chapter 8: Persistence in WFC3 IR
- Chapter 9: WFC3 Data Analysis
- Chapter 10: WFC3 Spatial Scan Data