8.3 Mitigating the Effects of Persistence

Two possible ways to mitigate persistence are to 1) exclude the affected pixels from the analysis, or 2) subtract an estimate of the persistence signal from the data and use the modified data for the analysis.

Simple procedures can be used to mark compromised pixels and modify the data quality extensions of the flt files. Typically one would use the persist.fits products (described in the previous section) to define an acceptable level of persistence, e.g. 0.01 e¯/sec, based upon a visual inspection of the data and the corresponding persistence images. The selected level can be used to flag all pixels in the persist.fits image that have larger predicted persistence values. The choice of level should be based on the science objective and the fraction of pixels impacted. For example, if the DQ extension of the flt.fits image is defined as im1 and the persist.fits product as im2, the following python code could be used:

	tol = 0.01
	Boolean_mask = im2 >= tol 
	im1[Boolean_mask] += 1024

The 1024 value is a currently unused data quality flag for the IR detector.

This method is typically very good at flagging persistence from previous observations in staring mode. However, the “A-gamma” model predictions show some discrepancy from the observed amount of persistence for images taken shortly before the image of interest (short-term persistence). The user should be aware that the estimated internal persistence may differ from what is observed. In a similar fashion, when persistence is from prior spatially scanned observations, the model significantly underestimates the level of persistence observed in the data. The “A-gamma” model relies on fluence estimates from the flt images of prior observations. Therefore it implicitly assumes that the estimated flux in the flt accurately reflects the fluence absorbed by specific pixels. The CRCORR step of calwf3 works by fitting a line to the accumulating charge between reads, but for scan-mode observations, the flux is only accumulating in a very short interval, thus the pipeline is unable to report the true flux (Section 10.2). This is true not only for the scans but also for other very rapidly moving targets such as like asteroid trails ( WFC3 ISR 2017-18) or bright light sources (e.g. cities) that streak across some of the dark-Earth IR flats used to monitor the IR blobs (Section 7.5). Persistence in those cases is very poorly estimated and can be inaccurate by orders of magnitude.

However, for both internal persistence and for persistence from moving targets, the “A-gamma” model does predict the location of persistence on the detector very well.  Therefore, the user is advised to tune the tolerance level to account for possible mismatches if moving targets are the suspected causes of persistence.

Once the pixels are flagged, down-stream analysis proceeds as it normally would, assuming the tools that are used take the data quality flags into account. Persistence is a property of the pixel: as long as the observer who planned the original observations created a set of appropriately-dithered images, using this procedure to flag bad-persistence pixels should not significantly impact the science.

As an example, using AstroDrizzle to exclude those flagged pixels would result in a combined image that is largely cleansed of the effects of persistence. In some cases, the threshold defined from inspection of the flt.fits may not be low enough to detect lower-level persistence, so the deeper, combined drz.fits images may be used to verify whether persistence was fully removed.

An alternative method is to use the persistence-subtracted files in down-stream analysis. If one adopts this approach, one should carefully inspect the _flt_cor.fits corrected data products (produced by simply subtracting the persist.fits model directly from the flt.fits image) to evaluate how well persistence has been subtracted from the image. Observers should inspect all of the images in a particular science observation as the persistence signal will decline with time. Typically in situations in which there is persistence, about 90% of the persistence signature is removed by the model. Because there are variations in persistence which are still being studied, the corrected data products can be either under- or over-subtracted, particularly in the case of internal persistence where the power law model is not optimal for estimating the short term decay (WFC3 ISR 2019-02).

The persistence-subtracted image of our example is shown in Figure 8.5. The resulting image is much cleaner, especially at the center of the image where the γ-ray burst had occurred. However, a careful inspection shows some residual signatures of the persistence, particularly due to the set of early images that produced the horizontally-stepped faint features in the original image. Sometimes one can do better by scaling the persistence model and producing one's own persistence-subtracted image. It is also, of course, possible to combine these approaches, flagging the worst pixels in the persistence-corrected images then using AstroDrizzle to produce a combined image.

In all cases in dealing with persistence, as is true of most problems associated with data analysis, first assess the severity of the problem and then choose a method of handling the issue that is consistent with the science being carried out. Users concerned about the effects of persistence on their data should check for updates on the WFC3 Persistence pages as well as the main WFC3 web page.