wiki:DC2ImageProcMetrics
Last modified 12 years ago Last modified on 05/14/2007 10:27:01 AM

This page contains a list of the QA metrics we will derive from the nightly Image Processing Pipeline during DC2. Such metrics may be subdivided into code performance metrics and data analysis metrics. The code performance metrics would measure speed and robustness, and are probably not a focus for DC2. The data analysis metrics will tell us how well we have been able to characterize our images. JK has requested that we initially focus on 3 core metrics. We propose these be on the processes of

  • Astrometry
  • Photometry
  • Difference Imaging

Astrometry

The natural metric to derive is how well we have been able to astrometrically register our images with an external reference catalog. Given the right (i.e. perfect) input catalog, this translates directly into our ability to characterize our focal plane distortions (with some contribution from centroiding error). We propose a metric derived from the RMS deviation of stellar positions derived from our astrometric solution compared to the positions in the reference catalog, in units of coordinates RA and Dec.

In summary, we will measure the RMS difference between our corrected positions and the nominal positions of astrometric standard stars, in RA and Dec.

Photometry

We will detail our ability to recover stellar brightnesses in our nightly pipeline. In practice, for bright stars, brightness is derived from aperture photometry. For fainter stars, this is derived from a PSF model fit performed at small aperture, plus an aperture correction to larger aperture. We propose a metric on relative photometry, where relative here means w.r.t. our input catalog. We will measure the width of the distribution of differences in magnitudes for the brightest stars in our images, for both aperture and PSF photometry.

For aperture photometry, this requires doing correct sky subtraction, compensating for other objects in your aperture, and making sure you treat the boundaries of the aperture correctly. For PSF photometry, this requires the additional steps of characterizing your PSF, and deriving aperture corrections to PSF flux and the spatial variation of these corrections.

In summary, we will measure the width of the distribution of *differences* in measured magnitude between bright stars in our science image and in our static catalog. We will use aperture photometry at a large radius (5-10"), and only stars more than XXX and less than YYY magnitudes below saturation, where an initial estimate of XXX is 1 and YYY is 3.

Difference Imaging

There are many possible metrics here. We will focus on a 0th order metric, which compares the actual residuals in the difference image w.r.t the expected residuals given the noise in the input images. Ideally, a histogram derived from all the pixels in the difference image, normalized by the input noise, will have a mean = 0 and standard deviation = 1. Since most pixels in the images will be sky pixels, we will focus on deriving this metric from pixels containing (non variable) sources. Another useful metric is the raw number of detections found in each difference image, some measure of the false detection rate.

In summary, we will measure the mean and the standard deviation of the residuals in each difference image, normalized by the propagated variance, for all MaskedImages? used to derive the image subtraction kernel. Ideally, this distribution has a mean of 0, standard deviation of 1, and skewness of 0.