SDQA Metrics for LSST

This document defines the metrics that are under consideration for the LSST Science Data Quality Assessment subsystem. SDQA is relevant for evaluating the scientific performance of the acquired data and scientific processing over many timescales (Alert Production, nightly summary processing, calibration processing, Data Release Production, and progress of the survey against design) and spatial scales (CCD amplifier, full CCD, raft, focal plane array, and source catalogs).

Related ticket: #409

Metric Attributes

The attributes for each metric are:

  • Name: name of the field in the database for each metric
  • Units: physical units of each metric; quantities with no physical units should be so indicated. Note that the units should follow the IAU recommendation for basic units, with additional allowed units and formatting guidelines as recommended in the FITS standard (see  http://fits.gsfc.nasa.gov/fits_standard.html, Table 4.2).
  • Type: data type to be used in the database for each metric
  • UCD: Uniform Content Descriptor for each metric (see  http://www.ivoa.net/Documents/REC/UCD/UCD-20050812.html for the definition of UCDs used in the Virtual Observatory); quantities with no UCD are indicated with a blank. [Note: this information TBD]
  • Production: the production(s) where the metric is computed. Value will be one or more of: * "Alert" for Alert-level processing
    • "Night" for nightly summary processing
    • "Cal" for Calibration processing
    • "DR" for Data Release processing
  • Entity: the entity or collection of entities over which the metric is computed. Possible values are:
    • "Amp" for a 512x2048 amplifier section of an image
    • "CCD" for a single 4Kx4K CCD sensor
    • "Raft" for a physical block of 3x3 co-mounted CCDs
    • "FPA" for the entire focal plane array of sensors
    • "Source" for all entries in the Source database
  • Description: Brief description of the meaning for each metric, and how it will be calculated

Exposure-level Metadata

Certain exposure-level metadata will be recorded for each image in order to make best use of the QA metrics. Most of the quantities listed below will be obtained directly (or derived) from the observing environment, as recorded in the Engineering & Facility database.

Name Units Type Production Description
ccdId [None] Int32 Alert Unique identifier for this CCD
offsetFpaX pixel Float Alert Separation between a reference point on the FPA and the first X pixel of a CCD
offsetFpaY pixel Float Alert Separation between a reference point on the FPA and the first Y pixel of a CCD
dateObs dateTime double Alert Timestamp for the start of this exposure, formatted per ISO8601
expTime s Float Alert Duration of the exposure
filterId [None] Int32 Alert Unique identifier for the filter (i.e., bandpass) in use
seeingDimm arcsec Float Alert Atmospheric seeing during the exposure, as determined from the median of DIMM measurements
seeingWfs arcsec Float Alert Atmospheric seeing during the exposure, as determined from the median of wave-front sensor measurements
angleMoon deg Float Alert Separation between the center of the lunar disk and the center of the image section at the start of the exposure
phaseMoon [None] Float Alert Phase of Moon, expressed as fraction illuminated: 1.0=full, 0.0=New
speedWind km/hr Float Alert Wind speed
directionWind deg Float Alert Wind direction, measured from North through East

DC3

Only a small set of metrics were captured for Data Challenge 3a (see below), since the main objective for SDQA in DC3 was to create the supporting infrastructure and prototype a status GUI. The list of metrics for subsequent data challenges will be expanded considerably.

  • ip.diffim.kernelSum
  • ip.diffim.residuals
  • ip.isr.numCosmicRayPixels -- Number of pixels affected by cosmic rays
  • ip.isr.numSaturatedPixels -- Number of pixels affected by detector saturation
  • phot.psf.numAvailStars -- Number of catalog stars within CCD footprint that are available for WCS solution
  • phot.psf.numGoodStars -- Number of catalog stars within CCD footprint actually used (i.e., not rejected) in WCS solution
  • phot.psf.spatialFitChi2
  • phot.psf.spatialLowOrdFlag

A precise definition of some of the above metrics is needed, and the names must be revised to conform to the nomenclature standard.

The complete set of metrics is still being compiled; the metrics listed in the tables below are meant to focus the discussion among all the stakeholders of SDQA.

Metrics for Basic Image Characterization

The following are basic, descriptive statistics for LSST raw images, computed on an Alert Production timescale. All quantities that describe native brightness (BUNIT) are anticipated to be in units of ADU for Amp, and in electrons for CCDs after ISR pipeline processing. Note that several of the metrics below (in italic) are naturally derived from image histograms. Image histograms could be persisted as a separate data product, and these QA quantities could be computed on-the-fly.

Name Units Type Production Entity Description
nGoodPix [None] Int32 Alert, DR Amp, CCD Number of good pixels, i.e., excluding nBadCalibPix, nSaturatePix, nSpurPix, nNonlinPix.
nBadCalibPix [None] Int32 Alert, DR Amp, CCD Number of pixels with compromised values (charge traps, hot pixels).
nSpurPix [None] Int32 Alert, DR Amp, CCD Number of pixels compromised by non-astronomical transients (Rad hits, mux bleeds, satellite trails, etc.).
nSaturatePix [None] Int32 Alert, DR Amp, CCD Number of pixels compromised by detector saturation.
nNonlinPix [None] Int32 Alert, DR Amp, CCD Number of pixels for which detector non-linearity cannot be corrected accurately.
fracGhostBg percent Float Alert, DR CCD Fraction of image area affected by ghost images with surface brightness gradients exceeding 0.333 of sky noise. [SRD: "GhostAF"]
overscanMean ADU Float Alert, DR Amp Mean of overscan pixels, with outlier rejection
overscanStdDev ADU Float Alert, DR Amp Standard deviation about mean of overscan pixels, computed from non-rejected values
overscanMedian ADU Float Alert, DR Amp Median of overscan pixels, with outlier rejection; used as robust estimate of CCD bias level
overscanMin ADU Float Alert, DR Amp Minimum value of overscan pixels
overscanMax ADU Float Alert, DR Amp Maximum value of overscan pixels
imageMean BUNIT Float Alert, DR CCD Mean of image, computed from good pixel values
imageStdDev [None] Float Alert, DR CCD Standard deviation about image mean, computed from good pixel values
imageMedian BUNIT Float Alert, DR CCD Median of image, computed from good pixel values
imageMode BUNIT Float Alert, DR CCD Mode of image, computed from good pixel values. This is an excellent measure of the background level unless there are very extended targets in the image
imageMin BUNIT Float Alert, DR CCD Minimum of image, computed from good pixel values
imageMax BUNIT Float Alert, DR CCD Maximum of image, computed from good pixel values
imagePtile01 BUNIT Float Alert, DR CCD Bottom 1% of pixel brightness distribution
imagePtile16 BUNIT Float Alert, DR CCD Bottom 16% of pixel brightness distribution
imagePtile84 BUNIT Float Alert, DR CCD Bottom 84% of pixel brightness distribution
imagePtile99 BUNIT Float Alert, DR CCD Bottom 99% of pixel brightness distribution
imageGradientX BUNIT/pixel Float Alert, DR CCD Image gradient, X-direction, computed from good pixel values in background regions
imageGradientY BUNIT/pixel Float Alert, DR CCD Image gradient, Y-direction, computed from good pixel values in background regions

Calibration Metrics

Calibration metrics are determined during the astrometric and photometric calibrations of each image.

Name Units Type Production Entity Description
wcsErrX arcsec Float Alert, DR CCD RMS of WCS solution, X-direction
wcsErrY arcsec Float Alert, DR CCD RMS of WCS solution, Y-direction
nWcsAvailStars arcsec Float Alert, DR CCD Number of available catalog stars for WCS solution
nWcsGoodStars arcsec Float Alert, DR CCD Number of non-rejected stars used in WCS solution
amplFringe [None] Float Alert, DR CCD Amplitude of fringe pattern, relative to subtraction template
photDepth ABmag Float Alert, DR CCD Median photometric depth, parameterized by filter/bandpass; threshold based on SNR=5 for point sources, an A05 star, 0.7 arcsec seeing, airmass=1.0, exposure=30 s, sky=21 mag/arcsec2. [SRD: "DB1"]
diffCatVsInst [mag] Float Alert, DR CCD Median of differences of catalog mag and instrumental mag for all detected sources. Used to detect attenuation across the FOV due to partial cloud cover.
photThruput percent Float Alert, DR CCD Implied system throughput. Note that this quantity is trivially derived from photDepth, and so could be considered virtual. [SRD: "DT1"]
photZeroPt mag Float Alert, DR CCD Photometric zero-point--i.e., magnitude corresponding to one count in the calibrated image.
psfFwhm arcsec Float Alert, DR CCD Median of delivered seeing (in r- and i-bands for SRD) for parameterized atm seeing. [SRD: "S1"]
psfEllipticity [None] Float Alert, DR CCD Median of point source ellipticity distribution (in r- and i-bands for ZD<10 deg and a reference atmospheric seeing of 0.69 arcsec). [SRD: "SE1"]

Other SRD Metrics

The following quantities map directly to the specifications in the Science Requirements Document (v4.3.3, as of 20-Oct-2008). These metrics are not yet assigned to one of the above categories of SDQA metrics, and some of the metric names need revision.

Name Units Type Production Entity Description
phot.star.rms mmag Float DR Source Upper limit of the point source magnitude distribution RMS about the mean. [SRD: "PA1"]
photZeroPtRms mmag Float DR Source Error distribution RMS of the internal photometric zero-point. [SRD: "PA3"]
phot.star.exErr percent Float DR Source Threshold for fraction by which photometric error from repeated observations can exceed noise from photon statistics and other expected sources. [SRD: "EPErr"]
psfEncirc(frac) arcsec Float DR CCD Radius within which frac of flux in contained for fiducial seeing=0.69 arcsec. [SRD: "SR''n''"]
psfEllipticityAcc [None] Float DR CCD Median of ellipticity auto- and cross-correlation functions, averaged over the detector must be less than this value within a parameterized distance from the center. [SRD: "SE3, SE4"]
astrm.distRms mas Float DR CCD, Raft, FPA Threshold for RMS of the distance distribution for stellar pairs with a parameterized separation. [SRD: AM''n'']
astrm.trans.bandErr mas Float DR FPA Accuracy of the astrometric transformations from band-to-band. [SRD: "AB1"]
astrm.transErr mas Float DR FPA Absolute accuracy of transformation of astrometric system to an external system (ICRF). [SRD: "AA1"]

===Data Quality Testing Metrics ===

The following matrics are to test data quality of LSST processing during DC3b (Aug, 2009 to May 2010).

* Production: the production(s) where the metric is computed. Value will be one or more of:
  • "Alert" for Alert-level processing
  • "Night" for nightly summary processing
  • "Cal" for Calibration processing
  • "DR" for Data Release processing
  • Entity: the entity or collection of entities over which the metric is computed. Possible values are:
    • "Amp" for a 512x2048 amplifier section of an image
    • "CCD" for a single 4Kx4K CCD sensor
    • "Raft" for a physical block of 3x3 co-mounted CCDs
    • "FPA" for the entire focal plane array of sensors
    • "Source" for all entries in the Source database
  • Description: Brief description of the meaning for each metric, and how it will be calculated

Data Quality Testing Metrics

The following matrics are to test data quality of LSST processing during DC3b (August 2009 to May 2010).

Subject Name Units Type Production Entity Description (contact for algorithm) Criteria
Background bkg_outlier none none Night Amp average DN number relative to the background numbers of adjacent (5) images. bkg_amp > bkg_avg(ccd)
Background bkg_high DN Float Night Amp average background DN number greater than 2/3 of saturation potential well bkg >> 2/3 bkg (saturation)
WCS wcs.dist.astronet arcsecFloat Night SDQA distance between expected WCS and performed WCS (Tim) wcs_dist > 1 arcsec
WCS wcs.dist.usno arcsecFloat Night SDQA distance between expected WCS (USNO catalog) and performed WCS wcs_dist > 1 arcsec
WCS wcs.dist.err arcsecFloat Night SDQA mean error of wcs_dist
WCS wcs.diff.hist noneFloat Night SDQA nomality of difference image histogram (Jeonghee)
WCS, photometry wcs.cal.photometry magFloat Night Amp (or CCD) photometry of a set of calibration stars compared with expected phtometry
WCS, astrometry wcs.disp arcsecFloat Night Amp (or CCD) Distribution of wcs.dist
photometrygrri.colorcolor magFloat Night CCD g-r and r-i color-color diagram photometry check and main-sequence population vs brown dwarf
photometrybvv.color.mag magFloat Night CCD B-V vs V color-magnitude diagram photometry check and main-sequence and pre-main-sequence population
photometrygrcolor.mag magFloat Night CCD g-r color vs Mr magnitude diagram photometry check and WD population
seeing/FWHMhist.mag magFloat Night CCD histogram of FWHM as a function of magnitude
photometrygrcolor.mag dmagFloat Night CCD g-r color vs Mr magnitude diagram photometry check and WD population