wiki:DC3bDataProducts

Version 11 (modified by ktl, 10 years ago) (diff)

Add more cross-links.

DC3b Data Products

The DC3b Data Release Production will generate a number of outputs. This page attempts to describe these outputs in more detail. Provisional class names, typically based on prior usage in DC3a, are given in brackets.

Some of these products may be merged in the course of generation. Some may be merged during catalog ingest. Some may not be persisted permanently.

Source-like measurements

S1. Measurements of (quite) bright stars on single CCDs (I3) for PSF and WCS determination. WcsSource

  • These are only stars.
  • These are produced internally to the Image Characterization Pipeline.
  • [RHL] We probably need to persist these too; you always ret not being able to recover this sort of information. It would be sufficient to just persist enough information to determine which stars were used, but probably not worth the trouble.
  • [RHL] Aren't we doing photometric calibrations here too?
  • [RHL] I don't like this class name. CalibSource?

S2. Measurements of astronomical objects over a given S/N threshold on single CCDs (I4). Source

  • [RHL] For DC3b, the S/N threshold will be 4.5 -- 5 sigma
  • These are produced by Single Frame Measurement.
  • Do we need to associate these before passing them to Astrometric Calibration? [RHL] Yes, I think so. Astrometry (== DGM) wants the set of Sources associated with a given Object; this implies matching across colours, I think, as he needs the object's colour
  • These are used by Astrometric Calibration.
  • These will be associated with Objects by Object Merge/Association?.

S3. Forced photometry measurements on single CCDs (I4) using the shapes from Multifit Measurement. Source and/or FaintSource

  • These include both stars and galaxies.
  • These may include measurements above and below the S/N threshold. The ones below the S/N threshold could perhaps have less information stored, or all of these could have less information stored. The ones above the S/N threshold might need to be associated with and combined with the measurements in S2, but it is more likely that this association will be indirect through their corresponding Object.
  • These are produced by Forced Photometry.
  • [RHL] I really don't like the name FaintSource. The point is that these are forced photometry, not that they are faint.

S4. Measurements on single-CCD difference images (I5) using positions from Difference Detection. DiaSource

  • These are only above a detection S/N threshold.
  • [RHL] Probably 5 sigma for DC3b. If we need to recover low S/N detections of moving objects for MOPS I can imagine pressure for lower significance detections, which may or may not need to be kept forever (as we'll be doing forced photometry later anyway; S5)
  • These are produced by Difference Measurement.
  • These are used by Alerts and MOPS.
  • These will be associated with Objects or used to create new Objects by Object Merge/Association?.

S5. Forced photometry measurements on single-CCD difference images (I5) using (all of) the positions/shapes from Deep Detection (O1). DiaSource and/or FaintDiaSource

  • These may include measurements above and below the S/N threshold. The ones below the S/N threshold could perhaps have less information stored, or all of these could have less information stored. The ones above the S/N threshold might need to be associated with and combined with the measurements in S4, but it is more likely that this association will be indirect through their corresponding Object or MovingObject.
  • Won't we also need to do forced photometry at orbit positions calculated by MOPS for MovingObjects? [RHL] Probably; this is related to my comment on S4

S6. S1-5 with additional per-Source photometric calibration information.

  • Will the DiaSources (S4-5) also have photo-cal information?
  • These are produced by Photometric Calibration.

Object-like measurements

O1. Detections on the panchromatic deep chi-square co-add (C3).

  • These are only above a detection S/N threshold.
  • These are produced by Deep Detection.
  • These may only be Footprints or positions with some shape information.

O2. Measurements of detections (O1) on each of the per-filter deep co-adds (C2). Detection

  • These may be produced by Deep Detection or by Multifit Measurement itself.
  • These will eventually need to be de-blended.
  • These are used by Multifit Measurement as the initial conditions for fitting.

O3. Model parameters based on stacks of postage stamps (St2) for all detections in O1.

  • These are produced by Multifit Measurement.
  • These are used by Forced Photometry.

O4. Astrometric models for all astronomical objects measured sufficient numbers of times in Single Frame Measurement (S2).

  • These are produced by Astrometric Calibration.
  • These are used by Multifit Measurement.

O5. Moving object models (i.e. orbits). [MovingObject]

  • These are produced by MOPS.

O6. Combinations of O1-5 with averages or other summaries of measurements from S1-5. Object

O7. O6 with additional per-Object photometric calibration information.

  • These are produced by Photometric Calibration.

CCD-level images

I1. Pre-ISR images in standard format with standard metadata.

  • Produced from input images and metadata by a metadata standardization stage as part of ISR.

I2. Post-ISR images.

  • Computed from pre-ISR images (I1) by ISR.

I3. Summed visit images.

  • Computed by adding two post-ISR images (I2) images per visit.

I4. Calibrated Science Exposures with PSF and WCS.

  • Produced by merging the PSF and WCS from Image Characterization on the summed visit image (I3).

I5. Difference images.

  • Computed from Calibrated Science Exposures (I4) and the per-filter template co-adds (C1) by Difference Imaging.

I6. PSF-matched, warped images for template co-addition.

  • Produced from the Calibrated Science Exposures (I4).
  • An alternate possibility is to preserve only the PSF-matching and warping kernels, generating needed sub-images on the fly by cutting out more pixels than required and then applying the preserved kernels.

I7. Calibrated Science Exposures with moving objects masked out.

  • Produced from the Calibrated Science Exposures (I4) and MovingObjects (O5) and/or the DiaSources (S4) associated with MovingObjects.

Co-adds

C1. Template co-adds, one per filter, using "good seeing" exposures (I4).

C2. Deep co-adds, one per filter, using all "reasonable" exposures (I4).

C3. Panchromatic deep chi-square co-add for deep detection.

  • Produced by combining the per-filter deep co-adds (C2).

Stacks

St1. Cutouts from PSF-matched, warped images (I6) for template co-addition.

St2. Cutouts from moving-object-masked Calibrated Science Exposures (I7) in the vicinity of each detection (O1) for Multifit Measurement and Forced Photometry.

  • [RHL] These may have to be larger than you'd like. What is your vision of 'vicinity'?

Possible Schema for Sources and Objects

Source

At least for DC3b we're making measurements down to 5ish sigma on each frame; this is called S2 above.

I assume that the pipeline is putting out uncalibrated quantities -- DN and pixel coordinates not fluxes/magnitudes and ra/dec. That's why the astrom quantities are float, not double, for example. I'm not sure how the DB is going to handle calibrations and I haven't worried about that here. The _id is an int for similar reasons; the DB needs more bits.

    int _id;
    boost::int64_t _flagForDetection;

    float _xAstrom;
    float _xAstromErr;
    float _yAstrom;
    float _yAstromErr;
    float _xyAstromErr;

    float _xDgm;
    float _xDgmErr;
    float _yDgm;
    float _yDgmErr;
    float _xyDgmErr;

    float _sky;
    float _skyErr;

    float _psfFlux;
    float _psfFluxErr;
    float _apFlux;
    float _apFluxErr;

    float _modelFlux;
    float _modelFluxErr;
    float _modelAb;		// Or do we store r_e, e_1, e_2 etc.?
    float _modelAbErr;
    float _modelPhi;
    float _modelPhiErr;
    float _modelRe;
    float _modelReErr;
    float _modelSersicN;
    float _modelSersicNErr;

    float _i;
    float _iErr;
    float _ix;
    float _ixErr;
    float _iy;
    float _iyErr;
    float _ixx;
    float _ixxErr;
    float _iyy;
    float _iyyErr;
    float _ixy;
    float _ixyErr;
  • We probably want to rename _flagForDetection; it's covering all of detection and measurement.
  • I've included extra astrometric quantities to test Dave Monet's preferred astrometric algorithms (as well as an astrometric covariance that is needed in theory but maybe not in practice, at least for DC3b). It isn't clear that Dave actually wants to calculate even errors, so we may gain some bytes here.
  • I've also included zeroth and first Gaussian-weighted moments as Tim asked for them. I suspect that the first moments will be very close to the other centroids, but maybe not. I imagine that the aperture corrections to the Gaussian-weighted flux are done via a Gaussian fit to the PSF and forcing the flux to match the PSF flux; that's certainly how I think that the models should be calibrated.
  • The _i* quantities are Gaussian-weighted moments, with the Gaussian chosen self-consistently.
  • I've omitted any Petrosian quantities (radius, flux + errors at the very least) that we may want later, but probably aren't in scope for DC3b
  • I also have not included any higher-moments for KSB "polarizability" corrections a l\'a Seljak; if someone wants these I think we should add them to some derived class.

WcsSource

    int _id;
    boost::int64_t _flagForDetection;

    float _xAstrom;
    float _xAstromErr;
    float _yAstrom;
    float _yAstromErr;
    float _xyAstromErr;

    float _psfFlux;
    float _psfFluxErr;
    float _apFlux;              // needed for aperture correction calculation
    float _apFluxErr;

    float _ixx;                 // probably none of _i* need be persisted
    float _ixxErr;              // but they are (currently) used in PSF determination
    float _iyy;
    float _iyyErr;
    float _ixy;
    float _ixyErr;

Source ====== DiaSource

    int _id;
    boost::int64_t _flagForDetection;

    float _xAstrom;
    float _xAstromErr;
    float _yAstrom;
    float _yAstromErr;
    float _xyAstromErr;

    float _psfFlux;
    float _psfFluxErr;

    float _i;
    float _iErr;
    float _ixx;
    float _ixxErr;
    float _iyy;
    float _iyyErr;
    float _ixy;
    float _ixyErr;
  • I'm assuming that the PSF aperture corrections are available from the WcsSource processing. The flux will still be in DN, but it'll have had an aperture correction applied (as is also true for Source)
  • The moments are primarily for the moving objects, but should carry some information about defects, cosmics, etc. too

FaintSource

    int _id;
    boost::int64_t _flagForDetection;
    boost::int64_t _objectId

    float _sky;
    float _skyErr;

    float _psfFlux;
    float _psfFluxErr;
    float _apFlux;
    float _apFluxErr;

    float _modelFlux;
    float _modelFluxErr;
    float _modelFlux_canonicalBand;
    float _modelFlux_canonicalBandErr;
  • Note that there's a 64 bit (input) objectId here. If the DB's associating things in some cleverer way I don't need to include it here.
  • I've included an aperture flux, but I think it's negotiable. Will our colleagues demand it?
  • I've also included a PSF flux. This may not be needed in addition to the model flux, but I think we should keep it for DC3b
  • I've included a flux in this band's model and a flux in some canonical band's model; I think that both are needed to get optimal fluxes and colours.
  • I'd be happy to remove the sky quantities. I like to estimate the sky globally (per CCD) rather than locally. It's helpful to report it at the position of each object (when deblending I included the siblings in SDSS) but I don't think it's critical. We don't yet know if this global-sky prejudice will hold up for LSST.

FaintDiaSource

  • I think that this is the same as FaintSource. It's possible that we can omit the model fluxes --- I think that they only matter for extended moving objects, and have no real idea how well they'll work. Maybe the _i and _iErr Gaussian fluxes would be better??
  • I'd certainly omit the sky here

Detection

??