wiki:DC3bDataProducts

Version 6 (modified by rhl, 10 years ago) (diff)

Added initial links to possible schema

DC3b Data Products

The DC3b Data Release Production will generate a number of outputs. This page attempts to describe these outputs in more detail. Provisional class names, typically based on prior usage in DC3a, are given in brackets.

Some of these products may be merged in the course of generation. Some may be merged during catalog ingest. Some may not be persisted permanently.

Source-like measurements

S1. Measurements of (quite) bright stars on single CCDs for PSF and WCS determination. Dc3bDataProducts?

  • These are only stars.
  • These are produced internally to the Image Characterization Pipeline.
  • [RHL] We probably need to persist these too; you always regret not being able to recover this sort of information. It would be sufficient to just persist enough information to determine which stars were used, but probably not worth the trouble.

S2. Measurements of astronomical objects over a given S/N threshold on single CCDs. Dc3bDataProducts?

  • [RHL] For DC3b, the S/N threshold will be 4.5 -- 5 sigma
  • These are produced by Single Frame Measurement.
  • Do we need to associate these before passing them to Astrometric Calibration?
  • These are used by Astrometric Calibration.
  • These will be associated with Objects by Object Merge/Association?.

S3. Forced photometry measurements on single CCDs using the shapes from Multifit Measurement. Dc3bDataProducts? and/or Dc3bDataProducts?

  • These include both stars and galaxies.
  • These may include measurements above and below the S/N threshold. The ones below the S/N threshold could perhaps have less information stored, or all of these could have less information stored. The ones above the S/N threshold might need to be associated with and combined with the measurements in S2, but it is more likely that this association will be indirect through their corresponding Object.
  • These are produced by Forced Photometry.

S4. Measurements on single-CCD difference images using the positions from Difference Detection. Dc3bDataProducts?

  • These are only above a detection S/N threshold.
  • [RHL] Probably 5 sigma for DC3b. If we need to recover low S/N detections of moving objects for MOPS I can imagine pressure for lower significance detections, which may or may not need to be kept forever (as we'll be doing forced photometry later anyway; S5)
  • These are produced by Difference Measurement.
  • These are used by Alerts and MOPS.
  • These will be associated with Objects or used to create new Objects by Object Merge/Association?.

S5. Forced photometry measurements on single-CCD difference images using the positions/shapes from Deep Detection (O1). Dc3bDataProducts? and/or Dc3bDataProducts?

  • These may include measurements above and below the S/N threshold. The ones below the S/N threshold could perhaps have less information stored, or all of these could have less information stored. The ones above the S/N threshold might need to be associated with and combined with the measurements in S4, but it is more likely that this association will be indirect through their corresponding Object or MovingObject.
  • Will all positions/shapes from Deep Detection be used or only a subset, and if a subset, how will that be determined? [RHL] I'd guess a PSF model, a galaxy model, and a galaxy model in a canonical band (maybe i).
  • Won't we also need to do forced photometry at orbit positions calculated by MOPS for MovingObjects? [RHL] Probably; this is related to my comment on S4

S6. S1-5 with additional per-Source photometric calibration information.

  • Will the DiaSources (S4-5) also have photo-cal information?
  • These are produced by Photometric Calibration.

Object-like measurements

O1. Measurements of detections on the panchromatic deep chi-square co-add. Dc3bDataProducts?

  • These are only above a detection S/N threshold.
  • These are produced by Deep Detection.
  • These are used by Multifit Measurement as the initial conditions for fitting.
  • [RHL] I'm not sure that this is what I'd call a Detection

O2. Model parameters based on stacks of postage stamps for all detections in O1.

  • These are produced by Multifit Measurement.
  • These are used by Forced Photometry.

O3. Astrometric models for all astronomical objects measured sufficient numbers of times in Single Frame Measurement (S2).

  • These are produced by Astrometric Calibration.
  • These are used by Multifit Measurement.

O4. Moving object models (i.e. orbits). [MovingObject]

  • These are produced by MOPS.

O5. Combinations of O1-4 with averages or other summaries of measurements from S1-5. Dc3bDataProducts?

O6) O5 with additional per-Object photometric calibration information.

  • These are produced by Photometric Calibration.

CCD-level images

I1. Pre-ISR images in standard format with standard metadata.

I2. Post-ISR images.

I3. Summed visit images.

I4. Calibrated Science Exposures with PSF and WCS.

I5. Difference images.

I6. PSF-matched, warped images for template co-addition.

  • An alternate possibility is to preserve only the PSF-matching and warping kernels, generating needed sub-images on the fly by cutting out more pixels than required and then applying the preserved kernels.

I7. Calibrated Science Exposures with moving objects masked out.

Co-adds

C1. Template co-adds, one per filter, using "good seeing" exposures.

C2. Deep co-adds, one per filter, using all "reasonable" exposures.

C3. Panchromatic deep chi-square co-add for deep detection and measurement.

  • [RHL] I don't think we'll be measuring on this

Stacks

St1. Cutouts from PSF-matched, warped images (I6) for template co-addition.

St2. Cutouts from moving-object-masked Calibrated Science Exposures (I7) in the vicinity of each detection (O1) for Multifit Measurement and Forced Photometry.

  • [RHL] These may have to be larger than you'd like. What is your vision of 'vicinity'?

Possible Schema for Sources and Objects

WcsSource

Source

DiaSource

FaintSource

FaintDiaSource

Detection

Source

For the class coming out of the deep measurement stage for DC3b (I can't

call it an object!) I think that the schema will be something like this. The NULLSUPPORT is a template argument to the class indicating that that field may be NULL; I have a naive implementation that costs 4 bytes per quantity (and can be turned off by specialising differently) --- I have ideas on how to make this less inefficient.

Note that I am putting out *UN*calibrated quantities -- DN and pixel coordinates not fluxes/magnitutes and ra/dec. That's why the astrom quantities are float, not double, for example. I'm not sure how the DB is going to handle calibrations and I haven't worried about that here. The _id is an int for similar reasons; the DB needs more bits.

    int _id;
    boost::int64_t _flagForDetection;

    int _xPeak;			// probably not persisted
    int _yPeak;			//  *   *    *    *   *
    float _xAstrom;
    float _xAstromErr;
    float _yAstrom;
    float _yAstromErr;
    float _yAstromErr;

    float _xDgm;
    float _xDgmErr;
    float _yDgm;
    float _yDgmErr;
    float _yDgmErr;

    float _sky;
    float _skyErr;

    float _psfFlux;
    float _psfFluxErr;
    float _apFlux;
    float _apFluxErr;

    float _modelFlux;
    float _modelFluxErr;
    float _modelAb;		// Or do we store r_e, e_1, e_2 etc.?
    float _modelAbErr;
    float _modelPhi;
    float _modelPhiErr;
    float _modelRe;
    float _modelReErr;
    float _modelSersicN;
    float _modelSersicNErr;

    float _ixx;
    float _ixxErr;
    float _iyy;
    float _iyyErr;
    float _ixy;
    float _ixyErr;

I've included extra astrometric quantities to test Dave Monet's preferred astrometric algorithms (as well as an astrometric covariance that is needed in theory but maybe not in practice, at least for DC3b)

I omitted any Petrosian quantities (radius, flux + errors at the very least) that we may want later, but probably aren't in scope for DC3b

I also have not included any higher-moments for KSB "polarisiblity" corrections a l\'a Seljak; if someone wants these I think we should add them to some derived class.