wiki:DC3bDataProducts

Version 21 (modified by ktl, 10 years ago) (diff)

Expanded object-like measurements, including transient objects.

DC3b Data Products

The DC3b Data Release Production will generate a number of outputs. This page attempts to describe these outputs in more detail. Provisional class names, typically based on prior usage in DC3a, are given in brackets.

Some of these products may be merged in the course of generation. Some may be merged during catalog ingest. Some may not be persisted permanently.

Some of these products may be stretch goals for DC3b but are expected to be present in production. Others are not expected to be present in production but are necessary for DC3b.

Source-like measurements

S1. Measurements of (quite) bright stars on single CCDs (I3) for PSF and WCS determination. CalibSource

  • These are only stars, but they include all those measured above a magnitude limit as possible PSF or WCS candidates.
  • These are produced internally to the Image Characterization Pipeline and will be persisted.
  • The matching between the catalogs and the CalibSource will be persisted in separate tables.

S2. Measurements of astronomical objects over the 4.5 -- 5 sigma threshold on single CCDs (I4). Source

  • These are produced by Single Frame Measurement.
  • These are used by Astrometric Calibration (and downstream science).
  • At least the brighter Sources need to be associated into groups before passing them to Astrometric Calibration.
  • These will be associated with Objects by Object Merge/Association?.
  • These may not be produced or may not be produced in the same manner in production.

S3. Forced photometry measurements on single CCDs (I4) using the shapes from Multifit Measurement. ForcedSource

  • These include both stars and galaxies.
  • These are inherently associated with Objects by virtue of being measured at the locations of deep Detections.
  • These may include measurements above and below the S/N threshold. The ones below the S/N threshold could perhaps have less information stored, or all of these could have less information stored. The ones above the S/N threshold will be associated with the measurements in S2 indirectly through their corresponding Object.
  • These are produced by Forced Photometry.

S4. Measurements on single-CCD difference images (I5) using positions from Difference Detection. DiaSource

  • These are only above a 5 sigma detection S/N threshold, unless MOPS requires lower S/N.
  • These are produced by Difference Measurement.
  • These are used by Alerts and MOPS.
  • These will be associated with Objects or used to create new Objects by Object Merge/Association?.

S5. Forced photometry measurements on single-CCD difference images (I5) using a list of positions including all predicted MovingObject positions from MOPS and all alerts issued prior to the commencement of Data Release processing. ForcedDiaSource

  • What happens if an alert was issued at a position that no longer has a DiaSource in the original alert-causing Exposure due to new ISR or CR removal or calibration or differencing algorithms?
  • These are inherently associated with Objects or MovingObjects by virtue of being measured either at the locations of DiaSources (S4) that are themselves associated with a MovingObject or (new) Object or else at the predicted location of a MovingObject.
  • These may include measurements above and below the S/N threshold. The ones below the S/N threshold could perhaps have less information stored, or all of these could have less information stored. The ones above the S/N threshold will be associated with the measurements in S4 indirectly through their corresponding Object or MovingObject.
  • [TA] For MOPS generated positions, we probably need to pass a "tweak up position" flag to the measurement process.

S6. Additional per-Source global photometric calibration information.

  • This includes gray extinction for each Source (S2) and ForcedSource (S3).
  • These will not be present for CalibSources (S1), DiaSources (S4) or ForcedDiaSources (S5).
  • These values will be maintained in the final catalog as additional columns for Source and ForcedSource.
  • These are produced by Photometric Calibration.

S7. Evaluations of each Object's Multifit shape model for each Exposure in which the Object could have been visible. EvaluatedModel

  • These will be present in production. They may not be available in DC3b.
  • In theory, these values can be produced from the shape model parameters in Object combined with the information in each Exposure, but it may be more robust and convenient to produce these explicitly.
  • These would be produced by Multifit Measurement.

Object-like measurements

All of these items will end up as columns in the Object or MovingObject table.

O1. Detections on the panchromatic deep chi-square co-add (C3).

  • These are only above a detection S/N threshold.
  • These are produced by Deep Detection.
  • These may only be Footprints or positions with some shape information.

O2. Measurements of detections (O1) on each of the per-filter deep co-adds (C2). Detection

  • These may be produced by Deep Detection or by Multifit Measurement itself.
  • These will eventually need to be de-blended.
  • These are used by Multifit Measurement as the initial conditions for fitting.

O3. Model parameters based on stacks of postage stamps (St2) for all detections in O1.

  • These are produced by Multifit Measurement.
  • These are used by Forced Photometry.

O4. Astrometric models for all astronomical objects measured sufficient numbers of times in Single Frame Measurement (S2).

  • These are produced by Astrometric Calibration.
  • These are used by Multifit Measurement.

O5. Moving object models (i.e. orbits). [MovingObject]

  • These are produced by MOPS.

O6. Measurements derived from one or more DiaSources that cannot be associated with a MovingObject or a deep Detection.

  • These are produced by DiaSource Association.

O7. Averages or other summaries of measurements from S1-5 and S7 (combined with all of O1-4 and O6). Object

O8. Additional per-Object photometric calibration information.

  • These are produced by Photometric Calibration.

CCD-level images

I1. Pre-ISR images in standard format with standard metadata.

  • Produced from input images and metadata by a metadata standardization stage as part of ISR.

I2. Post-ISR images.

  • Computed from pre-ISR images (I1) by ISR.

I3. Summed visit images.

  • Computed by adding two post-ISR images (I2) images per visit.

I4. Calibrated Science Exposures with PSF and WCS.

  • Produced by merging the PSF and WCS from Image Characterization on the summed visit image (I3).

I5. Difference images.

  • Computed from Calibrated Science Exposures (I4) and the per-filter template co-adds (C1) by Difference Imaging.

I6. PSF-matched, warped images for template co-addition.

  • Produced from the Calibrated Science Exposures (I4).
  • An alternate possibility is to preserve only the PSF-matching and warping kernels, generating needed sub-images on the fly by cutting out more pixels than required and then applying the preserved kernels.

I7. Calibrated Science Exposures with moving objects masked out.

  • Produced from the Calibrated Science Exposures (I4) and MovingObjects (O5) and/or the DiaSources (S4) associated with MovingObjects.

Co-adds

C1. Template co-adds, one per filter, using "good seeing" exposures (I4).

C2. Deep co-adds, one per filter, using all "reasonable" exposures (I4).

C3. Panchromatic deep chi-square co-add for deep detection.

  • Produced by combining the per-filter deep co-adds (C2).

Stacks

St1. Cutouts from PSF-matched, warped images (I6) for template co-addition.

St2. Cutouts from moving-object-masked Calibrated Science Exposures (I7) in the vicinity of each detection (O1) for Multifit Measurement and Forced Photometry.

  • [RHL] These may have to be larger than you'd like. What is your vision of 'vicinity'?

Possible Schema for Sources and Objects

Source

[RHL] At least for DC3b we're making measurements down to 5ish sigma on each frame; this is called S2 above.

[RHL] I assume that the pipeline is putting out uncalibrated quantities -- DN and pixel coordinates not fluxes/magnitudes and ra/dec. That's why the astrom quantities are float, not double, for example. I'm not sure how the DB is going to handle calibrations and I haven't worried about that here. The _id is an int for similar reasons; the DB needs more bits.

    int _id;
    boost::int64_t _flagForDetection;

    float _xAstrom;      --  x position computed by a centroiding algorithm for 
                         --  the purposes of astrometry using Dave Monet's algorithm
    float _xAstromErr;
    float _yAstrom;
    float _yAstromErr;
    float _xyAstromErr;

    float _xOther;       -- x position computed by another centroiding algorithm
    float _xOtherErr;
    float _yOther;
    float _yOtherErr;
    float _xyOtherErr;

    float _sky;
    float _skyErr;

    float _psfLnL;              // ln(Likelihood) of being a PSF

    float _psfFlux;
    float _psfFluxErr;
    float _apFlux;
    float _apFluxErr;

    float _modelFlux;
    float _modelFluxErr;
    float _modelAb;		// Or do we store r_e, e_1, e_2 etc.?
    float _modelAbErr;
    float _modelPhi;
    float _modelPhiErr;
    float _modelRe;
    float _modelReErr;
    float _modelSersicN;
    float _modelSersicNErr;

    float _i;
    float _iErr;
    float _ix;
    float _ixErr;
    float _iy;
    float _iyErr;
    float _ixx;
    float _ixxErr;
    float _iyy;
    float _iyyErr;
    float _ixy;
    float _ixyErr;
  • [TA] The various "_model" parameters should correspond to what DC3b multifit is actually measuring. I suspect this does not include Sersic parameters. UCD folks should comment.
  • We probably want to rename _flagForDetection; it's covering all of detection and measurement.
  • [RHL] I've also included zeroth and first Gaussian-weighted moments as Tim asked for them. I suspect that the first moments will be very close to the other centroids, but maybe not. I imagine that the aperture corrections to the Gaussian-weighted flux are done via a Gaussian fit to the PSF and forcing the flux to match the PSF flux; that's certainly how I think that the models should be calibrated.
  • The _i* quantities are Gaussian-weighted moments, with the Gaussian chosen self-consistently.
  • [RHL] I've omitted any Petrosian quantities (radius, flux + errors at the very least) that we may want later, but probably aren't in scope for DC3b
  • [RHL] I also have not included any higher-moments for KSB "polarizability" corrections a l\'a Seljak; if someone wants these I think we should add them to some derived class.

CalibSource

    int _id;
    boost::int64_t _flagForDetection;

    float _xAstrom;
    float _xAstromErr;
    float _yAstrom;
    float _yAstromErr;
    float _xyAstromErr;

    float _psfFlux;
    float _psfFluxErr;
    float _apFlux;              // needed for aperture correction calculation
    float _apFluxErr;

    float _ixx;                 // probably none of _i* need be persisted
    float _ixxErr;              // but they are (currently) used in PSF determination
    float _iyy;
    float _iyyErr;
    float _ixy;
    float _ixyErr;

Source

    int _id;
    boost::int64_t _flagForDetection;

    float _xAstrom;
    float _xAstromErr;
    float _yAstrom;
    float _yAstromErr;
    float _xyAstromErr;

    float _psfFlux;
    float _psfFluxErr;

    float _i;
    float _iErr;
    float _ixx;
    float _ixxErr;
    float _iyy;
    float _iyyErr;
    float _ixy;
    float _ixyErr;
  • [RHL] I'm assuming that the PSF aperture corrections are available from the WcsSource processing. The flux will still be in DN, but it'll have had an aperture correction applied (as is also true for Source)
  • The moments are primarily for the moving objects, but should carry some information about defects, cosmics, etc. too

DiaSource

see Source

ForcedSource

    int _id;
    boost::int64_t _flagForDetection;
    boost::int64_t _objectId

    float _sky;
    float _skyErr;

    float _psfFlux;
    float _psfFluxErr;
    float _apFlux;
    float _apFluxErr;

    float _modelFlux;
    float _modelFluxErr;
    float _modelFlux_canonicalBand;
    float _modelFlux_canonicalBandErr;
  • [RHL] Note that there's a 64 bit (input) objectId here. If the DB's associating things in some cleverer way I don't need to include it here.
  • [RHL] I've included an aperture flux, but I think it's negotiable. Will our colleagues demand it?
    • [TA] I think we need it.
  • [RHL] I've also included a PSF flux. This may not be needed in addition to the model flux, but I think we should keep it for DC3b
  • [RHL] I've included a flux in this band's model and a flux in some canonical band's model; I think that both are needed to get optimal fluxes and colours.
    • [TA] I'm not sure exactly what the "canonicalBand" flux is, but it certainly comes out of photometric calibration. It should (IMHO) be the flux in the LSST standard bandpass.
    • [RHL] It's the flux in this band using the model from the "canonical band". This will be specified by a Policy file --- I'd expect it to usually be i --- but you have to be careful for objects that aren't detected in that band.
  • [RHL] I'd be happy to remove the sky quantities. I like to estimate the sky globally (per CCD) rather than locally. It's helpful to report it at the position of each object (when deblending I included the siblings in SDSS) but I don't think it's critical. We don't yet know if this global-sky prejudice will hold up for LSST.
    • [TA] However it is determined, I think we need to report it locally.

ForcedDiaSource

  • [RHL] I think that this is the same as ForcedSource. It's possible that we can omit the model fluxes --- I think that they only matter for extended moving objects, and have no real idea how well they'll work. Maybe the _i and _iErr Gaussian fluxes would be better??
  • [RHL] I'd certainly omit the sky here

EvaluatedModel

There will be one table per model. In DC3b we will deal with PointSources only (PS). Other models may have ellipses (shape convolved with psf)

EvaluatedModelPS:

    float _x;
    float _xErr;
    float _y;
    float _yErr;
    float _xyErr;

    float _flux;
    float _obsTime;
    int8 _filterId;

Detection

??