wiki:DC3bProcessingStages
Last modified 9 years ago Last modified on 04/08/2010 08:20:25 PM

DC3b Processing Stages

This page lists the stages, aside from InputStages and OutputStages, used in the DC3b Data Release Production. The data flow is shown at DC3bProcessingFlow.

NOTE 9 March 2010: more current image of DC3B Processing Flow indicating proposed minimum PT1 scope and stretch PT1 scope.

Persisted outputs are marked with (P).

ISR

DC3b PT1 required.

This pipeline performs instrument signature removal. It also standardizes the input image and metadata into LSST format. Each slice operates on a single amplifier image.

Stage Owner Source Dictionary Stage UnitTests
ip.pipeline.IsrLinearityStage sk isrLinearityStage.py IsrLinearityStageDictionary.paf isrStageUnitTests.py
ip.pipeline.IsrSaturationStage sk isrSaturationStage.py IsrSaturationStageDictionary.paf isrStageUnitTests.py
ip.pipeline.IsrOverscanStage sk isrOverscanStage.py IsrOverscanStageDictionary.paf isrStageUnitTests.py
ip.pipeline.IsrBiasStage sk isrBiasStage.py IsrBiasStageDictionary.paf isrStageUnitTests.py
ip.pipeline.IsrDarkStage sk isrDarkStage.py IsrDarkStageDictionary.paf isrStageUnitTests.py
ip.pipeline.IsrFlatStage sk isrFlatStage.py IsrFlatStageDictionary.paf isrStageUnitTests.py
ip.pipeline.IsrFringeStage sk isrFringeStage.py IsrFringeStageDictionary.paf isrStageUnitTests.py
sdqa.pipeline.isrSdqaStage rrl isrSdqaStage.py isrSdqaStageDictionary.paf isrSdqaStageTest.py

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.isr sk

Events:

  • One event per input amplifier exposure containing its pathname. No particular order requirements.

Inputs:

  • amplifier image
  • image metadata
  • camera class instance
    • saturation
    • linearity lookup table (PAF) path and filename
    • overscansec
  • fwhm
  • biasExposure
  • overscanfittype (should be policy parameter)
  • darkexposure
  • exposurescale
  • darkscale
  • flatexposure
  • flatscalingtype
  • fringeexposure (not used)

Outputs:

  • various corrected images
  • (P) final corrected image

Todo:

  • Given exposure, determine and load the other clipboard inputs.
  • Implement fringe correction
  • Illumination correction?

CcdAssembly

DC3b PT1 required. This pipeline assembles amplifiers from ISR into CCD Exposures. It complete defect removal on the assembled exposure.

Stage Owner Source Dictionary Stage UnitTest
lsst.ip.pipeline.AssembleCcdStage sk not checked in yet
ip.pipeline.IsrDefectStage sk isrDefectStage.py IsrDefectStageDictionary.paf isrStageUnitTests.py

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.ccdAssembly sk

Events:

  • One event per input CCD containing 8 or 16 ISR-processed amplifier exposure pathnames. (Eventually, when STL collection persistence is available, this may be a directory path or a list of exposure paths in a single value.) No particular order requirements, other than the dependency on ISR.

Inputs:

  • amplifier image (x8/16)
  • defectList

Outputs:

  • (P) CCD image

Todo:

  • Need formatter for Saturation Defects - persisted from ISR pipeline.

CrSplit

DC3b PT1 required.

This pipeline handles cosmic ray removal using the two "split" exposures per visit, algorithmically rejecting cosmics on each one and then doing a difference between them to find more cosmics. Background estimation is in this pipeline because it is required before CrReject. Each slice operates on a visit pair of CCD images, assembling them into a single CCD exposure.

Note: SimpleDiffIm should not need to use a Kernel, since the seeing is presumed to be identical across the two exposures.

Similarly, SumStage should not need to do PSF-matching.

Stage Owner Source Dictionary Stage UnitTest
meas.pipeline.BackgroundEstimationStage rhl backgroundEstimationStage.py BackgroundEstimationStageDictionary.paf backgroundEstimationStageTest.py
ip.pipeline.CrRejectStage rhl crRejectStage.py CrRejectDictionary.paf crRejectStageTest.py
ip.pipeline.SimpleDiffImStage rhl simpleDiffImStage.py SimpleDiffImStageDictionary.paf diffImStageTest.py
meas.pipeline.SourceDetectionStage rhl sourceDetectionStage.py SourceDetectionStageDictionary.paf SourceDetectionStageTest.py
ip.pipeline.crSplitCombineStage rhl crSplitCombineStage.py CrSplitCombineStageDictionary.paf

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.CRSplit rhl crSplitStagesTest.py crSplitStages_policy.paf

Events:

  • One event per visit pair of input CCD exposures containing their pathnames. No particular order requirements, other than the dependency on CcdAssembly?.

Inputs:

  • exposure (x2) (mask modified in place)

Outputs:

  • (P) background (x2) (in processingInfo)
  • (P) nCR (x2) (in processingInfo)
  • exposure (x2)
  • differenceExposure
  • positiveDetection, negativeDetection
  • psf
  • backgroundSubtractedExposure
  • (P) combinedExposure

Todo:

  • Future: persist detections, maybe differenceExposure

ImgChar

DC3b PT1 required.

This pipeline handles image characterization, including determination of PSF and WCS. It also handles initial CCD-exposure-level photometric calibration and background subtraction. It combines all of this information into a single Exposure. The CalibSources used for the determinations are saved. Each slice operates on a single CCD image and produces a science CCD exposure that is used throughout the rest of the Data Release Production.

SourceDetection should be configured to accept only bright stars.

Stage Owner Source Dictionary Stage UnitTest
meas.pipeline.SourceDetectionStage rhl sourceDetectionStage.py SourceDetectionStageDictionary.paf SourceDetectionStageTest.py
meas.pipeline.SourceMeasurementStage rhl sourceMeasurementStage.py SourceMeasurementStageDictionary.paf sourceMeasurementStageTest.py
meas.pipeline.PsfDeterminationStage rhl psfDeterminationStage.py PsfDeterminationStageDictionary.paf psfDeterminationStageTest.py
meas.pipeline.WcsDeterminationStage frm wcsDeterminationStage.py WcsDeterminationStageDictionary.paf WcsDeterminationTest.py
meas.pipeline.WcsVerificationStage frm wcsVerificationStage.py WcsVerificationStageDictionary.paf WcsVerificationTest.py
meas.pipeline.PhotoCalStage frm photoCalStage.py PhotoCalStageDictionary.paf PhotoCalTest.py

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.imChar rhl

Events:

  • One event per input CCD exposure containing its pathname. No particular order requirements, other than the dependency on CrSplit?.

Inputs:

  • exposure (mask modified in place)
  • photoCalSourceSet
  • wcsVerificationSourceSet

Outputs:

  • background
  • positiveDetection
  • psf (rough)
  • (P) psf (measured)
  • (P) cellSet
  • (P) sdqa
  • (P) matchList (astrometric)
  • (P) wcs
  • (P) backgroundSubtractedExposure

Todo:

  • Write stage to perform preliminary photometric calibration (maybe later).
  • Given exposure and wcs on clipboard, load appropriate source set from photometric calibration catalog.
  • Given exposure and wcs (and psf?) on clipboard, create new exposure.
  • Given wcs, load appropriate source set from WCS verification catalog.
  • What will cellSet actually be on the clipboard and how will it be persisted?

SFM

DC3b PT1 required.

This pipeline handles measurement of Sources on a single frame. No deblending is done in DC3b. Each slice operates on a single science CCD Exposure.

Stage Owner Source Dictionary Stage UnitTest
meas.pipeline.SourceDetectionStage rhl sourceDetectionStage.py SourceDetectionStageDictionary.paf sourceDetectionStageTest.py
meas.pipeline.SourceMeasurementStage rhl sourceMeasurementStage.py SourceMeasurementStageDictionary.paf sourceMeasurementStageTest.py

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.sfm rhl

Events:

  • One event per input science CCD exposure containing its pathname. No particular order requirements, other than the dependency on ImgChar?.

Inputs:

  • exposure (mask modified in place)
  • psf

Outputs:

  • background
  • (P) positiveDetection
  • (P) sources

Todo:

AstroCal

DC3b PT2 required; PT1 stretch.

This pipeline handles astrometric calibration. Each slice operates on all Sources within a given sky tile across all epochs. Before calibration can be performed, each Source must be associated with a deep Detection.

Stage Owner Source Dictionary Stage UnitTest
datarel.SkyTileMetadataStage ktl
ap.pipeline.DetectionSourceAssocStage sm
astrocal.pipeline.AstroCalStage dgm+rhl

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.astroCal ??

Events:

  • One event per sky tile? Must be after all exposures for the sky tile have been processed.

Inputs:

  • Sources from database
  • Objects (or at least Detections) from database

Outputs:

  • Astrometric models for each Object (?)

Todo:

  • Define stages.
  • Write pipeline policy.

PSFMatch

DC3b PT2 required; PT1 stretch.

This pipeline handles the matching of PSFs between selected (excellent-seeing) individual Exposures and a reference model PSF for template co-adds. It also handles the mapping of Exposures into sky pixel space. There will be 6 reference model PSFs, one for each filter. Output is a sky map image and the PSF matching kernel. Each slice operates on a single science CCD Exposure.

Stage Owner Source Dictionary Stage UnitTest
coadd.pipeline.PsfMatchStage ro psfMatchStage.py psfMatchStage_dict.paf
lsst.datarel.GenerateSkyTileStage ro
coadd.pipeline.ExposureToHealPixStage ro

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.psfMatch ?ro?

Events:

  • One event per input science CCD exposure containing its pathname and a sky map identifier set covered fully by the exposure. No particular order requirements, except for the dependency on ImgChar?, but best if run on a per-sky-tile basis to enable coaddition to work optimally.

Inputs:

  • exposure
  • referencePsf

Outputs:

  • psfMatchedExposure
  • (P) psfMatchingKernel
  • psfMatchingKernelSum
  • (P) skyTile

Todo:

  • Write stage to generate blank sky tile given a sky map identifier set.
  • Write stage to map an exposure from rectilinear space to sky pixel space.
  • Write formatter for sky map image class.

TemplateGen

AKA: Co-Add Stack with Outlier Rejection

DC3b PT2 required; PT1 stretch.

This pipeline handles the generation of the difference imaging templates using outlier rejection. There will be 6 of these, one for each filter. Each slice operates on a stack of PSF-matched sky map images that overlap a single template-generation sky tile. The template-generation sky tile will be smaller than other sky tiles due to the need to keep an entire stack in memory.

Stage Owner Source Dictionary Stage UnitTest
lsst.datarel.GenerateSkyTileStage ro
coadd.pipeline.OutlierRejectionStage ro outlierRejectionStage.py outlierRejectionStage_dict.paf

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.templateGen ??

Events:

  • One event per sky tile containing the pathname of a directory containing the sky map images to be processed. Should be ordered spatially to reuse exposures at boundaries of template-generation sky tiles. Depends on PSFMatch.

Inputs:

  • skyMapImageStack

Outputs:

  • (P) skyTile

Todo:

  • Write stage to generate blank sky tile given a sky map identifier set.
  • Write formatter for sky map image class.

DiffIm

DC3b PT2 required; PT1 stretch.

This pipeline handles computation of difference images by subtracting templates from Exposures. It also detects and measures DiaSources on the resulting difference images. Each slice operates on a single science CCD Exposure.

Stage Owner Source Dictionary Stage UnitTest
datarel.IdentifySkyTilesStage ro
coadd.pipeline.HealPixToExposureStage ro
ip.pipeline.DiffImStage ab diffImStage.py DiffImDictionary.paf diffImStageTest.py
meas.pipeline.SourceDetectionStage rhl sourceDetectionStage.py SourceDetectionStageDictionary.paf sourceDetectionStageTest.py
meas.pipeline.SourceMeasurementStage rhl sourceMeasurementStage.py SourceMeasurementStageDictionary.paf sourceMeasurementStageTest.py

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.diffIm ??

Events:

  • One event per science CCD exposure containing its pathname. Should be ordered spatially to reuse templates.

Inputs:

  • skyMapImage
  • scienceExposure

Outputs:

  • skyMapIdSet
  • coaddExposure
  • (P?) differenceExposure
  • (P) spatialKernel
  • (P) differentialBackgroundModel
  • (P) positiveDetection, negativeDetection
  • psf
  • (P) sources

Todo:

  • Write stage to identify sky map id set covering a given exposure.
  • Write stage to map a sky map image from sky pixel space to rectilinear space, matching a given exposure.

MOPS

DC3b PT2 required; PT1 stretch.

This pipeline handles the linking of DiaSources into orbits, which are preliminary MovingObjects. A single slice repeatedly operates on all DiaSources from a single night.

Stage Owner Source Dictionary Stage UnitTest
mops.daymops.SetupStage jm (?new pex_h?) mops.daymops.SetupStage
mops.daymops.IntraNightLinkingStage jm (?new pex_h?) mops.daymops.IntraNightLinkingStage
mops.daymops.InterNightLinkingStage jm (?new pex_h?) mops.daymops.InterNightLinkingStage
mops.daymops.OrbitDeterminationStage jm (?new pex_h?) mops.daymops.OrbitDeterminationStage
mops.daymops.OrbitManagementStage jm (?new pex_h?) mops.daymops.OrbitManagementStage

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.dayMOPS ?jm?

jmyers: OrbitDeterminationStage will need to be parallelized; this can be done within Python. InterNightLinkingStage is mainly a call to linkTracklets, which may be parallelized in time.

Also NightMOPS, which is responsible for producing predicted object locations (see below in DiffPhotom).

Events:

  • One event per night. Pipeline runs with any number of slices, parallelizing orbit determination over the available slices.

Inputs:

  • diaSourceList from database

Outputs:

  • tracklets to database
  • tracks to database
  • movingObjects to database (updated in place after creation)

Todo:

  • Convert to new pex_harness Stage class.
  • Write pipeline policy.

MaskMovers

DC3b PT2 required; PT1 stretch.

This pipeline handles the masking of moving objects from science CCD Exposures that are to be used for deep detection and multifit measurement. Each slice operates on a single science CCD Exposure as well as (some type of) footprints for all moving objects that appear within that Exposure.

Stage Owner Source Dictionary Stage UnitTest
datarel.VisitMetadataStage ktl
ip.pipeline.MaskFootprintsStage md

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.maskMovers ?md?

Will this use DiaSources or Sources or something else to mask? The first is easiest, as MOPS can do the association. The other possibilities would likely require a separate association step.

Will the measured DiaSource have sufficient information to mask, or is the Footprint detected on the difference image better?

CoaddGen

DC3b PT2 required; PT1 stretch.

AKA: Co-Add Exposure to Deep Co-Adds.

This pipeline handles the co-addition of masked Exposures to form deep templates. There will be 6 of these, one per filter. Each slice operates on a single science CCD Exposure, adding it to the appropriate filter's deep co-add.

Stage Owner Source Dictionary Stage UnitTest
datarel.SkyTileMetadataStage ktl
coadd.pipeline.CoaddGenerationStage ro coaddGenerationStage.py coaddGenerationStage_dict.paf
coadd.pipeline.ExposureToHealpixStage ro

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.coaddGen ?ro?

Events:

  • One event per science CCD exposure or psfMatchedExposure containing its pathname. Must be ordered spatially within coadd sky tiles. First exposure defines the sky tile size and location. Last exposure for a given sky tile should have "isLastExposure" set to true in the event.

Inputs:

  • psfMatchedExposure

Outputs:

  • (P) coadd
  • (P) weightMap
  • (P) weight

Todo:

  • Write pipeline policy.
  • Given event on clipboard, load psfMatchedExposure.
  • Persist coadd, weightMap, and weight.

DeepDet

DC3b PT2 required; PT1 stretch.

This pipeline handles the generation of the deep panchromatic chi-square co-add and detection and measurement on that co-add. Each slice operates on a single sky tile.

Stage Owner Source Dictionary Stage UnitTest
datarel.SkyTileMetadataStage ktl
coadd.pipeline.ChiSquaredStage ro chiSquaredStage.py chiSquaredStage_dict.paf
meas.pipeline.SourceDetectionStage rhl sourceDetectionStage.py SourceDetectionStageDictionary.paf sourceDetectionStageTest.py
meas.pipeline.SourceMeasurementStage rhl sourceMeasurementStage.py SourceMeasurementStageDictionary.paf sourceMeasurementStageTest.py

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.deepDet ?rhl?

Are the existing meas.pipeline stages adequate for detection on HEALPix, or does a transformation to rectangular pixel space need to be done first?

Comment by Martin Dubcovsky 10/23/09

We determined a few pssibilities for how this might be done:

  1. specialize detection, and measurement code to work directly in HEALPix projection
  2. transform to rectangular pixels, and handle duplicate removal as part of association stage
  3. transform to rectangular pixels, but handle duplicate removal before multifit
  4. transform to rectangular pixels, but arrange it such that you never have duplicates, although you may miss some things.

As for reusing the detection/measurement stages, we will likely want them to do more than they did in DC3a. We certainly want to fit a model to each detection on the coadd. If we want to reuse the stages, they could be configured via policy to enable/disable such behaviors.

Events:

  • One event per science CCD exposure containing its pathname. Must be ordered spatially within coadd sky tiles. First exposure defines the sky tile size and location. Last exposure for a given sky tile should have "isLastExposure" set to true in the event.
  • One event per detection sky tile containing the pathnames of the deep and chi-squared coadd exposures covering it.

Inputs:

  • exposure
  • psf (optional)

Outputs:

  • coadd
  • weightMap
  • (P) positiveDetection
  • background
  • psf
  • (P) sources

Todo:

  • Rework pipeline into two pipelines, one to generate chi-squared coadd and one to do detection.
  • Detection pipeline may require deep coadds in addition to chi-squared coadd.
  • Write pipeline policy.
  • Given event on clipboard, load psfMatchedExposure.
  • Given event on clipboard, load chi-squared and deep coadd exposures.
  • Persist coadd, weightMap; positiveDetection and sources.

Multifit

DC3b PT3 required; PT2 stretch.

This pipeline handles the projection of a detection from a coadd to the all the exposures that overlap it, using the list of projected detections to input a stack of "postage stamp" images, and fitting of a model using the multifit algorithm. This includes forced photometry of the resulting shape (prototype Object) on each image of the stack to form ForcedSources. Each slice operates on a single detection. For efficiency, slices should be handed detections that come from the same region of sky.

Unlike normal input stages, the stage for getting a stack of postage stamps on the clipboard determines what to input from data on its clipboard, not from policy.

Stage Owner Source Dictionary Stage UnitTest
meas.pipeline.TransformDetection md transformDetectionStage.py TransformDetectionStageDictionary.paf
meas.pipeline.InputPostageStampStack md
meas.pipeline.MultifitStage md+jb multifitStage.py MultifitStageDictionary.paf
meas.pipeline.ForcedPhotomStage md+jb forcedPhotometryStage.py ForcedPhotometryStageDictionary.paf

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.multifit ?md?

Events:

  • One event per detection containing its key information. Should be ordered spatially to reuse exposures.

Inputs:

  • detection (model) (updated in place)
  • exposure subregion list

Outputs:

  • chisq
  • covarianceMatrix
  • forcedSources

Todo:

  • Write pipeline policy.
  • Given detection on clipboard, load science CCD exposure list and bbox list.
  • Persist model, chisq, and covarianceMatrix as preliminary Object, and persist forcedSources.

DiaSourceAssoc

DC3b PT3 required; PT2 stretch.

This pipeline handles association of DiaSources that have not already been associated with MovingObjects by DayMOPS. It associates them with prototype Objects from Deep Detection and Multifit. Unassociated DiaSources are taken to be transients and new Objects are created for them.

Stage Owner Source Dictionary Stage UnitTest
ap.pipeline.ObjectDiaSourceStage sm

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.diaSourceAssoc ???

DiffPhotom

DC3b PT3 required; PT2 stretch.

This pipeline handles forced photometry on difference images to form ForcedDiaSources. Each slice operates on a single CCD difference image as well as the MovingObjects and transient Objects that occur within it.

Stage Owner Source Dictionary Stage UnitTest
datarel.VisitMetadataStage ktl
mops.nightmops.MopsStage jm (?new pex_h?) pipeline.py ---none--- testMopsStage.py
meas.pipeline.SourceMeasurementStage rhl sourceMeasurementStage.py SourceMeasurementStageDictionary.paf sourceMeasurementStageTest.py

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.diffPhotom ???
  • We may need to use a more sophisticated measurement if we take Multifit models to measure on difference images.

ObjectAssoc

DC3b PT3 required; PT2 stretch.

This pipeline handles the association of Sources with MovingObjects.

Stage Owner Source Dictionary Stage UnitTest
mops.nightmops.MopsStage jm (?new pex_h?) pipeline.py ---none--- testMopsStage.py
ap.pipeline.MovingObjectSourceStage sm

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.objectAssoc ???

DbIngest

DC3b PT3 required; PT2 stretch.

This pipeline handles the ingestion of data products, including Objects, Sources, DiaSources, ForcedSources, ForcedDiaSources, and CalibSources, into the database.

Stage Owner Source Dictionary Stage UnitTest
cat.IngestStage ktl

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.ingestStage ktl

Events:

  • One event per sky tile?

Inputs:

  • Pathnames of files to be ingested

Outputs:

  • Loads to database

Todo:

  • Define stage.

PhotoCal

DC3b PT3 required; PT2 stretch.

This pipeline handles the global photometric calibration of all data products. It is expected to extract data from the database.

Stage Owner Source Dictionary Stage UnitTest
photocal.pipeline.GlobalPhotoCalStage tsa

--

Pipeline Owner Pipeline UnitTest Pipeline UnitTest Dictionary
datarel.tests.photoCal tsa

Events:

  • One event per sky tile?

Inputs:

  • Sources, ForcedSources, DiaSources, ForcedDiaSources, CalibSources(?), Objects, MovingObjects from database
  • Exposure information from database (?)

Outputs:

  • Updates for all inputs (?) applied to database

Todo:

  • Define stages.

Attachments