wiki:DC2NightlyProcessingUsecase
Last modified 12 years ago Last modified on 06/13/2007 12:05:57 PM

LSST Nightly Processing Usecase -- Wiki Format

Version 1.1

Introduction

This is a description of the nightly visit processing at the Base facility. This document is in Wiki format to accumulate the changes of all developers into one document. This page should reflect, as nearly and as clearly as possible, our current understanding of the processing. All developers are encouraged to keep this information up-to-date.

This page was initially based on http://dev.lsstcorp.org/htmldocs/NightlyPLUsecase.pdf.

The original diagram is at http://dev.lsstcorp.org/htmldocs/NightlyPLFlow.pdf; it hasn't been updated to reflect the most recent discussions.

There is also a source classification table at http://dev.lsstcorp.org/htmldocs/SourceClassificationTable.pdf which defines terms such as "Fast Mover".

Preparation For Observing

Prior to the observing night, a set of preparation steps have been performed:

  1. All fields F that are possible targets for tonight are obtained from the Scheduler
  1. For each Field F:
  1. Identify the current template image T for the Field F.
  1. If T is not at Base, fetch it from Archive
  1. If the version of T is later than the version used for DIASources, delete DIASources for F (note that this implies that a DIASource must be mappable to F).
  1. If DIASources for F does not contain at least the latest N nights, fetch the missing nights from Archive.
  1. MOPS Phase 2 is run at the Archive facility, incorporating all historical DIASources that are not matched to known variables in the Object Catalog. This produces an updated MovingObject? Catalog, which is transferred from Archive to Base.

Nightly Visit Processing

The LSST survey design is based on the "visit" as the unit of observation. A visit consists of a sequence of two camera exposures with identical filters and telescope pointing, separated in time only by the camera readout time. The principal motivation for this arrangement is to improve discrimination against cosmic rays and other noise events, and one of the tasks of the alert processing is to implement this discrimination in a computationally efficient fashion.

Nightly Visit Processing - MOPS Position Prediction

The MOPS ephemeris prediction code is run on the MovingObject? Catalog, given the telescope pointing and exposure epochs for each exposure in the visit. The result is a set of predicted DIASources, MOPSPred, including rough magnitudes, for the visit.

NOTE: If it is needed to reduce the processing time for this step, it can be done by running the ephemeris prediction code at the Archive center using the set of Fields and predicted exposure epochs from the Scheduler. The resulting ephemerides will be too inaccurate for direct use by the Nightly Pipeline, but they can be very rapidly corrected to the required accuracy once the exact exposure epochs are known.

Nightly Visit Processing - Image Processing and Detection Pipelines

The computational steps, illustrated in Figure http://dev.lsstcorp.org/htmldocs/NightlyPLFlow.pdf , are as follows:

  1. The two raw images from the visit, I1 and I2, are processed through the detrending stages of the Image Processing Pipeline (IPP), producing calibrated images C1 and C2. These have been corrected for instrumental response, had the atmospheric fringe image subtracted, and have been astrometrically and photometrically calibrated in a preliminary way (final calibration will occur at the Archive Center during Data Release processing). These steps make use of the calibration products produced at the Archive Center by the Calibration Pipeline, and an LSST catalog of photometric and astrometric standards.
  1. The template image for the field, T, which has been produced at the Archive Center, is subtracted from C1 and C2, producing difference images D1 and D2. Image subtraction requires that the template image be rotated and registered to the input image, and the PSF's matched. The algorithm will be an improved version of AlardLupton? [ref]. Note that T incorporates the information from a stack of images, and both cosmic ray events and moving objects have been removed from it by median filtering. The processing of steps 1 and 2 are logically independent for the two images I1 and I2, and this allows the processing for I1 to be completed while I2 is still being exposed.
  1. Image D2 is registered to D1, and added to it, producing D+. This is the last step of the IPP. Note the implicit assumption that the PSF is stable between the two images of the visit. If not true in practice, this step will need a slight modification during commissioning.
  1. The Detection Pipeline (DP) now begins, by detecting sources in D+, producing source collection S+. Note that the input image is a difference image, so that the sources in it (which may have negative flux in some cases) are mostly point sources or cosmic rays, the principal exception being streaks from rapidly moving objects. This processing stage is an evolved version of the code "sextractor" [ref].
  1. The source collection S+, and the images D1 and D2, are used by the next processing stage, "Classify Sources", in the DP to classify the type of the sources. To do this, the information in S+ is used to extract for each source a small subimage from each of D1 and D2 containing the source. The subimages are saved in an image collection for later use. The source fluxes and shapes in these subimages are analyzed according to the scheme in Table XX. Cosmic rays are recognized as such by their shape in a single image or, if present in both images, because they have shapes which do not match. A small fraction of cosmic rays will result in a PSF-like shape. If present in only a single image, these will be misclassified as a Flash. As a result, if Alerts are generated from Flashes a significant noise level will be present. The probability of two independent cosmic rays producing a PSF shape at the same location in the image is very small (the actual probability will not be known until commissioning data is available), and therefore the resulting contamination of "Positive Excursions" will also be small. Cosmic ray sources are removed from S+ and placed in the Junk Source Collection. The Junk Source Collection is utilized by the DQA system and periodically emptied. The remaining sources in DIA Source Collection are tagged with their type, added to the DIA Source Catalog, and passed to the Association Pipeline.

Nightly Visit Processing - Association Pipeline

  1. The incoming DIA Source Collection, S+, is cross matched with the Object Catalog. Note that the difference image context makes magnitude information useless in performing this cross-match. The ID of the matching Object (which may be NULL) is entered into the record for each Source, along with its variable/non-variable classification from the Object Catalog.
  1. Entries in S+ which do not match an Object that is a previously known variable are cross matched with the MOPS Predicted Source Collection, MOPSPred. Note that magnitude information may be useful for this cross match. This is because the difference flux from a moving object is usually the entire object flux, and thus may be taken into account by the cross-match. Note that each object will have a different MOPS-supplied error ellipse, and the cross match must return all matching objects within this ellipse. Sources in S+ which are successfully matched to a predicted source in MOPSPred are marked with their MOPS ID and deleted from S+. ISSUE: If a particular MOPS predicted position has a large error ellipse, there may be more than one entry in S+ that matches, even if magnitude information is taken into account. [Lynne Jones is evaluating how often this is likely to occur]. In this case, it is probably best to retain all matching sources in S+ for possible Alert generation.
  1. The Object Catalog is updated from S+: If the Object ID for a source in S+ is NULL, a new entry is made for it in the Object Catalog. Note that as a previously unknown moving object moves across the sky, new Objects will be formed each time it is detected until MOPS is able to predict its position with sufficient accuracy. These Object entries will be pruned after MOPS processing is completed at the Archive, and the Sources will be linked instead to a single entry in the MovingObject? Catalog. Note that the entire DIASource Collection will also be passed to MOPS at the Archive Center, though MOPS will ignore any DIASources which are associated with known variable Objects.
  1. The sources in S+, all of which now have an associated object, are passed to Alert Processing.

Nightly Visit Processing - Alert Generation

Editor's Note: (Ray Plante) This last section has been flagged as out of scope for DC2.

The nightly processing of a visit concludes with Alert Generation, which is performed on the full object entry. Each object's characteristics, including its prior history is checked against a set of rules that determines whether or not an alert will be issued, and if so, the type of alert. If an alert is to be generated, the object information, together with the subimages containing the sources are packaged into an alert structure, which is sent to the Archive center for dissemination to the community using the VOEvent mechanism [ref] Alert Processing opens the Source Collection (SC) sent by the Source Router. For each Source S in SC:

  1. Load the associated Object O. Note that this implies that there must be a map from S to O. O is guaranteed to have N epochs of DIASources available if it has been observed at least that many times by the survey to date (see section on preparation for observing).
  1. Whether or not an Alert is generated for S will depend on a potentially large number of factors which will be referenced by a set of rules. These rules may call on all the data associated with O such as time history, classification, and shape, as well as those of S itself, and the subimages associated with S. The rules are certain to change over time as we (and the community) gain experience, so it is important that the Alert generator be able to accept the rules in some readily changeable form.
  1. If an Alert is to be generated, an Alert will be created with all relevant data, including the two small images containing S, and sent to the Archive center for distribution via the VOEvent protocol.