DC2 Development and Execution Plan Outline

For draft paragraph descriptions, follow the links for each section heading.

1. General Description

1.1. Specific Goals?

  • Demonstrate use of astronomical algorithms for nightly processing
  • Establish an LSST simulated object/source database for testing by broader collaboration
  • Demonstrate integration of MOPS into LSST framework
  • Establish reusable code baseline for future DCs and LSST
  • Further develop and demonstrate common framework functionality
  • Update middleware implementation to enable application classes
  • Demonstrate a database partitioning scheme
  • Test the bandwidth utilization of the UDT protocol in the context of LSST data replication between Chile and the US
  • Pilot the development and testing process and tools for future DCs and LSST

1.2. Operations/Execution?

  • Plan for repeat runs of each part, document/analyze results with each run

1.3 Metrics?

1.4 Validation

We will have met our goals if:

  • We demonstrate a nightly processing pipeline that operates continuously on a set of input data. This processing includes at least image subtraction, source detection, association, and moving object detection using MOPS. As stretch goals, we would also include image linearization (flat-fielding, bias subtraction, etc.) and WCS determination.
  • We produce a persistant database of source detections and associations as a result of the nightly processing.
  • We execute MOPS automatically witin the nigthly pipeline.
  • We have checked in all original code used in DC2 into SVN.
  • Demonstrate the use of LSST framework and support classes
  • Employ in DC2 a harness that accesses MP functionality within a compiled layer but which can execute application code via a python interface.

2. Application Components

  • Nightly Processing Pipeline
    • Image Processing – NS, AB, RO
      • Common Dependencies (fw classes): Image, MaskedImage, Exposure, Kernel, WCS (and Catalog?)
        • Calibration
          • Inputs: raw images, calibration images (darks, biases, flats), masks (if available)
          • Outputs: calibrated MaskedImages
        • Astrometry (WCS Determination)
          • Inputs: calibrated Images, object detection, object Catalog
          • Outputs: calibrated Exposures
        • Image Subtraction
          • Inputs: calibrated Exposures, template archive
          • Outputs: difference Exposures
    • Detection – RHL, NS, AB, RO
      • For object detection immediately after Calibration
        • Inputs: calibrated MaskedImages
        • Outputs: new Objects for object Catalog
      • For object detection on the difference images
        • Inputs: difference Exposures, object Catalog
        • Outputs: new Objects
    • Association - SM, MNS, SN
    • MOPS - FP, JM, LJ
      • 64-bit support
        • Milani Wrappers
        • Milani libmopsephem
        • Compile Auton Code
        • Compile Extra S/W
        • Install on mops64
      • Ephem Service
        • Day-Time Service
        • Night-Time Service
        • X-Match Radius
      • Integration
        • Association Pipeline Interface
        • Data Transfer Service
      • Test
  • Application Framework
    • LsstData classes (astronomy data, algorithms) - TSA, RHL
      • MaskedImage - TSA
        • GIL vs Vision Workbench
      • Exposure - NS
      • Catalog
      • Match
      • Kernel - RO
      • WCS - UW?
    • Supporting classes (common framework behaviors, interface to Middleware)
      • Evaluate STL, boost classes for use in framework - All
      • Citizen class providing smart pointer, memory management, debug logging trace - RHL
      • Flexible tracing facility - RHL
      • DataProperty class providing Name Value pairs and collections of same - JK, JAB
      • Exception handling, assertion use - RA, SP
      • Metadata (DataProperty implementation, persistence via serialized object files) - JPK, JAB
      • Provenance (DataProperty implementation, persistence via serialized object files) - JPK, JB, RP, JAB
      • Policies (DataProperty implementation, persistence via serialized object files, 3-level policies: Default, Periodic, and Current, Configuration vs Trace provenance) - JPK, RP, JAB
      • File Persistence (Simple objects to serialized object files, composite and collection objects iterating over simple objects, recursively) - JPK, RP, JAB
      • DBMS Persistence (Database ingest of data and metadata, DBMSStorage & object -relational adapters, Table class) - JB, SM, RP
      • Formatters (fitsio, STIL formatters)
      • Events (Passed as XML a la OGRE, monitored/triggers a la DAGMAN, workflow control a la Chimera/Pegasus, remote monitoring/logging a la GRAM, interface to DMCS/OCS) - SP
  • Third Party
    • wcslib
    • GIL or Vision Workbench
    • C++ std (including STL)
    • numpy? There's a ? here, but RHL thinks that it's clear that we want to use numpy.
    • python plotting (sm? matplotlib? All of the above?)
    • Boost (which parts?)
    • fitsio, fics?
    • STIL?
  • Database Schemata? - JB

3. Middleware Components

  • Update MySQL and Ingest Services - JB, RO
  • Expand processing harness to support C++-based astronomical modules (Python/C++ interfaces, Gaia datatrain concept) - FP, GD, RHL
  • Logging (keep NetLogger?) - ???
  • Data Transport
    • DS on top of SRB/IRODS, http, Gridftp, Sector/UDT - CC, CW, SL, UIC

4. Infrastructure Components

  • Teragrid. other cluster as Processing Platform - RP, NCSA?
  • Development Platforms - RP, NCSA?
    • Lincoln development cluster
  • Database Deployment plan
  • MOPS Deployment plan

5. Input Data

  • CFHTLS raw images - TSA, RP
  • Simulated LSST raw images (DES, SkyMapper, Arcdata) - TSA, AJC, RP, CS, PP
  • LLNL Simulated sources - SN
  • Pan-STARRS Solar System Model generated sources - FP, LJ
  • Galactic simulation - KC, TSA
  • Extragalactic simulation - PP, TSA

6. Development Process and Tools?

7. Development Schedule