wiki:DC2PlanOutline
Last modified 12 years ago Last modified on 09/04/2007 03:24:47 PM

DC2 Development and Execution Plan Outline

For draft paragraph descriptions, follow the links for each section heading.

  1. General Description

1.1. Specific Goals?

  • Demonstrate use of astronomical algorithms for nightly processing
  • Establish an LSST simulated object/source database for testing by broader collaboration
  • Demonstrate integration of MOPS into LSST framework
  • Establish reusable code baseline for future DCs and LSST
  • Further develop and demonstrate common framework functionality
  • Update middleware implementation to enable application classes
  • Demonstrate a database partitioning scheme
  • Test the bandwidth utilization of the UDT protocol in the context of LSST data replication between Chile and the US
  • Pilot the development and testing process and tools for future DCs and LSST

1.2. Operations/Execution?

  • Plan for repeat runs of each part, document/analyze results with each run

1.3 Metrics?

1.4 Validation

We will have met our goals if:

  • We demonstrate a nightly processing pipeline that operates continuously on a set of input data. This processing includes at least image subtraction, source detection, association, and moving object detection using MOPS. As stretch goals, we would also include image linearization (flat-fielding, bias subtraction, etc.) and WCS determination.
  • We produce a persistant database of source detections and associations as a result of the nightly processing.
  • We execute MOPS automatically witin the nigthly pipeline.
  • We have checked in all original code used in DC2 into SVN.
  • Demonstrate the use of LSST framework and support classes
  • Employ in DC2 a harness that accesses MP functionality within a compiled layer but which can execute application code via a python interface.
  1. Application Components
    • Nightly Processing Pipeline
      • Image Processing – NS, AB, RO
        • Common Dependencies (fw classes): Image, MaskedImage, Exposure, Kernel, WCS (and Catalog?)
          • Calibration
            • Inputs: raw images, calibration images (darks, biases, flats), masks (if available)
            • Outputs: calibrated MaskedImages
          • Astrometry (WCS Determination)
            • Inputs: calibrated Images, object detection, object Catalog
            • Outputs: calibrated Exposures
          • Image Subtraction
            • Inputs: calibrated Exposures, template archive
            • Outputs: difference Exposures
      • Detection – RHL, NS, AB, RO
        • For object detection immediately after Calibration
          • Inputs: calibrated MaskedImages
          • Outputs: new Objects for object Catalog
        • For object detection on the difference images
          • Inputs: difference Exposures, object Catalog
          • Outputs: new Objects
      • Association - SM, MNS, SN
      • MOPS - FP, JM, LJ
        • 64-bit support
          • Milani Wrappers
          • Milani libmopsephem
          • Compile Auton Code
          • Compile Extra S/W
          • Install on mops64
        • Ephem Service
          • Day-Time Service
          • Night-Time Service
          • X-Match Radius
        • Integration
          • Association Pipeline Interface
          • Data Transfer Service
        • Test
    • Application Framework
      • LsstData classes (astronomy data, algorithms) - TSA, RHL
        • MaskedImage - TSA
          • GIL vs Vision Workbench
        • Exposure - NS
        • Catalog
        • Match
        • Kernel - RO
        • WCS - UW?
      • Supporting classes (common framework behaviors, interface to Middleware)
        • Evaluate STL, boost classes for use in framework - All
        • Citizen class providing smart pointer, memory management, debug logging trace - RHL
        • Flexible tracing facility - RHL
        • DataProperty class providing Name Value pairs and collections of same - JK, JAB
        • Exception handling, assertion use - RA, SP
        • Metadata (DataProperty implementation, persistence via serialized object files) - JPK, JAB
        • Provenance (DataProperty implementation, persistence via serialized object files) - JPK, JB, RP, JAB
        • Policies (DataProperty implementation, persistence via serialized object files, 3-level policies: Default, Periodic, and Current, Configuration vs Trace provenance) - JPK, RP, JAB
        • File Persistence (Simple objects to serialized object files, composite and collection objects iterating over simple objects, recursively) - JPK, RP, JAB
        • DBMS Persistence (Database ingest of data and metadata, DBMSStorage & object -relational adapters?, Table class) - JB, SM, RP
        • Formatters (fitsio, STIL formatters)
        • Events (Passed as XML a la OGRE, monitored/triggers a la DAGMAN, workflow control a la Chimera/Pegasus?, remote monitoring/logging a la GRAM, interface to DMCS/OCS) - SP
    • Third Party
      • wcslib
      • GIL or Vision Workbench
      • C++ std (including STL)
      • numpy? There's a ? here, but RHL thinks that it's clear that we want to use numpy.
      • python plotting (sm? matplotlib? All of the above?)
      • Boost (which parts?)
      • fitsio, fics?
      • STIL?
    • Database Schemata? - JB
  1. Middleware Components
    • Update MySQL and Ingest Services - JB, RO
    • Expand processing harness to support C++-based astronomical modules (Python/C++ interfaces, Gaia datatrain concept) - FP, GD, RHL
    • Logging (keep NetLogger?) - ???
    • Data Transport
      • DS on top of SRB/IRODS, http, Gridftp, Sector/UDT - CC, CW, SL, UIC
  1. Infrastructure Components
    • Teragrid. other cluster as Processing Platform - RP, NCSA?
    • Development Platforms - RP, NCSA?
      • Lincoln development cluster
    • Database Deployment plan
    • MOPS Deployment plan
  1. Input Data
    • CFHTLS raw images - TSA, RP
    • Simulated LSST raw images (DES, SkyMapper, Arcdata) - TSA, AJC, RP, CS, PP
    • LLNL Simulated sources - SN
    • Pan-STARRS Solar System Model generated sources - FP, LJ
    • Galactic simulation - KC, TSA
    • Extragalactic simulation - PP, TSA
  1. Development Process and Tools?
  1. Development Schedule