= DC2 Development and Execution Plan Outline = For draft paragraph descriptions, follow the links for each section heading. 1. General Description 1.1. [wiki:DC2PlanGoals Specific Goals] * Demonstrate use of astronomical algorithms for nightly processing * Establish an LSST simulated object/source database for testing by broader collaboration * Demonstrate integration of MOPS into LSST framework * Establish reusable code baseline for future DCs and LSST * Further develop and demonstrate common framework functionality * Update middleware implementation to enable application classes * Demonstrate a database partitioning scheme * Test the bandwidth utilization of the UDT protocol in the context of LSST data replication between Chile and the US * Pilot the development and testing process and tools for future DCs and LSST 1.2. [wiki:DC2PlanOperations Operations/Execution] * Multiple parts (each is separate test/execution environment, there is not one fully integrated DC execution) * [wiki:DC2DataTransfer Data Transport] * [wiki:DC2NightlyProcessingUsecase Nightly Processing pipeline] * [wiki:dbNightlyProcDC2 Data Ingest] * Plan for repeat runs of each part, document/analyze results with each run 1.3 [wiki:DC2PlanMetrics Metrics] * Scalabiity metrics (same as DC1 but higher) * Application/Data Quality (TBD) * [wiki:DC2ImageProcMetrics Image Processing Pipeline Metrics] 1.4 Validation We will have met our goals if: * We demonstrate a nightly processing pipeline that operates continuously on a set of input data. This processing includes at least image subtraction, source detection, association, and moving object detection using MOPS. As stretch goals, we would also include image linearization (flat-fielding, bias subtraction, etc.) and WCS determination. * We produce a persistant database of source detections and associations as a result of the nightly processing. * We execute MOPS automatically witin the nigthly pipeline. * We have checked in all original code used in DC2 into SVN. * Demonstrate the use of LSST framework and support classes * Employ in DC2 a harness that accesses MP functionality within a compiled layer but which can execute application code via a python interface. 2. [wiki:DC2PlanAppComp Application Components] * Nightly Processing Pipeline * Image Processing – NS, AB, RO * Common Dependencies (fw classes): Image, MaskedImage, Exposure, Kernel, WCS (and Catalog?) * Calibration * Inputs: raw images, calibration images (darks, biases, flats), masks (if available) * Outputs: calibrated !MaskedImages * Astrometry (WCS Determination) * Inputs: calibrated Images, object detection, object Catalog * Outputs: calibrated Exposures * Image Subtraction * Inputs: calibrated Exposures, template archive * Outputs: difference Exposures * Detection – RHL, NS, AB, RO * For object detection immediately after Calibration * Inputs: calibrated !MaskedImages * Outputs: new Objects for object Catalog * For object detection on the difference images * Inputs: difference Exposures, object Catalog * Outputs: new Objects * Association - SM, MNS, SN * MOPS - FP, JM, LJ * 64-bit support * Milani Wrappers * Milani libmopsephem * Compile Auton Code * Compile Extra S/W * Install on mops64 * Ephem Service * Day-Time Service * Night-Time Service * X-Match Radius * Integration * Association Pipeline Interface * Data Transfer Service * Test * Application Framework * !LsstData classes (astronomy data, algorithms) - TSA, RHL * !MaskedImage - TSA * GIL vs Vision Workbench * Exposure - NS * Catalog * Match * Kernel - RO * WCS - UW? * Supporting classes (common framework behaviors, interface to Middleware) * Evaluate STL, boost classes for use in framework - All * Citizen class providing smart pointer, memory management, debug logging trace - RHL * Flexible tracing facility - RHL * !DataProperty class providing Name Value pairs and collections of same - JK, JAB * Exception handling, assertion use - RA, SP * Metadata (!DataProperty implementation, persistence via serialized object files) - JPK, JAB * Provenance (!DataProperty implementation, persistence via serialized object files) - JPK, JB, RP, JAB * [PolicyDesign Policies] (!DataProperty implementation, persistence via serialized object files, 3-level policies: Default, Periodic, and Current, Configuration vs Trace provenance) - JPK, RP, JAB * File Persistence (Simple objects to serialized object files, composite and collection objects iterating over simple objects, recursively) - JPK, RP, JAB * DBMS Persistence (Database ingest of data and metadata, [wiki:DBMSStorage DBMSStorage & object -relational adapters], Table class) - JB, SM, RP * Formatters (fitsio, STIL formatters) * Events (Passed as XML a la OGRE, monitored/triggers a la DAGMAN, workflow control a la Chimera/Pegasus, remote monitoring/logging a la GRAM, interface to DMCS/OCS) - SP * [wiki:ToolSets Third Party] * wcslib * GIL or [wiki:VW Vision Workbench] * C++ std (including STL) * [wiki:numpy numpy]? There's a ? here, but RHL thinks that it's clear that we want to use numpy. * python plotting (sm? matplotlib? All of the above?) * [wiki:Boost Boost] (which parts?) * fitsio, fics? * STIL? * [wiki:dbSchema Database Schemata] - JB 3. [wiki:DC2PlanMiddleComp Middleware Components] * Update MySQL and [wiki:dbCIS Ingest Services] - JB, RO * [wiki:DC2DbPartitioningTests partitioning schemes] * ingest methods * [wiki:DBMSStorage adapters] * Expand processing harness to support C++-based astronomical modules (Python/C++ interfaces, Gaia datatrain concept) - FP, GD, RHL * Logging (keep !NetLogger?) - ??? * Data Transport * DS on top of SRB/IRODS, http, Gridftp, Sector/UDT - CC, CW, SL, UIC 4. [wiki:DC2PlanInfraComp Infrastructure Components] * Teragrid. other cluster as Processing Platform - RP, NCSA? * Development Platforms - RP, NCSA? * Lincoln development cluster * Database Deployment plan * MOPS Deployment plan 5. [wiki:DC2PlanInputData Input Data] * CFHTLS raw images - TSA, RP * [wiki:C2PlanInputDataTemplateRecipe Recipe for Template creation] * Simulated LSST raw images (DES, !SkyMapper, Arcdata) - TSA, AJC, RP, CS, PP * LLNL Simulated sources - SN * Pan-STARRS Solar System Model generated sources - FP, LJ * Galactic simulation - KC, TSA * Extragalactic simulation - PP, TSA 6. [wiki:DC2PlanDevProc Development Process and Tools] * [wiki:BuildSystem Build/deploy system] ([wiki:Eups EUPS], [wiki:SCons SCons]) - RHL, RP * Code management (SVN) - RA, RP * [wiki:CodeStandards Coding standards] ([wiki:C++CodeStandards C++], [wiki:PythonCodeStandards Python]) - RHL, AJC, JK, TSA, RA, RO * [wiki:CodeReviews Review process] - AJC, JK, TSA * UML model and code round-tripping - JK, TSA * Test management - TBD??? 7. [wiki:DC2PlanDevSched Development Schedule] * 20070101 - 20070228 [milestone:"DC2 Planning and Scoping" Planning and Scoping] * Create this document * Establish Development Process and Tools * Expand Nightly Processing use cases and do robustness * Develop patterns and class designs for Application Framework * Do library evaluations (STL, boost, GIL, Vision Workbench, STIL) * 20070201 - 20070331 - [milestone:"DC2 Initial Class Implementation" Develop initial implementation of LsstData classes and Support classes] * ... * 20070901 - 20071101 - [milestone:"DC2 End" Execute DC2 runs, analyze and repor