wiki:DC3Management

Version 49 (modified by jkantor, 10 years ago) (diff)

--

Data Challenge 3 Management

from: LSST Wiki Table of Contents.

Data Challenge 3 is being executed in two parts: Data Challenge 3a focused on improvements to the Application Framework, Middleware,and Alert Production, and implementation of a prototype SDQA system. Data Challenge 3b will focus primarily on the Data Release Production and improvements to the Alert Production algorithms and the SDQA system.

This page collects documents related to the management and execution of Data Challenge 3.

DC3 Startup Tasks

DC3 Planning

Development in DC3

  1. Developing with Virtual Machines
  2. Using the Pipeline Harness
  3. Configuring Predefined Stage Classes
  4. Example of A Simple Stage Tester
  5. Trac Ticket Workflow tuned for Data Challenge 3
  6. DC3 Software Tour

DATA CHALLENGE 3a

Pre- DC3a Integration Week

  • Developers:
    • resolve all your DC3a delivery Tickets; merge Ticket branches onto trunk; tag packages
  • Package Gurus whose package implements either a single stage or a pipeline with multiple stages:
    • test your package using either
      • the pipeline harness running single slice on a single node thru all stages; or
      • a script to advance the pipeline appropriately thru its stages.
    • load your pipeline policies into both
      • your local package: $SVN/DMS/<your package>/pipeline/ ; and
      • the DC3 policy repository: $SVN/DMS/ctrl/dc3pipe/pipeline/ .

DC3a Execution and Benchmarking

  • DC3aProductionsRuns
  • TBD Ray: Setting up the Run directory (out-of-date ref SetUpRun by DavidG)
  • Running the Pipeline Harness (current by GregD)
  • TBD DavidG: Accessing the LSST Cluster (out-of-date ref LSSTCluster by DavidG)
  • How to Run the Complete Set of DC3a Pipelines through ORCA ( DC3PipeHowto)
  • DC3a Test Data
    • Exposures and ancillary Calibration Artifacts (ref DC3TestData by DavidG)
      • CFHT (delivery: AndyB; installation: DavidG)
      • simulated LSST exposures (delivery: AndyC; installation: DavidG)
      • Access to images, templates, and calibration artifacts (KT)
    • Catalogs
      • catalog needed for WCS calculation, also known as Astrometry.net index tree (delivery: Fergal)
      • catalog needed for photometric zeropoint calculation (delivery: SteveB?; format and access unknown)
    • NightMOPS proprietary artifacts located in lsst9:/lsst/DC3/opt. See directory's README (by Francesco)

DC3a Report

  • Report being developed within $DMS/report/reportDC3a
  • Refer to DC3AReportOutline for your writing assignments.

DC3a Meetings

Reference Documents

DC3a Software at ROE

DATA CHALLENGE 3b

DC3b Meetings