wiki:DC3Management

Version 38 (modified by dgehrig, 10 years ago) (diff)

--

Data Challenge 3 Management

from: LSST Wiki Table of Contents.

Data Challenge 3 will focus primarily on the processing at the Archive Center that generates a Data Release.

This page collects documents related to the management and execution of Data Challenge 3.

DC3 Startup Tasks

DC3 Planning

Development in DC3

  1. Developing with Virtual Machines
  2. Using the Pipeline Harness
  3. Configuring Predefined Stage Classes
  4. Trac Ticket Workflow tuned for Data Challenge 3
  5. DC3 Software Tour

Pre- DC3a Integration Week

  • Developers:
    • resolve all your DC3a delivery Tickets; merge Ticket branches onto trunk; tag packages
  • Package Gurus whose package implements either a single stage or a pipeline with multiple stages:
    • test your package using either
      • the pipeline harness running single slice on a single node thru all stages; or
      • a script to advance the pipeline appropriately thru its stages.
    • load your pipeline policies into both
      • your local package: $SVN/DMS/<your package>/pipeline/ ; and
      • the DC3 policy repository: $SVN/DMS/ctrl/dc3pipe/pipeline/ .

DC3a Benchmarking

  • TBD Ray: Setting up the Run directory (out-of-date ref SetUpRun by DavidG)
  • Running the Pipeline Harness (current by GregD)
  • TBD DavidG: Accessing the LSST Cluster (out-of-date ref LSSTCluster by DavidG)
  • TBD Ray: How to Run the Complete Set of DC3a Pipelines through ORCA (out-of-date ref DC2PipeHowto)
  • TBD DC3a Test Data
    • Exposures and ancillary Calibration Artifacts (ref DC3TestData by DavidG)
    • Catalogs (might be single catalog)
      • catalog needed for WCS calculation (delivery: Fergal; DB setup policy: KT?)
      • catalog needed for photometric zeropoint calculation (delivery: SteveB?; DB setup policy: KT?)
    • NightMOPS proprietary artifacts located in lsst9:/lsst/DC3/opt. See directory's README (by Francesco)

Meetings

Reference Documents

DC3 Software at ROE