wiki:DC3bManagement
Last modified 8 years ago Last modified on 07/06/2011 05:57:28 PM

DATA CHALLENGE 3b MANAGEMENT

DC3b Scope

Overall Goals appear below on this page. Detailed pages for scope decisions, data requirements and data quality metrics, etc. are linked to DC3b Scoping Meeting III Virtual Meeting, July 27 - 28, 2009

These include:

DC3b Global goals

  • Improve middleware for continuous integration of applications into pipeline.
  • Upgrade middleware so that it can easily be used for algorithm development.
  • Scale database up in order to demonstrate ingest to 15% DR1 level for alert and data release productions.
  • Query DB which is running on multiple nodes transparently to user.
  • DC3b databases to include a combined database for pipeline outputs/SDQA and scalability tests.
  • Demonstrate technologies for file transfer (already demonstrated 15% rate in DC1).
  • Define a set of QA metrics we will meet and configure SDQA system to support those metrics.
  • Define pipeline validation metrics to encompass all stages and implement those metrics.
  • Implement deep detection and characterization using multifit with at least a simple galaxy/star model.
  • Integrate MOPS into LSST framework and run in that framework.
  • Improve alert production latency/throughput by x%. (x% should be set at a level that crosses a threshold on hardware from basic clusters to advanced technologies.)
  • General R&D goal of being knowledgable of advanced technologies for pipeline execution (at least GPUs). See the DC3b GPU development plan.
  • Demonstration of basic fault tolerance capability.

DC3b Data Quality Requirements

See also the main DM SDQA wiki page, where ongoing efforts are documented.

DC3b Plans

DC3b schedules are part of the overall DC3Schedule

DC3b Software Development Plans

The following plans will be updated for DC3b specifics.

DC3b User Support

DC3b Environment

TBD - Build environment, tools, platforms, etc.

DC3b Input Data

DC3b Processing Flow

DC3b Meetings

DC3b Telecons

DC3b Integration Stand-up Meetings

  • Starting the week of April 12th, 2010, we are having short daily meetings at 14:00 PDT. We are maintaining a status board for these meetings at DC3bPT1IntegrationStatus.

DC3b PT1 Execution and Benchmarking

DC3b PT1.1 aka 2 month plan

DC3b PT1.2

(previously known as "PT 1.5" as well as "6 month plan")

DC3b Reference Documents

Attachments