Last modified 10 years ago Last modified on 07/24/2009 05:45:58 PM

DC3b Scoping Meeting II, IPAC July 12, 2009

On Telecon line: Ray, David, + NCSA, Francesco, KT

In Room: Roc, Robert, Robert, Tim, Jeff, Gregory, Jacek, Dick Shaw, Vince, Deborah


Example of deliverable is DC2PlanOutline meeting Trac page. Need the DC3b Management version of this page. Use pages summarizing results linked from DC3APostMortemMeeting to assemble. DC3b plan due end of July. All known scope issues resolved by end of July.

DC3b Global goals

Improve middleware for continuous integration of applications into pipeline. Upgrade Middleware so that it can easily be used for algorithm development.

Scale database up in order to demonstrate ingest to 15% DR1 level for alert and data release productions. Query DB which is running on multiple nodes transparently to user. DC3b databases to include a combined database for pipeline outputs/SDQA and scalability tests, if possible. Need to evaluate hardware needed. Currently don’t have hardware to hit that spec. Demonstrate technologies for file transfer (already demonstrated 15% rate in DC1)

Define a set of QA metrics we will meet and configure SDQA system to support those metrics.

Define pipeline validation metrics to encompass all stages and implement those metrics.

Implement deep detection and characterization using multifit with at least a simple galaxy/star model.

Integrate MOPS into LSST framework and run in that framework. Any Alert Productions goals? Only as part of running in configuration for Data Release. We will not generate alerts (VOEvents) or look at characterization, false alert rates, etc. until DC4.

Improve alert production latency/throughput by x%. x% should be set at a level that crosses a threshold on hardware from basic clusters to advanced technologies.

General R&D goal of being knowledgable of advanced technologies for pipeline execution (at least GPUs).

Demonstration of basic fault tolerance capability. Assume we will have more than the NCSA LSST cluster available to us.

Disposition/Review? DC3a post mortem/DC3b meeting plenary/breakout results

Plenary 1 – progress on work (Tim, Robert). From trac page.


  • KT/Jacek to decide on approach to pre-sequence image. Part of data access wg discussion.
  • Implement cross talk module for ISR, as middleware needed is already planned. However, this is a candidate for descoping.
  • Don’t generate fringe fames and ghost images. This is work for image sim team and implementing some calibration products to generate and use them. Too much work and out of scope for DC3b.
  • Deblender not include within scope, but possible include as free work from Princeton. Treat same way as calsim, if anything useful, attempt to put it in, but not a requirement.
  • Deep detection process flow and visit processing flow. SAT to address this.
  • Handling multiple bands in deep detection. Assigned to Apps WG. SAT will review recommendation.
  • Star grids need for DC3b. Yes. Need to update input requirements document.
  • Tessellate the sky need in several places. Would like a single schema. Tim to drive selection process, proposal.

Rest of pages are assigned to working groups/teams

Assignments of DC3b scoping items to working groups/teams:

  • MOPS – Apps
  • Build/packaging – Middleware
  • Science Algorithms – Apps
  • SDQA – create as a peer working group. Actions bubble up to Apps WG.
  • Data Access – Data Access WG
  • Pipeline/Middleware? – Middleware
  • Metadata – define at SAT (too many different definitions currently)
  • Performance/scability/reliability – PMC

DC3b pacing items (due by early July)

  • Definition of input products
  • Change to middleware for continuous application testing
  • Availability of hardware
  • Statement of DC3b scope and corresponding changes to UML activity diagrams for productions.


  • Working Groups to finish resolving issues/actions on DC3b plenary/breakout pages in trac, so that scope decisions are complete. It is understood that new issues will arise during DC3b and will be addressed then. Any new issues should be added to the breakout pages and processed the same way. If any issues are not resolvable in WG, escalate to System Architecture Team (chaired by Gregory). Due July 10.
  • Tim update Plenary – Science I trac page with decisions from this meeting. Due June 29.
  • Jeff, Suzy, and David to create draft MS Project plan for DC3b. Due July 10. Final due by July 31.
  • Tim to create draft of DC3b Data Input requirements, Data Quality goals. Due June 22.
  • Tim to create draft of sky tessellation proposal. Due July 10.
  • Jacek, KT to create spec for architecture/hardware/constraints needed for DC3b databases, to include a combined database for pipeline outputs/SDQA and scalability tests, if possible. Due July 10.
  • Mike, Ray, Gregory to inventory available hardware/clusters at SLAC, NCSA, grid, etc. for DC3b uses, in anticipation of database needs and overall scope definition. Due July 10.
  • Gregory and Jeff to start working on open high-level scope items, e.g. What does 15% of DR1 mean, What improvement in Alert Production latency/throughput required, what fault-tolerance use cases will be required? Due June 29.
  • Gregory to document SAT process and schedule first meeting to examine overall production process flows, deep detection multiple band requirements, metadata definition. Due July 10.