Version 24 (modified by bbaker, 11 years ago) (diff)


Status Report for November 15, 2008

Previous Month Next Month

Summary of progress

Work activity; whether it reached associated milestones.


Bill (submitted on 11/14/2008)

Chris (submitted on 11/xx/2008)

David (submitted on 11/21/2008)

  • Continuing the process of setting up the pipelines on the NCSA cluster
  • Set up an SVN/Trac site for the LSST science book, and created the necessary accounts
  • Setup SC08 account and pipeline on NCSA cluster for Arun
  • Worked with Robyn on pdf export from LSST Trac for Code Standards document
  • On 11/18 the DC3 team held a telecon coordinating and updating the current schedules for the DC3a workgroups and verifying the division between DC3a and DC3b tasks. We have scheduled integration on the NCSA cluster to begin around January 15, and have informally decided on a face-to-face meeting similar to last year's "integration week", to be held the last week of January.

Greg (submitted on 11/24/2008)

  • Created ticket 423 branch of daf_persistence to build in support for boost MPI packed_oarchive/packed_iarchive Archive formats. Committed the necessary changes; the merge to the trunk awaits Boost MPI's addition to the standard build.
  • Within pex_harness (ticket 343 branch) coded up the first case of communicating DataProperty's between Slices using Boost.MPI. Succeeded in communicating DataProperty's using nonblocking isend/irecv from a single Slice to multiple neighbors.
  • Set up a test LSST software stack on the NCSA Abe Cluster in order to 1) debug portability issues with the software stack and 2) assist in the development of Orchestration layer design, via, for example, testing alternative Orchestration layer "adaptors" for submitting and monitoring jobs, file transfer, etc.

Steve (submitted on 11/19/2008)

Event Monitor:

  • Implemented MySQL reader/writer for Event Monitor for Log analysis
  • Fault system and log analysis configuration pushed back to deadline of 1/15/09

Orchestration Layer:

  • Working on preliminary redesign of orchestration layer using DC2 launchDC2 as a basis
  • Implemented policy file specified objects for subtasks in orchestration layer. This proof of concept design is working well so far.

Ray (submitted on 11/20/2008)

  • On Travel (10/27 - 11/7)
  • Met with lava community folks to discuss shared software opportunities
  • Worked on release packages in support of #471
    • dealt with various bugs in supporting packages
    • supported users in their upgrades (see UpdateCorePkgsNov2008)
  • DC3 management: worked on DC3 schedule refresh
Task %Done Task Title Comments
117 80% orchestration layer design Worked with Steve, Greg; should by done by EOM


Robyn (submitted on 11/10/2008)

Task %Done Task Title
7 SW Quality Assurance
8 Develop DC3 Software Development Plan
9 0% DC3 SQA Plan
10 10% DC3 V&V Plan
25 100% Specify test requirements for ea phase of Product dev
29 100% Specify pass/fail metrics for each test type
11 50% DC3 CM Plan
12 V & V: Acquire SQA support tools
13 Configurable tool to perform Lsst software standards checking
14 100% C++ LSST benchmark test suite packaged
100% Benchmark potential tools
18 0% Acquire
19 Coverage analysis tool
20 100% Benchmark
23 0% Release for developer use
30 100% V & V: Refine Ticket workflow to add more V & V testing.
168 non-DC3 activities
209 LSST Corp
210 Software Quality Assurance: Planning
211 Prepare LSST DM Software Development Plan
212 90% Software Quality Assurance Guideline
213 90% Verification and Validation Guideline
215 90% * Define process to track Requirements to Tested Release
218 100% Specify test requirements for ea development phase
214 90% * Configuration Management Guideline
216 90% Develop std labelling for products and provenance

Database Group (submitted on 11/13/2008)


  • UID 45 (build templates): Ticket #334, waiting on UID 33 (build system) [RHL]
  • UID 65 (DC2 policy capture): Ticket #315, stalled at 75%
  • UID 66 (DC3 provenance capture): design underway
  • UID 85 (Policy/DataProperty? integration): implementation 20%
  • UID 86 (exception improvements): design done, implementation 80%
  • UID 88 (shared pointer improvements): done by Serge
  • UID 89 (SDQA architecture support): more suggestions made
  • UID 129 (association improvements): looks like DC3B now
  • UID 143 (image metadata): proposal tentatively approved by CCB
  • UID 146 (image retrieval): design continues
  • UID 182 (SciDB steering): ongoing
  • UID 183 (SciDB design/impl): started
  • UID 189 (Bigtable schema): not yet started
  • Data Management internal review.
  • Helped complete XLDB2 workshop report.
  • Wrote script to translate from EA XML to SQL DML to load metadata tables.
  • Redesigned Exception class and nearly finished implementation.
  • Completed design of new PropertySet class to replace DataProperty.
  • Arranged telecon with Wil O'Mullane about data trains.
  • Fixed tickets #389 and #396 (getMetadata segfault).
  • Fixed ticket #371 (DateTime shouldn't hardcode leap seconds).
  • Gave Greg solution for problems serializing DataProperty to boost::mpi.
  • Debugged problem with missing Formatter in image subtraction for SC'08.
  • Pointed out needed modifications in IPD timeline.
  • Tried and failed to prepare poster on template storage for ADASS.
  • Added definitions to glossary Trac page; discussed other disambiguations.
  • Opened several tickets to request inter-slice communication use cases and discussed with ticket owners.
  • Discussed astrometric references and finding very high proper motion stars.
  • Discussed configuration information for rafts and the focal plane.
  • Discussed SDQA schema and metrics.
  • Discussed official documents policy including effect on Trac; wrote Trac page footer.
  • Discussed use/non-use of Numerical Recipes.
  • Discussed Camera/DM ICD background and Exception design with GPDF.
  • Discussed template storage with Russell.

SciDB items:

  • Edited NSF proposal.
  • Prepared requirements for SciDB language bindings.
  • Several weeks of reading and discussion on SciDB language bindings.
  • Phone-screened Daniel Wong.
  • Installed Mailman and created mailing lists.
  • Posted several documents to SciDB Trac.
  • Prepared slide for SC'08.

Maria, Ani

  • Ani attended 2 DataAccWG telecons

Note: Maria and Ani are further reducing their involvement. They will be attending the telecons only when invited / discussed topics requires their presence

Andy H

  • Away most of this month
  • We now have queries and input data Andy was waiting for last time, so he will resume large scale tests later this month.


  • discussion/review of MOPS schema merge (#406)
  • discussion of exception redesign proposal
  • looked over and cleaned up SWIG of middleware packages and afw new-image-api
  • fixes so that the new-image-api branch of afw compiles on Linux64 and with an up to date version of gcc
  • further work on ticket #374 (SWIG shared pointer enhancements)
    • found and investigated seg-fault arising when wrapping classes using multiple inheritance
    • proposed SWIG patch which handles this issue without ugly user-side workarounds


  • UIDs
  • work on schema
    • checked all DC3 usecases for implications on schema/database
    • discussions with Tim
    • researched tagged values in EA (to be used for units and ucds in schema)
    • #406 merged mops schema with baseline schema in EA
    • #409 Reviewed SDQA schema proposed by Russ L., added to EA
    • #402 - Added centroids for photometry and astrometry to database schema
    • #404 - Added to schema Gray_Surf, !NonGray_Extinction, DIA_Poly
    • #300 - cleaned up cx,cy,cz in schema
    • #405 - researched micro-sec support in schema
    • added 1st moments to schema
  • work on tickets
    • all database work for DC3a, DC3b and major tasks beyond DC3 revisited and tickets opened for each task (30+ tickets opened)
    • milestones, components, owners etc revisited
    • prepared slides, attended the conf, gave talk, side meetings
  • #443 -implemented schema browser, including integrating with new table metadata tables
  • statement of work for SLAC for FY'09 and '10
  • hiring
    • reviewed paperwork for ~7 applicants, checked references for some, scheduled interviews with 2, interviewed 1 so far
  • met with Andy Lawrence
  • telecons
    • attended many telecons (pmc, dc3, middleware, app)
    • organized 5 DataAccWG telecons
  • XLDB
    • report from the 2nd workshop written (with KT)
  • SciDB
    • work on the DataNet NSF proposal
    • connected with ncar
    • found connections with National Geospatial Intelligence Agency
  • Others
    • meetings related to initiating SLAC-PNNL collaboration on data intensive computing
    • planning to attend Scientific Challenges for Understanding the Quantum Universe and the Role of Computing at Extreme Scale

Major Accomplishments

Significant breakthroughs, issues resolved.

Objectives for the Next Period

What you expect to accomplish.


  • Bill:
    • Security doc preparation for PDR
    • Nightly build of packages in trunk, by TCT standards (currently only against release)
  • Chris:
  • David:
    • must complete standing up pipelines on NCSA cluster
  • Steve:
    • Meet with Greg and Ray to get further requirements for DC3 orchestration layer (ABE and UW clusters in particular)
    • Rework Events for the upcoming change from Data Properties to Data Sets
    • Implement event system failover
  • Greg:
    • Work on design and implementation for supporting multiple InterSlice? communication modes: send/recv between Slices, Bcast from Pipeline to Slices, Scatter from Pipeline to Slices, etc.
    • Finish Local Logging issue: make the "local file enabled Log" globally accessible through default log mechanism.
    • Examine incoming application use cases for interslice communication and ensure support for these cases.

  • Ray:


  • Robyn:
    • continue fleshing out the DC3 V&V and CM plans (task 8)

Database Group

Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues