Last modified 10 years ago Last modified on 06/15/2009 12:40:30 PM

Status Report for October 15, 2008

Next Month

Summary of progress

Work activity; whether it reached associated milestones.


Bill (submitted on 10/16/2008)

  • Finished deploying current software stack on virtual machines for distribution (38 - 100%)
  • Initial automated testing system: (43 - 90%)
    • The initial system is installed and running
    • Do I need to add CentOS 4 target(s)?
    • More work: the CCB has defined what "the trunk should always work" means, and it will be complicated to test. I would like to add a two-week 50% task (let's talk with Ray to get a good time estimate) to get it working properly.
    • Eventually, we'll also need to target major development branches, when they are created
  • Include V&V in automated testing: still waiting for V&V software (44 - 0%)
  • Pipeline orchestration: haven't started (121 - 0%)
  • Security document: edited for consistency; work on perimeter diagrams; still needs: (207 - 80%)
    1. some content and diagrams
    2. refinement and more consistency
    3. to be integrated & organized with rest of PDR docs

Chris (submitted on 10/20/2008)

• I attended the IEEE conference in Baltimore. I met with vendors from Blue Arc and Sun. We discussed their new products and their road map.
• I am currently updating the power and cooling for the Archive center, Base camp and DAC sites.
• I have changed the specs on the Infrastructure spreed-sheet to include 1TB drives for the med disk instead of 450GB disks.
• I am still working with Arun on the SC08 document. We have setup iRODS servers here at NCSA. We will have a tech call Monday to discuss the next step.

David (submitted on 10/21/2008)

  • Almost all my time in the last month went into standing the DC2/DMS pipeline up on the NCSA cluster. This has two purposes; the more immediate purpose is assisting Arun et al. in the SC08 project we are involved with, but this also involves debugging code that will be part of the DC3 pipeline, so there are DC3 benefits as well. The DC2/DMS stack on the NCSA cluster now runs the imageSubtraction sufficiently for SC08 purposes, but does not complete a run of the entire pipeline.
  • I also worked with Bill Baker on some build issues for the buildbot.

Greg (submitted on 10/21/2008)

  • Reworked the manner in which interSlice communication is enabled/scheduled in Pipeline Policy
  • Assisted D. Gehrig with getting DC2 pipelines running under DMS
  • Uncovered the issue that Boost.MPI is not currently building with the default Boost 1.36.0 of the build system. Organized notes on how Boost.MPI may be added to the Boost portion of the build.
  • Added Boost.MPI to my working LSST stack and started testing the using of Boost.MPI isend() and recv()'s to perform interSlice communication / share Clipboard data.

Steve (submitted on 10/17/2008)

  • Worked on version 1.0 of the event monitor, which is nearly complete. Met with Ray about this.
  • Working on the documentation and PowerPoint? for event monitor.
  • Assisted Francesco with transition his version of the DC2 tree to 64-bit.
  • Working on deploying a version of event monitor to Greg so we can test it in a "live" environment.

Ray (submitted on 10/20/2008)

General Comments
  • Much of my time this month was spent:
    • updating the infrastructure costing spreadsheets
    • preparing slides for the internal review (10/15-16)
    • catching up on NVO-related work
DC3 Tasks
Task 32: update software dist. server
Status: compete
Activity: Merged EUPS changes into trunk
Task 85: policy/data property integration
Status: not started
Task 120, 121: Orchestration Layer
Status: design discussions started with Pietrowicz and Daues
Other Tasks
  • worked with C. Cribbs and C. Beldica on update to Infrastructure costing
  • prepped for internal review.
  • assisted B. Baker with testing plaform issues
  • assisted D. Gehrig and A. Jagatheesan with getting DC2 pipelines running under DMS and assisting with SC08

LSSTC - Robyn (submitted on 10/16/2008)

Task    %Done   Task Title                                      
7	9%	SW Quality Assurance
8	13%	Develop DC3 Software Development Plan
9	0%	        DC3 SQA Plan
10	10%	        DC3 V&V Plan
25	100%	               Specify test requirements for ea phase of Product dev
29	100%	               Specify pass/fail metrics for each test type
11	20%	        DC3 CM Plan
12	50%	V & V: Acquire SQA support tools
13	50%	       Configurable tool to perform Lsst software standards checking
14	100%	               C++ LSST benchmark test suite packaged
	100%	                Benchmark potential tools
18	0%	               Acquire 
19	75%	        Coverage analysis tool 
20	100%	               Benchmark
23	0%	               Release for developer use
30	100%	V & V: Refine Ticket workflow to add more V & V testing. 
168	23%	non-DC3 activities
209	35%	LSST Corp
210	90%	Software Quality Assurance: Planning
211	90%	        Prepare LSST DM Software Development Plan
212	90%	                Software Quality Assurance Guideline
213	90%	                Verification and Validation Guideline
215	50%	                         Define process to track Requirements to Tested Release
218	100%	                         Specify test requirements for ea development phase 
214	90%	                Configuration Management Guideline
216	80%	                          Develop std labelling for products and provenance

Database Group (submitted on xx/xx/2008)


  • task 45 (build templates): Ticket #334, waiting on task 33 (build system) [RHL]
  • task 65 (DC2 policy capture): Ticket #315, stalled at 75%
  • task 66 (DC3 provenance capture): design underway
  • task 85 (Policy/DataProperty? integration): interface proposed
  • task 86 (exception improvements): not yet started
  • task 88 (shared pointer improvements): Serge has done most of this
  • task 89 (SDQA architecture support): some suggestions made
  • task 116 (fault tolerance): Done!
  • task 129 (association improvements): no actions needed yet
  • task 143 (image metadata): proposal tentatively approved by CCB
  • task 146 (image retrieval): design continues, test case writing and coding started
  • task 182 (SciDB steering): ongoing
  • task 183 (SciDB design/impl): started
  • task 189 (Bigtable schema): not yet started
  • Prepared for and held XLDB2 workshop; working on report.
  • Completed fault tolerance document.
  • Wrote up document on (extensive) discussion of low SNR Source reduction; discussed "Narrow Source Table" proposal.
  • Redesigned DataProperty, now PropertySet.
  • Reworked DC3 middleware use cases in EA.
  • Discussed image data train and image retrieval with Martin of UC Davis.
  • Discussed auxiliary telescope and calibration data schema and sizes.
  • Discussed WCS generation using astrometric standards.
  • Commented on objectId synchronization during data release processing.
  • Advised Serge on how to handle MySQL database setup for DC3 runs.
  • Evaluated SkyPixelization.
  • Commented on open source license selection.
  • Helped edit job description for new hire at SLAC.
  • Met with Cloudera.

SciDB items:

  • Presented SciDB at NSF, quite successfully.
  • Attended SciDB all-hands design meeting.
  • Read documents for SciDB API task force.
  • Set up Trac/Wiki?.
  • Edited CIDR paper.

Maria, Ani

  • nothing to report

Andy H

  • waiting for Jacek: need queries and sim data, then will continue with GridSQL tests


  • preliminary performance tests for DC3 database setup
  • attended XLDB
  • implemented support for new SWIG shared_ptr functionality in middleware packages (Trac ticket #374)
  • discussed DataProperty redesign
  • Helped Andy Becker with SWIG problems in deep detection code:


  • SciDB presentation for the LSST Board of Directors (prep, travel)
  • hiring: job requisition, several interviews
  • internal PDR review (prep, travel)
  • added storage estimates for "Narrow Source" table


  • preparation for xldb workshop
  • xldb workshop (2 full days)
  • scidb design meeting (1 day)
  • started writing xldb report
  • helped with SciDB paper for CIDR conference
  • organizing science advisory board
  • involved in writing DataNet NSF proposal, organizing SLAC PI for this proposal
  • article for SLAC Today:
  • meeting with Cloudera, Inc.

Major Accomplishments

Significant breakthroughs, issues resolved.

  • XLDB2 workshop organized

Objectives for the Next Period

What you expect to accomplish.


  • Bill:
    • Bring security doc closer to PDR readiness
    • Integrate V&V if it is ready
    • Continue to support VMs for development
    • Start work on automated testing of trunk according to guidance from CCB
  • Chris:
  • David
    • Finish standing up DC2/DMS stack
    • Work with Arun as necessary for SC08
  • Steve:
    • Misc bug fixes for anything Greg and I find for Event Monitor
    • Finalize event monitor 1.0 code base and documentation.
    • Meet with Ray about orchestration layer work.
  • Greg:
    • Debug and solve any problems with Boost.MPI use for the interSlice communication so that sharing of general DataProperty?'s on the Clipboard proceeds seamlessly.
    • Contribute to orchestration layer design and initial work.
    • Finish Local Logging ticket by ensuring access to the Local Log from Application Stages
  • Ray:
    • Attend IVOA Interop (10/26-31) and ADASS (11/3-5)
    • Complete orchestration layer design with Steve and Greg


  • Robyn:
    • setup a meeting of SDP-impacted parties for SDP review and revision (task 211).
    • continue fleshing out the DC3 V&V and CM plans (task 8)
    • bring the UnitTesting? Guidelines document to the CCB (task 23)

Database Group

  • DC3 tasks as planned
  • finish xldb2 report
  • submit DataNet proposal to NSF (main PI: Mike Stonebraker)
  • ADASS conference (prep talk, attending)

Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues

  • Robyn:
    • Over the next month, I have very limited time to work on acquiring, integrating and releasing a Coding Standards Checker. I will initiate conversations with Programming Research and Parasoft concerning licensing costs. (task 18)
  • David:
    • One of the communication switches associated with the cluster fried itself in early October, meaning several of the blades on the cluster were out of service for three days.