Last modified 10 years ago Last modified on 06/15/2009 12:37:32 PM

Status Report for April 15, 2009

Previous Month Next Month

Summary of progress

Work activity; whether it reached associated milestones.


Bill (submitted on 4/20/2009)

From MS Project Waterfall
description start end slice completion comments
27/569 Maintain automated testing structure 12/1/08 6/16/09 10% (ongoing) All my systems are functioning perfectly, Dave
28/41 (BB) support distributable VMs for new developers 8/11/08 7/9/09 5% 50% Waiting for Ray to install stack on latest VMs (I need an hour warning or so beforehand, to start the VMs from a clean state)
34/517 test that the trunk always works 3/9/09 3/20/09 50% 100% Currently functioning
35/74 configure test sytem for automated V&V 9/4/08 2/4/09 45% 50% Robyn and I are both working on it; me mostly the sysadmin and software install stuff; she the actual test configuration
Unlisted Assignments
  • Working on simple version of system to map data to slice workers in pex_harness.
  • Still trying to finish security document -- plan to give finished draft to Jeff before all-hands

David (submitted on 4/xx/2009)

2/194: DC3 coordination Start: 5/1/08; End 3/20/09; Assigned Gehrig David[20%] % Done: 85%

3/467: Infrastructure maintenance Start: 7/14/08; End 3/20/09; Assigned Gehrig David[20%] % Done: 80%

20/39: update software distribution server Start: 6/2/08; End 1/30/09; Assigned Gehrig David[61%],Plante Ray[50%] % Done: 88%

22/60: convert to new package build mechanism (remove dependency on pacman) Start: 6/16/08; End 5/5/09; Assigned Gehrig David[80%] % Done: 90%

39/46: build warnings if coding standards check fails Start: 7/14/08; End 5/7/09; Assigned Gehrig David[80%] % Done: 10%

40/47: build warnings if unittest doesn't satisfy DC3 coverage requirement Start: 7/16/08; End 5/12/09; Assigned Gehrig David[80%] % Done: 10%

42/49: build failure if unittest doesn't satisfy DC3 coverage requirement Start: 7/21/08; End 5/13/09; Assigned Gehrig David[80%] % Done: 10%

43/50: build failure if coding standards check fails Start: 7/23/08; End 5/18/09; Assigned Gehrig David[80%] % Done: 10%

45/52: any failures cause automatic 'Failed Nightly Build' ticket at high priority Start: 7/28/08; End 5/25/09; Assigned Gehrig David[80%] % Done: 10%

112/573: Prepare and stage test data Start: 3/23/09; End 3/23/09; Assigned Gehrig David % Done: 0%

120/579: Stage test data to software stack Start: 3/12/09; End 3/12/09; Assigned Gehrig David % Done: 0%

123/583: Stage data to Abe, UW Start: 3/16/09; End 3/17/09; Assigned Gehrig David % Done: 0%

Greg (submitted on 4/28/2009)

91/99: identify neighbors of a Slice: known in advance, or dynamic? Start: 5/27/08; End 10/15/09; Assigned Daues Greg[50%] % Done: 2%

95/106: Work to support communication of general serializable types over MPI Start: 11/20/08; End 2/5/09; Assigned Daues Greg[50%] % Done: 75%

96/107: Develop mechanisms for identifying/accessing Start: 2/5/09; End 3/5/09; Assigned Daues Greg[50%] % Done: 0%

  • Attended NCSA Accelerator Conference on application of GPUs and Cell technology based systems to the areas of the geosciences, chemistry and astronomy.
  • Minor pex_harness modifications and testing of examples in support of DC3a 'production' runs
  • Worked on installation and testing of orchestration infrastructure on an LSST cluster node to enable the launching of DC3a pipelines to other collaboration clusters (NCSA Abe, UW). This includes Globus and Condor installation and testing. This level of software will operate beneath, in support of, the ctrl_orca package.

Steve (submitted on 4/16/2009)

Start: 5/27/08; End 5/4/09; Assigned Pietrowicz Steve[25%] % Done: 94%

104/112: implement an orchestration layer Start: 3/23/09; End 4/9/09; Assigned Pietrowicz Steve[50%] % Done: 100%

105/536: migrate DC2 functionality Start: 3/23/09; End 4/9/09; Assigned Pietrowicz Steve % Done: 100%

106/229: ""life-support"" services Start: 4/10/09; End 7/24/09; Assigned Pietrowicz Steve % Done: 0%

119/582: Stand up DC3 services Start: 3/12/09; End 3/13/09; Assigned Pietrowicz Steve,Becla Jacek % Done: 0%

124/537: Add Abe, UW support for orchestration layer Start: 3/18/09; End 3/24/09; Assigned Pietrowicz Steve[25%],Daues Greg[75%] % Done: 0%

  • Additional post integration meeting changes to orchestration layer
  • Working on setting up event monitor scripts for event logging, and DC3 timing w/Ray
  • Fixed bugs within Orca
  • Additional objects added to the Event Monitor, and adding better error messages/handling

Ray (submitted on 3/18/2009)

  • Running integrated DC3a pipelines
  • Developed to launch pipelines via ctrl_orca and send data events (#740). This also involved:
    • overhauling logging in pex_harness and ctrl_orca to provide uniform logging for event monitor
  • Worked with Steve on integration of the event monitor to calculate performance measurements
  • Worked with Freemon and Cribbs on hardware infrastructure reference design
Task %Done* Task Title Comments
28/39 90% update distribution server little bit of clean up left
103/111 96% fill out the design of the orchestration layer...
111/572 100% Stand up DC2 pipeline under DMS on cluster
113/574 25% Perform Timing Runs
117/576 100% Install software stack on cluster
118/580 100% Stand up DC3 Pipelines Some tweaking in progress as expected.
122/577 10% Install software stack on Abe, UW Some planning and testing underway
126/586 25% DC3 Runs on NCSA Cluster In progress
127/587 0% DC3 Runs on UW Cluster
128/588 0% DC3 Runs on Abe


Robyn (submitted on 4/16/2009)

Status NL = Not Lead NW = No work done
Id PM = Project Management Plan DC = Data Challenge Plan AP = Applications Plan MW = Middleware Plan

NL NW PM50 ...Update SysML/UML models and prepare Technical Documents
98% PM51 ......Complete LSST Software Development Plan Waiting on final review
NL NW PM60 ...Complete DM Inputs to System Level Requirements and Design
NL NW PM67 ......Create document templates, recipes, tools for Enterprise Architect output/genas needed, prepared various EA Report templates
NL NW PM68 ...Complete Data Management Reference Design
70% PM69 ......Clean up DC3 UML Model, Update LSST Reference Design UML Model Merged old DC3 EA Models into restructured Model.
70% PM70 ......Update EA and UML Modeling Conventions Document for Activity Diagrams Wording OK'ed by UML Gurus
PM75 ...Create draft DM Integration Test Plan
10% PM77 ......Generate test cases Exercising prototype TestGen using DC3 model
NL NW PM86 Complete DM Inputs to Risk Management, System Integration, Commissioning, and Operations Plans
0% PM87 ...Update DMS Integration and Test Plan
0% PM90 ...Contribute DM sections to LSST Risk Management, Commissioning, I&T, and Operations Plans
DC56 Data Challenge 3 Software Development Plan
50% DC58 ...DC3 Configuration Management Plan no more on DC3a, will transition to DC3b
DC60 ...DC3 Verification and Validation Plan
75% ......DC3a Verification and Validation Plan Continued refinement: updated DC3a TestCases
0% ......DC3b Verification and Validation Plan

Becoming familiar with Parasoft C++ Coding Standard Checker which will probably be used for DC3b.

Database Group

K-T (submitted on 4/20/2009)

55/161: DC2 Policy capture Start: 7/23/08; End 3/30/09; Assigned Lim KT[10%] % Done: 100%

56/469: DC3 provenance capture [at DC3a level] Start: 9/8/08; End 7/10/09; Assigned Lim KT[10%] % Done: 100%

61/166: support persistence and retrieval of nightly pipeline products and inputs [i.e. Functions, Kernels, PSFs, image metadata, various image types] Start: 9/15/08; End 2/10/09; Assigned Lim KT[20%] % Done: 100%

71/531: Full integration of PropertySet? into packages Start: 9/1/08; End 4/27/09; Assigned Lim KT[20%],Plante Ray[10%] % Done: 100%

73/534: Full integration of exception changes Start: 1/29/09; End 2/19/09; Assigned Lim KT[20%] % Done: 100%

74/180: persistence framework refinement Start: 2/19/09; End 3/5/09; Assigned Lim KT[20%] % Done: 0% [postponed] 2008-03-24 to 2009-04-20:

44/43 (build templates): Ticket #334, stalled at 25%

Major accomplishments:

  • DC3a integration, including SDQA output and pipeline for simulated data.
    • Lots of minor fixes too numerous to list here.
    • #667: Properly persist pcaPsf objects via PSF base class pointer.
    • #671: Implemented TAN-SIP input from FITS.
  • Post-DC3a cleanups not yet merged to trunk:
    • Fix bad XML tag name in KernelFormatter.
    • #472: Update formatters and add CcdMetadataStage.
    • #745: Started thinking about persistence for self-registered objects.
  • Extracted list of "after-DC3a" discussion topics from chat logs.
  • #482: Implemented time conversion database stored functions.
  • Implemented generic SQL script execution from Python using DbStorage.
  • #706: Implement DateTime <-> ISO8601 and fix bug in constructor.
  • #731: Allow expressions for SELECT and FROM in queries using DbStorage.
  • Scalable database architecture
    • Discussed compression of MySQL tables.
    • Discussed partitioning of tables for near-neighbor queries.
    • Investigated Gearman.
    • Started implementing xrootd-based prototype.
  • More high-bandwidth database cluster hardware design (using Sun X4540s).

Minor accomplishments:

  • Made comments on SDQA persistence tests.
  • Discussed post-Davis tables, especially FaintSource?.
  • Discussed design of CFHT-specific astrometry indexes.
  • Edited documentation of Input/OutputStages.
  • Suggested splitting of cat package.
  • Discussed impact of removal of "test" MySQL account.
  • Participated in XLDB3 planning.
  • Participated in DC3a/DC3b All Hands planning.

Andy H (submitted on 4/xx/2009)

Serge (submitted on 4/xx/2009)

  • post-integration wrap-up (#669, #694)
  • scalable architecture discussion
  • DataAccWG telecons
  • add preliminary !Source matching capabilities to afw (#754)

Daniel (submitted on 4/17/2009)

Compression estimates for MySQL/myisam w/ mixed int/float and mostly float data

Scalable queries:

  • Discussions re: parallel work dispatcher (gearman or xrd)
  • High-level design of API
  • Skeleton implementation of:
    • Frontend server
    • Xrd file-based message-passing
    • sql parsing
    • chunk partition metadata table


  • observation group algorithm work
    • algorithm design, feasibility of moving objects
    • distributed execution plan
  • helping test and refine the storage manager
  • provenance model discussions

Jacek (submitted on 4/14/2009)

  • work related to the FRS Davis meeting and the follow-up DMS document
    • reviewed the DMS doc
    • reworking schema accordingly, captured at dbSchemaUpdateApr09
    • updating storage estimates accordingly
  • scalable architecture
    • design discussions and planning
    • testing various options and configurations
    • also tested postgres and finished gridsql tests
    • all documented, see dbScalableArch
  • work on splitting cat package
  • discussions with the infrastructure team about database hardware
  • work on standardizing names, proposal approved by TCT, see CcbMeeting20090401
  • various minor schema tweaks and admin scripts related to DC3a integration
  • organized four DataAccWG telecons
  • scidb
    • discussions related to NSF proposals
    • working on SLAC budget for the NSF proposal
    • discussed funding with slac management
    • discussions with PNNL on open provenance
    • discussions with Argonne Lab on collaboration
    • arranging quotes/statements from science for website
  • xldb
    • assembled draft agenda for XLDB3
    • compiling list of people to invite to XLDB3
    • organized the organizing committee meeting, discussing agenda/invites/logistics

Major Accomplishments

Significant breakthroughs, issues resolved.

Objectives for the Next Period

What you expect to accomplish.



Database Group

Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues