Last modified 10 years ago Last modified on 06/15/2009 12:37:51 PM

Status Report for March 15, 2009

Previous Month Next Month

Summary of progress

Work activity; whether it reached associated milestones.


Bill (submitted on 3/23/2009)

  • Supported build system & virtual machines for DC3a
  • Learned about harness, warm up to work on interslice choreography
  • Prepared new VM images -- blocked, waiting for Ray to install stacks on them
  • Made progress on Security doc, but didn't finish -- fortunately, deadline receded too

David (submitted on 3/23/2009)

  • Gathered raw images and additional calibration data for CFHTLS, all four Deep Survey areas
  • Created image/calibration repository layout and filled with test data
  • Working with Steven Lo at IPAC over network latency issues
  • Cluster, trac maintenance

Greg (submitted on 3/23/2009)

  • attended Integration Week at IPAC 3/8 - 3/12
  • Developed harness in support of DC3a integration:
    • coordinated development with ctrl_orca, especially wrt pipeline policy
    • coordinated development with daf_persistence, wrt handling of logical locations of files and db
    • added to Clipboard API
    • expanded examples and their documentation
    • introduced doxygen tutorial pages (under pex_harness/doc/*.dox) to summarize package and explain various examples

Steve (submitted on 3/18/2009)

  • worked on orchestration layer (orca) for "integration week" work
  • fixed bugs that came up in ctrl_orca and ctrl_events
  • added ctrl_evmon tree for event monitor

Ray (submitted on 3/18/2009)

  • worked on updates to Infrastructure cost model to address database performance for user queries
  • attended Integration Week at IPAC 3/8 - 3/12
  • I spent a fair amount of time managing new package releases, including
    • testing new releases
    • coordinating releases with package gurus
    • making releases available on the server
    • installing new releases on the cluster
    • keeping the buildbot up to date
    • building a stack based on gcc 4.3.3
  • assisted in the coordination and testing of the orchestration layer and the harness with Steve and Greg.
  • Complete DC3a coding tasks (see below); bug fixes
Task %Done* Task Title Comments
28/39 90% update distribution server little bit of clean up left
114/111 100% orchestration layer design Worked more with Steve, Greg
84/535 100% integrate logging and Trace
78/532 100% PropertySet? Integration into Policy


Robyn (submitted on 3/16/2009)

Note: Only my tasks, not listed as 100% completed in my LAST report, are included.

Status NL Not Lead
Status NW No work done
Id PM Project Management Plan
Id DC Data Challenge Plan
Id AP Applications Plan
Id MW Middleware Plan

NL NW PM50 ...Update SysML/UML models and prepare Technical Documents
NL NW PM51 ......Complete LSST Software Development Plan Waiting on Review
NL NW PM51 .........LSST Software Development Plan
NL NW PM52 .........LSST Configuration Management Guidelines
NL NW PM53 .........LSST Software Quality Assurance Guidelines
NL NW PM54 .........LSST Verification and Validation Guidelines
NL NW PM60 ...Complete DM Inputs to System Level Requirements and Design
NL NW PM67 ......Create document templates, recipes, tools for Enterprise Architect output/gen
NL NW PM68 ...Complete Data Management Reference Design
NL 10% PM69 ......Clean up DC3 UML Model, Update LSST Reference Design UML Model Updating existing Activity and UseCases? to
(1) conform to LSST UML standards definition;
(2) conform to DM Domain model.
Merging IPAC August Meeting EA Model, Original DC3 EA Models into the new 'Pipelines and Productions' EA Model.
NL NW PM70 ......Update EA and UML Modeling Conventions Document for Activity Diagrams
PM75 ...Create draft DM Integration Test Plan
10% PM77 ......Generate test cases Exercising TestGen? using DC3 model
NL NW PM78 Complete PM Control System Updates
NL NW PM81 ...Publish cost book, do team review
NL NW PM85 ...Conduct PM Internal Cost Review and updates
NL NW PM86 Complete DM Inputs to Risk Management, System Integration, Commissioning, and Operations Plans
0% PM87 ...Update DMS Integration and Test Plan
PM90 ...Contribute DM sections to LSST Risk Management, Commissioning, I&T, and Operations Plans
DC56 Data Challenge 3 Software Development Plan
50% DC58 ...DC3 Configuration Management Plan Need feedback from Gehrig, Baker, Plante
DC60 ...DC3 Verification and Validation Plan
75% ......DC3a Verification and Validation Plan Continuing refinement: added DC3a TestCases?
0% ......DC3b Verification and Validation Plan

Database Group (submitted on 3/xx/2009)

K-T (submitted on 3/23/2009)

2008-02-14 to 2009-03-23:

44/43 (build templates): Ticket #334, stalled 25% 64/161 (DC2 policy capture): Ticket #527 closed.

Major tasks:

  • DC3a integration.
  • Wrote up the fault tolerance requirements needed from system requirements.
  • Described GrayWulf hardware configuration and advantages (much lower cost per MB/sec and cost per disk TB) to Infrastructure team.

Minor tasks:

  • Fixed cat package installation.
  • #527: Provenance written to global database as well as per-run database.
  • #616: Completed boost::noncopyable warning fix.
  • #656: Fixed SWIG type lookups for runtime exception types.
  • #666: Major update to DateTime? interface, including persistence.
  • #667: Added ability to persist dgPsfs (but not yet pcaPsfs).
  • #668: Added template subimage retrieval (including WCS from FITS).
  • #671: Implemented TAN-SIP FITS header output.
  • #672: Added raw SQL execution for SDQA and AP.
  • #688: Added now() method to DateTime?
  • Debugging of nameserver issues.
  • Assistance with mysqldb package installation.
  • Prepared various design alternatives for more-images-than-cores issue.
  • Reviewed some of Russ Laher's SDQA code.
  • Discovered quibbles with coding standards.
  • Thinking about hierarchies of astronomical objects ("segmentation trees").
  • Suggestions to Martin about multifit implementation.
  • Coding style comments about C++ output arguments (non-const references).
  • Comments about detection vs. measurement vs. Source/DiaSource.
  • Discussion about pipeline vs. Pipeline vs. sub-pipeline vs. Stage terminology. Now resolved.
  • Comments on MySQLConfigurator for Orca.

Andy H (submitted on 3/xx/2009)

Serge (submitted on 3/xx/2009)

Daniel (submitted on 3/18/2009)

  • Tested packing performance on MySQL, compared packing time with/without indexes and query performance (high/low selectivity)
  • Literature survey for provenance, database
  • Help Jacek with shared-scan, gridsql partitioning
  • scidb
    • cooking algorithm udf implemenation and test code
    • working on observation grouping
    • discuss storage mgr/udf/query-executor api
    • vldb demo telecon hosting

Jacek (submitted on 3/20/2009)

  • scalability tests
    • testing postgres: feature set, performance benchmarks
    • testing different partitioning and distribution schemes with gridsql and postgres
    • attempt to put together a complete design of a scalable architecture implementable in DC3b timescale
    • related documents dbScalableArch, dbSpatialJoinPerf?
  • dc3a preps
    • added database support for global provenance
    • database related scripts: finalized and helped integrate with ctrl_orca
    • prepared database on lsst10
    • schema updates (#517, sdqa, few others)
  • updated storage estimates (WCSource, FaintSource)
  • discussed hardware for query access
  • assisted with network blocks at slac
  • cleaned up schema (*Err types. #649)
  • discussions about provenance
  • reviewed doc written after FRS Davis meeting
  • provided input for astro-informatics papers
  • organized 2 DataAccWG telecons
  • scidb
    • internal discussion about future funding
    • telecons with Emory Univ, demo telecons
    • collecting input for new
    • reviewed VLDB paper

Major Accomplishments

Significant breakthroughs, issues resolved.

  • Through March 20: Integration week onsite at IPAC March 8-12 and an additional week of intense offsite work brought the pipeline very close to completion. The imaging pipeline is essentially in place, although there are still some science-level (rather than framework level) issues to be addressed. The association pipeline should be in place by March 24. The MOPS pipeline is in place.

Objectives for the Next Period

What you expect to accomplish.


  • Bill:
    • Finish the security doc for reals
    • Continue to support infrastructure
      • Install V & V tool on trachea
    • Harness - interslice choreography
      • Consult on design with KT, Ray, and Greg
      • Prepare testbed
      • Try a simple algorithm
  • Chris:

  • David:
    • Transfer DC3a data to lustre file system when it becomes available
  • Steve:
    • Work on integration of event monitor for orca
    • Work on event monitor scripts for use with orca
    • Write unit tests for orca

  • Greg:

  • Mike:
  • Ray:
    • I will be spending the remainder of March on routine running of the DC3a


  • Robyn
    • continue exercising TestGen for completeness and accuracy when parsing EA models in order to build TestCase specifications.
    • continue to update the DC3a Integration Test Plan
    • complete the merge of the IPAC August Meeting EA DC3 model and the original DC3 model into the 'Pipelines and Productions' EA model.
    • monitor DC3a Integration and Test in preparation for cutting a validated DC3a Software Release.

Database Group

  • scalability tests
  • DC3b planning, in particular scalable architecture for DC3b
  • provenance
  • refining storage and disk io estimates

Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues

A city-wide power failure in Urbana burned out parts of the cluster backplane, taking four of its ten nodes offline. Since these were the nodes intended for testing the Lustre file system, it was not possible to implement Lustre during the integration, as replacing the backplane would have meant having the entire cluster out of service during part of the Integration Week(s). Chris Cribbs will be out of the country until April 1, and on his return we will replace the backplane and make Lustre available for testing on the cluster.