Last modified 10 years ago Last modified on 06/15/2009 12:38:58 PM

Status Report for January 15, 2009

Previous Month Next Month

Summary of progress

Work activity; whether it reached associated milestones.


Bill (submitted on 1/14/2009)

  • Made progress on nightly build of trunk, but having trouble with eups in complex packaging situations (#398, #525, #560, #565)
  • Supported cleanup of security breach, mainly installing one-time-password (OTP) authentication
  • Maintained development support infrastructure
    • Improving security to prevent future breaches
    • Help provide temporary home for development server

Chris (submitted on 1/22/2009)

  • Vacation / personal time

David (submitted on 1/22/2009)

  • Finished cleanup of security breach, with BillB
  • Transfered development server services back to their original host
  • Upgraded SVN repository to version 1.5
  • Working with Jeff K. on project plan

Greg (submitted on 1/22/2009)

  • Worked with KT on enabling support in daf_persistence (PropertySetFormatter) for boost::mpi::packed_oarchive/iarchive
  • Completed Clipboard-to-Clipboard interSlice communication (sending/receiving PropertySet::Ptr) for two dimensional "focalplane" topology. The dimensions of the 2D topology may now be set in policy. Most recent code is checked into pex_harness ticket 343.
  • Transition of pex_harness to new middleware (daf_base/pex_exceptions) and the associated software stack has progressed : 1) pipelines are receiving trigger events properly, 2) interSlice communication pipeline example is working, 3) EventStage (pipeline emits an outgoing event to next pipeline) example is working.

Steve (submitted on 1/22/2009)

  • Completed converting from DataProperty? to PropertySet? for ctrl_events for source, tests and examples #508
  • Did preliminary work for events failover

Ray (submitted on 1/xx/2009)


Robyn (submitted on 1/09/2009)

Converted %Done to new 5-Level percentage.

Task UniqID %Done Dropdead Task Title Current Status Note
7 1 SW Quality Assurance
8 7 Develop DC3 Software Development Plan DC3 Training Exercise; to be mandated in DC4
-- -- 75% DC4 DC3 Software Project Management (SPM) Plan
9 507 25% DC3b end DC3 SQA Plan
10 508 25% DC3b end DC3 V&V Plan
11 521 100% DC3a end Specify test requirements for ea phase of Product devdone
12 520 100% DC3a end Specify pass/fail metrics for each test typedone
13 509 50% DC3b end DC3 CM Plan
14 9 DC3a end V & V: Acquire SQA support tools
15 10 100% DC3b end Configurable tool to perform software standards checkingpurchased
16 479 100% DC3a start C++ LSST benchmark test suite packageddone
17 522 100% DC3a end Benchmark potential toolsdone
21 478 100% DC3b end Acquire Purchased but not installed yet
22 11 75% DC3a end Coverage analysis tool
23 485 100% DC3a end Benchmarkdone
24 488 75% DC3a end Release for developer useWaiting on TCT approval on 14 Jan 09
25 24 100% DC3a end V & V: Refine Ticket workflow to add more V & V testing Waiting on TCT approval on 14 Jan 09
170 429 03/15/2009 non-DC3 activities
210 493 LSST Corp
211 495 03/15/2009 Software Quality Assurance: Planning
212 496 03/15/2009 Prepare LSST DM Software Development PlanIn review by Lsst Subsystems
215 524 75% 03/15/2009 Software Development Plan Overview In review by Lsst Subsystems
213 503 75% 03/15/2009 Software Quality Assurance GuidelineIn review by Lsst Subsystems
214 497 75% 03/15/2009 Verification and Validation GuidelineIn review by Lsst Subsystems
-- --- 75% 03/15/2009 * Define process to track Requirements to Tested ReleaseVirtual process defined, need more concrete
216 523 100% 03/15/2009 * Specify test requirements for ea development phase done
217 499 75% 03/15/2009 Configuration Management GuidelineIn review by Lsst Subsystems
218 501 75% 03/15/2009 Develop std labelling for products and provenanceNeed to coord w/DB&Build teams

Database Group (submitted on 1/xx/2009)

K-T (2008-12-15 to 2009-01-22)

44/43 (build templates): Ticket #334, initial checkins started 25%
64/161 (DC2 policy capture):
        Redefining this to be incorporation of DC2 capture in DC3a context
        Ticket #315, design with Steve essentially complete 25%
65/469 (DC3 provenance capture):
        Redefining this to be capture of DC3b provenance (beyond policies) 0%
80/531 (PropertySet integration):
        daf_base done 100%
        daf_data/daf_persistence waiting on RP pex_logging 75%
82/534 (Exception integration):
        pex_exceptions, utils, daf_base done 100%
        daf_data/daf_persistence waiting on RP pex_logging 75%
  • Attended Friday Harbor meeting.
  • Presented LSST DM as an example of cyberinfrastructure to the iPlant Consortium.
  • Attended Conference on Innovative Database Research.
  • Listened in on some of functional requirements working session.
  • Updated and released pex_exceptions, utils, and daf_base.
    • Fixed #583 (PropertySet test failures on 64-bit).
    • Changed to boost::unordered_map from G++ extension to solve #548.
    • Reordered dependencies to solve #567.
    • Added generic Python get() to PropertySet to solve #562.
    • Implemented cycle prevention in PropertySets to solve #561.
    • Prevented leakage of boost::bad_any_cast in PropertySet to solve #564.
  • Got daf_persistence and daf_data to taggable state (waiting on pex_logging).
    • Integrated PropertySet into daf_persistence to complete #492.
    • Integrated PropertySet into daf_data to complete #531.
  • Replaced DataProperty references in pex_harness with PropertySet.
  • Started work on generic package templates in devenv/templates for #334.
    • Worked on SWIG files with several people (AndyB, Russell, Steve, Greg).
  • At Serge's suggestion, reimplemented PropertySet using custom "any" class to gain some performance and avoid type-switches (but add template specializations). Code is complete but will not be used for now, as benefits do not clearly outweigh costs.
  • Built and installed svn 1.5.4 locally.
  • Discussed design of orchestration/provenance interface.
  • Discussed design of orchestration/database setup interface.
  • Discussed design of database setup scripts/classes.
  • Got TCT approval for using Mysqldb Python package for admin tasks.
  • Debugged an issue with sending PropertySets in events with Steve.
  • Discussed design of improved FITS file retrieval/persistence.
  • Helped Greg with merging MPI archives into PropertySetFormatter.
  • Helped Russell with some build problems.
  • Answered some of Martin's questions about pex_harness/IOStage.
  • Wrote an order-insensitive comparison script for pex_logging tests.
  • Discussed WCS determination interslice communication use case.

SciDB items:

  • Design meeting at SLAC.
  • Ongoing API design work.
  • Made suggestions to improve proposed benchmark.
  • Packaged up sample astronomy data for Philippe.

Andy H

  • setup postgres on a small testbed at slac. Still waiting for the account at BNL to be set up


  • updated afw to use new PropertySet and exceptions (#547)


  • refreshed storage estimates
  • implemented database setup scripts (part of #460)
  • implemented automated run cleanup (#511)
  • worked out details of database permissions (#524)
  • finished work on SDQA schema (#409)
  • added magnitudes to schema (#401)
  • large scale tests: email exchange with GridSQL
  • SciDB/XLDB
    • SciDB meeting at SLAC
    • reviewed usecases
    • input about SciDB logo and SciDB website
    • organized SciDB steering telecon
    • side meeting with one of the SciDB developers
  • others
    • Kavli Symposium
    • first discussions with ASKAP about possible collaboration
    • hiring paperwork

Major Accomplishments

Significant breakthroughs, issues resolved.

Objectives for the Next Period

What you expect to accomplish.


  • Bill:
    • Continue support of & work on nightly builds (#398, #525)
    • Release new VM image with new release pre-installed
    • Support cleanup of security breach
    • Security doc for PDR
      • Integrate edits from team members
      • Draw remaining perimeter diagrams
  • Chris:

  • David:
    • Complete revision of DC3a project plan
      • Add system integration tasks, scalability testing tasks
    • Schedule DC3a "integration week"
  • Steve:
    • Work with Greg and Ray on Orchestration Layer design.
    • Preliminary implementation of Orchestration Layer.
    • Incorporate Orchestration Layer design requirements into the Events framework

  • Greg:
    • Transition of pex_harness to new middleware (daf_base/pex_exceptions) continues with IOStage testing, i.e., use of daf_persistence to retrieve input data and persist output.
    • Begin development/examination of a "sliceleaders" topology wherein, amongst the full set of Slices, there exist "Slice leaders" that receive/gather data thru interSlice communication from a group of Slices. The Slice leader may then perform a by-CCD calculation while other Slices in its group do no operation.

  • Ray:


  • Robyn:
    • PDR Preparation (highest priority)
      • bring LSST Software Development Plan, including DM specifics to completion
      • prepare the DM Integration and Test Plan;
      • prepare DM I&T Plan cost estimates
    • DMS Effort (in order of presumed occurence)
      • install revised Ticket Workflow, if approved
      • reinstall automatic notification to Developers of Ticket status
      • get the Coding Standards Checker installed;
        • will need assistence in getting the LSST ruleset implemented
      • install pre-commit block on all SVN source commits which do not refer to a Ticket.

Database Group

Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues