wiki:Report121508
Last modified 10 years ago Last modified on 06/15/2009 12:39:23 PM

Status Report for December 15, 2008

Previous Month Next Month



Summary of progress

Work activity; whether it reached associated milestones.

NCSA

Bill (submitted on 12/11/2008)

  • Supported recovery from security breach on ds33.ncsa.uiuc.edu
  • Worked on automated build system (slowed by security breach) #398, #433 (complete), #525

Chris (submitted on 12/xx/2008)

  • Installed Lustre on NCSA cluster to see if the new version had fixed the bug that bit us before; they haven't.
  • Reinstalled OS on ds33 and most of the cluster after breakin.
  • Investigating self-correcting (95%) disk drive technology

David (submitted on 12/18/2008)

  • Much work related to the security breach on the LSST cluster and the server dev.lsstcorp.org (ds33.ncsa.uiuc.edu)
    • Stood up new Trac/SVN on a virtual server setup by Bill B
    • Worked with Chris C to clear ds33 of possible compromise
    • Worked on clearing LSST cluster of possible compromise, with Chris C
  • Project management transition on reporting
  • Took over Microsoft Project files for DM planning
  • Support for SVN/Trac for LSST science book manuscript
  • Coordinated DC3 telecons

Greg (submitted on 12/15/2008)

  • Merged ticket 423 of daf_persistence to trunk prior to its low level moddleware upgrade
  • Worked on pex harness code incorporation of updated low level middleware. Pipeline and Slice catch LsstCppException when possible and display information on the Traceback; if a plain Exception is caught, available messages are displayed. InterSlice communication is reimplemented to send PropertySet's beteween Slices. Testing of all of these elements in real Pipeline execution remains to be done.
  • Had some discussions on ROE efforts to run some DC2 style Pipelines; some lessons learned there, but the effort halts for now and will recommence later with DC3 piplines.

Steve (submitted on 12/xx/2008)

  • Met with Greg and Ray to get further requirements for DC3 orchestration layer. This is on hold until I can get Events finished.
  • Reworking Events from Data Properties to PropertySets?. This is going well, but I've run into a couple of snags that I'm consulting others on. I expect to have this working in the #508 ticket by next week.

Ray (submitted on 12/17/2008)

  • Presented invited talk at the ESAC Grid Workshop (Madrid) on LSST Hardware Architecture (Travel: 12/10-12/12)
  • Worked with Gehrig and NCSA Security reacting to security breach on LSST machines
  • Updated pex_logging for #458
  • DC3 Management:
    • Worked with Gehrig and Freemon on project management transition
    • Assisted Gehrig with project plan modifications
  • Supported package distribution server
  • Made progress on tasks (see below):
    • distribution server
    • orchestration layer design
    • logging and trace
    • integration of PropertySet?

Task %Done* Task Title Comments
28/39 90% update distribution server little bit of clean up left
114/111 75% orchestration layer design Worked with Steve, Greg
84/535 25% integrate logging and Trace Just need to finish tests
79/531 100% Full integration of PropertySet?
78/532 0% PropertySet? Integration into Policy

*using new 5-level percentage on tasks where appropriate.


LSSTC

Robyn (submitted on 12/11/2008)

Task UniqID %Done Dropdead Task Title Current Status Note
7 1 SW Quality Assurance
8 7 Develop DC3 Software Development Plan DC3 Training Exercise; to be mandated in DC4
-- -- 90% DC4 DC3 Software Project Management (SPM) Plan
9 507 0% DC3b end DC3 SQA Plan
10 508 10% DC3b end DC3 V&V Plan
11 521 100% DC3a end Specify test requirements for ea phase of Product devdone
12 520 100% DC3a end Specify pass/fail metrics for each test typedone
13 509 50% DC3b end DC3 CM Plan
14 9 DC3a end V & V: Acquire SQA support tools
15 10 DC3b end Configurable tool to perform software standards checkingWaiting on funds
16 479 100% DC3a start C++ LSST benchmark test suite packageddone
17 522 100% DC3a end Benchmark potential toolsdone
21 478 0% DC3b end Acquire Jeff said maybe funds in start of new year
22 11 95% DC3a end Coverage analysis tool
23 485 100% DC3a end Benchmarkdone
24 488 90% DC3a end Release for developer useWaiting on TCT approval
25 24 100% DC3a end V & V: Refine Ticket workflow to add more V & V testing Waiting approval of proposed Ticket workflow
170 429 03/15/2009 non-DC3 activities
210 493 LSST Corp
211 495 03/15/2009 Software Quality Assurance: Planning
212 496 03/15/2009 Prepare LSST DM Software Development PlanIn review by Lsst Subsystems
215 524 90% 03/15/2009 Software Development Plan Overview In review by Lsst Subsystems
213 503 90% 03/15/2009 Software Quality Assurance GuidelineIn review by Lsst Subsystems
214 497 90% 03/15/2009 Verification and Validation GuidelineIn review by Lsst Subsystems
-- --- 90% 03/15/2009 * Define process to track Requirements to Tested ReleaseVirtual process defined, need more concrete
216 523 100% 03/15/2009 * Specify test requirements for ea development phase done
217 499 90% 03/15/2009 Configuration Management GuidelineIn review by Lsst Subsystems
218 501 90% 03/15/2009 Develop std labelling for products and provenanceNeed to coord w/DB&Build teams

Database Group (submitted on 12/xx/2008)

K-T

(This is incomplete.)

  • Did substantial work on 43/43 #334 (new templates for build/SWIG files). Maybe 75% done, rest waiting for RHL.
  • Generally completed #531 (daf_data update), but not yet merged to trunk, and FitsFormatter removed instead of updated. Contributes to 79/531 and 81/534.
  • Nearly finished #492 (PropertySetFormatter and update of daf_persistence) but not yet merged to trunk. Contributes to 79/531 and 81/534.
  • Fixed #536 (PropertySet check for nulls).
  • More fixes to 77/178 #480 (implement new PropertySet). A few more (64-bit) remain.

Andy H

  • setting up account at BNL - needed to run the large scale tests

Serge

  • reviewed DC3a/DC3b schema split (#519)
  • persisting classification information discussion (#415)
  • database permission discussion/review (#524)
  • DC3/DataAccWG telecons, data train telecon
  • run deletion proposal (#511)
  • proposal/discussion for generically supporting persistence of STL containers of Persistable objects
  • review of ticket #371 (don't hardcode leap-second table)
  • testing/review for new exception code (#458)
  • testing/review of new PropertySet code (#480)

Jacek

  • work on database schema
    • DC3 schema split into DC3a and DC3b (#519)
    • defined units
    • added schema for ISR (#407)
    • revisited exposure metadata schema (#513)
    • mops related improvements
  • released schema browser (#443)
  • work on database permissions (#524)
  • work on multi-run support (#447)
  • work on db views (#426)
  • discussions about persisting classification information (#415)
  • reading up about balanced systems / amdahl's law
  • discussions about DB selection for simulation
  • large scale tests planning
  • documented UniqueIds discussion in trac
  • interraction with Postgres
  • telecons: DC3, PMC, data train
  • hiring: interviews, candidate selected, paperwork
  • graphics for ADASS poster
  • organized 4 DataAccWG telecons
  • XLDB / SciDB
    • started organizing XLDB3
      • assembled organizing committee
      • dates/location selection
      • setting up website
      • connected with VLDB'09 organizers
      • working on logistics (conference center, hotel)
    • reviving xldb wiki
      • wiki contents updated
      • recruiting several authors
    • updated xldb entry in wikipedia
    • work on science benchmark
    • selecting scidb logo
  • Others


Major Accomplishments

Significant breakthroughs, issues resolved.



Objectives for the Next Period

What you expect to accomplish.

NCSA

  • Bill:
    • Continue support of & work on nightly builds (#398, #525)
    • Release new VM image with new release pre-installed
    • Support cleanup of security breach
    • Security doc for PDR
      • Integrate edits from team members
      • Draw remaining perimeter diagrams

  • Chris:
    • Finish clean-up of security incident
    • Vacation through mid-January
  • David:
    • Complete cleanup of security breach
    • Transition per developer request to Subversion 1.5
    • Project plan and emails
    • Vacation
  • Steve:
    • I'll be on vacation starting 12/20 until the end of the year.
    • Finish PropertySet? conversion in Events in the ticket by 12/19.
    • Start working on orchestration layer, consulting with Greg on the UW and ABE requirements.
  • Greg:
    • Work on design and implementation for supporting multiple InterSlice communication modes: send/recv between Slices, Bcast from Pipeline to Slices, Scatter from Pipeline to Slices, etc.
    • Examine incoming application use cases for interslice communication and ensure support for these cases.

  • Ray:
    • Complete Task: Integration of Trace and Logging
    • Complete Task: Integration of PropertySet? and Policy
    • Complete Task: Orchestration layer design

LSSTC

  • Robyn:
    • bring LSST Software Development Plan, including DM spcifics to completion

Database Group

  • See database related ticket
  • xldb3: agenda, logistics, who to invite
  • sciDB: face to face meeting, science benchmark, coordinating usecase collection


Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues

We had a serious problem regarding a security breach on two systems key for the DM team. We were quickly able to stand up a stand-in for the system holding the subversion code repository and Trac site. We are nearly done clearing up the LSST cluster, but its unavailability for two weeks put a roadblock on the effort to stand up the DC2 pipelines for comparative timing tests.

We have also made plans related to the loss of Cristina Beldica to the Blue Waters project; her management duties have been divided among Ray Plante, David Gehrig, and Mike Freemon, who has just joined the project.