wiki:Report051509

Version 11 (modified by daues, 10 years ago) (diff)

--

Status Report for May 15, 2009



Summary of progress

Work activity; whether it reached associated milestones.

NCSA

Bill (submitted on 5/19/2009)

  • No change to 27/569, 28/41, 34/517 from last month.
  • Working on #795 - a simple data mapper for pex_harness. Design is in flux, probably to be discussed at DC3a Tucson meeting.
  • 35/74 - Configure V&V - no substantial change since last month; I'm helping Robyn configure the software; I plan to hook it into the nightly build once we have a command-line interface working

David (submitted on 5/20/2009)

  • Agenda-wrangling for Tucson meeting
  • Prepared TeX skeleton for DC3a report
  • Moved missing calibration files to cluster
  • Continued administration of Trac, svn for LSST-DM
  • Continued administration of SVN for LSST Science Book

Greg (submitted on 5/21/2009)

  • Continued work on setup for LSST pipelines to execute on the NCSA Abe cluster; this work builds much of the infrastructure needed for submission to other collaboration clusters.
    • installation and testing of grid middleware infrastructure on an LSST cluster Condor/Globus?; test of job submissions/file transfers
    • development of ctrl_orca PipelineManager subclass suitable for Abe cluster

Steve (submitted on 5/20/2009)

  • Worked on build process of ctrl_evmon
  • Added several fixes to evmon objects and new objects
  • Added python object to parse db-auth.paf files so passwords are retrieved via that, instead of having cleartext passwords in code. This was necessary since jython can not import swigged classes.
  • Assisted Ray with duration calculations and wrote additional scripts to make one integrated script.

Ray (submitted on 5/xx/2009)

LSSTC

Robyn (submitted on 5/19/2009)

Status NL = Not Lead NW = No work done
Id PM = Project Management Plan DC = Data Challenge Plan AP = Applications Plan MW = Middleware Plan


.Status.Id...........Title.........................Comment..............
NL NW PM50 ...Update SysML/UML models and prepare Technical Documents
100% PM51 ......Complete LSST Software Development Plan Delivered to Deputy System Engineer
NL NW PM60 ...Complete DM Inputs to System Level Requirements and Design
NL NW PM67 ......Create document templates, recipes, tools for Enterprise Architect output/genas needed, prepared more EA Report templates
NL NW PM68 ...Complete Data Management Reference Design
100% PM69 ......Clean up DC3 UML Model, Update LSST Reference Design UML Model Completed my portion of Activity diagrams, robustness diagrams, reverse engineer of DC3a
100% PM70 ......Update EA and UML Modeling Conventions Document for Activity Diagrams Document updated and installed on Docushare
PM75 ...Create draft DM Integration Test Plan
10% PM77 ......Generate test cases Continued to exercise prototype TestGen as new releases were delivered
NL NW PM86 Complete DM Inputs to Risk Management, System Integration, Commissioning, and Operations Plans
0% PM87 ...Update DMS Integration and Test Plan
0% PM90 ...Contribute DM sections to LSST Risk Management, Commissioning, I&T, and Operations Plans
DC56 Data Challenge 3 Software Development Plan transitioning from DC3a focus to DC3b
0% DC58 ...DC3 Configuration Management Plan Will transition from DC3a focus to DC3b
0% DC60 ...DC3 Verification and Validation Plan Will transition from DC3a focus to DC3b
0% ...DC3 SQA Plan Will transition from DC3a focus to DC3b

Database Group

K-T (submitted on 5/21/2009)

2008-04-21 to 2009-05-20:

Major accomplishments:

  • Prepared for and attended DC3a/DC3b meeting.
  • Fixed #472: Persist/compute metadata for {Raw,Sci}_{Amp,CCD,FPA}_Exposure, including many fixes to ExposureFormatter and writing a new stage.
  • Support for DC3a pipeline production runs:
    • Investigated various astrometry/diffim issues, including #773, #775, #776, #783, and #801; fixed problem with BINNED1 WCS sources being ignored and cleaned up some other issues with the WcsDeterminationStage.
    • Investigated PSF determination issues #774 and #782.
    • Looked into missing calibration files and linearization problems for CFHT data.
    • Various build/tag chores for critical package releases, including daf_base 3.2.8/3.2.9, cat 3.3, afw 3.3.16, ctrl_dc3pipe 3.3.2, meas_astrom 3.0.7, ip_isr 3.3.9, and meas_pipeline 3.0.5.
  • Wrote data access and exceptions portions of DC3a report.
  • Completed C++ side of initial qserv/worker prototype.

Minor accomplishments:

  • Fixed:
    • #800: Check for out-of-range DateTimes and fix unit tests.
    • Initialize and persist DiaSource? obsCode correctly after #796 fix.
    • #792: Update all (my) Python classes to new-style.
    • #707: Update ctrl_orca Provenance_1.py test to match code.
    • #763: Fix microseconds returned from DateTime::timeval().
  • Worked on:
    • #808: Improve persistence of Exposure metadata to database.
    • #805: Enable persistence of more data types as PropertySet values.
    • #793: Investigated false report of persistence problems with PSFs.
    • #791: Investigated NaN retrieval problems.
    • Crude initial estimates for increased AOS/DPS database needs.
    • Began work on new database I/O spreadsheet.

Other:

  • Looked at Open Geospatial.
  • Discussions about multifit size estimates.
  • Discussions about database hardware and infrastructure Visio diagram.
  • Commented on LSST File Lifecycle Management requirements document.

Andy H (submitted on 5/xx/2009)

Serge (submitted on 5/xx/2009)

  • began investigating libraries that integrate (or could be used to implement) spatial primitives in mysql/postgres

Daniel (submitted on 5/xx/2009)

Jacek (submitted on 5/15/2009)

  • storage estimates (FaintSource, db compression, small tables)
  • disk io estimates (discussions, telecons, meeting with Mike Huffer)
  • attended mysql conference and drizzle meeting, see debrief posted to lsst-data
  • progress with schema updates
  • prepared for DC3b planning meeting
  • standardizing names (TCT proposal)
  • organized several DataAccWG telecons
  • XLDB
    • produced draft agenda
    • reached out to different groups to find best attendees (NASA, bio and communities in Europe, European Commission)
    • compiled list of invitees
    • started paperwork required by SLAC
    • coordinating logistics (catering, funding, lodging)
  • SciDB
    • did all the SLAC-related paperwork for the SciDB NSF proposal (budget, budget justification, statement of work, current and pending support, bios, no-conflict of interest forms, and others)
    • coordinated 9 letters of support for NSF proposal from science
    • rewrote science related sections of the NSF proposal
    • started officially building scidb community, including setting up scidb-news mailing list
    • wrote/submitted SLAC LDRD SciDB proposal
    • reviewed draft scidb.org mockup

Major Accomplishments

Significant breakthroughs, issues resolved.



Objectives for the Next Period

What you expect to accomplish.

NCSA

David Gehrig

  • Ride herd on getting the first draft of DC3a report finished by June 15.


LSSTC

Robyn Allsman

I will work on the following unless and until PM needs work on higher priority PDR tasks. Priority and percentage of time allocated to the following tasks has not been determined.

  • Will start work on Parasoft Coding Standards integration;
  • Will commence update of DC3b QA, Configuration, Validation & Test Plans (see Management Issues block below for Project Plan);
  • Will add all 3rd party software copyrights/licenses to the LSST DM Legal Notices directory and prepare Copyright notice to be inserted in each source file.


Database Group



Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues

Robyn ALlsman

DM Management needs to affirm the following intergroup participation:

  • Will turn over DC3b Project Plan preparation to App and MW Project Leads. The document should be prepared as part of the LSST DM Software Development Process.

MW Management needs to affirm the following intergroup participation:

  • Will need assistence from Ray Plante and/or David Gehrig on integration of Parasoft into Build system (scons) and possibly !Buildbot