wiki:Report061509
Last modified 10 years ago Last modified on 06/22/2009 01:45:43 PM

Status Report for June 15, 2009

Previous Month Next Month


Summary of progress

Work activity; whether it reached associated milestones.

NCSA

Bill (submitted on 6/15/2009)

  • Data mapping in Harness: Wrote basic Python interfaces
  • Security plan: Released major revision that for the first time has no blank sections; includes Jeff's edits & requests
  • Automated build system: functioning normally
  • Parasoft (automated V&V): supporting Robyn on tracer.ncsa.uiuc.edu
  • VMs for developers (2nd edition): waiting for Ray to put a stack on them

David (submitted on 6/22/2009)

  • Writing/coordinating/editing DC3a report
  • Agenda-wrangled and attended DC3a Post-Mortem
  • Upgrading Trac

Greg (submitted on 6/18/2009)

  • Developed classes/components needed for launching pipelines on the NCSA Intel 64 Abe cluster
    • Wrote AbePipelineManager.py of ctrl_orca for
      • deploying required files -- nodelist template, policy files, setup script -- to the remote platform via globus-url-copy
      • creating Condor job description file and performing associated job submission
    • Wrote launchPipelineAbe.sh of pex_harness for customized harness execution
      • sets up environment as first step
      • creates dynamic LSST node list based on template from ctrl_orca plus PBS batch system generated node list
    • Executed test single stage pipelines, performing sanity checks with a larger number of nodes. Test pipelines across 20 nodes are successful, show no issues.

Steve (submitted on 6/15/2009)

Steve was on vacation for the week of 6/8/09 - 6/13/09

  • Worked with Greg on resolving issue with ActiveMQ event broker where it disconnected during runs. This appears to be an issue with the broker itself, and a bug report has been filed. We have a possible work around for this issue, but implementing is on hold while runs are being completed.
  • Documented the ctrl_evmon code
  • Wrote the orchestration section for the report.

Ray (submitted on 6/18/2009)

  • Ran DC3a production runs; interspersed with continued debugging; recent results at wiki:DC3/DC3aProductionRuns
  • Working with Daues, prepped for runs on Abe (building stack; staging data)
  • Conducted detailed assessment of EUPS changes
  • DC3b planning via Middleware telecons
  • general middleware/infrastructure management
  • Travel:
    • DC3a Post-Mortem (Tucson, May 17-21)
    • IVOA Interop (Strasbourg, May 23-30)
    • Vacation (June 8-12)

LSST Corporation

Robyn (submitted on 6/12/2009)

Robyn took two weeks leave during period of 5/12/2009 - 6/15/2009. Primary focus during the other 2 weeks:

  • satisfying TCT requirement to develop list of 3rd party software usage, created the DM/ThirdPartySoftware table.
  • satisfying TCT requirement to prepare LSST Legal Notices for DM software licensing and copyrighting, created draft LSST Legal Notice and draft LSST Copyright.
  • started reviewing use of Parasoft C++Test for inclusion in LSST Build procedure.

Database Group

K-T (submitted on 6/18/2009)

2008-05-20 to 2009-06-17:

Major accomplishments:

  • Wrote up Data Access breakout from DC3b meeting.
  • Contributed to Build/Packaging? and Middleware breakout writeups.
  • Wrote provenance portion of DC3a report.
  • Drafted a proposal for Stage interfaces.
  • Wrote up and discussed redesign proposal for persistence framework.
  • Contributed to eups update checklist/proposal.
  • Researched hardware parameters for production database sizing.
  • Discussed SDS/DM interface with Mike Huffer.
  • Discussed scalable database implementation for PDR/DC3b.
  • #810: Discussed improving log messages and defined scope.
  • Discussed provenance for DC3b and defined scope.
  • Discussed image access for DC3b.
  • Pinned down problem with serializing denormalized floats on Mac OS.
  • #835: Lots of investigation, no firm conclusion yet.
  • Investigated pipeline failures from missing tables.

Minor accomplishments:

  • #818: Reported and did some investigation.
  • #793: Ruled out persistence as a cause.
  • #832: Did some investigation.
  • #834: Clarified NULL vs. NaN.
  • #807: Closed as non-repeatable.
  • #822: Discussed policy dictionaries.
  • Added initial code to generate subchunks to qserv/worker prototype.
  • Helped Andy Becker with running pipelines.
  • Helped clarify some photo-z schema issues.
  • Commented on data mapping document.
  • Discussed how to avoid event broker collisions.

Andy H (submitted on 6/xx/2009)

Serge (submitted on 6/21/2009)

  • Investigated AP failures arising from missing tables
  • Fixes to association pipeline to avoid repeated failures after a timed out visit
  • Discussed image access and scalable query architecture for DC3b

Daniel (submitted on 6/15/2009)

  • Attended DC3b meeting in Tucson
  • Discussed Gearman/drizzle possibilities for handling parallel queries
  • Scalable queries:
    • More testing with qserv framework
    • worked out user-level permission scheme
    • fixed query to include placeholder for subchunk number substitution
    • workaround lsst stack installation without proxy or outside network access.
  • Scidb:
    • Worked on udf api spec, convergence of udf api spec as operator.
    • Re-implemented more pieces of udf cooking algorithm using new api plus test harness.

Jacek (submitted on 6/12/2009)

  • lots of meetings
    • attended DC3b planning meeting in Tucson
      • including preparing, leading Data Products breakout and writing notes
    • attended OCS-DM Interfaces meeting in Tucson
    • attended DC3b planning meeting in Pasadena
      • including preparing scalable arch plan
    • attended SDQA meeting in Pasadena
      • including preparing db usecases
    • meeting with Drizzle/Gearman? developers
    • meeting with Andy Lawrence, discussing UK involvement
    • organized several DataAccWG telecons
    • progress with io estimates / infrastructure telecon
  • scalability tests: feasibility analysis
  • DC3a report - wrote my sections
  • db authorization related troubleshooting
  • XLDB
    • invites (compiling list, email correspondence)
    • slac paperwork related to international travel (for all going from slac)
  • SciDB
    • redesigned scidb.org released!
      • includes lots of last minute QA and cleanup
    • gave SciDB seminar at SDSC
    • meeting with Paul B - new developer
    • demo telecons

Major Accomplishments

Significant breakthroughs, issues resolved.



Ray (submitted on 6/18/2009)

Objectives for the Next Period

What you expect to accomplish.

NCSA

Ray (submitted on 6/18/2009)

  • DC3b planning
  • Large scale production runs on Abe covering entire focal plane

LSSTC

Robyn Allsman

During my summer part-time period, I will be working on the following, in the order shown:

  • complete preparation of the LSST copyright/license for inclusion in the LSST DM Legal Notices directory and prepare Copyright notice to be inserted in each source file.
  • continue work on Parasoft Coding Standards integration.

Database Group



Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues.

NCSA

LSSTC

Database Group