wiki:MWInfReport081510
Last modified 9 years ago Last modified on 08/26/2010 09:23:57 AM

Previous month Apps - MW/Inf - Mgt/SE

Current month Apps - Mgt/SE

Next month Apps - MW/Inf - Mgt/SE

Guide on Reporting Percentage Completed:

For software development tasks,

  • use 0% if the solution is still being designed;
  • use 25% if the solution's design is completed;
  • use 50% if the solution is implemented and tested within the Ticket branch;
  • use 75% if the implementation review is complete, the Ticket branch has been merged onto the Trunk branch, and the Ticket is closed.
  • use 100% if the affected Trunk packages are tagged and released.

For other tasks, still use these increments, but apply their meaning as best as possible.

Note: Please place an asterisk (*) after the percent if you updated this value in this report.

Middleware/Infrastructure? Status Report for August 15, 2010

Allsman Robyn

Reported xx/xx/xx

IDPercentTaskComments
MWI827100%*prototype buildbot script to run pipeline unittest

*Percent updated in this report

Comments:

  • There was no need to modify the DM buildbot infrastructure in order to run either the individual pipelines' unittests or the PT1 production pipeline unittest (an integration test of the individual pipelines); all that was needed was to add the package 'datarel' to buildbot's task list. The actual test was written by K-T & RHL. On request, Bill Baker added the package to the buildbot task list.
  • Last week I was also designated as Bill Baker's replacement on DM buildbot development.
  • As described in the Apps Monthly report, I am currently working on developing a new style of buildbot test which checks consistency against the development branch instead of the released branch. This job does involve modification to the DM buildbot implementation.

Other Activities

Baker Bill

Reported 8/21/10

IDPercentTaskComments
MWI56920% *Maintain automated testing structure Handing off to Robyn Allsman
MWI4125%Support distributable VMs for new developers
MWI85250%Web interface to Parasoft Spinning back up after long absence
MWI7450%configure test sytem for automated V&V
MWI63725%Automated standards checking script
MWI6380%Buildbot to detect ABI changes on trunknot started
MWI6610% *Advanced interslice communications support Handed off to Greg Daues
MWI6620%Machine-readable inputs/outputs manifestnot started
MWI70725% *Automating nightly DC3b run Handed off to Robyn Allsman

*Percent updated in this report

Comments:

Other Activities

Becla Jacek

Reported 08/23/10

IDPercentTaskComments
MWI6810%Updates: exposure-related tablesnot started
MWI69120%Partitioning: design partitioning of MovingObject? table
MWI69375%Tools for partitioning data
MWI83190%qserv: MyProxy?
MWI83275%DB backup and replication
MWI70025%Benchmarking
MWI16025%capturing full SW/HW info
MWI1590%demonstrating re-creation of Calibrated Science Images not started
MWI16594%support persistence and retrieval of deep detection products and inputs
MWI46540%implement objectId synchronization between data releases (not DC3)

*Percent updated in this report

Comments:

Other Activities

  • qserv - plan update
  • scalability tests plan - matrix
  • prepared for SSD tests at SDSC
  • reddnet servers and db servers at slac - paperwork and coordination
  • AHM
  • adass abstract
  • xldb
    • reaching out, arranging speakers, sponsors, memorabilia, and others
  • scidb
    • organizing 1st community meeting

Bosch Jim

Reported xx/xx/xx

IDPercentTaskComments
MWI6855%Near neighbor query over 2 or 3 nodes

*Percent updated in this report

Comments:

Other Activities

Daues Greg

Reported 08/24/10

IDPercentTaskComments
MWI65650%Pipeline shutdown mechanisms
MWI8530%Adding run filter for shutdown eventsnot started
MWI6640%Freeze-drying clipboards: Harness supportnot started
MWI1060%Work to support communication of general serializable types over MPI not started
MWI1070%Develop mechanisms for identifying/accessing not started
MWI1130%Improving harness exception handlingnot started
MWI70925%Improve simple, interactive harness execution: interface

*Percent updated in this report

Comments:

  • Worked in ticket 1398 to improve harness logging with respect to 1) formatting messages properly for event monitor system and 2) improve error detection reporting for faster traceback at middleware level.
  • Initiated running productions on Abe for middleware testing
  • Investigated TACC (Texas Advanced Computing Center) Ranger system as a backup computing system for PT1/2

Reported xx/xx/xx

IDPercentTaskComments

*Percent updated in this report

Comments:

Other Activities

Gehrig David

Reported xx/xx/xx

IDPercentTaskComments
MWI470%build warnings if unittest doesn't satisfy DC3 coverage requirement not started
MWI520%any failures cause automatic 'Failed Nightly Build' ticket at high priority not started
MWI540%any failure causes automatic 'Failed Production sanity check' ticket at high prioritynot started
MWI550%auxiliary automatic notification to highest authority on failure (separate from ticket notification) not started
MWI570%Standards: specify requirement to version and archive build's provenance;not started
MWI580%Configuration Management: define procedure to satisfy that requirement;not started
MWI590%Configuration Management: implement its instantiation. not started
MWI72325%Image file access server ready
MWI72230%Prepare LSST2 for iRODS
MWI84875%Prepare LSST1 as web data server / schema browser
MWI8590%Web Data server readynot started
MWI8340%Select CHFT Test Datanot started
MWI83750%Download CFHT Data

*Percent updated in this report

Comments:

Other Activities

Hanushevsky Andy

Reported xx/xx/xx

IDPercentTaskComments
MWI16594%support persistence and retrieval of deep detection products and inputs

*Percent updated in this report

Comments:

Other Activities

Lim KT

Reported xx/xx/xx

IDPercentTaskComments
MWI69120%Partitioning: design partitioning of MovingObject? table
MWI16025%capturing full SW/HW info
MWI1590%demonstrating re-creation of Calibrated Science Images not started
MWI16594%support persistence and retrieval of deep detection products and inputs
MWI46540%implement objectId synchronization between data releases (not DC3)
MWI1800%persistence framework refinementnot started
MWI6430%Upgrade OutputStage? to register outputsnot started
MWI6440%Implement simple, non-Policy retrievalnot started
MWI6450%App Class: remove need to declare friend classesnot started
MWI6460%App Class: remove intrusiveness of Boost persistencenot started
MWI6470%App Class: legacy file-based persistence/retrievalnot started
MWI6480%Python pers: investigate efficiency of methodsnot started
MWI6490%Python pers: define app class interfacenot started
MWI6500%Implement simple non-Policy persistencenot started
MWI6510%Upgrade InputStage?not started
MWI6520%Generic ORM: mapping attributes to columnsnot started
MWI6530%Generic ORM: collection persistence and retrievalnot started
MWI6630%Freeze-drying clipboards: designnot started
MWI6730%Determine, retrieve cutout areas over given sky tilenot started
MWI6750%Write to co-add template pixel store on sky tile basisnot started
MWI6740%Determine template pixels overlaying given coordsnot started
MWI6760%Retrieve template pixelsnot started
MWI6770%Investigate local disk use for postage stamp generationnot started

*Percent updated in this report

Comments:

Other Activities

Monkewitz Serge

Reported xx/xx/xx

IDPercentTaskComments
MWI69025%Partitioning: research htm/healpix/stomp/dif
MWI6920%Implement generic query partitionernot started
MWI69375%Tools for partitioning data
MWI46540%implement objectId synchronization between data releases (not DC3)
MWI6690%Determine segment images over a given sky tilenot started

*Percent updated in this report

Comments:

Other Activities

Pietrowicz Steve

Reported 08/23/10

IDPercentTaskComments
MWI6580%I/O stage for interactive pipeline launchnot started
MWI65975%Refactor Orca
25%life-support services
MWI5380%Event Monitor integrationnot started
MWI5890%Pipeline Fault Tolerancenot started
MWI57075%Events orchestration layer support

*Percent updated in this report

Comments:

Worked on issues for the startup script supplied by the user for orca runs. Added an enhancement to job office to start processing of data sooner. Both of these shortened the length of time the processing started by a considerable amount.

I was on vacation for the last two weeks.

Other Activities

Plante Ray

Reported 08/24/10

IDPercentTaskComments
MWI62475%User preferences directory
MWI62575%Support user/server tags
MWI6260%Improve packages installation/declarationnot started
MWI62725%Improve EUPS Docs
MWI6280%Meta releasesnot started
MWI6290%User publishing of packages to servernot started
MWI6310%As-built manifests into install directoriesnot started
MWI6320%Make SConsUtils dependency on EUPS optionalnot started
MWI6330%Upgrade SConsUtils for latest Sconsnot started
MWI6410%Logging tasksnot started
MWI84025%Update and test stack for latest product versions
MWI67050%LRU cache for raw segment images
MWI67150%LRU cache for calibrated science exposures
MWI67275%*Write/retrieve images to disk/tape I have software for pulling and putting; I just need to package some convenient tools.
MWI842100%*Develop and implement file bundling for Mass Store This is related to MWI672; due to my reduced availabilty during PT1, multiple ways of bundling data have crept into MSS, but we can deal with it.
MWI85575%JobOffice? Middleware, testing and debug I've picked up work on this again.
MWI848100%*Prepare LSST1 as web data server / schema browser data is now available via a web service on lsst1

*Percent updated in this report

Comments:

Other Activities

  • Attended All Hands Meeting 8/9-13.
  • Hosted visit from John Stevenson of Purdue to talk about running ImSim on GPU clusters at NCSA. We concluded that NCSA public and experimental platforms are an excellent match to the ImSim code.

Van Dyk Schuyler

Reported xx/xx/xx

IDPercentTaskComments
MWI72230%Prepare LSST2 for iRODS

*Percent updated in this report

Comments:

Other Activities

Wang Daniel

Reported 08/23/10

IDPercentTaskComments
MWI6961%*DC3b syntax parser and aggregationaggregation complete, lsst10 query logs to be processed
MWI16594%support persistence and retrieval of deep detection products and inputs working on ~100gb-data/many-node scalability, PT1a query support

*Percent updated in this report

Comments:

  • Qserv Development
    • Finish query sqashing on master and worker
    • Add/update basic commenting
    • Add support for convex hulls using geom pkg
    • (worker) fix various bugs that affected query squashing.
    • Implementing better configuration hooks to simplify things and aid testing multiple logical databases.
    • minor code cleanup
  • Testing
    • Implement data duplicator for PT1 data.

Status of PT1 query support: Parsing and execution appears robust, but query logs from lsst10 have not yet been sorted, characterized, and tested on qserv for syntax support. Deferred while scale tests are being done.

Other Activities

  • Attended AHM
  • Coordinate hive-testing.
  • qserv wiki updates

Objectives for the Next Period

What you expect to accomplish.

NCSA

LSSTC

Database Group



Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues.

NCSA

LSSTC

Database Group