wiki:MWInfReport051510
Last modified 9 years ago Last modified on 05/20/2010 12:09:31 PM

Previous month Apps Middleware/Infrastructure - Current Apps - Next month Apps Middleware/Infrastructure

Middleware/Infrastructure? Status Report for May 15, 2010

Guide on Reporting Percentage Completed:

For software development tasks,

  • use 0% if the solution is still being designed;
  • use 25% if the solution's design is completed;
  • use 50% if the solution is implemented and tested within the Ticket branch;
  • use 75% if the implementation review is complete, the Ticket branch has been merged onto the Trunk branch, and the Ticket is closed.
  • use 100% if the affected Trunk packages are tagged and released.

For other tasks, still use these increments, but apply their meaning as best as possible.

Note: Please place an asterisk (*) after the percent if you updated this value in this report.

Allsman Robyn

Reported xx/xx/xx

IDPercentTaskComments
MWI8270%prototype buildbot script to run pipeline unittest not started

*Percent updated in this report

Comments:

Other Activities

Baker Bill

Reported 4/19/10

IDPercentTaskComments
MWI56920%Maintain automated testing structureavg 3-5 hours a week
MWI4125%Support distributable VMs for new developers
MWI7450%configure test sytem for automated V&V
MWI63725%Automated standards checking scripthave spent about 50 hours on this
MWI6380%Buildbot to detect ABI changes on trunknot started
MWI6610%Advanced interslice communications supportnot started
MWI6620%Machine-readable inputs/outputs manifestnot started
MWI70725%Automating nightly DC3b runhave spent about 5 hours so far

*Percent updated in this report

Comments:

Other Activities

Time spent lately (I'm 50%, averaging 20-25 hours per week on LSST):

  • pex_policy - new features, maintenance, bug fixes - avg 5-8 hours a week for the last few months
  • latest big time sink has been developing a web interface for Parasoft -- spent 40-50 hours on it so far, 75% finished
  • Maintain buildbot (automated testing infrastructure) - about 3-5 hours a week
  • Misc (meetings, email, small tasks) - maybe 3-5 hours per week

Becker Andy

Reported xx/xx/xx

IDPercentTaskComments
MWI8380%Chop CFHT data by ampnot started

*Percent updated in this report

Comments:

Other Activities

Becla Jacek

Reported 05/20/10

IDPercentTaskComments
MWI68175%Updates: exposure-related tables
MWI69120%Partitioning: design partitioning of MovingObject? table
MWI69375%Tools for partitioning data
MWI83190%*qserv: MyProxy?
MWI83275%*DB backup and replication
MWI70025%Benchmarking
MWI16025%capturing full SW/HW info
MWI1590%demonstrating re-creation of Calibrated Science Images not started
MWI16541%support persistence and retrieval of deep detection products and inputs
MWI46540%implement objectId synchronization between data releases (not DC3)

*Percent updated in this report

Comments:

Other Activities

  • LSST
    • dc3b: updates to db scripts for ctrl/orca
    • dc3b: updates to db schema (sources/object)
    • dc3b: setup dc3b database
    • changes to mysql proxy to allow concurrent queries
    • proposed backup and replication plan, see dbBackupRepl?
    • discussions about mops architecture
  • XLDB4
    • website design and contents, agenda, dinner location, lots of other logistics
  • SciDB
    • website updates and big migration to a new hosting provider

Daues Greg

Reported 05/20/10

IDPercentTaskComments
MWI65650%Pipeline shutdown mechanisms
MWI6640%Freeze-drying clipboards: Harness supportnot started
MWI1060%Work to support communication of general serializable types over MPI not started
MWI1070%Develop mechanisms for identifying/accessing not started
MWI11350%Improving harness exception handling
MWI70950%Improve simple, interactive harness execution: interface

*Percent updated in this report

Comments:

  • For MWI656, within the context of ticket #1259, added support for the ShutdownThread? using event objects and filtering with runid. This is a crucial addition to prevent shutdown events from unintentionally affecting other running pipelines.
  • For MWI113, with repect tp tickets 1271/1272, added immediate Clipboard cleanup & deletion after failure stage execution to prevent any further use of that Clipboard.
  • Within the context of ticket #1259 added support to pex_harness for a command line option on run scripts to specify a unique log directory. Each worker Pipeline within a workflow/production needs to write to a unique location, and this is now ready and working within tag 3.6.8, having been tested with ctrl_orca.

Other Activities

Freemon Mike

Reported xx/xx/xx

IDPercentTaskComments
MWI8490%Prepare LSST10 for primary catalog databasenot started

*Percent updated in this report

Comments:

Other Activities

Gehrig David

Reported xx/xx/xx

IDPercentTaskComments
MWI72325%*Image file access server ready
MWI72425%*Unify account management procedures - svn mysql linux svn
MWI72290%Prepare LSST2 for iRODS
MWI84830%Prepare LSST1 as web data server / schema browser
MWI8490%Prepare LSST10 for primary catalog databasenot started
MWI83750%*Download CFHT Data

*Percent updated in this report

Comments:

Other Activities

  • Updated MWI project plan reflecting design review scoping changes

Hanushevsky Andy

Reported xx/xx/xx

IDPercentTaskComments
MWI16541%support persistence and retrieval of deep detection products and inputs

*Percent updated in this report

Comments:

Other Activities

Lim KT

Reported xx/xx/xx

IDPercentTaskComments
MWI69120%Partitioning: design partitioning of MovingObject? table
MWI16025%capturing full SW/HW info
MWI1590%demonstrating re-creation of Calibrated Science Images not started
MWI16541%support persistence and retrieval of deep detection products and inputs
MWI46540%implement objectId synchronization between data releases (not DC3)
MWI1800%persistence framework refinementnot started
MWI6430%Upgrade OutputStage? to register outputsnot started
MWI6440%Implement simple, non-Policy retrievalnot started
MWI6450%App Class: remove need to declare friend classesnot started
MWI6460%App Class: remove intrusiveness of Boost persistencenot started
MWI6470%App Class: legacy file-based persistence/retrievalnot started
MWI6480%Python pers: investigate efficiency of methodsnot started
MWI6490%Python pers: define app class interfacenot started
MWI6500%Implement simple non-Policy persistencenot started
MWI6510%Upgrade InputStage?not started
MWI6520%Generic ORM: mapping attributes to columnsnot started
MWI6530%Generic ORM: collection persistence and retrievalnot started
MWI6630%Freeze-drying clipboards: designnot started
MWI6730%Determine, retrieve cutout areas over given sky tilenot started
MWI6750%Write to co-add template pixel store on sky tile basisnot started
MWI6740%Determine template pixels overlaying given coordsnot started
MWI6760%Retrieve template pixelsnot started
MWI6770%Investigate local disk use for postage stamp generationnot started

*Percent updated in this report

Comments:

Other Activities

Monkewitz Serge

Reported xx/xx/xx

IDPercentTaskComments
MWI69025%Partitioning: research htm/healpix/stomp/dif
MWI6920%Implement generic query partitionernot started
MWI69375%Tools for partitioning data
MWI6960%DC3b syntax parser and aggregationnot started
MWI46540%implement objectId synchronization between data releases (not DC3)
MWI669100%Determine segment images over a given sky tile
? 75%Spatial support for qserv and corresponding MySQL UDFs

*Percent updated in this report

Comments:

Other Activities

Pietrowicz Steve

Reported 05/20/10

IDPercentTaskComments
MWI6580%I/O stage for interactive pipeline launchnot started
MWI65950%Refactor Orca Released and being used in testing. Working on Abe Plug-ins. Time Spent: 2 weeks
MWI5380%Event Monitor integrationnot started
MWI5890%Pipeline Fault Tolerancenot started
MWI57050%* Events orchestration layer support Implemented in Orca and in pex_harness. Time Spent: 2 weeks

*Percent updated in this report

Comments:

Other Activities

  • Tagged and released several version of ctrl_orca and ctrl_events.
  • Worked with Plante and Daues for orca/pex_harness updates.
  • Helped Rallsman with afw infinite loop issue.
  • Helped Rallsman come up to speed with how to use Orca.
  • Created trac pages for debugging running processes with no debugger attached and how to use Orca.

Plante Ray

Reported 05/15/10

IDPercentTaskComments
MWI62475%User preferences directory
MWI62575%Support user/server tags
MWI6260%Improve packages installation/declarationnot started
MWI62725%Improve EUPS Docs
MWI6280%Meta releasesnot started
MWI6290%User publishing of packages to servernot started
MWI6310%As-built manifests into install directoriesnot started
MWI6320%Make SConsUtils dependency on EUPS optionalnot started
MWI6330%Upgrade SConsUtils for latest Sconsnot started
MWI64150%Logging tasks initial pattern for logging established; some debugging/refinement needed
MWI84025%Update and test stack for latest product versions
MWI84175%Job scheduling via JobOffice/Blackboard? still needs data butler integration to support source association. Time spent: 2 weeks development, 1 week testing/integration
MWI67025%LRU cache for raw segment images
MWI67125%LRU cache for calibrated science exposures
MWI67225%Write/retrieve images to disk/tape
MWI725100%Define directory structure Time spent: ~3 days
MWI8480%Prepare LSST1 as web data server / schema browsernot started

*Percent updated in this report

Comments:

Other Activities

  • IVOA Interoperability conference: May 13-21
  • Consulted with Pietrowicz and Daues on completion of orca/harness updates
  • Integration testing: testing and debug joboffice middleware for actual pipelines
  • package maintenance on cluster and Abe.

Wang Daniel

Reported 04/20/10

IDPercentTaskComments
MWI16541%support persistence and retrieval of deep detection products and inputs

*Percent updated in this report

Comments:

Other Activities



Objectives for the Next Period

What you expect to accomplish.

NCSA

LSSTC

Database Group



Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues.

NCSA

LSSTC

Database Group