wiki:MWInfReport121509
Last modified 9 years ago Last modified on 12/21/2009 01:54:59 PM

Previous month Apps Middleware/Infrastructure - Current Apps - Next month Apps Middleware/Infrastructure

Middleware/Infrastructure? Status Report for December 15, 2009

Guide on Reporting Percentage Completed:

For software development tasks,

  • use 0% if the solution is still being designed;
  • use 25% if the solution's design is completed;
  • use 50% if the solution is implemented and tested within the Ticket branch;
  • use 75% if the implementation review is complete, the Ticket branch has been merged onto the Trunk branch, and the Ticket is closed.
  • use 100% if the affected Trunk packages are tagged and released.

For other tasks, still use these increments, but apply their meaning as best as possible.

Note: Please place an asterisk (*) after the percent if you updated this value in this report.

Baker Bill

Reported 12/18/09

IDPercentTaskComments
MWI56920%Maintain automated testing structure
MWI4125%*(BB) support distributable VMs for new developersA VM candidate has been waiting for review since September. An updated version (which would still need to be reviewed) is waiting for a working release build.
MWI7450%configure test sytem for automated V&V
MWI6370%Automated standards checking scriptnot started
MWI6380%Buildbot to detect ABI changes on trunknot started
MWI640100%*Validated policy filesTagged and released, but without thorough review.
MWI6610%Advanced interslice communications supportnot started
MWI6620%Machine-readable inputs/outputs manifestnot started
MWI7070%Automating nightly DC3a runnot started

*Percent updated in this report

Comments:

Other Activities

Becla Jacek

Reported 12/18/09

IDPercentTaskComments
MWI68075%*Updates: source-related tables, Object tableCurrent iteration done, I'm expecting more updates, waiting for input from Robert/Tim?
MWI6810%Updates: exposure-related tablesnot started
MWI6820%Synthetic sources of datanot started
MWI68475%*Standardizing names Mostly done, some fine-tuning still needed, related to covariance, tai etc, input from application people needed
MWI686100%*Evaluate hadoopDB Done all we want to do at this point. Not worthwhile doing full evaluation, we are in close contact with the HadoopDB developers
MWI6910%Partitioning: design partitioning of MovingObject? tablenot started
MWI69375%Tools for partitioning data
MWI69725%*Task manager: design presentationStarted
MWI70025%*BenchmarkingStarted
MWI7010%Data Release toolsnot started
MWI7020%Documentationnot started
MWI1600%capturing full SW/HW infonot started
MWI1590%demonstrating re-creation of Calibrated Science Images not started
MWI1650%support persistence and retrieval of deep detection products and inputsnot started
MWI4650%implement objectId synchronization between data releases (not DC3)not started

*Percent updated in this report

Comments:

Other Activities

  • NSF review: db-related slides, dry-runs, participated in the review
  • prepared documentation describing scalable db architecture, including tests and planning
  • updated DC3b schema in EA, updated schema browser
  • various mysql scalability related tests to determine best setup for dc3b and beyond
  • wrote 1 page test description for SSD tests
  • served on the review panel for the DES database review
  • organized/attended lsst-slac meeting about SciDB
  • meeting with Luiz (Observatorio Nacional/Brazil?)
  • XLDB
    • finalized writing the xldb report (exec summary, formatting)
  • SciDB
    • transfered organization of AGU town hall meeting from me/KT to others from SciDB

Bosch Jim

Reported xx/xx/xx

IDPercentTaskComments
MWI6855%Near neighbor query over 2 or 3 nodes
MWI6940%Basic generic query parser in placenot started
MWI6950%Demonstrate joining Object table with non-partitioned tablesnot started
MWI6960%DC3b syntax parser and aggregationnot started
MWI6980%Task manager: multiple queriesnot started
MWI6990%Task manager: interations with usernot started
MWI7010%Data Release toolsnot started
MWI7020%Documentationnot started

*Percent updated in this report

Comments:

Other Activities

Cribbs Chris

Reported xx/xx/xx

IDPercentTaskComments

*Percent updated in this report

Comments:

Other Activities

Daues Greg

Reported 12/21/09

IDPercentTaskComments
MWI65650%*Pipeline shutdown mechanisms
MWI6640%Freeze-drying clipboards: Harness supportnot started
MWI1060%Work to support communication of general serializable types over MPI not started
MWI1070%Develop mechanisms for identifying/accessing not started
MWI1130%Improving harness exception handlingnot started
MWI66050%*Improve simple, interactive harness execution: infrastructure
MWI70925%Improve simple, interactive harness execution: interface

*Percent updated in this report

Comments:

  • Shutdown event mechanism for modes 1--4 have been tested in the Python threaded harness ticket 1014

Other Activities

Hanushevsky Andy

Reported 12/18/09

IDPercentTaskComments
MWI1650%support persistence and retrieval of deep detection products and inputsnot started

*Percent updated in this report

Comments:

Other Activities

Started work on modifying xrootd - needed for scalable db architecture (node and task management)

Lim KT

Reported xx/xx/xx

IDPercentTaskComments
MWI6910%Partitioning: design partitioning of MovingObject? tablenot started
MWI6980%Task manager: multiple queriesnot started
MWI6990%Task manager: interations with usernot started
MWI7020%Documentationnot started
MWI1600%capturing full SW/HW infonot started
MWI1590%demonstrating re-creation of Calibrated Science Images not started
MWI1650%support persistence and retrieval of deep detection products and inputsnot started
MWI4650%implement objectId synchronization between data releases (not DC3)not started
MWI1800%persistence framework refinementnot started
MWI6430%Upgrade OutputStage? to register outputsnot started
MWI6440%Implement simple, non-Policy retrievalnot started
MWI6450%App Class: remove need to declare friend classesnot started
MWI6460%App Class: remove intrusiveness of Boost persistencenot started
MWI6470%App Class: legacy file-based persistence/retrievalnot started
MWI6480%Python pers: investigate efficiency of methodsnot started
MWI6490%Python pers: define app class interfacenot started
MWI6500%Implement simple non-Policy persistencenot started
MWI6510%Upgrade InputStage?not started
MWI6520%Generic ORM: mapping attributes to columnsnot started
MWI6530%Generic ORM: collection persistence and retrievalnot started
MWI6630%Freeze-drying clipboards: designnot started
MWI6690%Determine segment images over a given sky tilenot started
MWI6700%LRU cache for raw segment imagesnot started
MWI6710%LRU cache for calibrated science exposuresnot started
MWI6720%Write/retrieve images to disk/tapenot started
MWI6730%Determine, retrieve cutout areas over given sky tilenot started
MWI6750%Write to co-add template pixel store on sky tile basisnot started
MWI6740%Determine template pixels overlaying given coordsnot started
MWI6760%Retrieve template pixelsnot started
MWI6770%Investigate local disk use for postage stamp generationnot started

*Percent updated in this report

Comments:

Other Activities

Monkewitz Serge

Reported xx/xx/xx

IDPercentTaskComments
MWI69025%Partitioning: research htm/healpix/stomp/dif
MWI6920%Implement generic query partitionernot started
MWI69375%Tools for partitioning data
MWI6960%DC3b syntax parser and aggregationnot started
MWI4650%implement objectId synchronization between data releases (not DC3)not started

*Percent updated in this report

Comments:

Other Activities

Pietrowicz Steve

Reported 12/17/09

IDPercentTaskComments
MWI66550%Expose Event API
MWI6660%Scripts to monitor for node/pipeline failuresnot started
MWI6670%Python-based high-level event monitor scriptsnot started
MWI22825%Event system failover
MWI6580%I/O stage for interactive pipeline launchnot started
MWI65950%Refactor Orca
MWI5380%Event Monitor integrationnot started
MWI5890%Pipeline Fault Tolerancenot started
MWI5700%Events orchestration layer supportnot started

*Percent updated in this report

Comments:

Currently working on getting condor/DAGman functionality integrated with Orca, which is part of the "Refactor Orca" work.

Other Activities

Plante Ray

Reported 12/21/09

IDPercentTaskComments
MWI62475%*User preferences directory Lupton and Plante have been testing this heavily.
MWI62575%*Support user/server tags Lupton and Plante have been testing this heavily.
MWI6260%Improve packages installation/declarationnot started
MWI62725%Improve EUPS Docs
MWI6280%Meta releasesnot started
MWI6290%User publishing of packages to servernot started
MWI6310%As-built manifests into install directoriesnot started
MWI6320%Make SConsUtils dependency on EUPS optionalnot started
MWI6330%Upgrade SConsUtils for latest Sconsnot started
MWI6410%Logging tasksnot started

*Percent updated in this report

Comments:

Other Activities

  • Preparation and participation in FDP Review (15-17 Dec, Tucson)
  • posted package releases to server, including new 3rd party packages (with non-trivial build mechanisms), Healpix and minuit2
  • Non-LSST travel: NRAO Charlottesville, 2-4 Dec.
  • hosted visit from F. Pierfederici (Harvard), 16-20 Nov.; with assistance from Daues and Pietrowicz, worked on adapting IPSD pipeline to process Magellan data (see successes below).

Major Accomplishments

Significant breakthroughs, issues resolved.

  • Through a collaboration with F. Pierfederici (Harvard), we successfully used the LSST IPSD pipline (using the updated pex_harness and application packages) to calibrate Magellan images. We have also discussed plans for changes that would make LSST software more usable by outside projects.


Objectives for the Next Period

What you expect to accomplish.

NCSA

LSSTC

Database Group



Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues.

NCSA

LSSTC

Database Group