Status Report for September 15, 2009
Previous Month - Next Month Apps Middleware/Infrastructure
Guide on Reporting Percentage Completed:
For software development tasks,
- use 0% if the solution is still being designed;
- use 25% if the solution's design is completed;
- use 50% if the solution is implemented and tested within the Ticket branch;
- use 75% if the implementation review is complete, the Ticket branch has been merged onto the Trunk branch, and the Ticket is closed.
- use 100% if the affected Trunk packages are tagged and released.
For other tasks, still use these increments, but apply their meaning as best as possible.
Note: Please place an asterisk (*) after the percent if you updated this value in this report.
The "ID" number corresponds to the "Unique ID" column in the Middleware/Infrastructure/DB project plan, a PDF of which is attached at wiki:DC3bManagement.
Summary of progress
Work activity; whether it reached associated milestones.
Allsman Robyn
Reported xx/xx/xx
| ID | Percent | Task | Comments |
*Percent updated in this report
Comments:
Other Activities
* Started testing process/procedure/suitability of Automatic Coding Standards Checking when Tickets are designated as 'inStandardsCheck'.
Axelrod Tim
Reported xx/xx/xx
| ID | Percent | Task | Comments |
| 683 | 0% | Defining units | not started |
*Percent updated in this report
Comments:
Other Activities
Baker Bill
Reported 9/18/9
| ID | Percent | Task | Comments |
| 569 | 10% | Maintain automated testing structure | ongoing |
| 41 | 75%* | (BB) support distributable VMs for new developers | ongoing; latest VMs are ready & waiting for verification |
| 74 | 50%* | configure test sytem for automated V&V | software installed; Robyn is configuring |
| 637 | 0% | Automated standards checking script | not started |
| 638 | 0% | Buildbot to detect ABI changes on trunk | not started |
| 640 | 25%* | Validated policy files | implementation mostly complete |
| 661 | 0% | Advanced interslice communications support | not started |
| 662 | 0% | Machine-readable inputs/outputs manifest | not started |
*Percent updated in this report
Comments:
Other Activities
Becla Jacek
Reported xx/xx/xx
| ID | Percent | Task | Comments |
| 680 | 0% | Updates: source-related tables, Object table | discussions with App team will start Sep 18 |
| 681 | 0% | Updates: exposure-related tables | discussions with App team will start Sep 18 |
| 682 | 0% | Synthetic sources of data | not started |
| 684 | 25%* | Standardizing names | Implemented in schema, need to update formatters and test |
| 686 | 5%* | Evaluate hadoopDB | Discussed hadoopDB in details with hadoopDB developers |
| 687 | 25%* | MySQL: test scalability and architect | run some tests, still need to try at least binary bindings |
| 688 | 0% | Decide xrootd versus gearman | not started |
| 689 | 0% | Run near neighbor query on lsst10 | not started |
| 691 | 0% | Partitioning: design partitioning of MovingObject? table | not started |
| 693 | 0% | Tools for partitioning data | not started |
| 697 | 0% | Task manager: design presentation | not started |
| 700 | 0% | Benchmarking | not started |
| 701 | 0% | Data Release tools | not started |
| 702 | 0% | Documentation | not started |
| 160 | 0% | capturing full SW/HW info | not started |
| 159 | 0% | demonstrating re-creation of Calibrated Science Images | not started |
| 165 | 0% | support persistence and retrieval of deep detection products and inputs | not started |
| 465 | 0% | implement objectId synchronization between data releases (not DC3) | not started |
*Percent updated in this report
Comments:
Other Activities
LSST
- organized 1 DataAccWG telecon
- documented spec for testbed and DC3b hardware, see dbDC3bHardware
VLDB/XLDB
- prepared for XLDB, coordinated all logistics pre- and post-meeting (agenda, dinner, sponsors, badges, handouts, conf room arrangements, attaching photos and slides to website, mail list announcements, many others...)
- attended VLDB, helped with the SciDB demo
- organized XLDB3
- all paperwork related to the trip
- arranged article for "SLAC Today".
- started writing xldb report
SciDB
- organizing face to face SciDB developers meeting at SLAC (coordinating all logistics, invitations for international collaborators)
- discussions with slac management about funding and future proposals
- discussions with slac lawyers and CFO about bringing external funds to the lab
- reaching out to geoscience, including abstract for AGU conference
- meeting with CTO of Persistent Systems
- meeting with VCs
Daues Greg
Reported xx/xx/xx
| ID | Percent | Task | Comments |
| 655 | 25%* | Refactor harness | Stage splitting design is done and now being implemented |
| 656 | 0% | Pipeline shutdown mechanisms | not started |
| 664 | 0% | Freeze-drying clipboards: Harness support | not started |
| 106 | 75% | Work to support communication of general serializable types over MPI | |
| 107 | 0% | Develop mechanisms for identifying/accessing | not started |
| 113 | 0% | Improving harness exception handling | not started |
| 660 | 25%* | Improve simple, interactive harness execution: infrastructure | Thread based Pipeline designed and now being implemented |
| 709 | 0% | Improve simple, interactive harness execution: interface | not started |
*Percent updated in this report
Comments:
Other Activities
- Stage splitting changes have been implemented in Pipeline and Slice. Two pex_harness examples have been transformed and tested, including tests of the NoOp functionality for the case where a serialClass or parallelClass is not provided.
- In the context of simple, interactive harness execution, a Pipeline launching Slices using Python Threads (no MPI) is working in a primitive form with the beginnings of thread synchronization (duplicating MPI barrier functionality with base Python Events)
Gehrig David
Reported 09/21/09
| ID | Percent | Task | Comments |
*Percent updated in this report
Comments:
Other Activities
- Worked with Suzi and Jeff re DC3b project plan
- Received training in Primavera project planning/costing software
- Upgraded Trac installation on dc33
- Created anonymous access to SVN per Lupton's request
- Additional account management for NCSA cluster
- Vacation Sept. 10-11
Hanushevsky Andy
Reported xx/xx/xx
| ID | Percent | Task | Comments |
| 165 | 0% | support persistence and retrieval of deep detection products and inputs | not started |
*Percent updated in this report
Comments:
Other Activities
Laher Russ
Reported xx/xx/xx
| ID | Percent | Task | Comments |
| 482 | 30% | C++ | |
| 481 | 0% | Python | not started |
*Percent updated in this report
Comments:
Other Activities
Lim KT
Reported xx/xx/xx
| ID | Percent | Task | Comments |
| 688 | 0% | Decide xrootd versus gearman | not started |
| 691 | 0% | Partitioning: design partitioning of MovingObject? table | not started |
| 698 | 0% | Task manager: multiple queries | not started |
| 699 | 0% | Task manager: interations with user | not started |
| 702 | 0% | Documentation | not started |
| 469 | *100% | DC3 provenance capture | |
| 160 | 0% | capturing full SW/HW info | not started |
| 159 | 0% | demonstrating re-creation of Calibrated Science Images | not started |
| 166 | *100% | support persistence and retrieval of nightly pipeline products and inputs | |
| 165 | 0% | support persistence and retrieval of deep detection products and inputs | Design started |
| 465 | 0% | implement objectId synchronization between data releases (not DC3) | Not DC3 task |
| 180 | 0% | persistence framework refinement | Design started |
| 643 | 0% | Upgrade OutputStage? to register outputs | Design started |
| 644 | 0% | Implement simple, non-Policy retrieval | Design started |
| 645 | 0% | App Class: remove need to declare friend classes | DC3b stretch goal |
| 646 | 0% | App Class: remove intrusiveness of Boost persistence | DC3b stretch goal |
| 647 | 0% | App Class: legacy file-based persistence/retrieval | DC3b stretch goal |
| 648 | 0% | Python pers: investigate efficiency of methods | DC3b stretch goal |
| 649 | 0% | Python pers: define app class interface | DC3b stretch goal |
| 650 | 0% | Implement simple non-Policy persistence | DC3b stretch goal |
| 651 | 0% | Upgrade InputStage? | DC3b stretch goal |
| 652 | 0% | Generic ORM: mapping attributes to columns | DC3b stretch goal |
| 653 | 0% | Generic ORM: collection persistence and retrieval | DC3b stretch goal |
| 663 | 0% | Freeze-drying clipboards: design | Design started |
| 669 | 0% | Determine segment images over a given sky tile | Prototype started |
| 670 | 0% | LRU cache for raw segment images | not started |
| 671 | 0% | LRU cache for calibrated science exposures | not started |
| 672 | 0% | Write/retrieve images to disk/tape | not started |
| 673 | 0% | Determine, retrieve cutout areas over given sky tile | Prototype started |
| 675 | 0% | Write to co-add template pixel store on sky tile basis | Prototype started |
| 674 | 0% | Determine template pixels overlaying given coords | Prototype started |
| 676 | 0% | Retrieve template pixels | not started |
| 677 | 0% | Investigate local disk use for postage stamp generation | not started |
*Percent updated in this report
Comments:
Other Activities
Monkewitz Serge
Reported xx/xx/xx
| ID | Percent | Task | Comments |
| 690 | 0% | Partitioning: research htm/healpix/stomp/dif | not started |
| 692 | 0% | Implement generic query partitioner | not started |
| 693 | 0% | Tools for partitioning data | not started |
| 696 | 0% | DC3b syntax parser and aggregation | not started |
| 465 | 0% | implement objectId synchronization between data releases (not DC3) | not started |
*Percent updated in this report
Comments:
Other Activities
Pietrowicz Steve
Reported 09/15/09
| ID | Percent | Task | Comments |
| 665 | 75% | Expose Event API | |
| 228 | 0% | Event system failover | not started |
| 666 | 0% | Scripts to monitor for node/pipeline failures | not started |
| 667 | 0% | Python-based high-level event monitor scripts | not started |
| 658 | 0% | I/O stage for interactive pipeline launch | not started |
| 659 | 50% * | Refactor Orca | in branch, in testing phase |
| 229 | 0% | life-support services | not started |
| 538 | 0% | Event Monitor integration | not started |
| 570 | 0% | Events orchestration layer support | not started |
*Percent updated in this report
Comments:
Other Activities
Plante Ray
Reported xx/xx/xx
| ID | Percent | Task | Comments |
| 623 | 100%* | Summarize design for EUPS changes | Reviewed 9/2; see summary |
| 624 | 25%* | User preferences directory | Implementation is under way, 2/3 done |
| 625 | 25%* | Support user/server tags | Implementation is under way, 2/3 done |
| 626 | 0% | Improve packages installation/declaration | Design has been started |
| 627 | 25%* | Improve EUPS documentation | Mostly in the form of in-line python docs; user docs will not start until after Oct. 1 |
| 628 | 0% | Meta releases | not started |
| 629 | 0% | User publishing of packages to server | not started |
| 631 | 0% | As-built manifests into install directories | not started |
| 632 | 0% | Make SConsUtils dependency on EUPS optional | not started |
| 633 | 0% | Upgrade SConstUtils for latest SCons | not started |
| 641 | 0% | Logging tasks | Harness refactoring/scripting tasks depend on this; not started |
| 654 | 100%* | Update UML + docs for STage code usage | Reviewed 9/9; see summary |
| 111 | 100%* | Fill out design for orchestration layer | Done. see attached design documents to Orca refactor task page |
*Percent updated in this report
Other Activities
- assisted Gehrig with Data Challenge planning
- ran two Middleware telecons
- assisted Cribbs and Freemon with infrastructure cost spreadsheets
- participated in SAT: assisted with DC3b scoping
- assisted with hardware requirements/resource development
Other Activities
Wang Daniel
Reported xx/xx/xx
| ID | Percent | Task | Comments |
| 685 | 5%* | Near neighbor query over 2 or 3 nodes | Implementing parser updates (aliasing, stored procedure, subquery). |
| 688 | 0% | Decide xrootd versus gearman | not started |
| 694 | 0% | Basic generic query parser in place | not started |
| 695 | 0% | Demonstrate joining Object table with non-partitioned tables | not started |
| 696 | 0% | DC3b syntax parser and aggregation | not started |
| 698 | 0% | Task manager: multiple queries | not started |
| 699 | 0% | Task manager: interactions with user | not started |
| 701 | 0% | Data Release tools | not started |
| 702 | 0% | Documentation | not started |
*Percent updated in this report
Comments:
Other Activities
- SciDB
- Added no-nesting hack to cooking code
- Diagnosed Q5 problems
- Attended VLDB
- Fixed/helped-fix some last minute issues with SciDB for demo
- Staffed demo sessions, collected questions and feedback
- Attended XLDB
- Took photos,
- Moderated geoscience session
- Presented introduction to benchmark
- Various travel paperwork
(Vacation 8/31-9/2, 9/14-9/15)
Major Accomplishments
Significant breakthroughs, issues resolved.
Objectives for the Next Period
What you expect to accomplish.
NCSA
Ray Plante
- finishing Logging task
- finish EUPS implementation changes
LSSTC
Database Group
Problems Encountered and Solutions Being Pursued
Budget or schedule variance, technical issues, management issues.
NCSA
Ray Plante
I am encouraging in our group greater use of Trac in the documentation of progress on tasks. Several of us are maintaining pages like the following as a place to gather notes, design issues, and progress:
LSSTC
Database Group
