wiki:MWInfReport061510

Version 9 (modified by danielw, 9 years ago) (diff)

update for Daniel Wang

Previous month Apps Middleware/Infrastructure - Current Apps - Next month Apps Middleware/Infrastructure

Middleware/Infrastructure? Status Report for June 15, 2010

Allsman Robyn

Reported xx/xx/xx

IDPercentTaskComments
MWI8270%prototype buildbot script to run pipeline unittest not started

*Percent updated in this report

Comments:

Other Activities

Axelrod Tim

Reported xx/xx/xx

IDPercentTaskComments
MWI8340%Select CHFT Test Datanot started

*Percent updated in this report

Comments:

Other Activities

Baker Bill

Reported 06/21/10

IDPercentTaskComments
MWI56920%Maintain automated testing structure
MWI4125%Support distributable VMs for new developers
MWI85250%Web interface to Parasoft
MWI7450%configure test sytem for automated V&V
MWI63725%Automated standards checking script
MWI6380%Buildbot to detect ABI changes on trunknot started
MWI6610%Advanced interslice communications supportnot started
MWI6620%Machine-readable inputs/outputs manifestnot started
MWI70725%Automating nightly DC3b run

*Percent updated in this report

Comments:

Other Activities

I'm transitioning off of LSST, with two tasks to focus on:

  • finishing the security summary for PDR (an executive summary of the security document)
  • continuing to support nightly builds (buildbot)

Ergo, my list of tasks will probably stay where it is. There are some half-finished projects in there -- mainly the web interface to Parasoft -- that I can either hand over to someone else or just put into a dormant mode for now.

Becker Andy

Reported xx/xx/xx

IDPercentTaskComments
MWI8380%Chop CFHT data by ampnot started

*Percent updated in this report

Comments:

Other Activities

Becla Jacek

Reported 06/18/10

IDPercentTaskComments
MWI6810%Updates: exposure-related tablesnot started
MWI69120%Partitioning: design partitioning of MovingObject? table
MWI69375%Tools for partitioning data
MWI83190%qserv: MyProxy?
MWI83275%DB backup and replication
MWI70025%Benchmarking
MWI16025%capturing full SW/HW info
MWI1590%demonstrating re-creation of Calibrated Science Images not started
MWI16541%support persistence and retrieval of deep detection products and inputs
MWI46540%implement objectId synchronization between data releases (not DC3)

*Percent updated in this report

Comments:

  • lsst
    • qserv plan update
    • reddnet servers - logistics at slac incl security issues, server location, paperwork etc
    • reached out to students at Stanford and arranged one for the lsst-db group (part-time). Starts June 21
  • xldb4
    • finished website contents and released website
    • compiled list of speakers, invitees for the workshop
    • discussions with potential sponsors
  • scidb
    • changes to website
    • presentation for the ldrd proposal review committee
    • nsf proposal planning
  • slac - yearly performance reviews

Note: I am on vacation since June 09

Other Activities

Bosch Jim

Reported xx/xx/xx

IDPercentTaskComments
MWI6855%Near neighbor query over 2 or 3 nodes
MWI6960%DC3b syntax parser and aggregationnot started

*Percent updated in this report

Comments:

Other Activities

Cribbs Chris

Reported xx/xx/xx

IDPercentTaskComments

*Percent updated in this report

Comments:

Other Activities

Daues Greg

Reported xx/xx/xx

IDPercentTaskComments
MWI65650%Pipeline shutdown mechanisms
MWI8530%Adding run filter for shutdown eventsnot started
MWI6640%Freeze-drying clipboards: Harness supportnot started
MWI1060%Work to support communication of general serializable types over MPI not started
MWI1070%Develop mechanisms for identifying/accessing not started
MWI1130%Improving harness exception handlingnot started
MWI70925%Improve simple, interactive harness execution: interface

*Percent updated in this report

Comments:

Other Activities

Freemon Mike

Reported xx/xx/xx

IDPercentTaskComments
MWI8490%Prepare LSST10 for primary catalog databasenot started

*Percent updated in this report

Comments:

Other Activities

Gehrig David

Reported 06/21/10

IDPercentTaskComments
MWI72325%Image file access server ready
MWI724100%*Unify account management procedures - svn mysql linux svn
MWI72230%Prepare LSST2 for iRODS
MWI84825%*Prepare LSST1 as web data server / schema browser
MWI84925%*Prepare LSST10 for primary catalog databasenot started
MWI837100%*Download CFHT Data

*Percent updated in this report

Comments:

Other Activities

*Migrating data from lsst cluster to mass storage *Moved data browser from lsstbook to lsst1

Comments:

Other Activities

Hanushevsky Andy

Reported xx/xx/xx

IDPercentTaskComments
MWI16541%support persistence and retrieval of deep detection products and inputs

*Percent updated in this report

Comments:

Other Activities

Lim KT

Reported xx/xx/xx

IDPercentTaskComments
MWI69120%Partitioning: design partitioning of MovingObject? table
MWI16025%capturing full SW/HW info
MWI1590%demonstrating re-creation of Calibrated Science Images not started
MWI16541%support persistence and retrieval of deep detection products and inputs
MWI46540%implement objectId synchronization between data releases (not DC3)
MWI1800%persistence framework refinementnot started
MWI6430%Upgrade OutputStage? to register outputsnot started
MWI6440%Implement simple, non-Policy retrievalnot started
MWI6450%App Class: remove need to declare friend classesnot started
MWI6460%App Class: remove intrusiveness of Boost persistencenot started
MWI6470%App Class: legacy file-based persistence/retrievalnot started
MWI6480%Python pers: investigate efficiency of methodsnot started
MWI6490%Python pers: define app class interfacenot started
MWI6500%Implement simple non-Policy persistencenot started
MWI6510%Upgrade InputStage?not started
MWI6520%Generic ORM: mapping attributes to columnsnot started
MWI6530%Generic ORM: collection persistence and retrievalnot started
MWI6630%Freeze-drying clipboards: designnot started
MWI6730%Determine, retrieve cutout areas over given sky tilenot started
MWI6750%Write to co-add template pixel store on sky tile basisnot started
MWI6740%Determine template pixels overlaying given coordsnot started
MWI6760%Retrieve template pixelsnot started
MWI6770%Investigate local disk use for postage stamp generationnot started

*Percent updated in this report

Comments:

Other Activities

Monkewitz Serge

Reported xx/xx/xx

IDPercentTaskComments
MWI69025%Partitioning: research htm/healpix/stomp/dif
MWI6920%Implement generic query partitionernot started
MWI69375%Tools for partitioning data
MWI6960%DC3b syntax parser and aggregationnot started
MWI85415%Spatial support for qserv and corresponding MySQL UDFs
MWI46540%implement objectId synchronization between data releases (not DC3)
MWI6690%Determine segment images over a given sky tilenot started

*Percent updated in this report

Comments:

Other Activities

Pietrowicz Steve

Reported 06/18/2010

IDPercentTaskComments
MWI6580%I/O stage for interactive pipeline launchnot started
MWI65975% *Refactor Orca currently in testing for Abe
MWI5380%Event Monitor integrationnot started
MWI5890%Pipeline Fault Tolerancenot started
MWI57075% *Events orchestration layer support integrated and being used in orca and joboffice

*Percent updated in this report

Comments:

Other Activities

Plante Ray

Reported 06/18/2010

IDPercentTaskComments
MWI62475%User preferences directory
MWI62575%Support user/server tags
MWI6260%Improve packages installation/declarationnot started
MWI62725%Improve EUPS Docs
MWI6280%Meta releasesnot started
MWI6290%User publishing of packages to servernot started
MWI6310%As-built manifests into install directoriesnot started
MWI6320%Make SConsUtils dependency on EUPS optionalnot started
MWI6330%Upgrade SConsUtils for latest Sconsnot started
MWI6410%Logging tasksnot started
MWI84025%Update and test stack for latest product versions
MWI841100%*Job scheduling via JobOffice/Blackboard Basic functionality was in place and in use last month.
MWI6700%LRU cache for raw segment imagesnot started
MWI6710%LRU cache for calibrated science exposuresnot started
MWI67250%*Write/retrieve images to disk/tapeGeneral purpose software will save new data to tape via scp; will switch to globus-url-copy for improved performance. This month, I have been concentrating on supporting the ImSim team and the lsst2 server.
MWI8420%Develop and implement file bundling for Mass Storenot started
MWI85575%*JobOffice Middleware, testing and debugCurrently integrating use of databutler to support SrcAssoc pipeline
MWI8480%Prepare LSST1 as web data server / schema browsernot started

*Percent updated in this report

Comments:

Note that as of May 15, I am now 50% on LSST.

Other Activities

PT1 integration activities
  • My three most important activities:
    • Supporting SrcAssoc with the JobOffice (see MWI855 above)
    • Final testing of orca for routine launching of productions on Abe (with Pietrowicz)
    • Migrating ImSim data from lsst2 to mass storage
  • lsst cluster disk space maintenance

Van Dyk Schuyler

Reported xx/xx/xx

IDPercentTaskComments
MWI72230%Prepare LSST2 for iRODS

*Percent updated in this report

Comments:

Other Activities

Wang Daniel

Reported 06/24/10

IDPercentTaskComments
MWI16542%*support persistence and retrieval of deep detection products and inputs

*Percent updated in this report

Comments:

  • qserv-related:
    • Added db context support in proxy/Lua and qserv parser.
    • Check for garbage queries in frontend so that they don't crash the system.
    • Test Object-Source table joining.
    • Add more error checking in parser to catch bad (though not garbage) queries.
    • Accept non-lower-case functions in SQL.
    • Refactor parser-driving code.
    • Start stabilizing the testing code.
    • Implement synthesizer for fake Source data from usnob tables.
    • Increase robustness of result-merging code.
    • Add db-access checks in sql parser.
    • Add basic query squashing so that really early failures can avoid useless work.
    • Handle errors better when query hints are missing/malformed.
    • Write high-level overview of qserv.
    • Add result-table specification in subqueries, and dump/transmit/reload tables instead of databases to avoid excessive overhead from creation/destruction of databases.
    • Improve mutex protection on qserv master.
    • Add basic objectId support in query hints.
    • Switch subquery mgmt to conditions rather than polling -- shortest queries are now ~1s.
    • qserv-worker:
      • Fix worker to create overlap tables when subchunking.
      • Some code cleanup.
      • Improve error handling to pass execution errors during result signalling so that errors propagate up to xrootd.
      • Improve query write-buffering.

Other Activities



Objectives for the Next Period

What you expect to accomplish.

NCSA

LSSTC

Database Group



Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues.

NCSA

LSSTC

Database Group