wiki:Summer2012/WishList
Last modified 7 years ago Last modified on 03/29/2012 06:46:58 PM

THIS PAGE IS NOW FROZEN 3/16/2012 (by convention only)

  • There will soon be a new plan published for Summer2012 that replaces this page
  • Any desired changes to this page but before the plan is published can be emailed to Mike
  • There is now a wishlist page for Winter2012 that can be used to record this kind of information for that timeframe Winter2013/WishList

*

All tasks by anyone in DM during S2012 period should be part of single plan

More detailed definitions of these activities can be found at Summer2012/Deliverables

Applications

Immediate "start-up" tasks:

  • Stage S82 at NCSA
    • WHO: MJ + NCSA support
  • Write a butler mapper for S82
    • WHO: may have been done at UW, or ask K-T, or Paul if K-T is busy?
  • cameraGeom for SDSS
    • WHO: check if UW has done it; RHL otherwise
  • Run a weekly on S82
    • WHO: ? (needs NCSA support)

Tasks:

  • Improve background estimation
    • WHO: Simon, Martin
    • DURATION: 3 months
    • DEPENDENCIES: partial dep. on 'Co-addition of images' task
  • Extended source photometry on single exposures
    • WHO: Jim Bosch, Martin(?), Serge(?)
    • DURATION: 3 months
    • DEPENDENCIES: ImSim inputs(?)
  • Co-addition of images
    • WHO: Russell, w. help from Steve Bickerton w. HSC side
    • DURATION: 3 months (note: verify with Russell)
    • DEPENDENCIES: astrometry
  • Improve PSF estimation
    • WHO: RHL
    • DURATION: 2 months (note: very hard to estimate)
    • DEPENDENCIES:
  • Ubercalibration of S82 outputs
    • WHO: Peter Y. and calibration team (note: confirm with ZI and calibration team)
    • DURATION: 1 month (note: verify with Peter et al.)
    • DEPENDENCIES: Output S82 and/or ImSim catalogs
  • Stack-Fit Port
    • WHO: Perry Gee
    • DURATION: 3 months
    • DEPENDENCIES:
  • Prioritize the order of S82 processing
    • WHO: Mario
    • DURATION: 1 week
  • Detect on coadds
    • WHO: Paul Price
    • DURATION: 1 week
    • DEPENDENCIES: ImSim/S82 co-adds
  • Forced photometry based on coadd detections
    • WHO: Paul Price, Serge (?)
    • DURATION: 2 weeks
    • DEPENDENCIES: Detections on coadds
  • Produce a RGB coadds
    • WHO: RHL
    • DURATION: 1 week
    • DEPENDENCIES: ImSim/S82 co-adds
  • GPU code benchmarks
    • WHO: Kreso, Perry
    • DURATION: 2 months
  • Interactive analysis package
    • WHO: RHL
    • DURATION: ongoing
  • Image subtraction
    • WHO: AndyB
    • DURATION: 2 weeks (?)
  • Improve PipeQA - Steve & AndyB (talk to ASCOT, NCSA), help from Russell
    • WHO: SteveB, AndyB, with help from Russell
    • DURATION: ongoing
  • Processing some of the Pan-STARRS data using LSST stack
    • WHO: Peter Y., w. help from Paul P.
    • DURATION: 1 month
  • Add LSST-specific metadata to SDSS-derived output image headers
    • WHO: RHL
    • DURATION: 1 day
  • Relative astrometry
    • WHO: Peter Y., with help from RHL and/or Paul
  • Test compression of output images
    • WHO: Kreso, with help from Dustin
    • WHAT: Test whether
    • DURATION: 1 week
  • Various application infrastructure jobs
    • WHO: all
    • DURATION: ongoing

Middleware

  • Refactor persistence - K-T
  • Refactor logging - K-T
  • Refactor pipe_tasks - K-T
  • Replace pex_policy with pex_config - K-T
  • Automate file transfers to XSEDE - Greg (3 weeks)
  • Implement DRP+coadds+AP+ingest in pex_harness/ctrl_sched - ?
  • Implement DRP+coadds+AP+ingest as Condor DAG - ?
  • Implement DRP+coadds+AP+ingest in NHPPS - ?
  • Develop DRP mockup with resource consumers - ?
  • Demonstrate hot fail-over - low-priority - K-T
  • Demonstrate cluster monitoring - low-priority - K-T
  • Evaluate OWL, produce comparison document - Greg
  • Execute mini-productions in March - Greg
  • Execute mini-productions in April - Greg
  • Execute mini-productions in May - Greg
  • Increasing scale of runs on HPC platforms from Mar thru Jun - Mike (???)

Image Data Management

  • FUSE filesystem access to REDDnet - Mike
  • GridFTP access to NFS disks - Mike
  • Benchmark Globus Online performance - Mike
  • Scaling up the image retrieval service - Mike

qserv

  • Extend SQL coverage - Jacek
  • Unify metadata tables - Jacek
  • Add JHU HTM - Jacek
  • Performance/scale tests - Jacek

Modeling

  • UML DC model: Prepare EA:LSST_DM for next Data Challenge - LOE - Robyn
  • UML baseline model: Update EA components changed/added - LOE - Robyn
  • UML baseline model: Flow down DM System Requirements into upper level baseline use cases - LOE - Robyn
  • UML baseline model: Generate specifications for User & System Integration tests - LOE - Robyn
  • Infrastructure sizing model: Incorporate updates since PDR - Mike

SQA Development

  • Parasoft installation - asap to satisfy vendor agreement - Robyn
  • Implement ABI checker - Robyn
  • Update standards documents on Trac and docushare - LOE - Robyn
  • Review of DM policies - LOE - Robyn
  • Master document to organize all documents - LOE - Robyn

Buildbot

(in priority order)

  • Make integration tests run as separate pseudo-daemon - Robyn
  • Developers able to initiate builds via buildbot forms - Robyn
  • Build and Run Production Test against master or beta or stable tags - Robyn
  • On-demand integration tests - Robyn
  • Merge-triggering on well-known branches - Robyn

System Administration

  • ds33 replacement - Mike
  • System monitoring - LOE - Mike
  • System/cluster configuration tools - LOE - Mike
  • Review backup tools and strategy - LOE - Mike

MOPS

  • XSEDE allocation on gordon (ember?) - Mike
  • Update MOPS requirements to satisfy NEO science - LynneJ
  • Write report of MOPS results to date - LynneJ

System Engineering

  • N-squared diagram for external subsystem interfaces - Gregory
  • N-squared diagram for DM internal interfaces - Gregory
  • Risk Registry Update - Gregory
  • OSS Requirements Review - draft due Mar31 - final due May1 - K-T
  • Common standards working groups (SDP, electrical, networking) - Gregory (K-T? Mike? - need to split activity?)

Science User Interface

  • SUI Workshop - Schuyler
  • SUI use case expansion - Schuyler
  • Improved catalog and image access

Early Production Runs

Due before AHM.

  • XSEDE proposal due Apr. 15 - Mike
  • The target schedule is:
    • Code "freeze" date: 15-Jun-2012
    • Medium-scale runs and bug fixes: 18-Jun-2012 to 22-Jun-2012
    • Full-scale runs: 25-Jun-2012 to 13-Jul-2012
    • Results analysis, report writing, contingency: 16-Jul-2012 to 10-Aug-2012
  • Define input data for both sets of runs - Mario
  • Define technical requirements for both sets of runs - K-T
  • Acquire new storage if necessary for both sets of runs - Mike
  • Define SUI for S12 for both sets of runs - Schuyler
  • Integration testing on HPC platform
  • Production runs on HPC platform including data transfer and pipeQA - Greg
  • Load data into SUI systems (gator, etc.) - Schuyler
  • Data analysis of output data - Mario
  • Update DC handbook - Dick

Late Production Runs

Not due before the AHM.

  • Production runs on HPC platform - Greg
  • Load data into SUI systems (gator, etc.) - Schuyler
  • Data analysis of output data - Mario
  • Update DC handbook - Dick

Out-of-scope for Summer 2012

  • Apps package reorg
  • Pixel-class/footprint reimplementation

Deferred from Winter 2012

This is currently just a holding area for Mike, until these are slotted correctly.

Integrate new table classes and measurement overhaul into pipe_tasks Feb 15 Feb 22 "Jim, Paul" 0% added 2/14; Winter2012d
Add persistence API support for tables and update ingest Feb 27 Feb 27 Serge 0% added 2/14; Winter2012d
update app code to use new config, butler, pipette (AP) Jan17 Feb2 Feb10 Feb17 Feb28 Mar2 Serge 75% split out on 2/21; moved to Winter2012d
Apps package reorganization Oct11 Oct31 Jim 25% Informal agreement by apps team on proposal, but implementation has been reschedule to be one of the last tasks in Winter2012 and proposal will be formalized then (implementation is expected to be very easy, but would disrupt other work)
replacement server for ds33 Mike 5% reconfiguration of Trac depends on new distribution server
Develop stripped-down prototype pipeline for middleware R&D Nov1 Nov21 Greg, Steve, K-T 0%
afw pixel class reimplementation (footprint redesign; ticket #1836) Nov8 Dec12 Jim 5% Now a lower priority than source reimplementation; will resume work once that is complete.
Implement prototype pipeline on OWL or Condor Dec20 Jan16 Greg, Steve 0%

Notes from RHL's visit to UW, courtesy Simon

Simon's notes on Stripe82:

Coadd delivery:
deliver camcol 3 (or all) for central 40 deg of stripe82 all bands to
NCSA with PSF information for matched models. -- Simon
dump data quality database. (Not neede anymore, since the MSSQL
Server will be queried directly ???)
tan-sip headers to NCSA. -- Simon
Deliver reference catalog of mean object positions and fluxes in all
bands. -- Becker
Make astrometry indexes from reference catalog. -- Simon

Test data:
iterate with Mario on the 10,000 chips for the monthly runs -- Simon

Coadd fix-up:
Interpolate nans (in cores of stars) -- Russell
Bilinear interpolation for mask planes -- Russell
Merge these changes with Hadoop coadd code -- ????

obs_sdss package:
 --camera geom for SDSS. -- Paul/Simon
 --mapper -- K-T
 --fpm to mask image -- Becker
 --psfField to LSST PSF -- ????
 --asTrans to TAN--SIP -- Rob G.

Operations on delivered stripe82 coadds:
Define quality metrics for coadds (for deep and template coadds) -- Zeljko
Run single frame measurement on fpCs with zeropointing and astrometry -- Becker
run detection on the stack -- Becker
 --make up a reasonable object schema
merge and generate objects from stacks -- Becker/Paul
forced photometry on object -- Paul
detection and association on stacks -- Becker