wiki:PT1_1Report101510

Version 11 (modified by shaw, 9 years ago) (diff)

Updates from Dick Shaw

Table of Contents

  1. Other Activity

Previous month PT 1.1 - Mgt/SE

Current month Mgt/SE

Next month PT1.1 - Mgt/SE

Guide on Reporting Percentage Completed:

For software development tasks:

  • use 0% if the solution is still being designed;
  • use 25% if the solution's design is completed;
  • use 50% if the solution is implemented and tested within the Ticket branch;
  • use 75% if the implementation review is complete, the Ticket branch has been merged onto the Trunk branch, and the Ticket is closed.
  • use 100% if the affected Trunk packages are tagged and released.

For level of effort tasks, use % of effort expended rounded to nearest 5%:

For other tasks, still use these increments, but apply their meaning as best as possible:

  • use 0% if the deliverable is still being designed;
  • use 25% if the deliverable's design is completed;
  • use 50% if the deliverable is implemented awaiting review/approval;
  • use 75% if the deliverable has been reviewed and in revision;
  • use 100% if the deliverable is approved and posted to appropriate repository.

Note: Please place an asterisk (*) after the percent if you updated this value in this report.

Task Start Finish Resources (lead first) % Complete Comments Last Updated
Infrastructure Pietrowicz Steve
Acquire and configure storage 8/30/10 9/20/10 Pietrowicz Steve[20%] 100% done 9/29/10
Environment and Tools Plante Ray This effort is largely complete apart from the final upgrade of the stack, which has been postponed until after the Orchestration workshop and the completion of the JobOffice extensions.
Install, configure, publish environment/tool updates 8/24/10 9/17/10 Plante Ray[25%] 25% upgrade Boost, gcc are available now; stack builds under gcc 4.4.3; will resume this work after eups
Complete eups release 8/26/10 9/21/10 Plante Ray[25%] 50% this status represents work done prior to 8/26
Support trac, svn, systems administration 8/23/10 11/19/10 Pietrowicz Steve[15%] 47%
ImSim Data Krughoff Simon Scripts for staging complete and distributed to all three production sites (Purdue, SLAC, U. Washington). Inputs to simulator generated for seven fields for a year of opsim3_61. Generation of simulated images begun at all three sites. Transfer scripts tested at SLAC and ready for deployment to UW and Purdue. 10/08/2010
Make census of ImSim data at NCSA, request missing visits resend 9/16/10 9/16/10 Plante Ray[50%] 0%
Create ImSim Data Flow for PT 1.1 from all image generation sites 8/24/10 9/22/10 Plante Ray[25%],Krughoff Simon[25%] 25% SK: Fields selected, staging script set up.; RP: Waiting for contribution from Garrett
RP: received scripts; progress delayed by Orch. workshop
Prepare to generate 450 visits in 7 adjacent fields in 15 x 15 field 8/23/10 9/22/10 Krughoff Simon[30%] 50% These fields only had 450 observations in the first year. Generation at all three sites begun 10/08/2010
Generate reference galaxy catalog (turn on magnitudes, w/improved photometry) 8/30/10 9/28/10 Krughoff Simon[30%] 50% Testing complete, catalog generation in progress. 10/08/2010
Import reference catalogs into Gator for comparison to output catalogs 9/6/10 9/29/10 Monkewitz Serge[50%] 25%
Production-Pipeline-Stage Middleware Pietrowicz Steve Logging updates being integrated into the tree. Test runs are being executed now. 10/11/10
Improve Logging performance, reporting, error checking 8/23/10 9/29/10 Pietrowicz Steve[10%],Daues Greg[10%] 50% updated in trunk, additional testing happening now 10/11/10
Logging to per run database 8/23/10 10/4/10 Pietrowicz Steve[10%] 100% Done. 10/11/10
Update Event Monitor for execution statistics 9/6/10 10/6/10 Pietrowicz Steve[15%] 50% Updated, but additional testing required 10/11/10
Improve shutdown mechanism for data stack 9/16/10 10/6/10 Daues Greg[10%],Pietrowicz Steve[35%] 0%
Complete JobOffice? implementation to allow integration of Source Association 9/2/10 9/24/10 Plante Ray[50%] 50%
Automate transfer of data to/from MSS 9/24/10 10/29/10 Plante Ray[10%],Adams Darren 0%
Data Release Production Architecture and Performance Lim KT Performance improvements implemented resulting in convolution speedups of 4X-30X depending on kernel size and warping speedups of 1.5X-12X depending on kernel type and accuracy parameters. Ran unified pipeline (all stages through SFM for one CCD) via Orca using the files in datarel/trunk/pipeline/PT1Pipe and trunk versions of most other packages (except a hand-patched version of meas_astrom).
Implement speed-up for ISR metrics calculation 9/1/10 9/1/10 Krughoff Simon 100% done 10/08/2010
Merge 6 PT1 Pipelines into 2 8/25/10 9/20/10 Lim KT[50%] 25%
Data Access Middleware and Database Lim KT Some improvements to the data butler were determined to be unnecessary for PT1.1. However, a need was identified for additional design and implementation for the Source class and its persistence.
Subimage and WCS retrieval in Data Butler 8/30/10 9/22/10 Lim KT[50%] 25% No longer required for PT1.1
Pass through comments in FITS headers 9/23/10 10/21/10 Lim KT[25%] 0% Dates changed
Database schema updates (incl galaxy photometry) 9/29/10 10/19/10 Becla Jacek[5%] 0%
Point Source Photometry Lupton Robert 25%* Processed HSC simulated data through all PT1 stages. Achieved c. 0.009 mag RMS for standard stars in 3000 2kx4k CCDs (5 visits to 6 fields; 100 CCDs per visit)
Implement photometry/astrometry flag 9/16/10 9/29/10 Lang Dustin[50%] 0%
Identify and fix zeropoint calculation error 8/23/10 9/21/10 Bickerton Steve[30%] 25% At least part of this was due to poor measurement; this has now been fixed for the simplest case of a constant PSF and the fields (in HSC DC1) calibrate to under 1% for PSF mags.
Implement aperture corrections 9/16/10 10/13/10 Bickerton Steve[50%] 0%
PSF Estimation Lupton Robert
Implement Jarvis PSFs 8/23/10 9/22/10 Jarvis Mike[50%] 75% Code in trunk. More testing required.
Implement PCA PSFs 8/23/10 9/20/10 Lupton Robert[25%] 75% worked on algorithms; mostly behaves but some bad cases
Change meas_algorithms to support multiple PSFs 9/20/10 10/1/10 Lupton Robert[50%] 60% All book-keeping complete, now we just have to try it
Update psf/exposure formatters 9/23/10 9/29/10 Lim KT[50%] 0%
Background Determination Becker Andy Applying background subtraction and running detection on PT1.1 imsims. Background subtraction found to be most sensitive to background mesh size, and not on statistic used to calculate background (clipped mean vs median). Will next address spline type. We have the known imsim backgrounds which will be used as ground truth to compare our fitted models to.
Tune mesh size 8/23/10 9/27/10 Becker Andy[25%] 50%
Galaxy Photometry Bosch Jim Added a simpler galaxy model (exponential) as a fall-back option and to aid in testing our Sersic models. Added a Minuit-based fitter to replace our own custom fitter; this ensures we have a more robust and well-tested (if potentially slower) optimizer available for PT1.1.
Prepare for local test runs 9/16/10 9/22/10 Bosch Jim[10%],Dubcovsky Martin[50%] 0% Task description changed, and it's actually pretty trivial, so we're not bothered that we haven't started it.
Determine fitting code failure modes 9/9/10 9/24/10 Bosch Jim[10%],Dubcovsky Martin[50%] 30% Discovered two new problems we are presently working on: analytic gradient computations are incorrect (even though model derivatives seem to be correct), and Sersic lookup-table population is not robust for certain parameter values (already 40% resolved).
Determine fitting outputs to be stored in database 9/24/10 9/24/10 Bosch Jim[10%],Dubcovsky Martin[50%] 0%
Update Source class for galaxy photometry, incl formatters 9/23/10 9/28/10 Monkewitz Serge[50%],Lupton Robert[50%] 20% new meas::algorithms should support this, needs to be confirmed
Write stages for source fitting 9/27/10 10/1/10 Bosch Jim[10%],Dubcovsky Martin[50%] 40%
Source Association Leader
Define SDQA metrics for Source Association 8/23/10 9/20/10 Monkewitz Serge[50%],Axelrod Tim[25%] 30% initial proposal sent to Tim
Implement SDQA metrics for Source Association 8/26/10 9/29/10 Monkewitz Serge[50%] 25% Implemented coverage map generation (for determining the expected maximum number of sources for an object at a given position). Began implementing reference to output object catalog cross-matching - so far, I've got code that filters the reference catalog down to those entries that could actually have been observed in a particular production run.
Iteratively improve Source Association algorithm 9/29/10 10/13/10 Monkewitz Serge[50%] 0%
Change Source Association to use inverse variance weighting and calibrated fluxes 10/13/10 10/15/10 Monkewitz Serge[50%] 0%
Build and Test Allsman Robyn Implemented and deployed continuous integration of development branch and release branch.
Monitor buildbot integration and test 8/31/10 11/19/10 Allsman Robyn[25%] 7% Established new buildbot slave incorporating anonymous svn access; existing 2 slaves being converted. Upgrading slaves to use gcc 4.4 to bring into alignment with development software environment
Production Runs Daues Greg
Practice runs on Abe with CFHT night/sub-night 8/23/10 9/22/10 Daues Greg[10%] 50% Several good runs complete, scaling up in size and constituent pipelines. 1) ISR run thru 170 D2 visits; 2) ISR-ccdassembly-crsplit-imgchar thru 3 D2 visits
App debug/practice runs on lsst with ImSim data 9/23/10 11/3/10 Daues Greg[10%] 0%
Abe runs with ImSim data subsets 10/18/10 11/12/10 Daues Greg[50%] 0% First handful of ccds/sensors of ImSim visit 85661762 have been processed through Unified Pipeline on Abe with no apparent App errors. Next, scaling up to do all ccds, more visits.
User Support Shaw Dick
Prepare Data Challenge Handbook 8/23/10 9/30/10 Shaw Dick[50%] 60%* Input requested on details of processing algorithms. Technical review now planned for 18-Oct. 10/12/10
Organize DM User Support 9/20/10 10/11/10 Shaw Dick[50%] 50%* Wiki page now available to document User Support processes and identify unresolved issues 10/12/10
Define SDQA analysis process 10/12/10 10/25/10 Shaw Dick[50%] 0%
Configure/tune catalogs 10/26/10 11/22/10 Becla Jacek[10%] 0%
Data Analysis Axelrod Tim
DM Scientist/Engineer? data analysis 10/26/10 11/22/10 Axelrod Tim[50%],Lupton Robert[50%],Bosch Jim[50%],Becker Andy[50%],Van Dyk Schuyler[25%],Shaw Dick[50%] 0%
Science Collaboration data analysis 10/26/10 11/22/10 Science Collaboration 0%
Data analysis report 11/23/10 11/29/10 Axelrod Tim[50%],Lupton Robert[50%],Bosch Jim[50%],Becker Andy[50%],Van Dyk Schuyler[25%],Shaw Dick[50%],Science Collaboration 0%

Other Activity