wiki:PT1_1Report091510
Last modified 8 years ago Last modified on 10/07/2010 10:40:28 AM

Table of Contents

  1. Other Activity

Previous month Apps - MW/Inf - Mgt/SE

Current month Mgt/SE

Next month PT1.1 - Mgt/SE

Guide on Reporting Percentage Completed:

For software development tasks:

  • use 0% if the solution is still being designed;
  • use 25% if the solution's design is completed;
  • use 50% if the solution is implemented and tested within the Ticket branch;
  • use 75% if the implementation review is complete, the Ticket branch has been merged onto the Trunk branch, and the Ticket is closed.
  • use 100% if the affected Trunk packages are tagged and released.

For level of effort tasks, use % of effort expended rounded to nearest 5%:

For other tasks, still use these increments, but apply their meaning as best as possible:

  • use 0% if the deliverable is still being designed;
  • use 25% if the deliverable's design is completed;
  • use 50% if the deliverable is implemented awaiting review/approval;
  • use 75% if the deliverable has been reviewed and in revision;
  • use 100% if the deliverable is approved and posted to appropriate repository.

Note: Please place an asterisk (*) after the percent if you updated this value in this report.

Task Start Finish Resources (lead first) % Complete Comments Last Updated
ImSim Data Krughoff Simon 50% Scripts for staging complete and distributed to all three production sites (Purdue, SLAC, U. Washington). Inputs to simulator generated for seven fields for a year of opsim3_61. Generation of simulated images begun at 1 out of three sites (Purdue). 9/15/10
Make census of ImSim data at NCSA, request missing visits resend 8/23/2010 8/23/2010 Plante Ray[50%] 9/14/10
Create ImSim Data Flow for PT 1.1 from all image generation sites 8/24/2010 8/30/2010 Plante Ray[25%],Krughoff Simon[25%] 25% SK: Fields selected, staging script set up.; RP: Waiting for contribution from Garrett
RP: received scripts; progress delayed by Orch. workshop
9/24/10
Prepare to generate 450 (was 600) visits in 7 adjacent fields in 15 x 15 field 8/23/2010 8/27/2010 Krughoff Simon[30%] 10% These fields only had 450 observations in the first year. Generation at Purdue begun 09/14/2010
Generate reference galaxy catalog (turn on magnitudes, improved photometry) 8/30/2010 9/3/2010 Krughoff Simon[30%] 25% Code in place, testing in progress. 09/14/2010
Import reference catalogs into Gator for comparison to output catalogs 9/6/2010 9/7/2010 Monkewitz Serge[50%] 25% 8/27/2010
Environment and Tools Plante Ray This effort is largely complete apart from the final upgrade of the stack, which has been postponed until after the Orchestration workshop and the completion of the JobOffice extensions. 9/24/10
TCT approval for environment/tool updates 8/23/2010 8/23/2010 Allsman Robyn[50%] 100% 8/27/10
Install, configure, publish environment/tool updates 8/24/2010 8/25/2010 Plante Ray[25%] 25% upgrade Boost, gcc are available now; stack builds under gcc 4.4.3; will resume this work after eups 8/14/10
Complete eups release 8/26/2010 9/1/2010 Plante Ray[25%] 50% this status represents work done prior to 8/26
Support trac, svn, systems administration 8/23/2010 12/6/2010 Pietrowicz Steve[15%]
Document and improve Single Stage Tester, Stand-alone, Repeatable Failure scripts 8/30/2010 9/1/2010 Lim KT[50%] 100%* 9/2/2010
Production-Pipeline-Stage Middleware Pietrowicz Steve Logging updates being integrated into the tree. Test runs are being executed now. 9/15/2010
Improve Logging performance, reporting, error checking 8/23/2010 9/24/2010 Pietrowicz Steve[10%],Daues Greg[10%] 50%* workerid's now in database/logs using harness #1398 orca #1399; depends on Abe availability 9/20/2010
Restore logging to database 8/23/2010 9/3/2010 Pietrowicz Steve[10%] 100%* 9/14/2010
*new* Logging to per-run database 8/23/2010 9/24/2010 Pietrowicz Steve[10%] 50%* Need to test on Abe 9/20/2010
Update Event Monitor for execution statistics 9/6/2010 10/1/2010 Pietrowicz Steve[15%] 25%* 9/14/2010
Improve shutdown mechanism for data stack 8/23/2010 10/8/2010 Daues Greg[10%],Pietrowicz Steve[35%] 0%* 9/20/2010
Complete JobOffice implementation to allow integration of Source Association 9/2/2010 9/24/2010 Plante Ray[50%] 50%* 9/24/2010
Automate transfer of data to/from MSS 9/27/2010 11/1/2010 Plante Ray[10%]; Adams Darren[100%] 0% 9/20/2010
Data Release Production Architecture and Performance Lim KT Performance improvements implemented resulting in convolution speedups of 4X-30X depending on kernel size and warping speedups of 1.5X-12X depending on kernel type and accuracy parameters. Ran unified pipeline (all stages through SFM for one CCD) via Orca using the files in datarel/trunk/pipeline/PT1Pipe and trunk versions of most other packages (except a hand-patched version of meas_astrom). 9/27/2010
Improve speed of convolution and warping 8/23/2010 12/6/2010 Owen Russ[50%] 100% I implemented both planned convolution speedups (Kreso's algorithm and improved interpolation). The net improvement is shown in ticket #1411; the dramatic increase in speed for LinearCombinationKernel with delta function kernels is due to Kreso's algorithm (which will help any LinearCombinationKernel with many bases). RHL sped up warpExposure but I do not know by how much (no formal timing code exists yet); he described it as fast enough to not embarrass LSST. 9/17/2010
Implement speed-up for ISR metrics calculation 9/1/2010 9/1/2010 Krughoff Simon
Merge 6 PT1 Pipelines into 2 8/25/2010 9/30/2010 Lim KT[50%] 25%* 9/20/2010
Data Access Middleware and Database Lim KT Some improvements to the data butler were determined to be unnecessary for PT1.1. However, a need was identified for additional design and implementation for the Source class and its persistence. 9/15/2010
Subimage and WCS retrieval in Data Butler 8/30/2010 12/1/2010 Lim KT[50%] 25%* No longer required for PT1.1 8/31/2010
Pass through comments in FITS headers 9/3/2010 9/30/2010 Lim KT[25%] 0%* Dates changed 8/27/2010
Database schema updates (incl galaxy photometry) 9/21/2010 10/18/2010 Becla Jacek[5%] 9/10/2010
Update psf/exposure formatters 9/20/2010 9/24/2010 Lim KT[50%]
Update Source formatter 9/27/2010 10/1/2010 Lim KT[50%], Monkewitz Serge[50%] 0%* Depends on Apps work to redesign Source 9/10/2010
Point Source Photometry Axelrod Tim 25%* Processed HSC simulated data through all PT1 stages. Achieved c. 0.009 mag RMS for standard stars in 3000 2kx4k CCDs (5 visits to 6 fields; 100 CCDs per visit) 9/11/2010
Split photometry and astrometry catalogs 8/23/2010 9/3/2010 Lang Dustin[50%] We agreed this task is not necessary. But photometry/astrometry flag needs to be fully implemented
Identify and fix zeropoint calculation error 8/23/2010 8/27/2010 Bickerton Steve[30%] 0% At least part of this was due to poor measurement; this has now been fixed for the simplest case of a constant PSF and the fields (in HSC DC1) calibrate to under 1% for PSF mags.
Implement aperture corrections 8/23/2010 9/17/2010 Bickerton Steve[50%] 0%
PSF Estimation Lupton Robert Monthly
Implement Jarvis PSFs 8/23/2010 9/17/2010 Jarvis Mike[50%] 75%* Code in trunk. More testing required. 8/26/10
Implement PCA PSFs 8/23/2010 9/3/2010 Lupton Robert[25%] 75%* worked on algorithms; mostly behaves but some bad cases 2010-09-16
Change meas_algorithms to support multiple PSFs 9/20/2010 10/1/2010 Lupton Robert[50%] 60%* All book-keeping complete, now we just have to try it 2010-09-16
Background Determination Becker Andy Applying background subtraction and running detection on PT1.1 imsims. Background subtraction found to be most sensitive to background mesh size, and not on statistic used to calculate background (clipped mean vs median). Will next address spline type. We have the known imsim backgrounds which will be used as ground truth to compare our fitted models to. 20100915
Tune mesh size 8/23/2010 9/10/2010 Becker Andy[25%] 50% 9/28/2010
Examine statistics (mean vs median) 8/23/2010 9/10/2010 Becker Andy[25%],Bickerton Steve[50%] 100% 9/28/2010
Galaxy Photometry Bosch Jim Added a simpler galaxy model (exponential) as a fall-back option and to aid in testing our Sersic models. Added a Minuit-based fitter to replace our own custom fitter; this ensures we have a more robust and well-tested (if potentially slower) optimizer available for PT1.1. 9/28/2010
Prepare for local testing runs 8/27/2010 9/2/2010 Dubcovsky Martin[20%] 0% Task description changed, and it's actually pretty trivial, so we're not bothered that we haven't started it. 9/28/2010
Determine fitting code failure modes 9/3/2010 9/16/2010 Bosch Jim[10%],Dubcovsky Martin[50%] 30% Discovered two new problems we are presently working on: analytic gradient computations are incorrect (even though model derivatives seem to be correct), and Sersic lookup-table population is not robust for certain parameter values (already 40% resolved). 9/28/2010
Determine fitting outputs to be stored in database 9/16/2010 9/16/2010 Bosch Jim[5%],Dubcovsky Martin[5%] 0% 9/28/2010
Update Source class for galaxy photometry, incl formatters 9/17/2010 9/20/2010 Monkewitz Serge[50%],Lupton Robert[50%] 20%* new meas::algorithms should support this, needs to be confirmed 2010-09-16
Write stage for source fitting 9/21/2010 9/27/2010 Dubcovsky Martin[25%] 40% 9/28/2010
Source Association Monkewitz Serge Monthly
Define SDQA metrics for Source Association 8/23/2010 8/25/2010 Monkewitz Serge[50%],Axelrod Tim[25%] 30% initial proposal sent to Tim 8/27/2010
Implement SDQA metrics for Source Association 8/26/2010 9/8/2010 Monkewitz Serge[50%] 25% Implemented coverage map generation (for determining the expected maximum number of sources for an object at a given position). Began implementing reference to output object catalog cross-matching - so far, I've got code that filters the reference catalog down to those entries that could actually have been observed in a particular production run. 9/16/2010
Iteratively improve Source Association algorithm 9/9/2010 10/11/2010 Monkewitz Serge[50%]
Change Source Association to use inverse variance weighting and calibrated fluxes 10/12/2010 10/13/2010 Monkewitz Serge[50%]
Build and Test Allsman Robyn Implemented and deployed continuous integration of development branch and release branch. 9/23/10
Configure continuous integration - release branch 8/23/2010 8/23/2010 Allsman Robyn[50%] 100% 8/27/10
Configure continuous integration - development branch 8/24/2010 9/3/2010 Allsman Robyn[50%] 100% Implemented and deployed continuous integration of development branch. 9/23/10
Monitor buildbot integration and test 8/31/2010 12/6/2010 Allsman Robyn[25%] 5% Established new buildbot slave incorporating anonymous svn access; existing 2 slaves being converted. Upgrading slaves to use gcc 4.4 to bring into alignment with development software environment 8/27/10
Production Runs Daues Greg Monthly
Practice runs on Abe with CFHT night/sub-night 8/23/2010 9/3/2010 Daues Greg[10%] 50% Several good runs complete, scaling up in size and constituent pipelines. 1) ISR run thru 170 D2 visits; 2) ISR-ccdassembly-crsplit-imgchar thru 3 D2 visits
Abe runs with ImSim data subsets 10/18/2010 12/6/2010 Daues Greg[50%]
User Support Shaw Dick Monthly
Prepare Data Challenge Handbook 8/23/2010 9/30/2010 Shaw Dick[50%] 45%* Input requested on details of processing algorithms. Technical review now planned for 28-Sept. 23-Sep-2010
Organize DM User Support 9/20/2010 10/1/2010 Shaw Dick[50%] 30%* Wiki page now available to document User Support processes and identify unresolved issues 23-Sep-2010
Define SDQA analysis process 10/4/2010 10/15/2010 Shaw Dick[50%] 0% 27-Aug-2010
Configure/tune catalogs 10/18/2010 11/12/2010 Becla Jacek[10%] 9/10/2010
Data Analysis Axelrod Tim Monthly
DM Scientist/Engineer? data analysis 10/18/2010 11/12/2010 Axelrod Tim[50%],Lupton Robert[50%],Bosch Jim[50%],Becker Andy[50%],Van Dyk Schuyler[25%],Shaw Dick[50%]
Science Collaboration data analysis 10/18/2010 11/12/2010 Science Collaboration
Data analysis report 11/15/2010 11/19/2010 Axelrod Tim[50%],Lupton Robert[50%],Bosch Jim[50%],Becker Andy[50%],Van Dyk Schuyler[25%],Shaw Dick[50%],Science Collaboration
Infrastructure Pietrowicz Steve System has been ordered, and waiting for vendor to call to let us know the system is ready to be configured. We plan on a conference call with the vendor to discuss this.
Generate storage proposals 8/23/2010 8/27/2010 Pietrowicz Steve[20%] 100% 9/7/10
Acquire and configure storage 8/30/2010 9/10/2010 Pietrowicz Steve[20%]

Other Activity

Plante, Ray
9/10/10

  • Some support work for HTTP data access via lsst1
  • Logistics planning for the Middleware workshop

9/17/10

  • Logistics planning for the Middleware workshop

9/24/10

  • Middleware workshop: 9/20-22
  • subversion 1.6 now available as lsstpkg download (requested by Robyn)
  • Bug Fix: #1456