wiki:PT1_1Report101510

Version 21 (modified by Tim Axelrod, 9 years ago) (diff)

--

Table of Contents

  1. Other Activity

Previous month PT 1.1 - Mgt/SE

Current month Mgt/SE

Next month PT1.1 - Mgt/SE

Guide on Reporting Percentage Completed:

For software development tasks:

  • use 0% if the solution is still being designed;
  • use 25% if the solution's design is completed;
  • use 50% if the solution is implemented and tested within the Ticket branch;
  • use 75% if the implementation review is complete, the Ticket branch has been merged onto the Trunk branch, and the Ticket is closed.
  • use 100% if the affected Trunk packages are tagged and released.

For level of effort tasks, use % of effort expended rounded to nearest 5%:

For other tasks, still use these increments, but apply their meaning as best as possible:

  • use 0% if the deliverable is still being designed;
  • use 25% if the deliverable's design is completed;
  • use 50% if the deliverable is implemented awaiting review/approval;
  • use 75% if the deliverable has been reviewed and in revision;
  • use 100% if the deliverable is approved and posted to appropriate repository.

Note: Please place an asterisk (*) after the percent if you updated this value in this report.

Task Start Finish Resources (lead first) % Complete Comments Last Updated
Infrastructure Pietrowicz Steve
Acquire and configure storage 8/30/10 9/20/10 Pietrowicz Steve[20%] 100% done 9/29/10
Environment and Tools Plante Ray This effort is largely complete apart from the final upgrade of the stack, which has been postponed until after the Orchestration workshop and the completion of the JobOffice extensions.
Install, configure, publish environment/tool updates 8/24/10 9/17/10 Plante Ray[25%] 25% upgrade Boost, gcc are available now; stack builds under gcc 4.4.3; will resume this work after eups 10/12/10
Complete eups release 8/26/10 9/21/10 Plante Ray[25%] 50% this status represents work done prior to 8/26 10/12/10
Support trac, svn, systems administration 8/23/10 11/19/10 Pietrowicz Steve[15%] 47%
ImSim Data Krughoff Simon Scripts for staging complete and distributed to all three production sites (Purdue, SLAC, U. Washington). Inputs to simulator generated for seven fields for a year of opsim3_61. Generation of simulated images begun at all three sites. Transfer scripts tested at SLAC and ready for deployment to UW and Purdue. 10/08/2010
Make census of ImSim data at NCSA, request missing visits resend 9/16/10 9/16/10 Plante Ray[50%] 0% This target is not time-critical to PT1.1; postponing 10/12/10
Create ImSim Data Flow for PT 1.1 from all image generation sites 8/24/10 9/22/10 Plante Ray[25%],Krughoff Simon[25%] 25% SK: Fields selected, staging script set up.; RP: scripts received; in final testing 10/12/10
Prepare to generate 450 visits in 7 adjacent fields in 15 x 15 field 8/23/10 9/22/10 Krughoff Simon[30%] 50% These fields only had 450 observations in the first year. Generation at all three sites begun 10/08/2010
Generate reference galaxy catalog (turn on magnitudes, w/improved photometry) 8/30/10 9/28/10 Krughoff Simon[30%] 50% Testing complete, catalog generation in progress. 10/08/2010
Import reference catalogs into Gator for comparison to output catalogs 9/6/10 9/29/10 Monkewitz Serge[50%] 25%
Production-Pipeline-Stage Middleware Pietrowicz Steve Logging updates being integrated into the tree. Test runs are being executed now. 10/11/10
Improve Logging performance, reporting, error checking 8/23/10 9/29/10 Pietrowicz Steve[10%],Daues Greg[10%] 50% updated in trunk, additional testing happening now;
The harness issues log statements with CPU time, Sys time, Wall time for "visits" (tag 4.1.0.0)
10/11/10
Logging to per run database 8/23/10 10/4/10 Pietrowicz Steve[10%] 100% Done. 10/11/10
Update Event Monitor for execution statistics 9/6/10 10/6/10 Pietrowicz Steve[15%] 50% Updated, but additional testing required 10/11/10
Improve shutdown mechanism for data stack 9/16/10 10/6/10 Daues Greg[10%],Pietrowicz Steve[35%] 0%
Complete JobOffice? implementation to allow integration of Source Association 9/2/10 9/24/10 Plante Ray[50%] 50% Implementation is completed; includes unit tests; need to complete integration tests 10/12/10
Automate transfer of data to/from MSS 9/24/10 10/29/10 Plante Ray[10%],Adams Darren 25% plan in place 10/12/10
Data Release Production Architecture and Performance Lim KT Unified pipeline running (with various versions of apps packages) on SLAC machines and Abe. Runmaster responsibility turned over to NCSA.
Implement speed-up for ISR metrics calculation 9/1/10 9/1/10 Krughoff Simon 100% done 10/08/2010
Merge 6 PT1 Pipelines into 2 8/25/10 9/20/10 Lim KT[50%] 75% release awaiting tagging of apps packages 10/18/2010
Data Access Middleware and Database Lim KT Some improvements to the data butler were determined to be unnecessary for PT1.1. However, a need was identified for additional design and implementation for the Source class and its persistence.
Subimage and WCS retrieval in Data Butler 8/30/10 9/22/10 Lim KT[50%] 25% No longer required for PT1.1
Pass through comments in FITS headers 9/23/10 10/31/10 Lim KT[25%] 0% Some design done, unclear if this will make PT1.1 10/10/2010
Database schema updates (incl galaxy photometry) 9/29/10 10/19/10 Becla Jacek[5%] 0%
Update Source formatter 10/19/10 10/22/10 Lim KT[50%] 0% Added new task, blocked on apps completing class definition 10/18/2010
Point Source Photometry Lupton Robert 25%* Processed HSC simulated data through all PT1 stages. Achieved c. 0.009 mag RMS for standard stars in 3000 2kx4k CCDs (5 visits to 6 fields; 100 CCDs per visit)
Implement photometry/astrometry flag 9/16/10 9/29/10 Lang Dustin[50%] 0%
Identify and fix zeropoint calculation error 8/23/10 9/21/10 Bickerton Steve[30%] 25% At least part of this was due to poor measurement; this has now been fixed for the simplest case of a constant PSF and the fields (in HSC DC1) calibrate to under 1% for PSF mags.
Implement aperture corrections 9/16/10 10/13/10 Bickerton Steve[50%] 0%
PSF Estimation Lupton Robert
Implement Jarvis PSFs 8/23/10 9/22/10 Jarvis Mike[50%] 75% Code in trunk. More testing required.
Implement PCA PSFs 8/23/10 9/20/10 Lupton Robert[25%] 75% worked on algorithms; mostly behaves but some bad cases
Change meas_algorithms to support multiple PSFs 9/20/10 10/1/10 Lupton Robert[50%] 60% All book-keeping complete, now we just have to try it
Update psf/exposure formatters 9/23/10 10/22/10 Lim KT[50%] 0% Blocked on apps completing class definition 10/18/2010
Background Determination Becker Andy Background mesh size of 2048 does the best job of modeling the imsims, with the caveat that the background level may be under/over estimated in the very corners of the images, at the level of 1-2%. This should be sufficient for PT 1.1.
Tune mesh size 8/23/10 9/27/10 Becker Andy[25%] 100% 10/12/10
Galaxy Photometry Bosch Jim Sersic model evaluation bugs have been resolved. Starting SST-based testing on individual ImSim CCDs to look for additional pre-integration failure modes. 10/12/10
Prepare for local test runs 9/16/10 9/22/10 Bosch Jim[10%],Dubcovsky Martin[50%] 50% Trying to overcome our understanding of what information we can expect to be present in Source when our code is run. 10/12/10
Determine fitting code failure modes 9/9/10 9/24/10 Bosch Jim[10%],Dubcovsky Martin[50%] 30% We have resolved (Sersic lookup table creation) or have workarounds (derivatives) in place for previously discovered bugs. Now looking for more. 10/12/10
Determine fitting outputs to be stored in database 9/24/10 9/24/10 Bosch Jim[10%],Dubcovsky Martin[50%] 0% Need input from others; our email to the list has been largely ignored. We can make decisions on this ourselves, if necessary, but these are issues that will impact the usefulness of PT1.1 for end-users. 10/12/10
Update Source class for galaxy photometry, incl formatters 9/23/10 9/28/10 Monkewitz Serge[50%],Lupton Robert[50%] 20% new meas::algorithms should support this, needs to be confirmed
Write stages for source fitting 9/27/10 10/1/10 Bosch Jim[10%],Dubcovsky Martin[50%] 40% Implementation essentially done; testing. 10/12/10
Source Association Leader
Define SDQA metrics for Source Association 8/23/10 9/20/10 Monkewitz Serge[50%],Axelrod Tim[25%] 75% Initial set defined. Final set awaits testing on PT1_1 outputs
Implement SDQA metrics for Source Association 8/26/10 9/29/10 Monkewitz Serge[50%] 25% Implemented coverage map generation (for determining the expected maximum number of sources for an object at a given position). Began implementing reference to output object catalog cross-matching - so far, I've got code that filters the reference catalog down to those entries that could actually have been observed in a particular production run.
Iteratively improve Source Association algorithm 9/29/10 10/13/10 Monkewitz Serge[50%] 0%
Change Source Association to use inverse variance weighting and calibrated fluxes 10/13/10 10/15/10 Monkewitz Serge[50%] 0%
Build and Test Allsman Robyn Implemented and deployed continuous integration of development branch and release branch.
Monitor buildbot integration and test 8/31/10 11/19/10 Allsman Robyn[25%] 7% All slaves using anonymous svn access. Upgrading slaves to use gcc 4.4 to bring into alignment with development software environment 10/18/10
Production Runs Daues Greg
Practice runs on Abe with CFHT night/sub-night 8/23/10 9/22/10 Daues Greg[10%] 50% Several good runs complete, scaling up in size and constituent pipelines. 1) ISR run thru 170 D2 visits; 2) ISR-ccdassembly-crsplit-imgchar thru 3 D2 visits
App debug/practice runs on lsst with ImSim data 9/23/10 11/3/10 Daues Greg[10%] 0%
Abe runs with ImSim data subsets 10/18/10 11/12/10 Daues Greg[50%] 10%* A full ImSim visit 85661762 (189 CCDs) has been processed through Unified Pipeline on Abe with deprecated App tags. Runs with new tags will occur when available on Abe.
User Support Shaw Dick
Prepare Data Challenge Handbook 8/23/10 9/30/10 Shaw Dick[50%] 60%* Input requested on details of processing algorithms. Technical review now planned for 18-Oct. 10/12/10
Organize DM User Support 9/20/10 10/11/10 Shaw Dick[50%] 50%* Wiki page now available to document User Support processes and identify unresolved issues 10/12/10
Define SDQA analysis process 10/12/10 10/25/10 Shaw Dick[50%] 0%
Configure/tune catalogs 10/26/10 11/22/10 Becla Jacek[10%] 0%
Data Analysis Axelrod Tim
DM Scientist/Engineer? data analysis 10/26/10 11/22/10 Axelrod Tim[50%],Lupton Robert[50%],Bosch Jim[50%],Becker Andy[50%],Van Dyk Schuyler[25%],Shaw Dick[50%] 0%
Science Collaboration data analysis 10/26/10 11/22/10 Science Collaboration 0%
Data analysis report 11/23/10 11/29/10 Axelrod Tim[50%],Lupton Robert[50%],Bosch Jim[50%],Becker Andy[50%],Van Dyk Schuyler[25%],Shaw Dick[50%],Science Collaboration 0%

Other Activity