wiki:AppReport071510
Last modified 9 years ago Last modified on 08/26/2010 09:20:48 AM

Previous month Apps MW/Inf

Current Month MW/Inf - Mgt/SE

Next month Apps - MW/Inf- Mgt/SE

Apps Status Report for July 15, 2010

Guide on Reporting Percentage Completed:

For software development tasks,

  • use 0% if the solution is still being designed;
  • use 25% if the solution's design is completed;
  • use 50% if the solution is implemented and tested within the Ticket branch;
  • use 75% if the implementation review is complete, the Ticket branch has been merged onto the Trunk branch, and the Ticket is closed.
  • use 100% if the affected Trunk packages are tagged and released.

For other tasks, still use these increments, but apply their meaning as best as possible.

Note: Please place an asterisk (*) after the percent if you updated this value in this report.

Allsman Robyn

Reported 22 July 2010

IDPercentTaskComments
APP200100%*validate initial integration run successfully completed

*Percent updated in this report

Comments:

  • Science results from the PT1 software stack were analyzed and validated as consistent with the algorithms used.
  • The pipeline on ABE was integrated with the orchestration layer (job office, ctrl_orca, ctrl_events, etc).
  • Both abe and the lsst cluster were used for production runs on the CFHT dataset. Due to disk capacity restrictions on the lsst cluster, most of the CFHT production runs occurred on ABE.
  • The PT1 ImSim Production was run on the SLAC cluster which provided access to a large amount of disk space and compute cycles for a short window of opportunity.
  • Results of identical science runs on the lsst cluster and ABE, and later SLAC, were compared and determined to be equivalent.

Other Activities

Axelrod Tim

Reported xx/xx/xx

IDPercentTaskComments
APP1550%photometric self calibrationnot started
APP5680%defining units
APP26120%create calibration catalogs for CFHTLS
APP2910%create input data for ImSimnot started
APP26350%identify set of images for compression tests
APP26090%define DC3b data quality requirements
APP2040%identify scientists to analyze stage outputnot started
APP1990%PT1 best efforts science data analysis/validation not started
APP19150%Provide special case simulation needs (Axelrod)

*Percent updated in this report

Comments:

Other Activities

Becker Andy

Reported 07/22/2010

IDPercentTaskComments
APP24650%update Diff-Im post design review (ticket 1176) *
APP24525%implement correlation function for spatial kernal fit (ticket 1140)
APP1000%test difference imaging coaddsnot started
APP24740%code to measure quality of difference imaging coadds
APP123100%find CFHT fringe frames and move to NCSA*

*Percent updated in this report

Comments: Extensive improvements on regularization of delta function kernels

Other Activities

Chopping up CFHT images for PT1.

Becla Jacek

Reported xx/xx/xx

IDPercentTaskComments
APP5450%schema updates for all exposure related tables
APP550%synthetic sources of datanot started
APP5680%defining units

*Percent updated in this report

Comments:

Other Activities

Bickerton Steve

Reported xx/xx/xx

IDPercentTaskComments
APP29525%stellar photometry
APP2560%compute statistics on a sky pixel masked imagenot started

*Percent updated in this report

Comments:

Other Activities

Good John

Reported xx/xx/xx

IDPercentTaskComments
APP2260%code PT1 SUI toolsnot started

*Percent updated in this report

Comments:

Other Activities

Jarvis Mike

Reported 07/22/10

IDPercentTaskComments
APP10550%PSF for deep detection Code is complete, and passed a code review of the .h files only. However, I have not done much in the way of unit tests yet.

*Percent updated in this report

Comments:

Other Activities

Jones Lynne

Reported xx/xx/xx

IDPercentTaskComments

*Percent updated in this report

Comments:

Other Activities

Krughoff Simon

Reported xx/xx/xx

IDPercentTaskComments
APP840%apply fringe frame correctionnot started
APP83NAdevelop camera state classes (CCDinfo and Ampinfo)Butler does this
APP82NAdevelop calibration products database classesButler does this
APP253*100%implement ccd assembly code with appropriate stage code
APP252*100%test camera geometry classess with ImSim LSST focal plane model
APP251*100%implement datarel.IsrStageUnit? Test
APP250NAwrite stage dictionaries and unit tests for ValidateMetaData? and CalibrationDataProducts? Butler does this
APP249*100%create policy files for ImSim camera geometry
APP254*100%move saturation and defect correction to the CCD assembly stage

*Percent updated in this report

Comments:

Other Activities

Laher Russ

Reported xx/xx/xx

IDPercentTaskComments
APP900%implement SDQA tool functionality to support DC3b goalsnot started
APP890%query metadata and package in C++ containernot started
APP880%implement threshold comparisionnot started
APP21150%WCS verification code
APP21075%ATpy evaluation
APP2830%identify existing sdqa metricsnot started
APP2820%code to validate ISR pipeline outputsnot started
APP2810%code to validate IC pipeline outputsnot started
APP2800%code to validate image subtractionnot started
APP2880%code to validate detection of sources in subtracted imagenot started
APP2900%code to validate association pipelinenot started
APP2870%code to validate deep detection and measurement pipelinenot started

*Percent updated in this report

Comments:

Other Activities

Levine Deborah

Reported xx/xx/xx

IDPercentTaskComments
APP900%implement SDQA tool functionality to support DC3b goalsnot started
APP22925%identify tools to support PT1 data analysis
APP2270%manage PT1 data access tools tasknot started
APP2350%design and document SUI in UMLnot started

*Percent updated in this report

Comments:

Other Activities

Lim KT

Reported xx/xx/xx

IDPercentTaskComments
APP60%interslice communicationsnot started
APP70%post spatial matching (stretch goal)not started
APP19275%translate overall production into pipeline, stages, and policies

*Percent updated in this report

Comments:

Other Activities

Lupton Robert

Reported xx/xx/xx

IDPercentTaskComments
APP1390%integrate HEALPix into software stacknot started
APP1000%test difference imaging coaddsnot started
APP12150%adapt astrometry code for DC3b use

*Percent updated in this report

Comments:

Other Activities

Mannings Vince

Reported xx/xx/xx

IDPercentTaskComments
APP2350%design and document SUI in UMLnot started

*Percent updated in this report

Comments:

Other Activities

Monet Dave

Reported xx/xx/xx

IDPercentTaskComments
APP12150%adapt astrometry code for DC3b use

*Percent updated in this report

Comments:

Other Activities

Monkewitz Serge

Reported xx/xx/xx

IDPercentTaskComments
APP14450%move applications code out of stage codestill needs to be done for nightly association
APP16250%modify stage code to conform with new MW APIstill needs to be done for nightly association
APP244100%implement/wrap OPTICS source clustering algorithm
APP2430%implement Detection/Source? associationnot started
APP60%interslice communicationsnot started
APP70%post spatial matching (stretch goal)not started
APP740%characteriztion of and improvements to association cosmic ray rejection performancenot started
? 0%Assist Russel Owen with APP139 and APP94 as needednot started

*Percent updated in this report

Comments:

Other Activities

Mullally Fergal

Reported xx/xx/xx

IDPercentTaskComments
APP8660%implement improvements for WCS
APP15160%implement stage structure for support astrometry code

*Percent updated in this report

Comments:

Other Activities

Myers Jon

Reported xx/xx/xx

IDPercentTaskComments
APP14025%integrate MOPS stages with new C++ KD-Tree tools
APP12975%*run DayMOPS with solar system model Successful track generation for a large subset of our data has been completed.
APP14175%write linkTracklets and unit tests
APP26750%update DayMOPS for new DIASource Table Schema written and tested, stages not rewritten
APP2660%Fix C-linkTracklets Python bindingsnot started
APP2650% make DIASource the smallest unit of Tracks and MovingObject??not started
APP2640%update DayMOPS to stop using template tablesnot started
APP2700%remove DIASourceForTonight table, pass DIASource on clipboardnot started
APP2690%pass Tracks on clipboard rather than via Tracks tablenot started
APP2680%made a new initial stage which gets night numbers from Policynot started
APP2710%add debug printing to C find/linkTracklets and stage code for storingnot started

*Percent updated in this report

Comments:

As we await response from SAT about how to handle too-large-for-memory data sets, the current set of DayMOPS runs were performed using ad-hoc methods (command-lines scripts, later ingest to DB).

Other Activities

We have modified Kubica's version of linkTracklets, which was crashing when the number of tracks in memory became too large. It now writes out track data continuously. This version successfully ran for about 20 nights of data before we ran out of disk space.

Currently we are building estimates of the number of tracks to be generated and trying to calculate cost of doing IOD on them. Currently the numbers are quite large - we have about 1 billion tracks so far and ultimately expect on the order of 10s of billions. We need to reduce IOD costs, reduce our goals, or attempt a different approach if this is to be tractable for DC3b.

Ran experiments with ranging/LSL and examined possible methods to reduce the number of calls to ranging. Found an approach which looks to prevent > 90% of our calls.

Communicated with Larry Denneau about finding a faster IOD method; moving away from ranging could work.

Communication with Mikael Granvik about alternative ranging-based methods which may ultimately be cheaper than the current tracks->IOD approach, e.g. tracklets->ranging->tracklet linking->tracks->LSL

Communication with

Owen Russ

Reported xx/xx/xx

IDPercentTaskComments
APP77100%fix ticket #873
APP13925%integrate HEALPix into software stackthis has changed to implement sky pixelization
APP9425%code to warp images to/from sky pixel representation If we adopt my sky pixelization proposal then this is finished (existing code will work)
APP20950%implement outlier rejection
APP9850%code to create PSF-matched difference imaging coadds (no outlier rejection)
APP25750%code to create deep monochromatic coadds
APP10475%code to detect chi-squared deep detection coadds (no outlier rejection)
APP1030%create a set of chi-squared deep detection coaddsnot started
APP1020%code to measure quality of deep detection coaddsnot started

*Percent updated in this report

Comments:

Other Activities

Shaw Dick

Reported xx/xx/xx

IDPercentTaskComments
APP240*15%DC3b User Guide
APP2420%create uesr training programnot started
APP2340%develop Use Cases for SUInot started
APP239*100%CFHT-LS Calibration Reference Filesdone
APP208*100%define CFHT to be used Done

*Percent updated in this report

Comments:

Other Activities

  • Presented talk on SDQA at SPIE meeting in San Diego
  • Working on SDQA development plan for PDR
  • Responded to a few CFHT-LS calibration file issues encountered during PT1 processing

Van Dyk Schuyler

Reported xx/xx/xx

IDPercentTaskComments
APP22925%identify tools to support PT1 data analysis
APP2408%DC3b User Guide
APP2960%set up helpdesk systemnot started
APP2340%develop Use Cases for SUInot started
APP2350%design and document SUI in UMLnot started
APP26090%define DC3b data quality requirements
APP23050%complete DM system data product quality metrics document
APP1990%PT1 best efforts science data analysis/validation not started

*Percent updated in this report

Comments:

Other Activities

Major Accomplishments

Significant breakthroughs, issues resolved.

  • PT1 Production runs completed


Objectives for the Next Period

What you expect to accomplish.



Problems Encountered and Solutions Being Pursued

Budget or schedule variance, technical issues, management issues.

  • The problems of data accessibility (permissions), availability (location and quantity) and backup to mass storage, were addressed and new policies and procedures are being developed by the ABE and lsst cluster managers.
  • PT1 production pipeline stack startup time lengthens with increases in the number of raw input items. The short-term remedy of multiple shorter runs, was effective. The long-term issue is being addressed.