wiki:MinutesOfImageDetWG-2009
Last modified 8 years ago Last modified on 02/08/2010 12:25:45 PM

Minutes of the ImageDetWG Weekly 'Phone Con.s

Minutes for 2009

2009-01-07

Present: RHL, AJC, Rob Gibson, Martin Dubcovsky, David Wittman, Deborah Levine

U. Wahington:
   REO WCSmatch (i.e. warp).  What's next?  Finished up Kaiser image (but not deconvolution)
       		      	      	     	    Speed up convolution
   NMS: ISR. Running, not defringing.  Done by the end of Jan.
        Next look at what the calibration group are actually producing.  Will need a coding task too (50%?)
   ACB: Diffim kernels; desired number of kernels
   Image sims: WCS from sims is good to c. 2" (it comes from boresight + optical model)
   	RHL: use Fergal's stuff?
	AJC: want it this week.  No USNO-B stars, so need our own catalogs -- USNO-B in March?

Princeton:
   RHL
   Bickerton: Statistics, Interpolation, and Backgrounds.  Using pure noise images from
   	      Russ Laher (now in afwdata); would like simulated data
   Mullally:  Getting astrometry.net working.  Attempting to generate a Wcs (but failing)

   Next: PSF

Davis:
   Ryan worked on pixellisation library; just a set of projections.
   Martin: Measurement algorithms on a private branch
   	   Definition of "Source" is waiting on RHL review
   Dave: Two weeks with no email

Deborah:
   Russ is working on persistence for QA classes

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Next meeting:

RHL will be in Japan for the next two weeks; Andy Connolly will run meetings.

2009-01-14

Attending: nicole, roc, suzy, tim, martin, kt, becker, russell, russ  
laher, dave wittman, ajc, apologies if I missed anyone.

nicole - sick last week. Currently writing unit tests for ISR will  
update ISR to image api.

ajc - we have images from initial sim run and have refit an accurate  
wcs (so they align without needing the astrom code to be in place). We  
need to get them tested before the next image run to fix any bugs.   
Becker - need to warp the images using code formerly known as  
wcsMatch. Updated version not currently working so Nicole will look at  
getting an earlier version working.

becker - aas last week. currently refactoring spatial modelling,  
working on swig
issues for abstract classes and derived classes. KT offered to help  
with swig problem

russell - struggled with build system, now working, building
remapExposure with image api, exceptions etc. Few issues with const
iterators (filed bug) but no major issues. build system issues were   
dependencies
that are in the wrong order, not self-consistent (kt - build system is  
context
dependent so it works for some and not for others)

russell suggests always starting with a fresh package for any install

kt - working on new packages from server, need new eups (timeline for  
this not clear)

ajc - what about using DM packages within external code. martin - do a  
fresh build and create a new package in svn that is a wrapper around  
external code,  kt has templates for doing this

tim - im sim should use dm pipeline as it is beneficial for all (less  
code duplication and it exercises the build system). ajc - concerned  
about the overhead of doing this due to the build system headaches but  
will try this

tim -  ajc to email lsst-data to point to access to images, ajc - will  
do as soon as the new wcs's are in place (this week)

russ - rhl wants real frame (mask, image, variance) with known  
statistics to test background
estimation.  ajc/rhl will talk about how to specify stats of the  
images to see if this should be assigned to image sim or not.

martin - working on ticket 465 (definition of sources), fighting build  
system,  starting to work on deep detection pipeline

dave w - asked about the status of the psf module from penn. rhl  
talking to them but waiting for the build system to settle. Is there  
an interim psf module - not yet. What is the
penn scope for the psf and when is it supposed to start. ajc will ask  
bhuv and mike jarvis to attend next telecon for update

russ - has sdqa python wrappers, wants suggested names for the metrics  
that people will use. SDQA software is on the trunk and ready to be  
turned into an lsst package (dms/sdqa). Queston of how to put it into  
a build (so lsst package install does it). kt - just tag it and file a  
ticket to make it part of the  distribution. Russ on vacation next week

suzy - reviewing status of schedule, wants people responding to status  
requests that are sent out. IPAC working on independent cost analysis.  
Roc going to functional requirement meeting.

Tim - finished up a big update to the uml model which should make it  
more useful for everyone. Hope to roll it out early february. Version  
is using uses cases from ipac meeting so will request people to update  
in a few weeks after the new model is out.

ajc - asked can we get a priority list of tasks for DM so that as we  
finish up individual areas we know which ones should be taken first.  
Suzy will talk with rhl and roc about that.

2009-01-21

Attending: roc, russell, deborah, andy b, ajc, bhuv, martin, robyn,
jeff, mike jarvis

russell - continuing fighting the build - has a working
stack. Sometimes code from a ticket gets checked into the trunk but
not the complete code (i.e. not self consistent merge) which then
breaks the trunk. Martin ran into this with "svn merge" not
svnmerge.py. This might be the problem. David Gehrig looking into
this.

remapExposure is done - needed to generate the warped images. It will
be used for the simulated images but afw is currently broken (wcs
broken in the trunk - russell filed a ticket assigned to
fergal). Bilinear and lanczos are inplace for remapExposure.

andyb - dealing with swig and derived classes, possible bug in swig
shared pointer derived directive (serge tracked this down). Working on
difference imaging of the simulated images.

ajc - initial set of images in place and looking for feedback about
changes before generating the full data set (email sent to lsst-data
about this)

martin - dealing with swig problems with derived classes (inheriting
from derived class where derived class was not templated). There is a
swig FAQ on the wiki but still requires gaining experience with swig
to learn its issues.

deborah - russ making progress on sdqa. Wants to touch base with Bhuv
about psf requirements

roc - assembling requirements for visualization tools for DM (plus
camera and telescope). Will send note around about wish list for
requirements.

robyn - nothing specific for apps. Working on software development
plan across LSST.

bhuv and mike jarvis - Current PSF work is a module that takes a list
of pixels (image) and calculates the centroid and shapelet decomposes
a star (up to arbitary order). It allows for the scale size of the
decomposition to vary. Requires isolated stars currently. Output is a
new centroid, scale and coefficients (10th order typical - 66
numbers). Written in C++ and standalone. Becker - is this the model
for photometry and weak lensing?  Jarvis - not tested for photometry
yet but worth investigating (other groups have tried shapelet code for
photometry). In a position to try integration - needs to hook up with  
rhl
to get started. Not on near term work schedule to rewrite for the LSST
apis but it might be simple if he wraps the footprint classes

2009-01-28

RHL REO Roc Suzy Levine Bickerton Mullally AJC Dubcovsky Wittman


10am Pacific; 1pm Eastern


Washington:
	REO:  Warp exposure;  swigged but not tested
	      RHL: Speed?  Who knows -- using gcc 4.0
	ACB:  Using simulated images.  Verified FITS files, produced MaskedImage inputs
	      Tracked down swig + derived classes with diffim. --- running on sims
	      Waiting for released daf_data for API conversion
	NMS:  Out for the last fortnight.
	      ISR is checked in.
	AJC:  Delivered sims.  Initial WCS fixup.  Want feedback for production run within a week.
	      DGM: stars too round
	      Initial WCS on boresight (10").  Fitting polynomial WCS to chips
	      Sims available from UW;  will be from NCSA.  AJC is point of contact

Princeton:
	RHL:  Japan, Speed, CR is fast enough. Looking for things to.
	Bickerton:  Statistics/Interpolation/Background all in afw, even doxygen
	      Problems with installed scons due to envAppend.  Fixed sconsUtils to
	      use envPrepend.  Pulls system python into path
	      Getting randoms etc. into afw.  Need to control seeds. -- low priority
	Mullally: Astrometry.net to build WCS from scratch.  In meas/astrom/net
	      Need to get astrometry.net in distrib system
	      Building on ubuntu system -- problems.
IPAC:
	Levine:  Mostly on PDR related stuff.  Not much to report
	Roc:  Meeting with Jeff Kantor + SysML

Davis:
	Dubcovsky: Definition of DIASource and Source (#465) -- inherit from baseclass.  Scared of merge
	      REO: We already had DIASource?  No, it was called "Source" but no deep Source
	      Working on detection pipeline.  Read sets of images, ..., back to measure
	      Centroids?  Shapes? No, not yet.
	      REO: Working middleware?  Martin: I don't yet know
	      
	
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

DC3 Schedule	Deadline April 1st --- all testing done (but report not written)
    		Week of 9--13 March:  Integration week @ IPAC  -- maybe 10-12 people.
		All code checked in/tested by 1st March
		? Do we need to move apps from DC3a to DC3b ?

Speed		Diffim.
			Convolution	REO
			Matrices esp. eigensystems	Fergal & Toby.  Inform ACB
		gcc 4.3:
		    Washington: AJC will investigate
		    Princeton:  RHL will investigate
		    Davis:      Doesn't care about speed for DC3.

WCS		AJC:
		What WCS is supported?  Per-chip tangent plane not good enough (c. 1pixel)
		2nd and 3rd order polynomials get to 0.1pixel
		Haven't looked at colour dependent terms in astrometry
		We're using wcslib 4.2 internally; AJC will check.

		RHL will talk to Axelrod on generating good WCS from linear

		Mullally will investigate building indices from AJC's catalogs

		Mullally:  need 20ish good sources per field
		
Matrices	boost::ublas?  gsl + good BLAS?
		ACB would be happy with GSL.

PSF		Bickerton + ACB
		AJC: sims good enough for this.

2009-02-04

February 4, 2009 1:00 pm : Applications phone con

Dave Wittman, Martin, RHL, Suzy Dodd, REO, Roc, ACB, Deborah Levine

Princeton:
       RHL:     afw updates. meas_algorithms/meas_astrom with afw 3.3
		Working on Centroid code (using Naive and SDSS;  run-time configurable)
       Fergal:  On mountain.  Working on BLAS.  WCS: Use SIP?  Roc: can help; it's in the FITS standard
       		Roc will look into libraries.
		SIP is the standard to beat.
       AJC:	Need forward and reverse for DS9
       Steve:   Thinking about PSFs;  pry away from SDSS

Davis: Martin: Playing with Hello World in pipelines.  trac:PipelinesFramework
       	       Using latest version of all but pex_harness
	       Source/DiaSource on trunk -- ugly to write; OK to use.  Will be in afw 3.3.1

	       Post-DC3a review:  Design for DB or Applications?

IPAC:  Deborah: PDR stuff;  SDQA requirements
       Russ Laher: 
       Suzy:    Will everyone please look at the DC3a schedule;
       		http://dev.lsstcorp.org/trac/attachment/wiki/DC3Schedule/LSST-DC3-udated-020209.pdf

U. Washington:  ACB: Working on MaskedImages
      		     Diffim works on simulated images.  Better than for real data...
		     RHL: scary..
   		AJC: Why ?
		RHL: Because I thought we had most of the bad things in.

		AJC: Two DC3a runs. No USNO-B stars.  Time to start thinking about next round (e.g. asteroids)
		     Roc: can we do sims to move MOPS off their sims?
		     RHL: catalogue is good enough, but detection efficiency?

		REO: Working on warp exposure.  Nearly there...

		NMS AJC reports: was in yesterday; will be working at some level.  Will get together a to-do
		    list this week.  RHL: We need lists of defects.
		AJC: should keep in DC3s

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Roc: There's a meeting in Davis next week to discuss DM requirements and scope
     RHL feels conflicted:  Science Council v. member of DM team --- are the requirements ambitious enough?




REO: when are the boost 1.37 packages due?
     RHL: no idea

Suzy: Schedule is very daunting --- code has to be tested and working by March 9th.

2009-02-11

Present: RHL, REO, Russ Laher, Martin Dubcovsky, ACB, AJC, Suzy,

RHL: Report on Davis meeting.  Working on measuring centroids/shapes.   
Fixed some WCS stuff.

REO: ISR -> new APIs; went quickly,  Back to Nicole for now.
     Working on warp -- need to grow masks by 1 pixel.
     RHL: speed?  REO: still on g++ 4.0

ACB: Converting to new Image API, using new classes (e.g. Footprint  
Functor).
     Tracking down bugs;  confused by Footprint code.
     Working on gcc43 in eups.

Suzy will talk to NCSA and see if they can help with getting gcc43  
versions
of the framework deployed.  Note that they'll need new version numbers,
so we can mix-and-match the two sets of packages (this is caused by the
boost library's soname conventions)

AJC: Sims are going.  Data out at the end of the week.


Martin:  Playing with pex_harness stuff.  A long way towards being  
usable; see #582.
	 KTL has parallelization concerns.

	 Moving towards, "Compute coadd" stage.

Russ: Distracted for Last week or two with Palomar Transient Factory,
      but now back (and other project is helpful).  Implemented LSST
      SDQA schema in PTF database.  Started using Java tools from
      Spitzer project in PTF.  No problems (yet) in LSST schema ---
      implemented two schema, both LSST and a simple table to compare
      usefulness.

      In LSST: have tagged SDQA package, with a few rudimentary
      classes.  Developers should be able to put SDQA objects onto
      the clipboard now.

      Looking forward, need bare-bones SDQA pipeline --- takes data
      of clipboards;  thresholds, formats for DB.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 
=-=-=-=-

DC3 discussion of delay from IPAC to NCSA cluster, can be up to 2s.   
IPAC
network group is investigating.  Dave Gehrig is looking at the NCSA end.
Suzy hopes to have an answer by late Thursday.

RHL: 2s is a show-stopper.


Factories. I'm beginning to implement creation of e.g. Centroid
algorithms via a concrete factory, createCentroid


Status:
	Diffim:  in a week and a bit
	ISR:     Unknown
	WCS:	 SIP.  No number on performance
	PSF:	 Almost all pieces written
	2nd moments	Will be there

2009-03-25

RHL, David Wittman, Russell, KTL, Deborah, Vince, Suzy, Martin


KTL: Bottomlevel middleware is stable (but not yet a metapackage)
     Harness etc. is also OK.
     Ctrl/orca needs a provenance fix --- maybe one more top-level

     Francesco:  Night MOPS
     Serge:      AP with faked up DIA sources
     Subtraction/Sims both running
	ISR  -- flat fields, biases, linearity, defect removal
	WCS/PSF on one exposure
	Difference both exposures wrt template
	Add differences and detect.
	Measure on each exposure
	Persist DIASources
	Persist SDQA ratings

     Timings:  diffim 80s per amp single core (was 3mins)
               100s for coadd-and-detect
	       30s  dia-source measurement

     Some dipole residuals --- are we ignoring SIP corrections?

     Warping is slower that swarp by...

Analysis:
	Diffim             ACB
	ISR                AJC
	PSFs               RHL
	SDQA               Verify that it's persisted
	Diffim photometry  (only 1 real source in first CCD?)
	Classification:    Bickerton
	WCS:		   Fergal

Prepare for DC3a debrief:
	What algorithms should we be using?

	Is the QA good enough?  What more do we need?

	What do we need in sims for next generation?  Look at CFHT and ask if
	we need it in Sims.

2009-04-01

Lee Armus, Suzy,  RHL,  Roc, Vince, AJC

RHL:  Worried that e.g. we weren't handling -ve detections.  I fixed all the
      ones I found, but they were pretty low level.

IPAC: QA's doing what we asked it to do

RHL:  Did you look at Sims?  AJC: No.

RHL: Need stable arrangement of disk layout: Roc yes.

RHL: Are we using the SIP distortions?  Does anyone know?  I'll ask Fergal to look.


Suzy: Postmortem meeting.  Please send comments to Suzy.

Roc:  Please start thinking about TCT recommendations.
      1/ How to encode a NULL in a row in the DB
      2/ Can column names include units?

AJC: What's the best machine at NCSA?  RHL: I use lsst8

2009-04-08

Present: REO, Roc, Deborah, RHL, AJC, ACB, Martin, Steve Bickerton, Fergal

RHL:
	PSF

Steve Bickerton:
        SDSS-III.

	Was working on aperture fluxes with tapered circular apertures.  Needs c. 1 week
	more work.

	RHL: A standard aperture is 1 iff r < a, but that's
	an arbitrary choice.  In SDSS we calculated the exact
	weights for an arbitrary aperture (assuming band-limited
	data), but that required us to sinc shift the data to be
	centered in a pixel.  If you smooth the top-hat filter
	with a band-limiting function (e.g. the PSF) you can 
	instead shift the coefficients.

	There's a discussion of the SDSS apertures at
		http://www.astro.princeton.edu/~rhl/photomisc/aperture.pdf

Fergal:
	The problem appears to be in the astrometry.net SIP solutions. We tried backing off from 3rd order to 2nd order and things improve. This afternoon we'll try using more stars for the SIP than the initial stages; if this doesn't work, RHL thinks that we'll have to run an object detector on the template to have enough stars to do a good distortion map. 
	
REO:
	Kaiser coadds, and fighting badly factored e.g. sourceDetection

AJC:
	CMU (Chris Genevieve) is looking at the bright-source case.  It doesn't
	seem to be much better than Kaiser's faint-source limit.

	They are doing the deconvolution step.

	Have something in 2-3 weeks.   Will discuss on the 6th of May.

	REO:  what are Pan-STARRS doing?  AJC: an expedient naive coadd (according to Paul Price)

	RHL:  can use for detection anyway.  REO/AJC:  we knew that.

ACB:
	Working with REO/AJC on PSFs for coadds.  Need to work with RHL.

Roc:
	The latest TCT measure on variable names:
	    Naming conventions --- names should not contain units (e.g. Mag)
	    	   Motivation:  define units in metadata.
	    	   Roc:  pushback is probably warranted

		   Please review proposal (URL) and let's discuss next week

	    How do we represent bad values?  External representations to be
	    loaded should be explicit.

	    E.g. WISE prints out file with NULL in the column (but how does
	    the C work?)

Vince was working with Robyn on UML model for SDQA.

Martin:
	Timing on diffims?  I was looking at the code, and wondered why
	QR not SVD?

ACB:    The matrix maths is sub-dominant to the convolution code (c. 10%)

RHL:    Please add metadata as to what method you used.

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Tim passed two questions on from Jacek:

1.  He is carrying space for four subtraction templates for every field and filter.
    This is to allow for templates to differ based on seeing and airmass.

    ACB:  We need the airmass more than the seeing.  Can test the seeing with
    	  different CFHT templates.

    RHL:  Speed penalties for a large dynamic range in the input seeing,
	  as we need a larger kernel

    ACB:  Need large kernel anyway for bad seeing (as core/wing vary separately)

    RHL:  I don't see a way to be clever with different airmasses as red-blue
          pairs separate.
	  
2.  Does the detection coadd need to be per-filter or not?

    AJC:  Are we going to only detect on a chi^2 coadd?

    RHL: This depends on the detection prior.

    AJC: chi^2 optimal for sources close to colour of the sky (or noise or the sky?)

    Roc: we do a multi-band detection in WISE by detecting to a fainter limit (e.g.
    4.5sigma not 5sigma) in a coadd of some sort.  Simulations show that this works.


RHL:  So we don't really know the answer to either of Tim's questions.  We'll 
    continue to think about getting him an answer.

3.  AJC:  SEND ME THINGS IN THE NEXT GEN. SIMS BEFORE NEXT WEEK'S SIMS MEETING!

2009-04-15 (Tax Day)

RHL, Martin, Gregory, Suzy, Craig Loomis, ACB, Fergal, Russ Laher, Roc

ACB:
	Nothing much.   Preparing DC3a PSFs for CMU

RHL:
	Mostly SDSS-III;  working on WCS with Fergal, which brings us to:

Fergal:
	Bad WCS is causing bad subtractions.

	Distortion terms from templates are different from those in the science images.  We're trying using catalogues
	derived from the templates to define our science-frame WCSs.  This requires us to
	generate new index files for astrometry.net --- he's got Dustin working on it

	Roc:  should we be solving for the distortions on every exposure?  RHL/ACB: probably not
	in the long run

Suzy:
	Is this convergent?  RHL: I think so.  If needs be, we'll write our own polynomial fitter
	to fit catalogues from the template images to the science images.

Martin:
	Looking at speedup on diffim (failed).  Coming up to speed on mathematics of multifit

Russ:   Yesterday I got back on the SdqaRating formatter (-> DB; needs to work on other
	formatters)

Suzy:
	Sent out first draft on application writing assignments --- now's the time to
	volunteer for more sections.

	Please write apps. monthly report!

	Breakout sessions at the DC3[ab] meeting.  E.g. MOPS; what else do we want?  SDQA, ...

	RHL:  Please look at http://dev.lsstcorp.org/trac/wiki/DC3APostMortemMeeting page
	and edit by the close of work Sunday.  Then prepare list of breakouts for the
	PM meeting on 2009-04-23

RHL:
	Did anyone read the TCT on code standard (deafening silence)

	This will be discussed next week, so please read
	     http://dev.lsstcorp.org/trac/wiki/CcbMeeting20090401

2009-04-22

Minutes were written and lost.  Mea culpissima

2009-04-29

Present: RHL, Suzy, Lee Armus, Martin, Deborah


RHL:  Fixing DC3a problems and recovering from a broken laptop.

      Is anyone looking at DC3a outputs?
      Suzy:  I'm holding off getting IPAC people involved until the new astrometric runs are done

      Suzy:  do you have DC3a goals?
      RHL:
       We could do Photometric calibration (at least on CFHT D2, D3 where I have SDSS catalogues)

       Split-exposure processing;  probably worth a factor of 2 in speed, but tricky
       to test scientifically as the current "exposures" are identical + fake CRs

       Whether to implement this for DC3a is probably a political issue for Tim and Jeff

IPAC:
	Suzy:  working with David Gehrig on DC3a post-mortem agenda + breakouts
	       taking with Jonathan Meyers and Robert Jedicke re MOPS

	Deborah:  Working on cost estimation by end of month

Martin:
	Talking with Jim Boesch about multifit architecture proposal;  try to get something out soon

2009-05-06

RHL, David Wittman, Fergal, Deborah, Roc, Russ Laher, Martin, Russell, Robyn, AJC, Steve Bickerton, Suzy

RHL:
	Speedup on Kernel estimation/convolution speedup
	AJC: Would FFTs help?

Fergal:
	Working on WCS.  Bug until last night in new index file generation
	Looking into current problems (namely that subtractions look v. poor)
	REO: Does this explain #801 (BBox outside template)?
	Fergal:  didn't see any catastrophic errors

Bickerton:
	I'm back on LSST as of today

Suzy:
	I'm totally behind on DC3 breakout agenda.  We can't cram this into two days; prioritize
	things that we need to discuss face-to-face.

Deborah:
	Working on costing

Russ:
	Nothing to report

Roc:
	At DES director's review.  In pretty good shape for spring 2011, using lots
	of existing software.
	AJC:  Science groups are doing all of basic code (e.g. SNe are doing diffim)

Russell:
	Talked to Simon Krughoff about ISR.  Working on #801

Martin:
	Background subtraction:  are we persisting (ask KT?)
	WCS QA?  (RHL: not much --- we need more).

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Template Generation:
	 AJC:  With Chris Genevieve and Darren Homrighausen at CMU.  Trying 
	 to do better than Kaiser.  Implemented variation of Kaiser,  comparing
	 with straight average (no PSF convolution) --- preliminary analysis says
	 no improvement.  Will implement within LSST code; have built under Linux.

RHL:
	What's the best way to prepare for DC3 meeting?

	We propose to hold the following ordered set of meetings:
		Tucson	May
		Phone
		Aspen	June
		Phone
		Davis	August/September (if we need to; subject to funding)
	to get our acts together for DC3b

2009-07-15

Present:
	RHL, Suzy, Tim, David Wittman, Russell, Steve Bickerton, Lee, AJC, Fergal, Simon, Jim Bosch

Agenda:
	Status and confessions:  No one spoke up
		Ticket #835 (Wcs) -- Merged to mainline;  required hack to class and
		also the formatter.  Fergal's adding a unittest

		Will change Wcs to only pass "proper" keywords to wcslib

	-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
			
	Status of DC3a runs (is rlp1234 good?); if this is covered in the DC3 phone con we can skip it

	Tim:  no gross problem in D3. 	Will D4 too.  No difference between 1233 and 1234; am waiting
	on IPAC (maybe faxe-to-face on monday).  Offsets c. 20mas
	RHL:  I should have saved the catalogues I created from the templates to generate the
	astrometry.net indexes

	What's Becker say about diffim?
	Tim:  I don't think it's working well
	Becker thinks that deconvolution is a problem, but why is there any deconv.?
	RHL:  Deconv kernels look smooth but convolution looks noisy; this is weird;
	      Steve Bickerton will be visiting Seattle to work on this in early August
	AJC:  Same problems with simulated data;  used sky-free images as templates

	Work left over from DC3a.  In particular,
		Astrometry:   Is it good enough?  Reliable enough?  Do
			      we need a new game-plan for determining the
			      SIP distortions?

		RHL:  need 3-5 mas for week-lensing
		Tim/RHL:  let's write our own SIP fitter.
		Minuit?  Minuit2?  Matrix inversion?  (N.b. minuit2 has problems on Linux --- assigned to ACB)

		QA:  Are the proper metrics being generated?
		     Astrometry:  Tim: No.  
		     		  Fergal:  Would need to hack the astrom.net code
				  RHL:  If we wrote our own SIP fitter that'd do the QA
		     PSF:  RHL: Probably not.  I did a bad job
		     Diffim:  Nothing in for DC3a is useful, with possible exception of kernel sums

		Summary:  We need to test QA metrics as unit tests.
			  RHL'll check with Robyn about adding to QA testing

	PSF/multifit meeting plans
                11 --13 August

	Resolution of #806 (should we set edges of convolved images to NaN):
		http://dev.lsstcorp.org/trac/ticket/806

	Russell: Pixels that we can't set should be NaN (as well as a mask bit,
		 variance == Inf).  Fine for warping.

		 We also need to handle pure Images (no masks)

		 For convolution we could do this or copy the input pixels.

		 RHL:  it's useful to keep the raw pixels around the edges,
		 e.g. so you can see them

		 ACB in absentia: If kernel offsets the image, then the edge pixels are not well defined.

		 Steve:  Statistics operators (probably) ignore NaNs.  Masks are not (yet) supported

		 RHL:  I'll probably cave in on this.  Post meeting:  I thought about external, non-LSST,
		 users of our stack, and I don't see how we can force NaNs on them.  So I, as afw-coordinator,
		 have decided to:
		      1/ Make NaN the LSST default
		      2/ Provide an option to p

	How should we handle sub-image coordinates?  See Russell's proposal at
		http://dev.lsstcorp.org/trac/wiki/TCT_PixelCoordinates

	RHL: Did anyone read this?  No.  Let's do this by email.
	Russell:  Is this TCT?
	RHL: I'll check with Robyn (answer: we don't need to pass it by the TCT)
	Jim B:  What's a Parent image?  Will annotate document (He's done now so)

	Suzy: What do we need to do in preparation for 27/28 meeting?
	      Do we need an applications proposal prior to the meeting?
	Tim: Yes.  From the IPAC meeting minutes.
	Suzy: I'll make a summary

	Next week: I'll run meeting from APO
	     Main topic:  DC3b planning 

2009-07-29

Present: RHL, Roc, Suzy, Jum, David W, Martin, KT, Simon, AJC, ACB, PzW, Deborah

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Suzy:
	Estimates should just be coding/meeting/test writing time -- not debugging

	Look at http://dev.lsstcorp.org/trac/wiki/DC3bScopingMeetingIII

	Links on this page should be updated (although they refer back to the
	May meeting);  Suzy has added lists at the bottom of the page.

	We want milestones at least once a month (clocktime);  if delivering/receiving
	something that should be a milestone too.

	KT:  Jeff would like tasks in terms of durations, rather than due dates;
	Suzy OK; durations are fine.

	Suzy:  I did this for MOPS and SDQA, everything else is under "Science/Algorithms"
	as far as I know it's complete, but please check and edit the page:
	   http://dev.lsstcorp.org/trac/wiki/DC3APostMortemMeetingScienceAppsI

	You should add your work estimates to this page -- with the correct names. Please call
	out dependencies.  Due date:  today (29th of Sept)!  We'll discuss it on the 5th August,
	so please try to get it done ASAP.

2009-07-05

Roc, RHL, David Wittman, REO, KTL

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Work for DC3b
     Roc:  Just do what needs to be done; nothing more.
     	   What are the key things?  Can we Triage?

We looked at Suzy's plan and the following thoughts came up:
   	Don't generate fringe frames --- this is thought to be an error in the plan

	Is AP in apps?

	Need deep coadds for first pass to multifit.   Coadds need to be defined better
	REO: No Kaiser coadds, just PSF matched using diffim kernels

	REO/ACB: need more info about PSF fits.
	Wants to fit bright and faint stars separately for QA

	WE need these for diagnostics on coadds

	Sky pixellation:  REO:  generate in tan projection?
	    		  KTL:  no,  please generate coadds in Ryan's space-filling-curve
			  grid (all: scary for DC3b)
			  RHL:  is this scope creep?  DW: yes
			  KTL:  for image access this is roughly the same --- I just want one
			  	pixel per point on the sky
			  Roc:  agree with RHL

			  Where does HEALpix fit in?   Or should we go with
 			  Overlapping tan projections, with overlap width ~ 1 amp/CCD.

			  We'll decide for DC3b next Wednesday;  RHL will send mail to list
			  asking for input

			  Who is going to choose inputs for coadds/define tan planes.  RHL: TSA! 

	PSF photometry:  It's basic, and we need this for S/G separation.   Bickerton
	    		 has most of the aperture code written

	Shapes/Multifit: Shear + galaxy photometry? UCDavis thinks they'll be OK if their dependencies
			 are available

	Roc:  Suzy and I can work on dependencies.  She's (probably) back at the end of the week.
	      Roc'll start now.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
				  
Agenda for PSF/Multifit.  RHL: Please send agenda items

2009-08-26

Present:  Roc, KT, RHL, Suzy, Simon, Steve Bickerton, Deborah, PzW, Martin, Fergal

Simon:
	ACB (in absentia):  diffim.  Some progress, still noisy kernels.  On vacation
	Simon.  What needs to be done for ISR?
		Fringe frame corrections --- is imSim going to generate fringes?
		Metadata to describe the camera (biassec etc.) -- also want a class
REO:
	Implemented normalising linear combination kernels (just waiting for a review)
	Problems with variance in linear combination kernel

RHL:
	SDSS3 + prep for PSF/Lensing meeting
Steve:
	Nothing.  Back to LSST on Monday at latest
Fergal:
	Extract internals from Astrometry.net for QA.  Preparing false inputs
	It'd be nice to have external images with bad distortions
Martin:
	Back from vacation;  basic models in multifit
Deborah:
	Waiting on DC3b quality goals --- i.e. waiting on Tim
Suzy:
	DC3b plan.
	
Pixellation
	KT: there are space-filling code packages out of Russia and ???
	REO: for warping all I need is a pixel --> sky and a scale
	RHL:  HEALPIX it is for DC3b.  RHL: get this into stack
	Simon: will HEALPIX need more than 64 bits to get to the needed resolution of ~0.1"?

Suzy DC3b.  Preconditions and when we need what

List of tickets
        Everyone should triage tickets and add DC3b ones to task list, maybe
	as "fix my 10 tickets"

Please move code out of middleware (i.e. stages) and into apps

2009-09-02

Present: RHL, Martin, Suzy, Russell, Deborah, Dave Wittman, Gregory, AJC, Jon Meyers,
	 Lee Armus, Roc, Steve, ACB, Jim, Simon


RHL:
	SDSS.  BOSS red CCDs (from LBL) have, umh, interesting CRs

Suzy:
	Sent out request on Monday for comments on DC3b schedule --- still need RHL, Simon,
	Tim, Jon Meyers.  Wanted by Thursday, please.

	Also working on Dave Silva's operations plan

	N.b. Science synergy between LSST and PTF (Palomar Transient Factory) meeting tomorrow

	Tim sent out DC3b SDQA requirements -- comments by Tuesday, please.

Martin:
	Waiting on RHL's PSF API comments.

	Designing Fourier classes, trying to base on afw classes.   Want to replace gil::image
	Will write a ticket

	PSF: belongs to Martin.  Some debate between Russell and Jarvis as to where convolve goes
	RHL will investigate by email and resolve next meeting if not before.

Jim:  would like to put ellipse code into afw.  Ticket please.

Fergal
	Working on SIP code.  Using SourceMatch class to match up list of Sources, and
	rematching as the WCS changes.  "The QA is going to be absolutely great".  A fortnight.

Steve:  Working with ACB on diffims.  RHL: do you need support?  No, we're OK.
	No --- the installed Python dumps core
	RHL:  use the system python and lie to eups

ACB:
	Unhappiness about os/x installs.  RHL will find out NCSA's position.  Gregory
	will talk to Jeff etc. if needs be

Russell:
	Trying to stop being sick.  Little bits and pieces

Simon:
	Writing a proposal to describe the camera.  Based on Jim Pisagno's imageSim
	team description of the camera.  Will provide a class to describe the camera

	Gregory:  This is a system engineering group issue.  Please provide a proposal

2009-09-09

Present:

RHL, REO, Gregory, Martin, AJC, Becker-Bickerton, Deborah, Fergal, Suzy, Dave Wittman

State of DIA:

Sent out emails before phone con.  Regularisation constrains delta function kernels
to look like Alard-Lupton
RHL:  so the variance of the diffim goes up?
ACB:  Mean of diffim within the footprints is smaller with a delta-function kernel

AJC: The deconvolution used to look better;  does it still?
ACB: I don't know;  needs spatial modelling.  Propose switching to new spatial models.

Coadds for diffim.   TSA didn't think that he owns it.
REO:  I will write the code.  Someone has to decide how to choose input images;
      we gave this to TSA.   RHL will check with TSA

Glocal photometry, "Ubercal".   Suzy: I thought it was out of scope,
but Tim wants to do it.   Gregory:  Tim really wants to do this.

AJC:  Can't you do this from calSim?
Gregory:  Maybe; or maybe a hybrid.   TSA, Gregory, and AJC will discuss.  TSA's
	  thought to be worried about subtle spatial variations

Photometry didn't make it into image characterisation, but should be added;
good to 1%.   Also need to spec the input catalogues --- AJC will ask Lynne Jones.

Measuring the DN is in scope --- probably 1% but check with TSA.

Fergal will stagify Dave Monet code

Stellar pops working group are worried about multifit;  RHL will talk to Abi Saha
Gregory:  What are the requirements for scientific use of DC3b?  What do we need
to do to get science working group buy-in?   Move this to SAT.

RHL: Are PSF APIs settled?
Martin: Negotiating with KT.  Other stsake-holders are happy.
RHL:  I'm concerned about factory functions

RHL: Multifit's not using afw's Image classes for DC3b as RHL/KT want images
to always own their data.

Jim/Martin:  Want sign off on ellipses

REO:  normalisation of linear combination kernels is in your inbox.  RHL: give it to someone else.

REO:  When should we cut afw?
RHL:  I'll send out a call for inclusion and cut by the end of the week.
      Don't need to wait for ellipses

2009-09-16

Present:

RHL, TSA, KTL, Roc, Fergal, REO, Gregory, Suzy, Russ Laher, ACB, Simon,
Martin, David Wittman, Jim, Jacek

No-one from Penn;  RHL will ping.

Interesting things.

UW:
	ACB:  difference code speedups,  probably SSE due to Eigen
	Implement Alard-Lupton kernels.  No spatial modelling yet.
	Regularise delta function basis functions.

	Lessons learned:  Either Alard/Lupton or regularised Alard/Lupton
	can be tuned.  We don't yet know how to do the tuning.

	Steve:  can regularise lots of ways;  e.g. 1st, 2nd, 3rd derivatives.
	Must worry about boundary conditions.  Some are better than others,
	e.g. diagonal banding in Kernels --- maybe problems in cross-term
	determination.

	RHL: please get the spatial stuff working.

	All code's on #876

	Bickerton:  get AL working, then fiddle.

Simon:
	Nothing.  Get camera description out today.
	RHL: are you responsible for ISR?

RHL: please sign up for trac categories

RHL:
	Put Subaru data through LSST python

Fergal: SIP distortion code.  Using afw::SourceMatch; and LSQ fitter.
	Untested.  Will chat with Dustin.

Steve:	Cleaning out all numerical recipes code from the stack...

	Gave up on GSL integrators as they don't understand functors;
	they use some C hack (struct with a function pointer and void *data)

	Gregory:  could you post the details to lsst-data?

Dustin:
	Hello.

UCDavis.   New PSF API proposal
	We think it's nice, and people will like it.  Please read.
	Meeting Thursday/Friday to sign off if we can't get signoff by email.

IPAC:
Suzy:  Last call today for inputs on DC3b schedule.
       MONTHLY REPORTS are due

Russ Laher:
	Trying to ramp back up on LSST and escape PTF.  We want to do 2.1,
	WCS errors.  RHL:  Please work with Fergal --- Tim's the point man
	for defining what we want.

Roc:
	Started to listen in on MOPS.  They've made progress in getting
	day-mops into framework,  with C, openOrb, python.
	
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

GPUs.  Meeting with folks at Split, Croatia.  Still need to identify
       individuals in apps.  Will conduct a poll on lsst-data.  TSA:
       I have experience!  Mario recommends CUDA at this point.  

Failures.  Please ensure that all your products are passing tests.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

KT:	Wants input on inputs...

	What are the pipeline/stage inputs;  do they come from a database?
	E.g. how are the astrometric source lists split up?  How are their
	sky tiles defined?

	Do we need things that need to be inherently updated in place in
	a DB?

	RHL:  how is calibration done?

	KT: astrometric calib adds columns.

	TSA:  photometric calib could work the same way, adding calibrated
	magnitudes.  

	KT:  I don't hear any need for live access to the DB during
	pipeline operations.

	TSA:  One exception in LSST (not DC3b) is alert production.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Can Roc run meeting next week?  Yes.

2009-10-07

Present:
	Gregory, RHL, ACB, Mike Jarvis, Bickerton, Fergal, Jacek, REO, Russ Laher,
	KT, Meyers, Martin, Wittman, Wozniak, Simon, AJC

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

PSF  API

Martin:
	Kernel is changed.  #905

Mike Jarvis:   I can work with what what we have.

Martin:
	Have we had trouble with std::complex and swig?
	[Silence]  Will post example.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

AFW

Outstanding changes are:
	#905	Change Kernel API
	#955	Removal of NR (1 hour)
	#956	DetectionSet -> FootprintSet

RHL will cut today or tomorrow.

N.b. will break other packages; owners should expect to fix or face
the wrath of the BuildBot.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Coadds:

Russell:  No outlier rejection.

TSA:  How do we handle moving objects?

RHL:  Just use masks from diffims?
TSA:  What about variables?
KTL:  We don't have diffims before we build templates
RHL:  Ah.

RHL:  I'm concerned about the cores of bright stars, but PSF matching helps.

RHL: The course of least resistance is to do outlier rejection.

KT: Have to worry about memory issues.

RHL: How many pixels do we lose at the edge of the CCD?  23 pixels.

REO: Will add this to the statement of work.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

State of diffim.

ACB:  Got spatial stuff up, but tested.  No numbers yet.

RHL:  state of ip_diffim tests?

Wozniak:  Could compare with standard A&L code.

RHL: Please make this happen

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Focal plane description.

RHL:  Document, not code, from Simon?

Simon:  Yes.  Need more comments (got them from KT and TSA).

RHL:  on disk?
Simon:  from FITS
RHL:  No;  I think from config file

KTL/TSA: Split per-exposure from configuration from static camera description

KT:  Probably don't want to use Andy Rasmussen's code as is for now.

RHL: Do we need CCD heights etc. for physical models of PSF?

Mike:  We don't currently use this

AJC:  Should feed back to Andy Rasmussen

RHL:  Let's concentrate on focal plane geometry.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

WCS:  Wcs should use coordinates relative to the origin of the CCD

TSA:  Would be nice to do it now...

Fergal:  OK, OK, I'll do it.  Please send email traffic (REO will do)

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Source for DC3

Jacek: Schema for big tables for Source and FaintSource.  My FaintSource schema
       is 4 columns (id, exposure id, flux, sky)

TSA: FaintSource and forced photom are 1-to-1.
     Also Source 5-sigma full-up measuring for at least DC3b

Let's take 10am Pacific on Friday 2009-10-9 to further discuss.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Suzy:
	Can you remind folks to take a look at this Apps schedule and send comments to me?  I have already received
	comments from Russell and Andy B.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

The TCT decided to use radians internally and in APIs.  Externally (files, DBs) use degrees

Some dissent, which should be aired to the list.  Basically,  things like ra, degrees,
angular tolerances are not [usually] passed to trig functions, so it's much easier to
say, "Match within 5arcsec" than "Match within 2.42e-5 radians"

2009-10-21

Present: Jon Meyers, David Wittman, RHL, TSA, KTL, Suzy Dodds, Fergal, Deborah, Martin, ACB, Roc, Russ Laher, REO, Steve Bickerton, AJC

RHL: Talking with Monet.  RHL will get his centroider into the stack,  
and then
     Monet expects to query DB for stars.

     TSA:  Need to get output back into LSST stack.

     RHL:  Will talk to Monet about integrating his analysis code.

Fergal:
     One function away from SIP from list of points --- then debugging.

     Need to be able to calculate forward matrices (A and B); need Ap  
and Bp
     maybe stealable from Dustin.

     Technical issue:  SourceMatch is a std::tr1::tuple and Serge wrote
     custom typemaps in one direction.

Bickerton:
     Hand-coded (perl) parasoft-like checker.

     Suzy: state of stellar photometry?

     Got the aperture mags more-or-less right.  Now to see if  
everything works.

     RHL:  How do we go about running photometry on e.g. all the  
exposures
     of some CCD

     TSA:  Install orca etc. and run DC3a framework.  TSA has blog/ 
notes
     that he could post.

UC Davis:
     Suzy:  Multifit looked like 3 serial tasks, but you're part way  
through 3.
     Martin:  Yes, that's right.  Part of this is iterative design

     Martin:  porting non-LSST code into LSST (first an ellipse  
library,
     then some models).  Then multifit-fitter.

     Jim:  I'm a bit more optimistic.

UW:
     ACB:  ip_diffim unit tests and doxy.

     RHL:  What's the chi^2 of the difference image?

     ACB:  We're not there yet.  Working with Wozniak

     Simon [in absentia]:  Internal review --- he's a juror this  
week.  No
     thoughts yet on disk format.

     Policy files are acceptable

     AJC:  What do we want from design review?  (Suzy'll try to  
clarify)

     AJC:  Starting on 2e4 deg^2 of stellar catalogue data.

     	   Ray tracing coming along;  new atmosphere, Zeljko's cloud  
model.  Working
	   with Split group on GPUs --- images in a fortnight.

	   TSA: Is Zeljko's cloud model too good --- I'm worried.

	   RHL: Is the seeing bad enough?  DGM's says it's too good, so let's
	   check as soon as images are available.

    REO: I've gotten chi^2 coadds running.  I'm not convinced that  
things make sense.
         KTL: did you see my email?  REO: did you get my reply?

	 The question is, on the psf-matched coadd how do we get a uniform PSF
	 everywhere.  RHL: so you don't match to some defined PSF?  AJC: I  
haven't
	 done that; it would be a bit tricky as you'd need isolated stars.

	 We could generate overlapping images that provide at least one amp- 
sized
	 patch.

	 RHL:  What's the scale of Balkanisation?  REO:  We can always use a
	 single CCD as a reference image.

IPAC:
	Russ: Working on atpy package, writing unit tests.  The package makes
	it possible to convert data from e.g. mysql and put it in binary fits.

	Russ: I'd like to work with KT/Serge to get USNO-B1 installed on lsst10

	RHL: AJC: does your 2e4 deg^2 catalogue include USNO-B1?
	AJC: No, but we could.  Do you want them?
	RHL: Do we want to use the same catalogues for sims and CFHT data?

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- 
=-=-=-=-

Suzy: Code reviews for Camera Classes --- Simon's on Jury duty.  Use  
Apps phone con
      on Wednesday.   30 minutes presentation, then 15-25 min  
discussion.  Probably
      in a fortnight.

Suzy: KTL's skeleton production.  KT's preparing a stage listing  
linked off
      http://dev.lsstcorp.org/trac/wiki/DC3bProcessingFlow.  I'm  
about 1/2 way
      thorugh; when I'm done individual pipeline owners will have to  
pay close
      attention.

      Check on developer response next week.

2009-11-04

Present:  REO, RHL, TSA, Martin, Suzy, Russ Laher, ACB, David Wittman, Lee Armus, AJC, Simon

Russell:
	Reading on PSF QA?  Needed for coadds
	RHL:  what exactly do you need.

	State of Healpix (in stack?)	RHL will check on status with KTL
					Answer:  Andy Connolly is looking into desired resolution

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

The main topic is a review of Simon's camera geometry proposal at
	http://dev.lsstcorp.org/trac/wiki/CCB_CCDCoordSystem

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

I'd also like to discuss moving to the new middleware stage code as  
there seems to have been a misunderstanding between middleware and  
applications (or at least me).  The links are
	http://dev.lsstcorp.org/trac/wiki/CreateStageImplementation
	svn+ssh://svn.lsstcorp.org/DMS/pex/harness/tickets/924/examples
	http://dev.lsstcorp.org/trac/wiki/DC3bProcessingStages

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

(RHL:  I didn't do a very good job of capturing this discussion.  Sorry)

CurvatureModel needs to be default constructable and be built using a factory with a name specified in a policy file.

Use opaque types instead of e.g. std::pair<int, int> to index the rafts.

Come back to WCS (add boresight, optical model, ...) and  Camera::_origin in a future discussion.

Camera is constructed from a policy file

string name -> string const& name

Define getShape in more detail --- is this a free function or parallel hierarchy

2009-11-18

TSA, Roc, RHL, Deborah, Ray, Martin, David Wittman, Simon, ACB, KTL, Suzy, Russ Laher, REO, Jim, Steve, Fergal

Stages:

KT:
	Almost every stage for apps is designed to run in parallel (multiple copies of stage running at once)
	Expect only DB access stages to be serial -- e.g. global astrometry.

	Stages code should be small, although may have much apps code (KT hopes not).

	2 pieces of code in addition to apps code:
	  Stage
	  Default dictionary
	  	  Plumbing (e.g. image keys) without defaults
		  Parameters with defaults

	  Default parameters can be installed with package --- developers please use this

	KT added adaptors to hide the clipboard from the users.

	RHL: Why is a clipboard not a dictionary?  Ray: because data items can be shared, but it always looks like a dictionary to a user.

	Martin:  could use Adaptors to validate clipboards.  Ray:  yes, we're trying to do this

	Martin: I don't like magically mapping fooKey -> foo (All: this is a good idea but a detail)

Where do these files go:

         RHL: where do these stage/detection files go:

      	 KT:  meas/pipeline, ip/pipeline		RHL task to create/reorganise

	      meas/pipeline/policy			Policies
	      meas/pipeline/python/lsst/meas/pipeline	Stage code
			  
	 Ray/RHL:  That's OK.

	 Martin:  KT, how far does the skeleton pipeline run?
	 KT:      in datarel

Towards an LSST Manual:

	 We only have doxygen-level micro comments.

	 TSA:  There's lots of good stuff on trac, but it's hard to find.

	 Steve:  Manual level should ship with the code.

	 Ray:  Start an outline, and put in links when available.

	 RHL:  Can this be done with volunteer labour?

	 Ray:  My initial outline can be done that way;  filling gaps is harder.

	 RHL:  Need to hire someone as we ramp up
	 TSA:  Maybe 18months from now...

Other code reviews:

	Suzy:  Design reviews for key areas, e.g.:
	       	      Diffim
		      Multifit
		      Interslice
	       Is this a good idea?  What areas do we want to review?

	       How about diffim?
	       TSA: Yes.  Algorithmic
	       ACB: I'm not ready --- need to run some data

	       PSFs & Models

	       Multifit

	       How did camera classes go?  Well.
	       RHL: when will there be an implementation?  Simon:  ask me next week.

2009-12-2

Present:
RHL, Russ Laher, David Gehrig, Martin, John Meyers, Schuyler, ACB, David Wittman, KTL, Simon, Suzy, ACB, Deborah, REO, Steve Bickerton, Fergal

	Agenda

David proposed switching this phone con with the DCn phone con on Tuesdays at 11 PST.  The basic idea is that there's a major overlap, and at least sometimes DCn generates a question to be answered at Apps.
KT: Apps discussions affect DCn more often than vice-versa.

Resolution: don't change things now, but if the opportunity comes up let's move one of the phone cons by a couple of days.

What can we get running by Jan 4th?

Status:
Princeton:
Steve:
	worked on a few tickets; done some SDSS work.  Was observing last week.

	RHL: state of photometry codes?
	Steve:  pretty good;  aperture mags no longer need a Psf.

	RHL: Plan for testing, and will it be ready for January?
	Steve: no SEGVs.  Need to test if flux measurements are accurate on simulations
	RHL: plans for scientific testing?
	Steve: what do you want?  RHL: meet the SRD

	Suzy:  action to figure out where we'll be at the end of the month
	David: Need to get some sort of CPU estimate by 2010-01-15
	KTL: don't need good science, just expensive science

	RHL: do you owe any stage code?  Steve: No

RHL:
	Misc loose ends for new afw release.

	Writing some stage code

	For Jan 4:  Problems with minuit

	Suzy:  Dave Monet?
	RHL:   Centroids are in

Fergal:
	Astrometry is basically working.  RMS of WCS solution c. 0.1 asec

	Waiting on APIs for astrometry.net to get photometry from KD catalogues

	Working on Tim's CTIO problems (failing on 50% of frames)

	RHL:    Stage code?
	Fergal: Yes, but waiting on API.
	RHL:    Could we write the stage code now?
	Fergal: It's mostly done

IPAC:

Suzy:
	Serge has been doing WISE, not LSST.  KT, do we need him?
	KTL: yes, there are stages that match catalogues with billions of entries
	Suzy:  I'll talk with him today

Russ:
	AWOL on PTF, should be back this month.

	Worked on meeting coding standards.

	Skeleton stages for WCS verification if we need it, seeing Fergal's work

Deborah:
	Need to work with Tim on SDQA development, maybe just before Christmas?

	RHL: Do the stages generate SDQA data, or just metadata?
	Deborah:  I thought that we'd use the metadata;  let's do that for the first of the year
	Russ: OK

	RHL:  Are you all happy?
	Deborah: not really.  We thought that Tim wanted us to provide independent checks.
	RHL:  OK, let's do that.
	
U Washington:
	RHL: We're out of time, what shall we do now?

	REO: chi^2 coadd staging is wrong.  KTL: I permit this to change

David:  I'll be asking for the big picture next Tuesday
RHL:  I'll be trying to get it during the rest of this week

2009-12-9

Applications WG minutes for 12/9/09.

Attendees:  Suzy, Schuyler, Tim, Jim, David Gehrig, Dave Wittman, KT, Jon, Fergal, Mike Jarvis, Deborah, Russell, Russ, Steve, Andy

This meeting was dedicated to reviewing the status of the skeleton stage implementation.    Each developer was asked to provide status on the stages assigned to them (see attachment).

Tim mentioned that he thought there were two items being mixed together:
1. the need to assemble the skeleton stages into something that runs
2. producing science output that gives a reasonable estimate of compute resources needed from Teragrid.

Will the resource estimation question be answered with running skeleton stages linked together or just based on running individual stages? It’s not clear who is in charge of collecting resource estimation info or how it's being done.  Not a lot of work days left. Tim to talk with Jeff about this.

[** In a special DM management meeting held 12/10,  Jeff clearly stated that item 1 above this the highest priority, and item 2 is a secondary priority.  Please refer to Jeff’s email notes of that meeting for specific actions.]

Developers are asked to put their stage code in the trunk and to link it to the DC3b processing page.  Below are two emails from KT on how to do this.

“I have added an example of how to link your completed stage and dictionary into the Trac stage list at http://dev.lsstcorp.org/trac/wiki/DC3bProcessingStages.  The example is for the meas.pipeline.MultifitStage.  Please do this for all skeleton stages as you complete them.  Thanks.”

“Information on where various files should live for DC3b has been added to the page on creating a stage implementation:
http://dev.lsstcorp.org/trac/wiki/CreateStageImplementation”

Suzy's assessment is that most stages are on track for a build by Jan 4.  The exceptions to that are association pipeline, multifit, and perhaps diff-im.    More than one developer appears to be waiting on a new version of AFW.    We need Robert to give his status on that.    Also, Robert is asked to review Andy and Simon’s code before they get committed to the trunk.

There are still five unassigned stages.

We should adhere to the notion that the trunk can run - don't break the trunk.

Finally, the group felt using the straight interpretation of the JPL Healpix code was the way to proceed for now.




2009-12-16

Present: RHL, Lynne, Russell, Martin, KT, Dustin, Jim, Simon, Fergal, Steve, ACB

KT:
	Trying to put a pipeline together this afternoon

RHL: 	Has anyone got stages that aren't on KT's page?
	RHL: I do
	Simon:  mine are working on a branch, but the tests fail
	RHL: Merge ip_isr to the trunk and dummy the tests

RHL: OK, over to Martin;  his presentation (aspossible updated after this phonecon) is at
http://dev.lsstcorp.org/trac/attachment/wiki/MartinDubcovsky/PSF_Multifit_Design_Review.pdf

Note: RHL was too busy harassing Martin to take good notes on the presentation.  I think it basically went quite well, and I await the resulting code.  Two details that came out were:
 1. Multifit will get phometrically calibrated Exposures
 1. We need a "Heavy" Footprint with pixels included.  As a sidebar, there was some thought that Footprints should be able to handle e.g. HEALPix coordinates.

We will be having a meeting on 2009-12-23

2009-12-23

Present: RHL, Mike Jarvis, Schuyler, Deborah, Jim Bosch, TSA, Steve, ACB, Martin

Agenda:  Healpix

NSF review
    TSA:  Went very well.  We expect a report in January.

Status:
    RHL:  State of ISR stages?
    ACB:  Mostly done (RHL: but maybe not checked in)
    Jim:  The multifit stages are very dummy...
    TSA:  Any progress on getting me a number?  Jim:  we're getting close.
    	  Martin: tests running by the end of the week.
	  Stuck on Wcs ticket -- need afw::geom::affine transform.
	  Mike Jarvis:  In DES I return the Jacobean, and always represent the Wcs as a polynomial
	  Swig by the end of next week.

    RHL:  Where are the Camera classes?
    ACB:  I think that Simon didn't use them in the ISR, so they are bottom of his list
    TSA:  I just hacked up the DC3a stuff

    RHL:  Healpix.  Andy?  ACB:  I don't know
    TSA:  I need to know the size of the sky tiles rather than the 

    ACB:  There are PSF-matched template images, and non-PSF matched per-band coadds, and chi^2 images
          The PSF-matched ones assume isolated images and an ideal PSF (currently a single Gaussian).

AOB?
    Jim: Box code is in ^/DMS/afw/tickets/1095

We will be having a meeting on 2009-12-30

2009-12-30

Present: RHL, REO

We discussed pressing items of mutual interest.  You just had to be there.