wiki:MinutesOfImageDetWG-2008
Last modified 11 years ago Last modified on 12/09/2008 09:13:13 PM

Minutes of the ImageDetWG Weekly 'Phone Con.s

Minutes for 2008

2008-06-09

Present: RHL, Roc Cutri, Deborah Levine, TSA, ACB
MIA: The rest of the LSST collaboration

1/ Status reports [inc. Mea Culpas] (All)
   RHL: Nothing to report
   
   TSA: calibration
	association standalone (i.e. DC2 stack)
	    i.e. can be run as a standalone pipeline, but not generally usable

   ACB: Diff Im; two tickets:
	a. Kernel has constant volume
	b. Spatial variation by kriging

	Waited on code cleanup until someone can do a code review

   Deborah: Nothing to report.

2/ Report from CCB (Robyn)
   See CCB page (http://dev.lsstcorp.org/trac/wiki/CCB). Basically,
   we:
	adopted some typedef name change conventions

3/ DC3 schedule for Applications (Suzy Dodd)
   Postponed due to confusion about time zones.

4/ LSST algorithms on GPUs
   RHL: Has writeup on MaskedImage convolution
   TSA: I thought that we said the entire image subtraction pipeline?
   RHL: Will investigate

   TSA: Also Chris Hoffman at Purdue is involved in a new, secret,
   intel processor and wants to try out image subtraction.

   Roc: Do we need to do this?
   TSA: Probably; we're a factor of 5 too slow and Moore's law
   is mostly going into cores, not clock speed, these days

5/ Any other business
   Tim and I are otherwise engaged next week, so the 2008-06-16
   meeting is hereby cancelled.  I'll send out a reminder.

2008/06/23

Present: RHL, REO, TSA, RRA, Roc Cutri, NMS, ACB, AJC, Deborah Levine,

Status report:
       REO's working on separable kernels
       NMS is coding up the ISR stuff, but not testing it due to perceived
       difficulties with the DMS stack (it's not clear if we're really in trouble)

       ISR will ingest CFHT data initially (NMS will talk to Lynne)

       ACB's helping with EPO and light curves
       He's also looking at a geostat kriging library to experiment
       with spatial variation of the diffim kernel (basically a linear
       predictive code?)

       AJC reports that John Peterson's added Mie scattering.  He's talking
       to Dave Monet about DGM's needs.

       He expects to deliver simulations in a month or so BUT NEEDS REQUIREMENTS

       TSA has been working on photometric calibration in a custom python/mysql
       framework (e.g. deriving grayExtinction(x, t) to 1%).

       Q: where does generation of flats belong?
       A: RHL should ask David Burke
       

2008/06/30

Jacek, K-T, RHL, Roc, Suzy Dodd, NMS, REO, ACB, AJC, Deborah Levine

REO: Separable kernels. API changes in afw allowed a build all the way to DiffIm
NMS: Debugging ISR with all the world built
ACB: Wrapper about gstat.  Wrestling with build system on 64.  AJC notes that this dates from January

RHL: Pixel coords proposal (http://dev.lsstcorp.org/trac/wiki/BottomLeftPixelProposalII)
     Russ Laher will head up Abraxis evaluation
     Image -- not done (REO : wants input into pixel access APIs)

Suzy Dodd: sent out DC3 schedule proposal on the 2008-06-20.  Will start pinging you on schedule in Monthly
     Still accepting comments.
     RHL: use trac

AJC: Recovering from lost laptop
     Roc + Armus looking at sims
     Wants a guinea pig:  Proposes Russell and template generation
     (Send AJC and Roc Nick's paper)

RHL: Penn group will join DM

K-T Lin: Image metadata.  Persisted to DB, but what about in memory?
     DataProperty?  Member variables?  Strings
     Deborah: is there a list on trac?  A: No
     KTL:
	* Metadata passed through
	* Immutable metadata
	* Mutable metadata
     REO: Set flags rather than consider new metadata.

     So we have:
	* Dictionary metadata
	* Interpreted as member variables
	* Interpreted as a class

     Currently Image has metadata, and Exposure has an Image and WCS --- two places

     If an Image is part of something more complex, does it acquire metadata when
     written to a FITS file?

     KTL: Images will be saved as minimal FITS + DB (RHL: is this a good idea?)

     RHL: Always read a FITS file into an Exposure --- Exposure is the fundamental image object
     RHL: If you need metadata, inherit from Image and add members (not a dictionary!)

     Does Provenance belong to objects, or is it global?  It's global

     REO will write up for CCB

Any other business:

Francesco:
     Will join this telecon

K-T Lim:  Storage for templates --- not just images. E.g. how we tile the sky
     AJC: keep Davis group in the loop re multifit?  No.

2008-07-07

Present: REO, RHL, AJC, ACB, NMS, Francesco, Bhuvnesh, Roc, Suzy Dodd, Deborah Levine, Dave Wittman, Mike Jarvis

ACB still having trouble with build system.  Waiting on RHL

NMS Looking for data from Ray;  Ray is on holiday (as is Lynne)

REO Posted metadata proposal at
    http://dev.lsstcorp.org/trac/wiki/ImageMetadataProposal
Started on template generation.

RHL nothing relevant

Suzy Dodd: Aug 21-22 for DC3 use case meeting; Rosenberg will be present.

There was some discussion of whether we all needed to go to this meeting,
and whether discussing use cases was really the best way to spend the
whole meeting.   Please send comments to Roc, Suzy, and me.

AJC: Template formats.  Initial plan is to save tangent plane tilings;
questions remain about footprint sizes and overlaps.  Will make a proposal.
Q: Who cares outside applications and needs to sign off?

Bhuvnesh and Mike.  Discussed value of shapelets in context of shear
measurement: 2nd order shapelets are Gaussian-weighted second moments 
for shear. Advantage of shapelet PSF is that convolutions are matrix
operations so can measure deconvolved shapes.

Q: is this optimal for non-shape things?   A: don't know.
Dave Wittman:  There's a Koiken paper on PSF and aperture-matched photometry
RHL: (obnovious comment that SDSS has done this for years)

Measurement of PSF assumes an isolated object, and fit for the model coefficients;
RHL expressed concern about blending.

Bhuv:  we only promised to deliver our weak-lensing model;  so we need to
write the Penn SOW carefully.

Bhuv: We expect to have an at-a-point PSF model integrated into the LSST
system by March 31st, which is the deadline for the data release pipeline (spatially
varying PSFs may need to be deferred to DC4).

REO/ACB:  So we'll need another implementation?
RHL:  Yes;  I'd like to get Steve Bickerton/Fergal Mullally here in 
P'ton + ACB + input from Dave Wittman to do it.  Then we can figure 
out what we need, and work with the Penn group on a design that works
for LSST.

Roc: We need to avoid a "my PSF's better" war.  RHL's well-known diplomatic 
skills will be employed to keep us safe.

2008-07-14

RHL NMS AJC Jacek Roc KTL Francesco REO ACB Deborah

ACB:  I have the build system working on 64-bit systems;
      RHL found a problem with ldopen flags on 64-bit
      machines in the build product

      He's also converting CFHT bad masks to LSST Images

      There followed a discussion on where this information comes
      from.  There were various proposals as to the correct classes
      (e.g. Masks; lists of Footprints) but apart from these
      technicalities there's the question of where this sort of
      data should be saved.  NMS was given the job of sorting
      out what sorts of inputs will be needed to the ISR, and
      deciding what sort of database queries they need to support.

      Back to ACB's geo-statistical library...  should he wrap it in C++
      and then swig that, or swig the C directly.  RHL pointed out
      that there are pros and cons of C (e.g. telling swig what's
      a constructor; possibly needing typedefs for containers).  He'll
      send the headers to the mailing list.

      (Parenthetically, are the lists functional?  No one seems to use
      the keywords.  At some point RHL was going to transfer our mail
      to hypernews, but he failed to figure out a way to import our
      archived mail.  Maybe we should just set up a new list...)
      
NMS:  Can build the ISR on 32 bits.

      Found CFHT raw data at NCSA (thank you David Gehrig) and is
      writing scripts to mangle the data into LSST formats.

      RHL complained this sounds a bit ad-hoc; we expect to import
      data from many cameras before LSST (CFHT, SDSS?, suprimeCam, ...)
      and we should try to re-use as much of the effort as possible.

      AJC: So this is just stuffing the clipboard?

      KTL: We have generic persistence, but this is more specific

      AJC: NMS will look at how specialised her design is, and
      how hard it'd be to generalise.

REO:  Thinking abour template generation

RHL:  Working on the build system and on SDSS



IPAC: Russ Laher is thinking about how to store SDQA data on the
      clipboard.  This is currently a learning exercise, and he'll
      collaborate with e.g. KTL.  It's also probably a topic
      of discussion for this phone con

DC3 Design/ES meeting on 21-22 August
      Roc: If we have to go, what would be useful?

      KTL: What's in DC3 scope?  What algorithms are going to be
      implemented? What new classes (e.g. bad pixel lists); this
      could be done via EA.

      There was a general discussion about what this group would
      really like to get out of the meeting. We felt that we needed
      a meeting, but that this one might not be it, as it'll take at
      least 1.5--2 days to do the EA part and we'd like time to
      discuss everything else too.

Should we put SDQA (and other) information gleaned during processing
into the Facilities database

      Terry Schalk wants information on timescale of c. alerts on
      camera performance, and load facilities database.  E.g. PSFs;
      S/N; bad pixels.   Can we do this?  Do they access clipboards?

      KTL:  A replica of SDQA DB into his DB; performance is OK. We
      don't need to worry about de-normalisation if we just copy
      [part of] the DB.

      Jacek: Terry was thinking about a publish/subscribe style, but
      he's flexible
      
Simulations (Roc)
      Roc and Lee Armus will be contacting people individually;
      some people (e.g. AJC + REO) already have lists of desiderata.

REO will run next meeting as RHL'll be in Japan.

2008-07-21

Minutes of the LSST Applications phone con held 2008-07-21 10:00 Pacific time

Attendees:
- Russell Owen
- Nicole Sylvestri
- Andy Becker
- Suzy Dodd
- Roc Coutri
- Ray Plante
- K-T Lim
- Deborah Levine
- Jacek Becla

*** Progress Reports

   Andy made progress in two areas:
1) Wrote an LSST wrapper around the gstat geospatial library. It 
compiles and he is now testing it.
2) Talked to Andy Connolly about image simulation requirements for 
difference imaging. They agreed that initially they wish to 
investigate the effects of the optics and variable seeing. After that 
they would like to study the effects of different airmass and 
differential chromatic refraction.

Ray asked if gstat was to become a new required package for the LSST 
software stack. Andy said it was too early to say; for now he is just 
testing.


   Nicole worked on the code that preprocesses data from various 
surveys into the format required for the LSST ISR pipeline.

Initially this code was written to handle CFHT data but Robert Lupton 
asked about making it more general at last week's Applications phone 
con. Nicole wrote a proposal for how to accomplish this and first 
circulated it within the UW group. Having incorporated that feedback 
she will post the proposal to the Trac wiki shortly.

She expects to have the software itself running by the end of the 
week. Depending on reactions to her proposal it may need some work, 
but it will certainly work with the CFHT data.


   Deborah debriefed Robert on what diagnostics will be output by the 
various pipelines he is responsible for. These include detection, PSF 
fitting and measurement of object parameters after detection. She 
plans to debrief Nicole next.


   Russell mainly worked on other projects last week, but set up 
meetings this week to detail out how we will implement Nick Kaiser's 
paper on Template generation.


*** Discussion of the Proposal for Metadata For Image-Like Objects
<http://dev.lsstcorp.org/trac/wiki/ImageMetadataProposal>

(please feel free to post corrections and additions; it was hard to 
keep up with the discussion and write notes at the same time)

Russell wants this proposal discussed and agreed to as soon as 
possible so we can design the ISR with it in mind. To that end we 
spent the rest of the meeting discussing it.

* Does it make sense for LSST standard metadata to be represented as 
metadata-specific member variables, rather than as name:value pairs 
in a general DataProperty. (Note: name:value DataProperty metadata 

will still be supported for experimental code, test code, etc.)

Andy Becker worries about needing to modify the class every time we 
add a bit of metadata.

On the whole people seemed to agree that this was acceptable as long 
as the updates occurred occasionally and at specific times (that we 
could run for awhile using new metadata in name:value pairs).


* Should new metadata should be persisted in the database and if so, how?

The obvious way to handle new metadata is to add it to a Character 
Large Object (CLOB) or something similar. This gets the data *into* 
the database, but it is not easily queried.

Jacek proposed that if we initially use a CLOB for new metadata that 
we actually want to persist (much of it can be reconstructed and so 
may not need to be persisted). Then during the next data release we 
turn that metadata into new database fields (which *can* easily be 
queried).

The consensus on persisting seemed to be that Jacek's proposal was 
the best solution, offering a good balance of power, ease of use and 
ease of implementation.

However, K-T brought up the issue of backwards compatibility issues 
cause by the database changing:
- queries on newer databases may not work on older databases.
- There are compatibility issues for persistence, and thus running 
different versions of the LSST software on different versions of the 
data release database.
Everyone agreed this is a serious issue that needs further thought. 
But it does seem to be largely independent of the image metadata 
proposal.


* Should the Image and Mask classes have metadata (beyond the bare 
minimum required for the classes)?

Advantages to not having metadata (all metadata in Exposure only):
- There is only one obvious place to look for the metadata
- If one gets an Image out of an Exposure there is no issue about 
whether to copy the metadata from the Exposure to the Image

Disadvantages:
- There is no afw object that is a really direct representation of a 
simple FITS file. Thus it is a bit harder to read and write simple 
FITS images (e.g. have to carry around the variance and mask of the 
Exposure in afw).

In the end it was agreed that if the applications developers were 
comfortable giving up the metadata in Image and Mask then it was 
fine. Nicole, Andy and Russell all feel it is OK, but Andy would like 
to give Tim Axelrod a chance to weigh in.


* Should we keep MaskedImage or combine it with Exposure into one 
class? Also, do we want more than one kind of Exposure?

Nicole and Andy could not think of any use case for a bare 
MaskedImage, but neither they nor Russell have carefully looked 
through the EA model to be sure.

K-T then asked if we should have different kinds of Exposures for 
different kinds of images, e.g. bias vs science image. This would be 
sensible if the metadata is very different (especially if the 
metadata is in instance variables). Andy felt that the different 
kinds of images weren't different enough to justify this.

Ray pointed out that we can avoid the need for different kinds of 
Exposures if we use a DataProperty for most metadata (as we do now). 
In other words, using dynamically typed metadata.

However, Andy was pretty sure we only need one kind of Exposure in any case.

The consensus was that we probably only need one kind of Exposure and 
that we can probably do without MaskedImage. But given the 
uncertainties it makes sense to change Exposure to inherit from 
MaskedImage (rather than contain a MaskedImage, as now) and prefer 
Exposure to maskedImage and see how that works out. If it works well 
then we can get rid of Exposure with minimal pain later.


* What about obscure metadata that is only needed by one or two tiny 
bits of code?

K-T brought up the issue that there is a lot of metadata associated 
with an image and some of it may only be needed by one bit of code, 
perhaps only sometimes. He pointed out that there was a proposal for 
lazy evaluation of such metadata and he felt this would still be a 
good idea. He proposed that we add a new metadata member variable 
"lazyMetadata" for such metadata (since it would probably want to be 
some new kind of object).


We wrapped up by summarizing the conclusions:
- Get rid of metadata in Image provided Tim agrees.
- Avoid duplicate metadata. There should only be one obvious place to 
look for an item of metadata and it should never be duplicated in a 
given object.
- Combine MaskedImage and Exposure only if MaskedImage truly not 
needed and there is only one kind of Exposure. (Change Exposure to 
inherit from MaskedImage otherwise).
- It is OK to put standard LSST metadata in member variables. But we 
probably need to add a new lazyMetadata member variable.

2008-07-28

RHL wrote minutes, but restarted his laptop before posting them,
so they're lost.  Mea Culpa.

2008-08-04

RHL TSA REO RRA Roc ACB NMS KTL Francesco Deborah DaveWittman
-------------------------------------------------------------

Missing agenda

Status Reports

Ray Plante:  working on Linux64 build;  expects to get it out by later today
    ACB: vw 1.0.1+2;  boost 1.34.1

NMS: writing ISR code. Use cases will define inputs
     Expect to have ISR code running by end of August

REO: Metadata proposal; Kaiser coadds

ACB: Send out email; worked on coadds

RHL: Build systems.

TSA: I'm back

RRA: CCB meeting.
     Abraxis/Parasoft comparison;  more impressed by parasoft [who'll be at IPAC]

     Metadata proposal.  REO will go ahead and develop the headers that define
     the metadata that'll appear as C++

Deborah: working on Use Cases

== How are we going to maximise the usefulness of the DC3 meeting at IPAC? ==

TSA: we have lots of bottom-up stuff OK, but haven't made progress in top-down

Need to take Use Cases (Nightly and Data Release) and move forward until can
be used for the design.

TSA intends to read use cases in EA.  Where there are place holders
for e.g.  multifit , expand them.

KTL:  need to know inputs and outputs for all pipelines, enough to be persisted.

TSA: need use cases in advance with all nouns mapping to classes.

TSA: if you don't use EA, write the use cases as text and post the list

Roc:  Suzy and I are planning the spare 1/2- day;  including integrates schedule


== How do we want to build templates/coadds for detection? ==

REO AJC thinks image subtraction coadd;  we think for object detection

REO: Convolve all images by its own PSF and add.
     Do same for PSFs

These two sums are jointly a sufficient statistic for the true sky

Templates:  Probably need 2 (low airmass and high airmass)
	    AJC: Wants good and bad seeing templates;  the bad seeing
	    	 templates have better S/N

TSA: Are they a significant impact on the storage?

KTL: they can be; and they're stored at higher resolution (2x?) and at
the base camp

TSA: Are they worse than 10-15% at base camp?

TSA: Need coadds for at least EPO, in addition to colour JPGs.  Not at
base.

We'll try Kaiser, and see how it goes.  Phone con. tomorrow
     
AJC:  Would Dave Wittman use this to warm-start for detection?

Dave: it sounds interesting, but I'll wait until I can play with it


NEXT MEETING
    RHL's in the UK starting tomorrow, returning on the 18th.

    Next week (11th August) there's a meeting in Tucson so this phone con's cancelled
  
    The following week (18th) is the week of the DC3 meeting, but we decided
    that we should have a meeting anyway.  Tim will officiate.


2008-08-25

RHL, Russ Laher, KTL, Suzy, Roc, REO, NMS, ACB, Vince, Wittman, Deborah

Status Reports:
        RHL:  Worked on domain model/robustness.
              Went through task list
        Suzy: Nag: outstanding work due by Friday (29th)
              Nag: progress reports (but I did that at the meeting)
        Roc:  Task list discussion at meeting was surprising!  Make
              sure you know what you're on the hook for

Technical issues:
        ACB:  Two issues from apps meeting:

        1. Deborah + KTL and I talked about SDQA.  How do we bring
        SDQA into line with the rest of the project. KTL: after each
        stage have an SDQA stage.  ACB: have each stage include its
        own sub-stage.

        Roc:  Are there shades of gray?  Or just good/fatal.  ACB: the latter

        RHL:  Isn't an extra stage able to accept a 0/1 flag, or be flexible?

        Roc:  Didn't expect that SDQA would have an operational role (Vince: SDQA is
              passive)

        KTL:  Is SDQA running in realtime? (Yes)

        RHL:  I don't care if there's an associated SDQA stage, or an SDQA module
              that's called by the stage.

        ACB:  With an in-stage SDQA we need to mix policy files (processing + SDQA)

        RHL:  We may want to call this qa-info gathering information
              something other than SDQA, but we do need it.  We may
              need some qa-aware piece of middleware.  Call is Pipeline QA

        KTL:  So each app stage generates SDQA data in clipboard.
              There's another consumer PQA (also called pipeline executive),
              which also persists SDQA information.

              Where do we document this?  In "How Pipeline Runs" doc; KTL
              gets middleware, Deborah gets SDQA.


       2. Jeff is unhappy about OO nature of Image differencing code.  He wants
          more methods, less subroutines.

          KTL/RHL:  Are subroutines stateless?  Do you have lots of routines with
          a common set of arguments?

          ACB:  Not really.

          KTL:  We could use controller classes.

          ACB:  Should we rewrite diffim to learn this?

          RHL:  Or should we make ISR a learning example?  Probably.  Roc/RHL: we
          can't afford to reimplement just now.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

        There's a meeting tomorrow/Wednesday to decide middleware/apps interfaces.
        KTL will send an agenda and the meeting's open --- but it's not a discussion
        meeting, it's to take decisions.

2008-09-01 -- 2008-09-15

RHL played hookie

2008-09-08

Apps Telecon 9/8/08                                                                              

Present:  Jacek, Andy B, Russell, Dave W, Deborah, Vince, Russ L,
Martin, Nicole, David Burke, Tim                                 

---

We discussed Jeff's need to have controller complexities assigned in
the EA model.   The status on a per-package basis:                  

Astrometric Calibration:  => TA to complete

CoAdd Images:  Kaiser CoAdd completed by RO; Unclear what we will do
with the general CoAdd Images use cases (further discussion below)  

Day MOPS: No robustness or controllers at this point.  => FP to add
what's needed for Jeff's estimate                                  

Deep Detection: DW has already assigned complexities.

ISR: NS has already assigned complexities

Photometric Calibration:  => TA to complete

Produce Data Release Image Products:  This is mostly an "umbrella use
case" package with the controllers existing in other packages.  We   
agreed that "Produce RGB Images for Visualization" will remain a     
placeholder for DC3.  "Produce Template Images for Subtraction" needs
to be modeled.   => AB to do this, using the Kaiser CoAdd model as a 
starting place                                                       

Produce Data Release:  Again, mostly an umbrella with the
content elsewhere. A package that needs to be fleshed out is "Run
Image Data Train", which has been under active discussion.  Not clear
at the telecon who will do this, though JB indicated it is mostly a  
middleware task.                                                     

Run Nightly Pipeline for DR:  => TA to complete

=> Note that all assigned complexities need to be reviewed for
consistency with JK's revised guidelines (1 week, 1 month, 3 months)

The question came up of how to assign complexity for controllers that
are already implemented.   AB suggested changing the status for these
to "Implemented" from "Proposed".  NS noted that it is not possible to
leave the complexity annotation off entirely, as JK suggested for     
trivial controllers => JK to advise                                   

Another modeling issue that came up is how we treat parts that we will
have for LSST, but not for DC3.  There was general agreement that (a) 
we need to handle this uniformly; and (b) it is probably best to show 
the full LSST case when we know it, and mark in some way those bits   
that are not included in DC3.  => JK to advise how he wants this done 

----

Nightly processing for Data Release and how it differs from regular
nightly processing for alerts:                                     

Does DD or NP come first?   TA argued that DD should come first, AB
that NP should.  A use case to keep in mind here is how a supernova
occurring within an extended galaxy will be handled.   In particular,
when doing DD, would the galaxy model used by multifit contain a time
variable point source for the supernova?   Would the time varying    
source be subtracted off before DD?                                  

TA raised the issue of NN2 processing for light echoes and
similar time varying diffuse structures.  There was general agreement
that this should not be part of DR processing, but instead be a "value
added" product from science collabs or similar.

TA asked whether we should do forced photometry on difference images
for all Objects, or perhaps a subset, such as those with significant
variability. There was general agreement that this should be done at
some level.  Details are not clear.

----

Deep detection issues:

JB brought up the issue of DD runs in between data releases.  TA
argues that DD will consume the majority of processing time for a data
release (DR), and that if we need DD done more frequently it would be
simpler to just speed up the DR cycle.  There was not much dissent
from this view at the telecon, though JB expressed concern for the
storage budget if we speed up the DR cycle.   => TA to send out email
proposing that the intermediate DD runs be dropped, and await
reaction.

TA asked whether "Assemble Relevant Pixels" needs a WCS match step.
DW replied that his current thinking is that the geometric remap will
NOT be applied to the pixels, but rather kept as part of the model.
If we do this, the remap is not part of "Assemble..."

DW suggests that we should mark "Evaluate Model on DataCube" and
following controllers "not for DC3"

TA noted that we need CoAdds in several different forms throughout the
DR use cases.  This cries out to be carefully modeled as a shared
capability.

While discussing CoAdds we discovered a lack of clarity on how
transient objects (such as solar system objects) will be handled in
producing the CoAdd used by DD for object detection.  TA assumed that
sigma clipping would get rid of such things, while AB assumed that
explicit masking based on prior image differencing would be required.
=> action needed to resolve this.   DW/UCD team?

----

The telecon ran out of time with lots left to discuss.

2008-09-15

Apps Telecon 9/15/08                                                                             

Present: Andy B, Russell, Dave W, Deborah, Russ L,
Martin, Nicole, Robyn, Andy C, Serge, Roc, Suzy, Tim, RHL (late)

---

We discussed the issue raised by Martin of whether the multiple
uses of detection in the UML model really differ in their      
requirements.  TA pointed out that in addition to detection on 
difference images and detection on coadds for deep detection, we have
detection on bright stars in unsubtracted images for WCS             
determination.                                                       

There was general agreement that detection on a coadd will need a
different, and significantly more sophisticated, algorithm than for
detection on difference images.  Among other capabilities, it needs
good PSF determination, sky estimation, and deblending.  Detection on
image subtraction usually needs none of these.  AB, however, suggested
that deblending might sometimes be needed for crowded fields like the LMC,
and agreed to supply some examples of this (he's now done so).            

---

We began to go through the UML for "Detect on CoAdd Exposure".  The
first sentence of the use case ("Crudely estimate background from the
input Polychromatic Coadd Exposure, generating a Background Model    
Set") raised several points that needed clarification, and it seemed 
likely that subsequent ones would be similar.   Martin agreed to email
a list of points that need clarification, since there wasn't time to  
go much further on the telecon.                                       

---

Suzy repeated her email request for input to the Applications Monthly
Report by the end of the day.                                        

---

TA discussed the need to get AAS poster titles and authors in by the
end of Tuesday.   AC reported that there will be one on image       
simulation.  DW felt that there had not been enough progress on deep
detection to warrant a poster at this point.  It looks like we will
have one on data release (TA lead author), one on cosmic ray rejection
(Russ L), and one on new developments in image subtraction (AB)

---

AB brought up the desire to have a C++ math library API that looks
like the python API.  TA pointed out that this falls within the
purview of the group set up at the IPAC meeting in February.  This
group has never met, and clearly needs to.  RHL, who is the chair of
the group, joined the telecon at this point and agreed to call a
meeting.

---

At this point we went to status reports.  RHL reported his progress on
reimplentation of the image classes:

   Implemented Image and DecoratedImage.  Fits I/O code
   works (written in an ACT meeting; it's much cleaner than
   the hack I had to do for vw.  Uses boost::mpl)  Metadata is
   probably OK.   Swig works, but needed a change to daf_utils
   to get proper shared_ptr support (i.e. use the code that's
   now in swig).  This is a general issue -- the new code seems
   to just work, and we should use it.

   Mask isn't written, but char Images are, so it should
   be easy.

   Masked image next.  It'll use DecoratedImage and will
   be a simple wrapper of a set of 3 images/masks for now.
   We might consider an interleaved implementation later,
   but it should be invisible.

   Needs to be doxygenated.  I'll then merge all this
   code to an afw branch and review etc.

Serge reported that he is working on implementing the changes to
shared pointer support.  He and KT are working on the changes needed
to DataProperty, which is one of the bigger tasks.

AB reported that he is still blocked by the build system not working
on his machine.   We agreed to discuss at Tuesday's DC3 telecon.



2008-09-22

	RHL, ACB, TSA, REO, Dave Wittman, Martin, \v{Z}I, AJC, RRA, Ryan Scranton

	Achievements du semain:

	ACB:  writing code with build system working on 64-bit machine
	      Installed ...

	      Kernel testing package.
	      Study kernel basis functions and spatial variation

	      TSA: Is WCS on your plate?
	      ACB: Yes; given positions of stars, solve for WCS. Simple once an initial guess is there
	      TSA: Consider astrometry.net

	REO:  Getting CFHT data for coadd tests.

	NMS:  Writing ISR code. Five stages done, three to go.

	RHL:  Image classes almost there

	TSA:  Mostly photom calib;  April CTIO data.  Systematics in DoPhot analysis.
	RHL:  Apps layer?
	TSA:  Need attention to detection pipeline for DC4...  E.g. sky subtraction

	Davis:  Fighting build system with Ray's help.

	AJC:  Delivered catalog generation for Jacek.  John Peterson sent TSA an image (TSA:  it looks great)

	-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


	Are we duplicating Detection in EA? (Martin)
	Martin:  we seem to have to same code duplicated everywhere.
	TSA:  we resolved this last week.  Coadds and diffims are really different, but will share code

	Do we understand detection in coadd (Martin)
	REO will propagate PSF into coadded Exposures
	Princeton will handle the detection coadds

	All you wanted to know about Footprints and ExposurePlanes (Martin)
	REO: Would like Footprint in afw.  RHL will do.  REO will file ticket.

	Martin: Will Footprints be contiguous?
	RHL:  issues with blending

	ExposurePlane: Footprint + sky pixels around it
	RHL:  Match individual exposures.
	Dave: Triage for DC3?

	RHL:  I'd like to see a non-local sky (e.g. estimated on 100" scale)

	Calibration issues that David Burke needs clarified
	Unknown; RHL will investigate
	
	Should we move this telecon?

	Should image API use col/row or x/y (RHL)
	REO:  sets us back a month?
	RHL:  that's a lot;  I'd have said yes.  RHL will send around a note

	Any Other Business:

	REO:  Who owns SkyRegion stuff?
	TSA:  No-one yet.  
	Ryan: Maybe me.  RHL: will you decide?
	REO:  VO wants to use HTM, but healpix is better.  Should we pay attention?
	Ryan: VO not setup for depth masks
	TSA:  It's in the model.  I'll help Ryan

2008-09-29

Present:
	RHL, Suzy Dodd, AJC, TSA, ACB, NMS, RRA, REO, Vince Manning, Ryan Scranton, Martin

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Achievements and Confessions

	RHL: Image classes

	IPAC:
		Suzy:  Status + indendent cost estimates.
		       Next week is DM review.
		       Preparing to prepare for construction estimates
	
		TSA: DM review is not a dry run for the PDR, but to pick external brains
		     and see where we are.  Jeff Kantor will probably do most of presenting

		Vince: Next week or two should generate SDQA report, maybe with a preliminary
		       wiki post

	UW:
		NMS: Coding^n on ISR.  Splitting CFHT form MEF to single amplifier files.
		     Not checked in so an not to break the trunk
	
		REO: Working with NMS.

		ACB: Mostly on vacation, and fighting templates with Russell

		AJC: Being a professor, and catalogues for image sims.  A few CCDs come next

	TSA: Conversation with transient group on catalogues.
	         1. How we link together difference-image-Sources and Sources
		 2. What we do with many lowish S/N objects
	     Need to converge soon

	     Detection -- really source categorisation.

	     Cf. Trac: http://dev.lsstcorp.org/trac/wiki/DetectionAlgorithms
	     Summary: Can't just use multifit, but will share lots of code

	RRA: DC3 integration test plan
	
	UCDavis:
		Ryan: will do image footprint stuff

		Martin: talking with KT on data chain --- please use email

2008-10-17

Present: RHL Robyn Suzy NMS Laher  Wittman Dubcovsky ACB TSA Burke Deborah.

RHL: worked on afw Image classes, and closing tickets

Robyn:  Worked on the DM Review

TSA:  DM Review.  Calib review next week

     Went very well.  David Silva (Director of NOAO).  Peter Lee (CS@CMU).
     Silva pleased that we're so organised;  went very well.  Comments that
     have most bearing were about map-reduce v. sciDB, and for apps as well
     as DBs.  RHL: who's going to pay attention to this?

Suzy: Inputs on monthly report, and construction workforce -- esp. RHL

ACB   kernel generation process, including errors in kernels.  Sending
	set of kernels to Genevieve to choose good basis functions. Good
	metrics for kernels.  Iterating using noise to re-determine kernels.

	TSA: WCS? ACB:  Monet's worried about how we're doing WCSs, but we
	don't have code yet.  TSA: we need to move forward. RHL: shall I
	take this on for a new post-doc?  Yes.

	TSA: astometry.net is robust -- but want initial guess.  Monet
	probably shouln't care -- this is just the initial guess, and
	association has to work.  Burke: this is correct.  ACB: why do
	we need astrometry.net?  TSA: telescope may not point well. enough.

	David Burke: Check that the second stage is happy with the first stage QA.
	
	RHL: I'll synthesise and consult.

NMS	I was sick last week.  Just about got images through ISR, and fixed
	up policy files a la Ray Plante.  Scaling fringe patterns.

	Test suite?  Grabbed raw/calibs for CFHT data in afw.

REO:	Kaiser coadds just-about working (images have been seen by ACB)

Russ Laher:	Sent round SDQA document this morning. Will read TSA architecture blurb

Wittman:	I've been sick

Martin:		Working with Ryan Scranton on pixellisation code (STOMP).

Move to 10am Wednesday

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Image classes.

RHL: shall we update detection then merge to the mainline
TSA: Yes.
  Branch: svn+ssh://svn.lsstcorp.org/DMS/afw/branches/new-image-api
  Gil tutorial: http://opensource.adobe.com/wiki/display/gil/Generic+Image+Library
(afw supports the same API, even for MaskedImage)


Image stats.   Means, std.devs, etc....

Russ Laher will file ticket.  afw/math   RHL will talk to Gene Magnier
at Pan-STARRS about getting their stats library.

Robyn says: could you add to the agenda the final selection of the
directory names/loc for detection/deepdetection. DC3 meeting pushed it
over to App meeting since so many folks were gone.

Robyn:  I want a name for measureObj
	TSA: how to avoid Balkanisation.
	RHL: Devote next Wednesday meeting to this issue.
	Robyn:  Martin NEEDS this, but he can use the
	current detection pending the decision, and we'll
	move the current code into afw

2008-10-20 Special meeting on detection code layout

We held a meeting to decide how to layout the code that
measures things;  we need to make progress on DC3, but
we don't want to hobble our later efforts either.

Executive summary:  We'll have
	DMS/measureObjects/algorithms
	DMS/measureObjects/deblender
	DMS/measureObjects/plumbing
unless there are better names suggested by Wednesday.  N.b.
the deblender will probably be basically a NOP for DC3.


Present: TSA RHL ACB Roc(==RMC) DaveBurke Deborah Martin

TSA: How do we want to organize ourselves to do
detection and object properties.  There's a trac
page:
	  http://dev.lsstcorp.org/trac/wiki/DetectionAlgorithms


Having detected all objects in a coadd, how do we measure
their properties. Extremal proposals:
      1/ Measure all objects in one code in image stacks
      2/ Have a set of specialist measurement codes (e.g.
      stars only).

RHL:  in SDSS we used one measurement code for everything,

TSA... but people weren't happy with SDSS photometry

RMC:  you want simple things too

TSA:  what would you like to do, Robert?

RHL: let's agree that we'll measure faint properties
     on all objects

TSA:  Do we need to merge dophot and multifit?

RHL:  yes, certainly.

TSA:  but for DC3?  Can't we do a crowded stellar code + isolated galaxy multifit

Martin:  I'm not afraid of large, isolated objects.

TSA: RHL's proposal was:
algorithms (e.g. centroiding) would end up in e.g.
	DMS/measureObjects/algorithms
	
the deblender would probably go in
	DMS/measureObjects/deblender
	
the multifit plumbing would be in
	DMS/measureObjects/multifit

RCM: should the plumbing go up a level?

TSA:  Let's split the plumbing from multifit in this hieararchy.  We need
a name:
  plumbing
  Joe
  pixelDelivery
  framework
  middleware
  infrastructure
  utils

These will be cast in stone on Wednesday.  Plumbing unless something else wins my Wednesday

ACB:  Shouldn't the plumbing go up a level?

RCM:  There's probably no need for 
	DMS/measureObjects/middleware
	DMS/measureObjects/science
as only plumbing would in the former.

2008-10-29

RHL, Deborah Levine, NMS, David Burke, ACB, Steve Bickerton, Fergal Mullally, Roc, REO, AJC, Suzy

UW:
	ACB:  Nothing
	NMS:  ISR.  C++ issues.  No unit tests
	REO:  Improve background subtraction;  worked with KT.  CCB proposal
	AJC:  Compare image sims with SDSS images.  Working on an aproximate WCS in
	      image headers. There are no real sources, so problem for WCS... will
	      insert bright end from USNO-B

Princeton:
	RHL: Welcome to Steve and Fergal.

	     I'm finishing putting detection in new API; I filed a bug
	     report against doxygen this morning...
	     
	Fergal:  Built LSST stack in Princeton on Centos 64-bit
		 Built astrometry.net  Written in C with python front end.
	ACB:  need to inject our own reference catalogue.
	Fergal: uses USNO-B stored as a K-D tree

Deborah Levine:  Nothing to report this week

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Image classes.  RHL: proposes moving to mainline as soon as there are some docs.
      REO:  Let's do it -- but first check with Serge as to his swig changes.

Numerical Recipes & stats libraries.
	RHL:  what do we need?
	REO:  what's the overlap with GSL?
	ACB:  Matrix math?
	RHL: from Lapack? ACB: OK.

              Also minimisation from minuit, and
	      e.g. probabilities from GSL, FFTs from fftw. Possibly
	      Image stats from Pan-STARR's psLib, but it isn't clear
	      how well this will map to our data structures.

	      I was assigned #395 (Russ Laher's going to provide test
	      cases; #400) and put some initial functionality into the
	      new AFW branches as I needed it to move the Footprint
	      code over.  Russel reports that he has some fast median
	      code that could go in the same place.

	      We need a list of functionality from NR that LSST should
	      provide, and an assesment of what we gain from NR as
	      opposed to other sources of algorithms.  The packaging
	      into good idiomatic C++ may not be a trivial effort.  We
	      hope that Russ Laher will investigate.

2008/11/05 Guy Fawkes' Day

Present: RHL, REO, Roc, KTL (Lurking), Suzy, NMS, RRA
MIA: UCDavis, Mullally, Bickerton, Deborah/Vince

Progress RHL: Image API #374. eups distrib issues + os/x
              Steve/Fergal (in absentia) astrometry.net is built.  Learning build system

	 NMS: Working on getting swig working.  Some trivial unit-tests already written
	      Thinking about cross-talk EA modelling (from two new tickets)
	      RHL: how long will this take?  NMS: not long
	      RHL: If the cross-talk takes more than 1/2 a day, please check with RHL/Roc

	 REO: Background subtraction;  Naive use of PSF fitting to estimate background
	      doesn't work.
	      Paying attention to Exceptions

	 Suzy/Roc: Possible changes to Applications organisation with TSA/RHL/JeffK
	      Concerned about DC3 schedule

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Pixel Coordinates
      PhysicalCcd
      For now

Image Statistics
      Naive in afw-new-image-api

Background Subtraction
      REO: Estimate background as a functional form.
           Background is represented as afw::math::Function2
	   Background estimator is kludge

      RHL: Where does the background estimator belong?
      RRA: Becker/Lupton
      RHL: Rats.

2008/11/12

Present: RHL, ACB, AJC, NMS, RAA, Roc, Jacek, Martin, KTL, REO, TSA, Suzy

Template storage:
	KTL: Hyperatlas? Healpix? HTM?
	AJC: Why not tangent plane?
	KTL: Worried about needing large edge overlaps
	AJC: So a storage issue?  Yes.
	Roc: Is a tangent plane the current default?
	KTL: Yes; due to TSA
	KTL: Who's going to decide?
	RHL: Should we privide firm requirements, and allow you to choose?
	Roc: We should make a proposal.
	Martin: Talk about tomorrow at SLAC? Yes.
	TSA: Two issues:  Tesselation and Projection
	RHL:  TSA, AJC, Martin Dubcovsky, KTL will define requirements and make a proposal.

Milestones:
	The question is, where are we for DC3a?
	RHL cedes control to Suzy...

2008/11/21

No meeting

2008/11/26

RHL, David Burke, Dave Wittman, Roc, NMS, Martin Dubcovsky, KTL, Suzy, ACB, Gregory, AJC

Reports
	RHL: Misc afw/build system stuff
	NMS: Moving code from C++ to python.  Some unit-tests.
	ACB: Full time diffim.  Up on cluster;  3e5 kernels sent to CMU for modelling
	     Two models for spatial variation:
	     	     1. Fit for each pixel separately.
		     2.  Use eigen kernels (try 1..n*n)
	     Run all DC2 data on using cluster directly.
	Dave/Martin: Source/DIASource classes
	Suzy: Working with Ray Plante on DC3 schedule (Cristina's more-or-less off project)
	
	Status of Image API
	Operational, but may not be fast enough.  Possible Serge involvement.
	
	Status of Source (DIASource etc.)
	Defined from DB schema and reverse engineered; we can change this but keep KT/Jacek in the loop
	RHL: why not use a base class?
	Gregory: polymorphism or common base class.  Can use included member
	KTL: pieces of data are common to Source and DIASource; could be factored out
	RHL: how do do merge?
	Martin: Merge 465 by end of day
	KTL: Merge then review as per Robyn yesterday
	Consensus:  try that

	Thoughts on Matrices
	Naive GSL slow using CBLAS.
	C++ matrix classes don't seem to exist (but e.g. Blitz)
	Gregory:  how far does gsl permeate code?  Should we wrap?
	RHL: can we just use lapack?
	RHL: Talk to Steve Meyers at NRAO.
	ACB: not being held up; am using GSL.  Takes 33% longer to fill matrices;
	RHL: can you generate a simple test program?  ACB: certainly
	Gregory:  we have the tools here...	
	
	PSF spatial variation/Kernel synergies; QA.
	Meeting at 11am Pacific on 2008/12/1
	
	Calib point of view:  Understand dome screen flats in ISR
	David Burke:  Take monochromatic flats:
	      Spatial flat field for ISR
	      Filter band passes
	      Fringe frame inputs?
	Also white-light screens (quartz)
	Writing simulation for DC3; how do we use it?

	NMS:  Who provides the illumination correction?  From a dithered star pattern?
	I've implemented something for CFHT.  NMS will write up on trac

	Any Other Business

2008-12-03

RHL, ACB, NMS, RAA, MD, Roc, REO, TSA, Suzy, Ryan, Fergal, Gregory, AJC, Bickerton

Status:
	ACB  Really basic matrix test code (in svn). GSL: really slow wrt vw::lapack (slow BLAS)
	     Working on a class to handle spatial modelling of Kernels/PSF, with SDQA thingies. Need ticket number
	     Will be late on diffim deliverable.

	NMS  Starting to test ISR code. 
	     Talking with David Burke about calib images on Friday at 1:30 Pacific

	RAA  Reverse engineering code into EA

	RHL  Converting measure objects to new Image API, and fixing bugs.  New baseclass

	MD   Baseclass for Sources/DIASource

	Roc  Interface meeting with camera team requested a list of metadata (everything
	    except pixels) needed for pipelines.  Should be inclusive, not just camera.
	    
	    Meeting mostly concentrated on interface docs, not interfaces.

	Gregory  Camera/OCS interface meeting at SLAC raised questions about DM/OCS interface
	    Meeting notes will be on camera hypernews groups.

	    Will DM need to pull bits out of the facility DB and put it into some DM DB?
	    E.g. facility DM don't plan to support more than one set of cross-talk corrections.

	RHL  Will camera/DM share code?

	TSA  Camera team can share image display/analysis algorithms, but it isn't clear that
	     much code will be shared.

	     Gregory will try to get some end-to-end tests sorted out. He wants to have an
	     appropriate level of reuse, but not too closely coupled.
	
	REO  Working on Kaiser coadd.  RHL: when will it sort-of-work?  A month to do the
	     deconvolve step.

	     Need Exposure class.  Need to speed up "WCS Match" (warp) and convolution.

	TSA  No DC3a responsibilities, but photom calib for DC3b.
	     Stalled by inability to run DC2 (or DC3) pipelines. Ray says it'll be really soon.

	Suzy  Updating apps schedule, not back in Ray's court. We'll be getting weekly
	     pings on status.

	     We're behind --- need to get things put together by the middle of January.
	
	Ryan  Mainly starting to work on pixellization stuff, as per K-T. Check in when svn's back
	
	Fergal  Working on astrometry.net package.  Code compiles.
	        Data needs to be distributed in some complicated way (permissions?)

		Working on specifying an input WCS.

		Ticket?

	AJC  First 10% of DC3 data.  150 CCDs; 1 pointing. r band. No USNO bright stars.
	     Please look at it! It'll be at NCSA.

	     No ISR related data applied or needed; constant gain. No CRs, no bad columns

	     Will put up an overview on trac.  Use tickets to get comments?

	Steve Working on Statistics class adding medians/quantiles/clipped statistics.

	      Do we want to handle the Mask plane?

	      All needed to provide background estimates --- started by next week.

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Next meeting
     TSA RHL ACB Ryan AJC Grillmair

     2 or 3 Pacific on Thursday 11th December 2008