Minutes of the ImageDetWG Weekly 'Phone Con.s
- Minutes of the ImageDetWG Weekly 'Phone Con.s
-
Minutes for 2008
- 2008-06-09
- 2008/06/23
- 2008/06/30
- 2008-07-07
- 2008-07-14
- 2008-07-21
- 2008-07-28
- 2008-08-04
- 2008-08-25
- 2008-09-01 -- 2008-09-15
- 2008-09-08
- 2008-09-15
- 2008-09-22
- 2008-09-29
- 2008-10-17
- 2008-10-20 Special meeting on detection code layout
- 2008-10-29
- 2008/11/05 Guy Fawkes' Day
- 2008/11/12
- 2008/11/21
- 2008/11/26
- 2008-12-03
Minutes for 2008
2008-06-09
Present: RHL, Roc Cutri, Deborah Levine, TSA, ACB MIA: The rest of the LSST collaboration 1/ Status reports [inc. Mea Culpas] (All) RHL: Nothing to report TSA: calibration association standalone (i.e. DC2 stack) i.e. can be run as a standalone pipeline, but not generally usable ACB: Diff Im; two tickets: a. Kernel has constant volume b. Spatial variation by kriging Waited on code cleanup until someone can do a code review Deborah: Nothing to report. 2/ Report from CCB (Robyn) See CCB page (http://dev.lsstcorp.org/trac/wiki/CCB). Basically, we: adopted some typedef name change conventions 3/ DC3 schedule for Applications (Suzy Dodd) Postponed due to confusion about time zones. 4/ LSST algorithms on GPUs RHL: Has writeup on MaskedImage convolution TSA: I thought that we said the entire image subtraction pipeline? RHL: Will investigate TSA: Also Chris Hoffman at Purdue is involved in a new, secret, intel processor and wants to try out image subtraction. Roc: Do we need to do this? TSA: Probably; we're a factor of 5 too slow and Moore's law is mostly going into cores, not clock speed, these days 5/ Any other business Tim and I are otherwise engaged next week, so the 2008-06-16 meeting is hereby cancelled. I'll send out a reminder.
2008/06/23
Present: RHL, REO, TSA, RRA, Roc Cutri, NMS, ACB, AJC, Deborah Levine,
Status report:
REO's working on separable kernels
NMS is coding up the ISR stuff, but not testing it due to perceived
difficulties with the DMS stack (it's not clear if we're really in trouble)
ISR will ingest CFHT data initially (NMS will talk to Lynne)
ACB's helping with EPO and light curves
He's also looking at a geostat kriging library to experiment
with spatial variation of the diffim kernel (basically a linear
predictive code?)
AJC reports that John Peterson's added Mie scattering. He's talking
to Dave Monet about DGM's needs.
He expects to deliver simulations in a month or so BUT NEEDS REQUIREMENTS
TSA has been working on photometric calibration in a custom python/mysql
framework (e.g. deriving grayExtinction(x, t) to 1%).
Q: where does generation of flats belong?
A: RHL should ask David Burke
2008/06/30
Jacek, K-T, RHL, Roc, Suzy Dodd, NMS, REO, ACB, AJC, Deborah Levine
REO: Separable kernels. API changes in afw allowed a build all the way to DiffIm
NMS: Debugging ISR with all the world built
ACB: Wrapper about gstat. Wrestling with build system on 64. AJC notes that this dates from January
RHL: Pixel coords proposal (http://dev.lsstcorp.org/trac/wiki/BottomLeftPixelProposalII)
Russ Laher will head up Abraxis evaluation
Image -- not done (REO : wants input into pixel access APIs)
Suzy Dodd: sent out DC3 schedule proposal on the 2008-06-20. Will start pinging you on schedule in Monthly
Still accepting comments.
RHL: use trac
AJC: Recovering from lost laptop
Roc + Armus looking at sims
Wants a guinea pig: Proposes Russell and template generation
(Send AJC and Roc Nick's paper)
RHL: Penn group will join DM
K-T Lin: Image metadata. Persisted to DB, but what about in memory?
DataProperty? Member variables? Strings
Deborah: is there a list on trac? A: No
KTL:
* Metadata passed through
* Immutable metadata
* Mutable metadata
REO: Set flags rather than consider new metadata.
So we have:
* Dictionary metadata
* Interpreted as member variables
* Interpreted as a class
Currently Image has metadata, and Exposure has an Image and WCS --- two places
If an Image is part of something more complex, does it acquire metadata when
written to a FITS file?
KTL: Images will be saved as minimal FITS + DB (RHL: is this a good idea?)
RHL: Always read a FITS file into an Exposure --- Exposure is the fundamental image object
RHL: If you need metadata, inherit from Image and add members (not a dictionary!)
Does Provenance belong to objects, or is it global? It's global
REO will write up for CCB
Any other business:
Francesco:
Will join this telecon
K-T Lim: Storage for templates --- not just images. E.g. how we tile the sky
AJC: keep Davis group in the loop re multifit? No.
2008-07-07
Present: REO, RHL, AJC, ACB, NMS, Francesco, Bhuvnesh, Roc, Suzy Dodd, Deborah Levine, Dave Wittman, Mike Jarvis
ACB still having trouble with build system. Waiting on RHL
NMS Looking for data from Ray; Ray is on holiday (as is Lynne)
REO Posted metadata proposal at
http://dev.lsstcorp.org/trac/wiki/ImageMetadataProposal
Started on template generation.
RHL nothing relevant
Suzy Dodd: Aug 21-22 for DC3 use case meeting; Rosenberg will be present.
There was some discussion of whether we all needed to go to this meeting,
and whether discussing use cases was really the best way to spend the
whole meeting. Please send comments to Roc, Suzy, and me.
AJC: Template formats. Initial plan is to save tangent plane tilings;
questions remain about footprint sizes and overlaps. Will make a proposal.
Q: Who cares outside applications and needs to sign off?
Bhuvnesh and Mike. Discussed value of shapelets in context of shear
measurement: 2nd order shapelets are Gaussian-weighted second moments
for shear. Advantage of shapelet PSF is that convolutions are matrix
operations so can measure deconvolved shapes.
Q: is this optimal for non-shape things? A: don't know.
Dave Wittman: There's a Koiken paper on PSF and aperture-matched photometry
RHL: (obnovious comment that SDSS has done this for years)
Measurement of PSF assumes an isolated object, and fit for the model coefficients;
RHL expressed concern about blending.
Bhuv: we only promised to deliver our weak-lensing model; so we need to
write the Penn SOW carefully.
Bhuv: We expect to have an at-a-point PSF model integrated into the LSST
system by March 31st, which is the deadline for the data release pipeline (spatially
varying PSFs may need to be deferred to DC4).
REO/ACB: So we'll need another implementation?
RHL: Yes; I'd like to get Steve Bickerton/Fergal Mullally here in
P'ton + ACB + input from Dave Wittman to do it. Then we can figure
out what we need, and work with the Penn group on a design that works
for LSST.
Roc: We need to avoid a "my PSF's better" war. RHL's well-known diplomatic
skills will be employed to keep us safe.
2008-07-14
RHL NMS AJC Jacek Roc KTL Francesco REO ACB Deborah
ACB: I have the build system working on 64-bit systems;
RHL found a problem with ldopen flags on 64-bit
machines in the build product
He's also converting CFHT bad masks to LSST Images
There followed a discussion on where this information comes
from. There were various proposals as to the correct classes
(e.g. Masks; lists of Footprints) but apart from these
technicalities there's the question of where this sort of
data should be saved. NMS was given the job of sorting
out what sorts of inputs will be needed to the ISR, and
deciding what sort of database queries they need to support.
Back to ACB's geo-statistical library... should he wrap it in C++
and then swig that, or swig the C directly. RHL pointed out
that there are pros and cons of C (e.g. telling swig what's
a constructor; possibly needing typedefs for containers). He'll
send the headers to the mailing list.
(Parenthetically, are the lists functional? No one seems to use
the keywords. At some point RHL was going to transfer our mail
to hypernews, but he failed to figure out a way to import our
archived mail. Maybe we should just set up a new list...)
NMS: Can build the ISR on 32 bits.
Found CFHT raw data at NCSA (thank you David Gehrig) and is
writing scripts to mangle the data into LSST formats.
RHL complained this sounds a bit ad-hoc; we expect to import
data from many cameras before LSST (CFHT, SDSS?, suprimeCam, ...)
and we should try to re-use as much of the effort as possible.
AJC: So this is just stuffing the clipboard?
KTL: We have generic persistence, but this is more specific
AJC: NMS will look at how specialised her design is, and
how hard it'd be to generalise.
REO: Thinking abour template generation
RHL: Working on the build system and on SDSS
IPAC: Russ Laher is thinking about how to store SDQA data on the
clipboard. This is currently a learning exercise, and he'll
collaborate with e.g. KTL. It's also probably a topic
of discussion for this phone con
DC3 Design/ES meeting on 21-22 August
Roc: If we have to go, what would be useful?
KTL: What's in DC3 scope? What algorithms are going to be
implemented? What new classes (e.g. bad pixel lists); this
could be done via EA.
There was a general discussion about what this group would
really like to get out of the meeting. We felt that we needed
a meeting, but that this one might not be it, as it'll take at
least 1.5--2 days to do the EA part and we'd like time to
discuss everything else too.
Should we put SDQA (and other) information gleaned during processing
into the Facilities database
Terry Schalk wants information on timescale of c. alerts on
camera performance, and load facilities database. E.g. PSFs;
S/N; bad pixels. Can we do this? Do they access clipboards?
KTL: A replica of SDQA DB into his DB; performance is OK. We
don't need to worry about de-normalisation if we just copy
[part of] the DB.
Jacek: Terry was thinking about a publish/subscribe style, but
he's flexible
Simulations (Roc)
Roc and Lee Armus will be contacting people individually;
some people (e.g. AJC + REO) already have lists of desiderata.
REO will run next meeting as RHL'll be in Japan.
2008-07-21
Minutes of the LSST Applications phone con held 2008-07-21 10:00 Pacific time Attendees: - Russell Owen - Nicole Sylvestri - Andy Becker - Suzy Dodd - Roc Coutri - Ray Plante - K-T Lim - Deborah Levine - Jacek Becla *** Progress Reports Andy made progress in two areas: 1) Wrote an LSST wrapper around the gstat geospatial library. It compiles and he is now testing it. 2) Talked to Andy Connolly about image simulation requirements for difference imaging. They agreed that initially they wish to investigate the effects of the optics and variable seeing. After that they would like to study the effects of different airmass and differential chromatic refraction. Ray asked if gstat was to become a new required package for the LSST software stack. Andy said it was too early to say; for now he is just testing. Nicole worked on the code that preprocesses data from various surveys into the format required for the LSST ISR pipeline. Initially this code was written to handle CFHT data but Robert Lupton asked about making it more general at last week's Applications phone con. Nicole wrote a proposal for how to accomplish this and first circulated it within the UW group. Having incorporated that feedback she will post the proposal to the Trac wiki shortly. She expects to have the software itself running by the end of the week. Depending on reactions to her proposal it may need some work, but it will certainly work with the CFHT data. Deborah debriefed Robert on what diagnostics will be output by the various pipelines he is responsible for. These include detection, PSF fitting and measurement of object parameters after detection. She plans to debrief Nicole next. Russell mainly worked on other projects last week, but set up meetings this week to detail out how we will implement Nick Kaiser's paper on Template generation. *** Discussion of the Proposal for Metadata For Image-Like Objects <http://dev.lsstcorp.org/trac/wiki/ImageMetadataProposal> (please feel free to post corrections and additions; it was hard to keep up with the discussion and write notes at the same time) Russell wants this proposal discussed and agreed to as soon as possible so we can design the ISR with it in mind. To that end we spent the rest of the meeting discussing it. * Does it make sense for LSST standard metadata to be represented as metadata-specific member variables, rather than as name:value pairs in a general DataProperty. (Note: name:value DataProperty metadata will still be supported for experimental code, test code, etc.) Andy Becker worries about needing to modify the class every time we add a bit of metadata. On the whole people seemed to agree that this was acceptable as long as the updates occurred occasionally and at specific times (that we could run for awhile using new metadata in name:value pairs). * Should new metadata should be persisted in the database and if so, how? The obvious way to handle new metadata is to add it to a Character Large Object (CLOB) or something similar. This gets the data *into* the database, but it is not easily queried. Jacek proposed that if we initially use a CLOB for new metadata that we actually want to persist (much of it can be reconstructed and so may not need to be persisted). Then during the next data release we turn that metadata into new database fields (which *can* easily be queried). The consensus on persisting seemed to be that Jacek's proposal was the best solution, offering a good balance of power, ease of use and ease of implementation. However, K-T brought up the issue of backwards compatibility issues cause by the database changing: - queries on newer databases may not work on older databases. - There are compatibility issues for persistence, and thus running different versions of the LSST software on different versions of the data release database. Everyone agreed this is a serious issue that needs further thought. But it does seem to be largely independent of the image metadata proposal. * Should the Image and Mask classes have metadata (beyond the bare minimum required for the classes)? Advantages to not having metadata (all metadata in Exposure only): - There is only one obvious place to look for the metadata - If one gets an Image out of an Exposure there is no issue about whether to copy the metadata from the Exposure to the Image Disadvantages: - There is no afw object that is a really direct representation of a simple FITS file. Thus it is a bit harder to read and write simple FITS images (e.g. have to carry around the variance and mask of the Exposure in afw). In the end it was agreed that if the applications developers were comfortable giving up the metadata in Image and Mask then it was fine. Nicole, Andy and Russell all feel it is OK, but Andy would like to give Tim Axelrod a chance to weigh in. * Should we keep MaskedImage or combine it with Exposure into one class? Also, do we want more than one kind of Exposure? Nicole and Andy could not think of any use case for a bare MaskedImage, but neither they nor Russell have carefully looked through the EA model to be sure. K-T then asked if we should have different kinds of Exposures for different kinds of images, e.g. bias vs science image. This would be sensible if the metadata is very different (especially if the metadata is in instance variables). Andy felt that the different kinds of images weren't different enough to justify this. Ray pointed out that we can avoid the need for different kinds of Exposures if we use a DataProperty for most metadata (as we do now). In other words, using dynamically typed metadata. However, Andy was pretty sure we only need one kind of Exposure in any case. The consensus was that we probably only need one kind of Exposure and that we can probably do without MaskedImage. But given the uncertainties it makes sense to change Exposure to inherit from MaskedImage (rather than contain a MaskedImage, as now) and prefer Exposure to maskedImage and see how that works out. If it works well then we can get rid of Exposure with minimal pain later. * What about obscure metadata that is only needed by one or two tiny bits of code? K-T brought up the issue that there is a lot of metadata associated with an image and some of it may only be needed by one bit of code, perhaps only sometimes. He pointed out that there was a proposal for lazy evaluation of such metadata and he felt this would still be a good idea. He proposed that we add a new metadata member variable "lazyMetadata" for such metadata (since it would probably want to be some new kind of object). We wrapped up by summarizing the conclusions: - Get rid of metadata in Image provided Tim agrees. - Avoid duplicate metadata. There should only be one obvious place to look for an item of metadata and it should never be duplicated in a given object. - Combine MaskedImage and Exposure only if MaskedImage truly not needed and there is only one kind of Exposure. (Change Exposure to inherit from MaskedImage otherwise). - It is OK to put standard LSST metadata in member variables. But we probably need to add a new lazyMetadata member variable.
2008-07-28
RHL wrote minutes, but restarted his laptop before posting them, so they're lost. Mea Culpa.
2008-08-04
RHL TSA REO RRA Roc ACB NMS KTL Francesco Deborah DaveWittman
-------------------------------------------------------------
Missing agenda
Status Reports
Ray Plante: working on Linux64 build; expects to get it out by later today
ACB: vw 1.0.1+2; boost 1.34.1
NMS: writing ISR code. Use cases will define inputs
Expect to have ISR code running by end of August
REO: Metadata proposal; Kaiser coadds
ACB: Send out email; worked on coadds
RHL: Build systems.
TSA: I'm back
RRA: CCB meeting.
Abraxis/Parasoft comparison; more impressed by parasoft [who'll be at IPAC]
Metadata proposal. REO will go ahead and develop the headers that define
the metadata that'll appear as C++
Deborah: working on Use Cases
== How are we going to maximise the usefulness of the DC3 meeting at IPAC? ==
TSA: we have lots of bottom-up stuff OK, but haven't made progress in top-down
Need to take Use Cases (Nightly and Data Release) and move forward until can
be used for the design.
TSA intends to read use cases in EA. Where there are place holders
for e.g. multifit , expand them.
KTL: need to know inputs and outputs for all pipelines, enough to be persisted.
TSA: need use cases in advance with all nouns mapping to classes.
TSA: if you don't use EA, write the use cases as text and post the list
Roc: Suzy and I are planning the spare 1/2- day; including integrates schedule
== How do we want to build templates/coadds for detection? ==
REO AJC thinks image subtraction coadd; we think for object detection
REO: Convolve all images by its own PSF and add.
Do same for PSFs
These two sums are jointly a sufficient statistic for the true sky
Templates: Probably need 2 (low airmass and high airmass)
AJC: Wants good and bad seeing templates; the bad seeing
templates have better S/N
TSA: Are they a significant impact on the storage?
KTL: they can be; and they're stored at higher resolution (2x?) and at
the base camp
TSA: Are they worse than 10-15% at base camp?
TSA: Need coadds for at least EPO, in addition to colour JPGs. Not at
base.
We'll try Kaiser, and see how it goes. Phone con. tomorrow
AJC: Would Dave Wittman use this to warm-start for detection?
Dave: it sounds interesting, but I'll wait until I can play with it
NEXT MEETING
RHL's in the UK starting tomorrow, returning on the 18th.
Next week (11th August) there's a meeting in Tucson so this phone con's cancelled
The following week (18th) is the week of the DC3 meeting, but we decided
that we should have a meeting anyway. Tim will officiate.
2008-08-25
RHL, Russ Laher, KTL, Suzy, Roc, REO, NMS, ACB, Vince, Wittman, Deborah
Status Reports:
RHL: Worked on domain model/robustness.
Went through task list
Suzy: Nag: outstanding work due by Friday (29th)
Nag: progress reports (but I did that at the meeting)
Roc: Task list discussion at meeting was surprising! Make
sure you know what you're on the hook for
Technical issues:
ACB: Two issues from apps meeting:
1. Deborah + KTL and I talked about SDQA. How do we bring
SDQA into line with the rest of the project. KTL: after each
stage have an SDQA stage. ACB: have each stage include its
own sub-stage.
Roc: Are there shades of gray? Or just good/fatal. ACB: the latter
RHL: Isn't an extra stage able to accept a 0/1 flag, or be flexible?
Roc: Didn't expect that SDQA would have an operational role (Vince: SDQA is
passive)
KTL: Is SDQA running in realtime? (Yes)
RHL: I don't care if there's an associated SDQA stage, or an SDQA module
that's called by the stage.
ACB: With an in-stage SDQA we need to mix policy files (processing + SDQA)
RHL: We may want to call this qa-info gathering information
something other than SDQA, but we do need it. We may
need some qa-aware piece of middleware. Call is Pipeline QA
KTL: So each app stage generates SDQA data in clipboard.
There's another consumer PQA (also called pipeline executive),
which also persists SDQA information.
Where do we document this? In "How Pipeline Runs" doc; KTL
gets middleware, Deborah gets SDQA.
2. Jeff is unhappy about OO nature of Image differencing code. He wants
more methods, less subroutines.
KTL/RHL: Are subroutines stateless? Do you have lots of routines with
a common set of arguments?
ACB: Not really.
KTL: We could use controller classes.
ACB: Should we rewrite diffim to learn this?
RHL: Or should we make ISR a learning example? Probably. Roc/RHL: we
can't afford to reimplement just now.
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
There's a meeting tomorrow/Wednesday to decide middleware/apps interfaces.
KTL will send an agenda and the meeting's open --- but it's not a discussion
meeting, it's to take decisions.
2008-09-01 -- 2008-09-15
RHL played hookie
2008-09-08
Apps Telecon 9/8/08 Present: Jacek, Andy B, Russell, Dave W, Deborah, Vince, Russ L, Martin, Nicole, David Burke, Tim --- We discussed Jeff's need to have controller complexities assigned in the EA model. The status on a per-package basis: Astrometric Calibration: => TA to complete CoAdd Images: Kaiser CoAdd completed by RO; Unclear what we will do with the general CoAdd Images use cases (further discussion below) Day MOPS: No robustness or controllers at this point. => FP to add what's needed for Jeff's estimate Deep Detection: DW has already assigned complexities. ISR: NS has already assigned complexities Photometric Calibration: => TA to complete Produce Data Release Image Products: This is mostly an "umbrella use case" package with the controllers existing in other packages. We agreed that "Produce RGB Images for Visualization" will remain a placeholder for DC3. "Produce Template Images for Subtraction" needs to be modeled. => AB to do this, using the Kaiser CoAdd model as a starting place Produce Data Release: Again, mostly an umbrella with the content elsewhere. A package that needs to be fleshed out is "Run Image Data Train", which has been under active discussion. Not clear at the telecon who will do this, though JB indicated it is mostly a middleware task. Run Nightly Pipeline for DR: => TA to complete => Note that all assigned complexities need to be reviewed for consistency with JK's revised guidelines (1 week, 1 month, 3 months) The question came up of how to assign complexity for controllers that are already implemented. AB suggested changing the status for these to "Implemented" from "Proposed". NS noted that it is not possible to leave the complexity annotation off entirely, as JK suggested for trivial controllers => JK to advise Another modeling issue that came up is how we treat parts that we will have for LSST, but not for DC3. There was general agreement that (a) we need to handle this uniformly; and (b) it is probably best to show the full LSST case when we know it, and mark in some way those bits that are not included in DC3. => JK to advise how he wants this done ---- Nightly processing for Data Release and how it differs from regular nightly processing for alerts: Does DD or NP come first? TA argued that DD should come first, AB that NP should. A use case to keep in mind here is how a supernova occurring within an extended galaxy will be handled. In particular, when doing DD, would the galaxy model used by multifit contain a time variable point source for the supernova? Would the time varying source be subtracted off before DD? TA raised the issue of NN2 processing for light echoes and similar time varying diffuse structures. There was general agreement that this should not be part of DR processing, but instead be a "value added" product from science collabs or similar. TA asked whether we should do forced photometry on difference images for all Objects, or perhaps a subset, such as those with significant variability. There was general agreement that this should be done at some level. Details are not clear. ---- Deep detection issues: JB brought up the issue of DD runs in between data releases. TA argues that DD will consume the majority of processing time for a data release (DR), and that if we need DD done more frequently it would be simpler to just speed up the DR cycle. There was not much dissent from this view at the telecon, though JB expressed concern for the storage budget if we speed up the DR cycle. => TA to send out email proposing that the intermediate DD runs be dropped, and await reaction. TA asked whether "Assemble Relevant Pixels" needs a WCS match step. DW replied that his current thinking is that the geometric remap will NOT be applied to the pixels, but rather kept as part of the model. If we do this, the remap is not part of "Assemble..." DW suggests that we should mark "Evaluate Model on DataCube" and following controllers "not for DC3" TA noted that we need CoAdds in several different forms throughout the DR use cases. This cries out to be carefully modeled as a shared capability. While discussing CoAdds we discovered a lack of clarity on how transient objects (such as solar system objects) will be handled in producing the CoAdd used by DD for object detection. TA assumed that sigma clipping would get rid of such things, while AB assumed that explicit masking based on prior image differencing would be required. => action needed to resolve this. DW/UCD team? ---- The telecon ran out of time with lots left to discuss.
2008-09-15
Apps Telecon 9/15/08
Present: Andy B, Russell, Dave W, Deborah, Russ L,
Martin, Nicole, Robyn, Andy C, Serge, Roc, Suzy, Tim, RHL (late)
---
We discussed the issue raised by Martin of whether the multiple
uses of detection in the UML model really differ in their
requirements. TA pointed out that in addition to detection on
difference images and detection on coadds for deep detection, we have
detection on bright stars in unsubtracted images for WCS
determination.
There was general agreement that detection on a coadd will need a
different, and significantly more sophisticated, algorithm than for
detection on difference images. Among other capabilities, it needs
good PSF determination, sky estimation, and deblending. Detection on
image subtraction usually needs none of these. AB, however, suggested
that deblending might sometimes be needed for crowded fields like the LMC,
and agreed to supply some examples of this (he's now done so).
---
We began to go through the UML for "Detect on CoAdd Exposure". The
first sentence of the use case ("Crudely estimate background from the
input Polychromatic Coadd Exposure, generating a Background Model
Set") raised several points that needed clarification, and it seemed
likely that subsequent ones would be similar. Martin agreed to email
a list of points that need clarification, since there wasn't time to
go much further on the telecon.
---
Suzy repeated her email request for input to the Applications Monthly
Report by the end of the day.
---
TA discussed the need to get AAS poster titles and authors in by the
end of Tuesday. AC reported that there will be one on image
simulation. DW felt that there had not been enough progress on deep
detection to warrant a poster at this point. It looks like we will
have one on data release (TA lead author), one on cosmic ray rejection
(Russ L), and one on new developments in image subtraction (AB)
---
AB brought up the desire to have a C++ math library API that looks
like the python API. TA pointed out that this falls within the
purview of the group set up at the IPAC meeting in February. This
group has never met, and clearly needs to. RHL, who is the chair of
the group, joined the telecon at this point and agreed to call a
meeting.
---
At this point we went to status reports. RHL reported his progress on
reimplentation of the image classes:
Implemented Image and DecoratedImage. Fits I/O code
works (written in an ACT meeting; it's much cleaner than
the hack I had to do for vw. Uses boost::mpl) Metadata is
probably OK. Swig works, but needed a change to daf_utils
to get proper shared_ptr support (i.e. use the code that's
now in swig). This is a general issue -- the new code seems
to just work, and we should use it.
Mask isn't written, but char Images are, so it should
be easy.
Masked image next. It'll use DecoratedImage and will
be a simple wrapper of a set of 3 images/masks for now.
We might consider an interleaved implementation later,
but it should be invisible.
Needs to be doxygenated. I'll then merge all this
code to an afw branch and review etc.
Serge reported that he is working on implementing the changes to
shared pointer support. He and KT are working on the changes needed
to DataProperty, which is one of the bigger tasks.
AB reported that he is still blocked by the build system not working
on his machine. We agreed to discuss at Tuesday's DC3 telecon.
2008-09-22
RHL, ACB, TSA, REO, Dave Wittman, Martin, \v{Z}I, AJC, RRA, Ryan Scranton
Achievements du semain:
ACB: writing code with build system working on 64-bit machine
Installed ...
Kernel testing package.
Study kernel basis functions and spatial variation
TSA: Is WCS on your plate?
ACB: Yes; given positions of stars, solve for WCS. Simple once an initial guess is there
TSA: Consider astrometry.net
REO: Getting CFHT data for coadd tests.
NMS: Writing ISR code. Five stages done, three to go.
RHL: Image classes almost there
TSA: Mostly photom calib; April CTIO data. Systematics in DoPhot analysis.
RHL: Apps layer?
TSA: Need attention to detection pipeline for DC4... E.g. sky subtraction
Davis: Fighting build system with Ray's help.
AJC: Delivered catalog generation for Jacek. John Peterson sent TSA an image (TSA: it looks great)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Are we duplicating Detection in EA? (Martin)
Martin: we seem to have to same code duplicated everywhere.
TSA: we resolved this last week. Coadds and diffims are really different, but will share code
Do we understand detection in coadd (Martin)
REO will propagate PSF into coadded Exposures
Princeton will handle the detection coadds
All you wanted to know about Footprints and ExposurePlanes (Martin)
REO: Would like Footprint in afw. RHL will do. REO will file ticket.
Martin: Will Footprints be contiguous?
RHL: issues with blending
ExposurePlane: Footprint + sky pixels around it
RHL: Match individual exposures.
Dave: Triage for DC3?
RHL: I'd like to see a non-local sky (e.g. estimated on 100" scale)
Calibration issues that David Burke needs clarified
Unknown; RHL will investigate
Should we move this telecon?
Should image API use col/row or x/y (RHL)
REO: sets us back a month?
RHL: that's a lot; I'd have said yes. RHL will send around a note
Any Other Business:
REO: Who owns SkyRegion stuff?
TSA: No-one yet.
Ryan: Maybe me. RHL: will you decide?
REO: VO wants to use HTM, but healpix is better. Should we pay attention?
Ryan: VO not setup for depth masks
TSA: It's in the model. I'll help Ryan
2008-09-29
Present: RHL, Suzy Dodd, AJC, TSA, ACB, NMS, RRA, REO, Vince Manning, Ryan Scranton, Martin -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Achievements and Confessions RHL: Image classes IPAC: Suzy: Status + indendent cost estimates. Next week is DM review. Preparing to prepare for construction estimates TSA: DM review is not a dry run for the PDR, but to pick external brains and see where we are. Jeff Kantor will probably do most of presenting Vince: Next week or two should generate SDQA report, maybe with a preliminary wiki post UW: NMS: Coding^n on ISR. Splitting CFHT form MEF to single amplifier files. Not checked in so an not to break the trunk REO: Working with NMS. ACB: Mostly on vacation, and fighting templates with Russell AJC: Being a professor, and catalogues for image sims. A few CCDs come next TSA: Conversation with transient group on catalogues. 1. How we link together difference-image-Sources and Sources 2. What we do with many lowish S/N objects Need to converge soon Detection -- really source categorisation. Cf. Trac: http://dev.lsstcorp.org/trac/wiki/DetectionAlgorithms Summary: Can't just use multifit, but will share lots of code RRA: DC3 integration test plan UCDavis: Ryan: will do image footprint stuff Martin: talking with KT on data chain --- please use email
2008-10-17
Present: RHL Robyn Suzy NMS Laher Wittman Dubcovsky ACB TSA Burke Deborah.
RHL: worked on afw Image classes, and closing tickets
Robyn: Worked on the DM Review
TSA: DM Review. Calib review next week
Went very well. David Silva (Director of NOAO). Peter Lee (CS@CMU).
Silva pleased that we're so organised; went very well. Comments that
have most bearing were about map-reduce v. sciDB, and for apps as well
as DBs. RHL: who's going to pay attention to this?
Suzy: Inputs on monthly report, and construction workforce -- esp. RHL
ACB kernel generation process, including errors in kernels. Sending
set of kernels to Genevieve to choose good basis functions. Good
metrics for kernels. Iterating using noise to re-determine kernels.
TSA: WCS? ACB: Monet's worried about how we're doing WCSs, but we
don't have code yet. TSA: we need to move forward. RHL: shall I
take this on for a new post-doc? Yes.
TSA: astometry.net is robust -- but want initial guess. Monet
probably shouln't care -- this is just the initial guess, and
association has to work. Burke: this is correct. ACB: why do
we need astrometry.net? TSA: telescope may not point well. enough.
David Burke: Check that the second stage is happy with the first stage QA.
RHL: I'll synthesise and consult.
NMS I was sick last week. Just about got images through ISR, and fixed
up policy files a la Ray Plante. Scaling fringe patterns.
Test suite? Grabbed raw/calibs for CFHT data in afw.
REO: Kaiser coadds just-about working (images have been seen by ACB)
Russ Laher: Sent round SDQA document this morning. Will read TSA architecture blurb
Wittman: I've been sick
Martin: Working with Ryan Scranton on pixellisation code (STOMP).
Move to 10am Wednesday
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Image classes.
RHL: shall we update detection then merge to the mainline
TSA: Yes.
Branch: svn+ssh://svn.lsstcorp.org/DMS/afw/branches/new-image-api
Gil tutorial: http://opensource.adobe.com/wiki/display/gil/Generic+Image+Library
(afw supports the same API, even for MaskedImage)
Image stats. Means, std.devs, etc....
Russ Laher will file ticket. afw/math RHL will talk to Gene Magnier
at Pan-STARRS about getting their stats library.
Robyn says: could you add to the agenda the final selection of the
directory names/loc for detection/deepdetection. DC3 meeting pushed it
over to App meeting since so many folks were gone.
Robyn: I want a name for measureObj
TSA: how to avoid Balkanisation.
RHL: Devote next Wednesday meeting to this issue.
Robyn: Martin NEEDS this, but he can use the
current detection pending the decision, and we'll
move the current code into afw
2008-10-20 Special meeting on detection code layout
We held a meeting to decide how to layout the code that
measures things; we need to make progress on DC3, but
we don't want to hobble our later efforts either.
Executive summary: We'll have
DMS/measureObjects/algorithms
DMS/measureObjects/deblender
DMS/measureObjects/plumbing
unless there are better names suggested by Wednesday. N.b.
the deblender will probably be basically a NOP for DC3.
Present: TSA RHL ACB Roc(==RMC) DaveBurke Deborah Martin
TSA: How do we want to organize ourselves to do
detection and object properties. There's a trac
page:
http://dev.lsstcorp.org/trac/wiki/DetectionAlgorithms
Having detected all objects in a coadd, how do we measure
their properties. Extremal proposals:
1/ Measure all objects in one code in image stacks
2/ Have a set of specialist measurement codes (e.g.
stars only).
RHL: in SDSS we used one measurement code for everything,
TSA... but people weren't happy with SDSS photometry
RMC: you want simple things too
TSA: what would you like to do, Robert?
RHL: let's agree that we'll measure faint properties
on all objects
TSA: Do we need to merge dophot and multifit?
RHL: yes, certainly.
TSA: but for DC3? Can't we do a crowded stellar code + isolated galaxy multifit
Martin: I'm not afraid of large, isolated objects.
TSA: RHL's proposal was:
algorithms (e.g. centroiding) would end up in e.g.
DMS/measureObjects/algorithms
the deblender would probably go in
DMS/measureObjects/deblender
the multifit plumbing would be in
DMS/measureObjects/multifit
RCM: should the plumbing go up a level?
TSA: Let's split the plumbing from multifit in this hieararchy. We need
a name:
plumbing
Joe
pixelDelivery
framework
middleware
infrastructure
utils
These will be cast in stone on Wednesday. Plumbing unless something else wins my Wednesday
ACB: Shouldn't the plumbing go up a level?
RCM: There's probably no need for
DMS/measureObjects/middleware
DMS/measureObjects/science
as only plumbing would in the former.
2008-10-29
RHL, Deborah Levine, NMS, David Burke, ACB, Steve Bickerton, Fergal Mullally, Roc, REO, AJC, Suzy
UW:
ACB: Nothing
NMS: ISR. C++ issues. No unit tests
REO: Improve background subtraction; worked with KT. CCB proposal
AJC: Compare image sims with SDSS images. Working on an aproximate WCS in
image headers. There are no real sources, so problem for WCS... will
insert bright end from USNO-B
Princeton:
RHL: Welcome to Steve and Fergal.
I'm finishing putting detection in new API; I filed a bug
report against doxygen this morning...
Fergal: Built LSST stack in Princeton on Centos 64-bit
Built astrometry.net Written in C with python front end.
ACB: need to inject our own reference catalogue.
Fergal: uses USNO-B stored as a K-D tree
Deborah Levine: Nothing to report this week
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Image classes. RHL: proposes moving to mainline as soon as there are some docs.
REO: Let's do it -- but first check with Serge as to his swig changes.
Numerical Recipes & stats libraries.
RHL: what do we need?
REO: what's the overlap with GSL?
ACB: Matrix math?
RHL: from Lapack? ACB: OK.
Also minimisation from minuit, and
e.g. probabilities from GSL, FFTs from fftw. Possibly
Image stats from Pan-STARR's psLib, but it isn't clear
how well this will map to our data structures.
I was assigned #395 (Russ Laher's going to provide test
cases; #400) and put some initial functionality into the
new AFW branches as I needed it to move the Footprint
code over. Russel reports that he has some fast median
code that could go in the same place.
We need a list of functionality from NR that LSST should
provide, and an assesment of what we gain from NR as
opposed to other sources of algorithms. The packaging
into good idiomatic C++ may not be a trivial effort. We
hope that Russ Laher will investigate.
2008/11/05 Guy Fawkes' Day
Present: RHL, REO, Roc, KTL (Lurking), Suzy, NMS, RRA
MIA: UCDavis, Mullally, Bickerton, Deborah/Vince
Progress RHL: Image API #374. eups distrib issues + os/x
Steve/Fergal (in absentia) astrometry.net is built. Learning build system
NMS: Working on getting swig working. Some trivial unit-tests already written
Thinking about cross-talk EA modelling (from two new tickets)
RHL: how long will this take? NMS: not long
RHL: If the cross-talk takes more than 1/2 a day, please check with RHL/Roc
REO: Background subtraction; Naive use of PSF fitting to estimate background
doesn't work.
Paying attention to Exceptions
Suzy/Roc: Possible changes to Applications organisation with TSA/RHL/JeffK
Concerned about DC3 schedule
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pixel Coordinates
PhysicalCcd
For now
Image Statistics
Naive in afw-new-image-api
Background Subtraction
REO: Estimate background as a functional form.
Background is represented as afw::math::Function2
Background estimator is kludge
RHL: Where does the background estimator belong?
RRA: Becker/Lupton
RHL: Rats.
2008/11/12
Present: RHL, ACB, AJC, NMS, RAA, Roc, Jacek, Martin, KTL, REO, TSA, Suzy Template storage: KTL: Hyperatlas? Healpix? HTM? AJC: Why not tangent plane? KTL: Worried about needing large edge overlaps AJC: So a storage issue? Yes. Roc: Is a tangent plane the current default? KTL: Yes; due to TSA KTL: Who's going to decide? RHL: Should we privide firm requirements, and allow you to choose? Roc: We should make a proposal. Martin: Talk about tomorrow at SLAC? Yes. TSA: Two issues: Tesselation and Projection RHL: TSA, AJC, Martin Dubcovsky, KTL will define requirements and make a proposal. Milestones: The question is, where are we for DC3a? RHL cedes control to Suzy...
2008/11/21
No meeting
2008/11/26
RHL, David Burke, Dave Wittman, Roc, NMS, Martin Dubcovsky, KTL, Suzy, ACB, Gregory, AJC Reports RHL: Misc afw/build system stuff NMS: Moving code from C++ to python. Some unit-tests. ACB: Full time diffim. Up on cluster; 3e5 kernels sent to CMU for modelling Two models for spatial variation: 1. Fit for each pixel separately. 2. Use eigen kernels (try 1..n*n) Run all DC2 data on using cluster directly. Dave/Martin: Source/DIASource classes Suzy: Working with Ray Plante on DC3 schedule (Cristina's more-or-less off project) Status of Image API Operational, but may not be fast enough. Possible Serge involvement. Status of Source (DIASource etc.) Defined from DB schema and reverse engineered; we can change this but keep KT/Jacek in the loop RHL: why not use a base class? Gregory: polymorphism or common base class. Can use included member KTL: pieces of data are common to Source and DIASource; could be factored out RHL: how do do merge? Martin: Merge 465 by end of day KTL: Merge then review as per Robyn yesterday Consensus: try that Thoughts on Matrices Naive GSL slow using CBLAS. C++ matrix classes don't seem to exist (but e.g. Blitz) Gregory: how far does gsl permeate code? Should we wrap? RHL: can we just use lapack? RHL: Talk to Steve Meyers at NRAO. ACB: not being held up; am using GSL. Takes 33% longer to fill matrices; RHL: can you generate a simple test program? ACB: certainly Gregory: we have the tools here... PSF spatial variation/Kernel synergies; QA. Meeting at 11am Pacific on 2008/12/1 Calib point of view: Understand dome screen flats in ISR David Burke: Take monochromatic flats: Spatial flat field for ISR Filter band passes Fringe frame inputs? Also white-light screens (quartz) Writing simulation for DC3; how do we use it? NMS: Who provides the illumination correction? From a dithered star pattern? I've implemented something for CFHT. NMS will write up on trac Any Other Business
2008-12-03
RHL, ACB, NMS, RAA, MD, Roc, REO, TSA, Suzy, Ryan, Fergal, Gregory, AJC, Bickerton
Status:
ACB Really basic matrix test code (in svn). GSL: really slow wrt vw::lapack (slow BLAS)
Working on a class to handle spatial modelling of Kernels/PSF, with SDQA thingies. Need ticket number
Will be late on diffim deliverable.
NMS Starting to test ISR code.
Talking with David Burke about calib images on Friday at 1:30 Pacific
RAA Reverse engineering code into EA
RHL Converting measure objects to new Image API, and fixing bugs. New baseclass
MD Baseclass for Sources/DIASource
Roc Interface meeting with camera team requested a list of metadata (everything
except pixels) needed for pipelines. Should be inclusive, not just camera.
Meeting mostly concentrated on interface docs, not interfaces.
Gregory Camera/OCS interface meeting at SLAC raised questions about DM/OCS interface
Meeting notes will be on camera hypernews groups.
Will DM need to pull bits out of the facility DB and put it into some DM DB?
E.g. facility DM don't plan to support more than one set of cross-talk corrections.
RHL Will camera/DM share code?
TSA Camera team can share image display/analysis algorithms, but it isn't clear that
much code will be shared.
Gregory will try to get some end-to-end tests sorted out. He wants to have an
appropriate level of reuse, but not too closely coupled.
REO Working on Kaiser coadd. RHL: when will it sort-of-work? A month to do the
deconvolve step.
Need Exposure class. Need to speed up "WCS Match" (warp) and convolution.
TSA No DC3a responsibilities, but photom calib for DC3b.
Stalled by inability to run DC2 (or DC3) pipelines. Ray says it'll be really soon.
Suzy Updating apps schedule, not back in Ray's court. We'll be getting weekly
pings on status.
We're behind --- need to get things put together by the middle of January.
Ryan Mainly starting to work on pixellization stuff, as per K-T. Check in when svn's back
Fergal Working on astrometry.net package. Code compiles.
Data needs to be distributed in some complicated way (permissions?)
Working on specifying an input WCS.
Ticket?
AJC First 10% of DC3 data. 150 CCDs; 1 pointing. r band. No USNO bright stars.
Please look at it! It'll be at NCSA.
No ISR related data applied or needed; constant gain. No CRs, no bad columns
Will put up an overview on trac. Use tickets to get comments?
Steve Working on Statistics class adding medians/quantiles/clipped statistics.
Do we want to handle the Mask plane?
All needed to provide background estimates --- started by next week.
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Next meeting
TSA RHL ACB Ryan AJC Grillmair
2 or 3 Pacific on Thursday 11th December 2008
