Minutes of the ImageDetWG Weekly 'Phone Con.s
- Minutes of the ImageDetWG Weekly 'Phone Con.s
Minutes for 2009
- 2009-04-15 (Tax Day)
Minutes for 2009
Present: RHL, AJC, Rob Gibson, Martin Dubcovsky, David Wittman, Deborah Levine U. Wahington: REO WCSmatch (i.e. warp). What's next? Finished up Kaiser image (but not deconvolution) Speed up convolution NMS: ISR. Running, not defringing. Done by the end of Jan. Next look at what the calibration group are actually producing. Will need a coding task too (50%?) ACB: Diffim kernels; desired number of kernels Image sims: WCS from sims is good to c. 2" (it comes from boresight + optical model) RHL: use Fergal's stuff? AJC: want it this week. No USNO-B stars, so need our own catalogs -- USNO-B in March? Princeton: RHL Bickerton: Statistics, Interpolation, and Backgrounds. Using pure noise images from Russ Laher (now in afwdata); would like simulated data Mullally: Getting astrometry.net working. Attempting to generate a Wcs (but failing) Next: PSF Davis: Ryan worked on pixellisation library; just a set of projections. Martin: Measurement algorithms on a private branch Definition of "Source" is waiting on RHL review Dave: Two weeks with no email Deborah: Russ is working on persistence for QA classes -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Next meeting: RHL will be in Japan for the next two weeks; Andy Connolly will run meetings.
Attending: nicole, roc, suzy, tim, martin, kt, becker, russell, russ laher, dave wittman, ajc, apologies if I missed anyone. nicole - sick last week. Currently writing unit tests for ISR will update ISR to image api. ajc - we have images from initial sim run and have refit an accurate wcs (so they align without needing the astrom code to be in place). We need to get them tested before the next image run to fix any bugs. Becker - need to warp the images using code formerly known as wcsMatch. Updated version not currently working so Nicole will look at getting an earlier version working. becker - aas last week. currently refactoring spatial modelling, working on swig issues for abstract classes and derived classes. KT offered to help with swig problem russell - struggled with build system, now working, building remapExposure with image api, exceptions etc. Few issues with const iterators (filed bug) but no major issues. build system issues were dependencies that are in the wrong order, not self-consistent (kt - build system is context dependent so it works for some and not for others) russell suggests always starting with a fresh package for any install kt - working on new packages from server, need new eups (timeline for this not clear) ajc - what about using DM packages within external code. martin - do a fresh build and create a new package in svn that is a wrapper around external code, kt has templates for doing this tim - im sim should use dm pipeline as it is beneficial for all (less code duplication and it exercises the build system). ajc - concerned about the overhead of doing this due to the build system headaches but will try this tim - ajc to email lsst-data to point to access to images, ajc - will do as soon as the new wcs's are in place (this week) russ - rhl wants real frame (mask, image, variance) with known statistics to test background estimation. ajc/rhl will talk about how to specify stats of the images to see if this should be assigned to image sim or not. martin - working on ticket 465 (definition of sources), fighting build system, starting to work on deep detection pipeline dave w - asked about the status of the psf module from penn. rhl talking to them but waiting for the build system to settle. Is there an interim psf module - not yet. What is the penn scope for the psf and when is it supposed to start. ajc will ask bhuv and mike jarvis to attend next telecon for update russ - has sdqa python wrappers, wants suggested names for the metrics that people will use. SDQA software is on the trunk and ready to be turned into an lsst package (dms/sdqa). Queston of how to put it into a build (so lsst package install does it). kt - just tag it and file a ticket to make it part of the distribution. Russ on vacation next week suzy - reviewing status of schedule, wants people responding to status requests that are sent out. IPAC working on independent cost analysis. Roc going to functional requirement meeting. Tim - finished up a big update to the uml model which should make it more useful for everyone. Hope to roll it out early february. Version is using uses cases from ipac meeting so will request people to update in a few weeks after the new model is out. ajc - asked can we get a priority list of tasks for DM so that as we finish up individual areas we know which ones should be taken first. Suzy will talk with rhl and roc about that.
Attending: roc, russell, deborah, andy b, ajc, bhuv, martin, robyn, jeff, mike jarvis russell - continuing fighting the build - has a working stack. Sometimes code from a ticket gets checked into the trunk but not the complete code (i.e. not self consistent merge) which then breaks the trunk. Martin ran into this with "svn merge" not svnmerge.py. This might be the problem. David Gehrig looking into this. remapExposure is done - needed to generate the warped images. It will be used for the simulated images but afw is currently broken (wcs broken in the trunk - russell filed a ticket assigned to fergal). Bilinear and lanczos are inplace for remapExposure. andyb - dealing with swig and derived classes, possible bug in swig shared pointer derived directive (serge tracked this down). Working on difference imaging of the simulated images. ajc - initial set of images in place and looking for feedback about changes before generating the full data set (email sent to lsst-data about this) martin - dealing with swig problems with derived classes (inheriting from derived class where derived class was not templated). There is a swig FAQ on the wiki but still requires gaining experience with swig to learn its issues. deborah - russ making progress on sdqa. Wants to touch base with Bhuv about psf requirements roc - assembling requirements for visualization tools for DM (plus camera and telescope). Will send note around about wish list for requirements. robyn - nothing specific for apps. Working on software development plan across LSST. bhuv and mike jarvis - Current PSF work is a module that takes a list of pixels (image) and calculates the centroid and shapelet decomposes a star (up to arbitary order). It allows for the scale size of the decomposition to vary. Requires isolated stars currently. Output is a new centroid, scale and coefficients (10th order typical - 66 numbers). Written in C++ and standalone. Becker - is this the model for photometry and weak lensing? Jarvis - not tested for photometry yet but worth investigating (other groups have tried shapelet code for photometry). In a position to try integration - needs to hook up with rhl to get started. Not on near term work schedule to rewrite for the LSST apis but it might be simple if he wraps the footprint classes
RHL REO Roc Suzy Levine Bickerton Mullally AJC Dubcovsky Wittman 10am Pacific; 1pm Eastern Washington: REO: Warp exposure; swigged but not tested RHL: Speed? Who knows -- using gcc 4.0 ACB: Using simulated images. Verified FITS files, produced MaskedImage inputs Tracked down swig + derived classes with diffim. --- running on sims Waiting for released daf_data for API conversion NMS: Out for the last fortnight. ISR is checked in. AJC: Delivered sims. Initial WCS fixup. Want feedback for production run within a week. DGM: stars too round Initial WCS on boresight (10"). Fitting polynomial WCS to chips Sims available from UW; will be from NCSA. AJC is point of contact Princeton: RHL: Japan, Speed, CR is fast enough. Looking for things to. Bickerton: Statistics/Interpolation/Background all in afw, even doxygen Problems with installed scons due to envAppend. Fixed sconsUtils to use envPrepend. Pulls system python into path Getting randoms etc. into afw. Need to control seeds. -- low priority Mullally: Astrometry.net to build WCS from scratch. In meas/astrom/net Need to get astrometry.net in distrib system Building on ubuntu system -- problems. IPAC: Levine: Mostly on PDR related stuff. Not much to report Roc: Meeting with Jeff Kantor + SysML Davis: Dubcovsky: Definition of DIASource and Source (#465) -- inherit from baseclass. Scared of merge REO: We already had DIASource? No, it was called "Source" but no deep Source Working on detection pipeline. Read sets of images, ..., back to measure Centroids? Shapes? No, not yet. REO: Working middleware? Martin: I don't yet know -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- DC3 Schedule Deadline April 1st --- all testing done (but report not written) Week of 9--13 March: Integration week @ IPAC -- maybe 10-12 people. All code checked in/tested by 1st March ? Do we need to move apps from DC3a to DC3b ? Speed Diffim. Convolution REO Matrices esp. eigensystems Fergal & Toby. Inform ACB gcc 4.3: Washington: AJC will investigate Princeton: RHL will investigate Davis: Doesn't care about speed for DC3. WCS AJC: What WCS is supported? Per-chip tangent plane not good enough (c. 1pixel) 2nd and 3rd order polynomials get to 0.1pixel Haven't looked at colour dependent terms in astrometry We're using wcslib 4.2 internally; AJC will check. RHL will talk to Axelrod on generating good WCS from linear Mullally will investigate building indices from AJC's catalogs Mullally: need 20ish good sources per field Matrices boost::ublas? gsl + good BLAS? ACB would be happy with GSL. PSF Bickerton + ACB AJC: sims good enough for this.
February 4, 2009 1:00 pm : Applications phone con Dave Wittman, Martin, RHL, Suzy Dodd, REO, Roc, ACB, Deborah Levine Princeton: RHL: afw updates. meas_algorithms/meas_astrom with afw 3.3 Working on Centroid code (using Naive and SDSS; run-time configurable) Fergal: On mountain. Working on BLAS. WCS: Use SIP? Roc: can help; it's in the FITS standard Roc will look into libraries. SIP is the standard to beat. AJC: Need forward and reverse for DS9 Steve: Thinking about PSFs; pry away from SDSS Davis: Martin: Playing with Hello World in pipelines. trac:PipelinesFramework Using latest version of all but pex_harness Source/DiaSource on trunk -- ugly to write; OK to use. Will be in afw 3.3.1 Post-DC3a review: Design for DB or Applications? IPAC: Deborah: PDR stuff; SDQA requirements Russ Laher: Suzy: Will everyone please look at the DC3a schedule; http://dev.lsstcorp.org/trac/attachment/wiki/DC3Schedule/LSST-DC3-udated-020209.pdf U. Washington: ACB: Working on MaskedImages Diffim works on simulated images. Better than for real data... RHL: scary.. AJC: Why ? RHL: Because I thought we had most of the bad things in. AJC: Two DC3a runs. No USNO-B stars. Time to start thinking about next round (e.g. asteroids) Roc: can we do sims to move MOPS off their sims? RHL: catalogue is good enough, but detection efficiency? REO: Working on warp exposure. Nearly there... NMS AJC reports: was in yesterday; will be working at some level. Will get together a to-do list this week. RHL: We need lists of defects. AJC: should keep in DC3s -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Roc: There's a meeting in Davis next week to discuss DM requirements and scope RHL feels conflicted: Science Council v. member of DM team --- are the requirements ambitious enough? REO: when are the boost 1.37 packages due? RHL: no idea Suzy: Schedule is very daunting --- code has to be tested and working by March 9th.
Present: RHL, REO, Russ Laher, Martin Dubcovsky, ACB, AJC, Suzy, RHL: Report on Davis meeting. Working on measuring centroids/shapes. Fixed some WCS stuff. REO: ISR -> new APIs; went quickly, Back to Nicole for now. Working on warp -- need to grow masks by 1 pixel. RHL: speed? REO: still on g++ 4.0 ACB: Converting to new Image API, using new classes (e.g. Footprint Functor). Tracking down bugs; confused by Footprint code. Working on gcc43 in eups. Suzy will talk to NCSA and see if they can help with getting gcc43 versions of the framework deployed. Note that they'll need new version numbers, so we can mix-and-match the two sets of packages (this is caused by the boost library's soname conventions) AJC: Sims are going. Data out at the end of the week. Martin: Playing with pex_harness stuff. A long way towards being usable; see #582. KTL has parallelization concerns. Moving towards, "Compute coadd" stage. Russ: Distracted for Last week or two with Palomar Transient Factory, but now back (and other project is helpful). Implemented LSST SDQA schema in PTF database. Started using Java tools from Spitzer project in PTF. No problems (yet) in LSST schema --- implemented two schema, both LSST and a simple table to compare usefulness. In LSST: have tagged SDQA package, with a few rudimentary classes. Developers should be able to put SDQA objects onto the clipboard now. Looking forward, need bare-bones SDQA pipeline --- takes data of clipboards; thresholds, formats for DB. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- =-=-=-=- DC3 discussion of delay from IPAC to NCSA cluster, can be up to 2s. IPAC network group is investigating. Dave Gehrig is looking at the NCSA end. Suzy hopes to have an answer by late Thursday. RHL: 2s is a show-stopper. Factories. I'm beginning to implement creation of e.g. Centroid algorithms via a concrete factory, createCentroid Status: Diffim: in a week and a bit ISR: Unknown WCS: SIP. No number on performance PSF: Almost all pieces written 2nd moments Will be there
RHL, David Wittman, Russell, KTL, Deborah, Vince, Suzy, Martin KTL: Bottomlevel middleware is stable (but not yet a metapackage) Harness etc. is also OK. Ctrl/orca needs a provenance fix --- maybe one more top-level Francesco: Night MOPS Serge: AP with faked up DIA sources Subtraction/Sims both running ISR -- flat fields, biases, linearity, defect removal WCS/PSF on one exposure Difference both exposures wrt template Add differences and detect. Measure on each exposure Persist DIASources Persist SDQA ratings Timings: diffim 80s per amp single core (was 3mins) 100s for coadd-and-detect 30s dia-source measurement Some dipole residuals --- are we ignoring SIP corrections? Warping is slower that swarp by... Analysis: Diffim ACB ISR AJC PSFs RHL SDQA Verify that it's persisted Diffim photometry (only 1 real source in first CCD?) Classification: Bickerton WCS: Fergal Prepare for DC3a debrief: What algorithms should we be using? Is the QA good enough? What more do we need? What do we need in sims for next generation? Look at CFHT and ask if we need it in Sims.
Lee Armus, Suzy, RHL, Roc, Vince, AJC RHL: Worried that e.g. we weren't handling -ve detections. I fixed all the ones I found, but they were pretty low level. IPAC: QA's doing what we asked it to do RHL: Did you look at Sims? AJC: No. RHL: Need stable arrangement of disk layout: Roc yes. RHL: Are we using the SIP distortions? Does anyone know? I'll ask Fergal to look. Suzy: Postmortem meeting. Please send comments to Suzy. Roc: Please start thinking about TCT recommendations. 1/ How to encode a NULL in a row in the DB 2/ Can column names include units? AJC: What's the best machine at NCSA? RHL: I use lsst8
Present: REO, Roc, Deborah, RHL, AJC, ACB, Martin, Steve Bickerton, Fergal RHL: PSF Steve Bickerton: SDSS-III. Was working on aperture fluxes with tapered circular apertures. Needs c. 1 week more work. RHL: A standard aperture is 1 iff r < a, but that's an arbitrary choice. In SDSS we calculated the exact weights for an arbitrary aperture (assuming band-limited data), but that required us to sinc shift the data to be centered in a pixel. If you smooth the top-hat filter with a band-limiting function (e.g. the PSF) you can instead shift the coefficients. There's a discussion of the SDSS apertures at http://www.astro.princeton.edu/~rhl/photomisc/aperture.pdf Fergal: The problem appears to be in the astrometry.net SIP solutions. We tried backing off from 3rd order to 2nd order and things improve. This afternoon we'll try using more stars for the SIP than the initial stages; if this doesn't work, RHL thinks that we'll have to run an object detector on the template to have enough stars to do a good distortion map. REO: Kaiser coadds, and fighting badly factored e.g. sourceDetection AJC: CMU (Chris Genevieve) is looking at the bright-source case. It doesn't seem to be much better than Kaiser's faint-source limit. They are doing the deconvolution step. Have something in 2-3 weeks. Will discuss on the 6th of May. REO: what are Pan-STARRS doing? AJC: an expedient naive coadd (according to Paul Price) RHL: can use for detection anyway. REO/AJC: we knew that. ACB: Working with REO/AJC on PSFs for coadds. Need to work with RHL. Roc: The latest TCT measure on variable names: Naming conventions --- names should not contain units (e.g. Mag) Motivation: define units in metadata. Roc: pushback is probably warranted Please review proposal (URL) and let's discuss next week How do we represent bad values? External representations to be loaded should be explicit. E.g. WISE prints out file with NULL in the column (but how does the C work?) Vince was working with Robyn on UML model for SDQA. Martin: Timing on diffims? I was looking at the code, and wondered why QR not SVD? ACB: The matrix maths is sub-dominant to the convolution code (c. 10%) RHL: Please add metadata as to what method you used. #-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Tim passed two questions on from Jacek: 1. He is carrying space for four subtraction templates for every field and filter. This is to allow for templates to differ based on seeing and airmass. ACB: We need the airmass more than the seeing. Can test the seeing with different CFHT templates. RHL: Speed penalties for a large dynamic range in the input seeing, as we need a larger kernel ACB: Need large kernel anyway for bad seeing (as core/wing vary separately) RHL: I don't see a way to be clever with different airmasses as red-blue pairs separate. 2. Does the detection coadd need to be per-filter or not? AJC: Are we going to only detect on a chi^2 coadd? RHL: This depends on the detection prior. AJC: chi^2 optimal for sources close to colour of the sky (or noise or the sky?) Roc: we do a multi-band detection in WISE by detecting to a fainter limit (e.g. 4.5sigma not 5sigma) in a coadd of some sort. Simulations show that this works. RHL: So we don't really know the answer to either of Tim's questions. We'll continue to think about getting him an answer. 3. AJC: SEND ME THINGS IN THE NEXT GEN. SIMS BEFORE NEXT WEEK'S SIMS MEETING!
2009-04-15 (Tax Day)
RHL, Martin, Gregory, Suzy, Craig Loomis, ACB, Fergal, Russ Laher, Roc ACB: Nothing much. Preparing DC3a PSFs for CMU RHL: Mostly SDSS-III; working on WCS with Fergal, which brings us to: Fergal: Bad WCS is causing bad subtractions. Distortion terms from templates are different from those in the science images. We're trying using catalogues derived from the templates to define our science-frame WCSs. This requires us to generate new index files for astrometry.net --- he's got Dustin working on it Roc: should we be solving for the distortions on every exposure? RHL/ACB: probably not in the long run Suzy: Is this convergent? RHL: I think so. If needs be, we'll write our own polynomial fitter to fit catalogues from the template images to the science images. Martin: Looking at speedup on diffim (failed). Coming up to speed on mathematics of multifit Russ: Yesterday I got back on the SdqaRating formatter (-> DB; needs to work on other formatters) Suzy: Sent out first draft on application writing assignments --- now's the time to volunteer for more sections. Please write apps. monthly report! Breakout sessions at the DC3[ab] meeting. E.g. MOPS; what else do we want? SDQA, ... RHL: Please look at http://dev.lsstcorp.org/trac/wiki/DC3APostMortemMeeting page and edit by the close of work Sunday. Then prepare list of breakouts for the PM meeting on 2009-04-23 RHL: Did anyone read the TCT on code standard (deafening silence) This will be discussed next week, so please read http://dev.lsstcorp.org/trac/wiki/CcbMeeting20090401
Minutes were written and lost. Mea culpissima
Present: RHL, Suzy, Lee Armus, Martin, Deborah RHL: Fixing DC3a problems and recovering from a broken laptop. Is anyone looking at DC3a outputs? Suzy: I'm holding off getting IPAC people involved until the new astrometric runs are done Suzy: do you have DC3a goals? RHL: We could do Photometric calibration (at least on CFHT D2, D3 where I have SDSS catalogues) Split-exposure processing; probably worth a factor of 2 in speed, but tricky to test scientifically as the current "exposures" are identical + fake CRs Whether to implement this for DC3a is probably a political issue for Tim and Jeff IPAC: Suzy: working with David Gehrig on DC3a post-mortem agenda + breakouts taking with Jonathan Meyers and Robert Jedicke re MOPS Deborah: Working on cost estimation by end of month Martin: Talking with Jim Boesch about multifit architecture proposal; try to get something out soon
RHL, David Wittman, Fergal, Deborah, Roc, Russ Laher, Martin, Russell, Robyn, AJC, Steve Bickerton, Suzy RHL: Speedup on Kernel estimation/convolution speedup AJC: Would FFTs help? Fergal: Working on WCS. Bug until last night in new index file generation Looking into current problems (namely that subtractions look v. poor) REO: Does this explain #801 (BBox outside template)? Fergal: didn't see any catastrophic errors Bickerton: I'm back on LSST as of today Suzy: I'm totally behind on DC3 breakout agenda. We can't cram this into two days; prioritize things that we need to discuss face-to-face. Deborah: Working on costing Russ: Nothing to report Roc: At DES director's review. In pretty good shape for spring 2011, using lots of existing software. AJC: Science groups are doing all of basic code (e.g. SNe are doing diffim) Russell: Talked to Simon Krughoff about ISR. Working on #801 Martin: Background subtraction: are we persisting (ask KT?) WCS QA? (RHL: not much --- we need more). -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Template Generation: AJC: With Chris Genevieve and Darren Homrighausen at CMU. Trying to do better than Kaiser. Implemented variation of Kaiser, comparing with straight average (no PSF convolution) --- preliminary analysis says no improvement. Will implement within LSST code; have built under Linux. RHL: What's the best way to prepare for DC3 meeting? We propose to hold the following ordered set of meetings: Tucson May Phone Aspen June Phone Davis August/September (if we need to; subject to funding) to get our acts together for DC3b
Present: RHL, Suzy, Tim, David Wittman, Russell, Steve Bickerton, Lee, AJC, Fergal, Simon, Jim Bosch Agenda: Status and confessions: No one spoke up Ticket #835 (Wcs) -- Merged to mainline; required hack to class and also the formatter. Fergal's adding a unittest Will change Wcs to only pass "proper" keywords to wcslib -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Status of DC3a runs (is rlp1234 good?); if this is covered in the DC3 phone con we can skip it Tim: no gross problem in D3. Will D4 too. No difference between 1233 and 1234; am waiting on IPAC (maybe faxe-to-face on monday). Offsets c. 20mas RHL: I should have saved the catalogues I created from the templates to generate the astrometry.net indexes What's Becker say about diffim? Tim: I don't think it's working well Becker thinks that deconvolution is a problem, but why is there any deconv.? RHL: Deconv kernels look smooth but convolution looks noisy; this is weird; Steve Bickerton will be visiting Seattle to work on this in early August AJC: Same problems with simulated data; used sky-free images as templates Work left over from DC3a. In particular, Astrometry: Is it good enough? Reliable enough? Do we need a new game-plan for determining the SIP distortions? RHL: need 3-5 mas for week-lensing Tim/RHL: let's write our own SIP fitter. Minuit? Minuit2? Matrix inversion? (N.b. minuit2 has problems on Linux --- assigned to ACB) QA: Are the proper metrics being generated? Astrometry: Tim: No. Fergal: Would need to hack the astrom.net code RHL: If we wrote our own SIP fitter that'd do the QA PSF: RHL: Probably not. I did a bad job Diffim: Nothing in for DC3a is useful, with possible exception of kernel sums Summary: We need to test QA metrics as unit tests. RHL'll check with Robyn about adding to QA testing PSF/multifit meeting plans 11 --13 August Resolution of #806 (should we set edges of convolved images to NaN): http://dev.lsstcorp.org/trac/ticket/806 Russell: Pixels that we can't set should be NaN (as well as a mask bit, variance == Inf). Fine for warping. We also need to handle pure Images (no masks) For convolution we could do this or copy the input pixels. RHL: it's useful to keep the raw pixels around the edges, e.g. so you can see them ACB in absentia: If kernel offsets the image, then the edge pixels are not well defined. Steve: Statistics operators (probably) ignore NaNs. Masks are not (yet) supported RHL: I'll probably cave in on this. Post meeting: I thought about external, non-LSST, users of our stack, and I don't see how we can force NaNs on them. So I, as afw-coordinator, have decided to: 1/ Make NaN the LSST default 2/ Provide an option to p How should we handle sub-image coordinates? See Russell's proposal at http://dev.lsstcorp.org/trac/wiki/TCT_PixelCoordinates RHL: Did anyone read this? No. Let's do this by email. Russell: Is this TCT? RHL: I'll check with Robyn (answer: we don't need to pass it by the TCT) Jim B: What's a Parent image? Will annotate document (He's done now so) Suzy: What do we need to do in preparation for 27/28 meeting? Do we need an applications proposal prior to the meeting? Tim: Yes. From the IPAC meeting minutes. Suzy: I'll make a summary Next week: I'll run meeting from APO Main topic: DC3b planning
Present: RHL, Roc, Suzy, Jum, David W, Martin, KT, Simon, AJC, ACB, PzW, Deborah -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Suzy: Estimates should just be coding/meeting/test writing time -- not debugging Look at http://dev.lsstcorp.org/trac/wiki/DC3bScopingMeetingIII Links on this page should be updated (although they refer back to the May meeting); Suzy has added lists at the bottom of the page. We want milestones at least once a month (clocktime); if delivering/receiving something that should be a milestone too. KT: Jeff would like tasks in terms of durations, rather than due dates; Suzy OK; durations are fine. Suzy: I did this for MOPS and SDQA, everything else is under "Science/Algorithms" as far as I know it's complete, but please check and edit the page: http://dev.lsstcorp.org/trac/wiki/DC3APostMortemMeetingScienceAppsI You should add your work estimates to this page -- with the correct names. Please call out dependencies. Due date: today (29th of Sept)! We'll discuss it on the 5th August, so please try to get it done ASAP.
Roc, RHL, David Wittman, REO, KTL -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Work for DC3b Roc: Just do what needs to be done; nothing more. What are the key things? Can we Triage? We looked at Suzy's plan and the following thoughts came up: Don't generate fringe frames --- this is thought to be an error in the plan Is AP in apps? Need deep coadds for first pass to multifit. Coadds need to be defined better REO: No Kaiser coadds, just PSF matched using diffim kernels REO/ACB: need more info about PSF fits. Wants to fit bright and faint stars separately for QA WE need these for diagnostics on coadds Sky pixellation: REO: generate in tan projection? KTL: no, please generate coadds in Ryan's space-filling-curve grid (all: scary for DC3b) RHL: is this scope creep? DW: yes KTL: for image access this is roughly the same --- I just want one pixel per point on the sky Roc: agree with RHL Where does HEALpix fit in? Or should we go with Overlapping tan projections, with overlap width ~ 1 amp/CCD. We'll decide for DC3b next Wednesday; RHL will send mail to list asking for input Who is going to choose inputs for coadds/define tan planes. RHL: TSA! PSF photometry: It's basic, and we need this for S/G separation. Bickerton has most of the aperture code written Shapes/Multifit: Shear + galaxy photometry? UCDavis thinks they'll be OK if their dependencies are available Roc: Suzy and I can work on dependencies. She's (probably) back at the end of the week. Roc'll start now. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Agenda for PSF/Multifit. RHL: Please send agenda items
Present: Roc, KT, RHL, Suzy, Simon, Steve Bickerton, Deborah, PzW, Martin, Fergal Simon: ACB (in absentia): diffim. Some progress, still noisy kernels. On vacation Simon. What needs to be done for ISR? Fringe frame corrections --- is imSim going to generate fringes? Metadata to describe the camera (biassec etc.) -- also want a class REO: Implemented normalising linear combination kernels (just waiting for a review) Problems with variance in linear combination kernel RHL: SDSS3 + prep for PSF/Lensing meeting Steve: Nothing. Back to LSST on Monday at latest Fergal: Extract internals from Astrometry.net for QA. Preparing false inputs It'd be nice to have external images with bad distortions Martin: Back from vacation; basic models in multifit Deborah: Waiting on DC3b quality goals --- i.e. waiting on Tim Suzy: DC3b plan. Pixellation KT: there are space-filling code packages out of Russia and ??? REO: for warping all I need is a pixel --> sky and a scale RHL: HEALPIX it is for DC3b. RHL: get this into stack Simon: will HEALPIX need more than 64 bits to get to the needed resolution of ~0.1"? Suzy DC3b. Preconditions and when we need what List of tickets Everyone should triage tickets and add DC3b ones to task list, maybe as "fix my 10 tickets" Please move code out of middleware (i.e. stages) and into apps
Present: RHL, Martin, Suzy, Russell, Deborah, Dave Wittman, Gregory, AJC, Jon Meyers, Lee Armus, Roc, Steve, ACB, Jim, Simon RHL: SDSS. BOSS red CCDs (from LBL) have, umh, interesting CRs Suzy: Sent out request on Monday for comments on DC3b schedule --- still need RHL, Simon, Tim, Jon Meyers. Wanted by Thursday, please. Also working on Dave Silva's operations plan N.b. Science synergy between LSST and PTF (Palomar Transient Factory) meeting tomorrow Tim sent out DC3b SDQA requirements -- comments by Tuesday, please. Martin: Waiting on RHL's PSF API comments. Designing Fourier classes, trying to base on afw classes. Want to replace gil::image Will write a ticket PSF: belongs to Martin. Some debate between Russell and Jarvis as to where convolve goes RHL will investigate by email and resolve next meeting if not before. Jim: would like to put ellipse code into afw. Ticket please. Fergal Working on SIP code. Using SourceMatch class to match up list of Sources, and rematching as the WCS changes. "The QA is going to be absolutely great". A fortnight. Steve: Working with ACB on diffims. RHL: do you need support? No, we're OK. No --- the installed Python dumps core RHL: use the system python and lie to eups ACB: Unhappiness about os/x installs. RHL will find out NCSA's position. Gregory will talk to Jeff etc. if needs be Russell: Trying to stop being sick. Little bits and pieces Simon: Writing a proposal to describe the camera. Based on Jim Pisagno's imageSim team description of the camera. Will provide a class to describe the camera Gregory: This is a system engineering group issue. Please provide a proposal
Present: RHL, REO, Gregory, Martin, AJC, Becker-Bickerton, Deborah, Fergal, Suzy, Dave Wittman State of DIA: Sent out emails before phone con. Regularisation constrains delta function kernels to look like Alard-Lupton RHL: so the variance of the diffim goes up? ACB: Mean of diffim within the footprints is smaller with a delta-function kernel AJC: The deconvolution used to look better; does it still? ACB: I don't know; needs spatial modelling. Propose switching to new spatial models. Coadds for diffim. TSA didn't think that he owns it. REO: I will write the code. Someone has to decide how to choose input images; we gave this to TSA. RHL will check with TSA Glocal photometry, "Ubercal". Suzy: I thought it was out of scope, but Tim wants to do it. Gregory: Tim really wants to do this. AJC: Can't you do this from calSim? Gregory: Maybe; or maybe a hybrid. TSA, Gregory, and AJC will discuss. TSA's thought to be worried about subtle spatial variations Photometry didn't make it into image characterisation, but should be added; good to 1%. Also need to spec the input catalogues --- AJC will ask Lynne Jones. Measuring the DN is in scope --- probably 1% but check with TSA. Fergal will stagify Dave Monet code Stellar pops working group are worried about multifit; RHL will talk to Abi Saha Gregory: What are the requirements for scientific use of DC3b? What do we need to do to get science working group buy-in? Move this to SAT. RHL: Are PSF APIs settled? Martin: Negotiating with KT. Other stsake-holders are happy. RHL: I'm concerned about factory functions RHL: Multifit's not using afw's Image classes for DC3b as RHL/KT want images to always own their data. Jim/Martin: Want sign off on ellipses REO: normalisation of linear combination kernels is in your inbox. RHL: give it to someone else. REO: When should we cut afw? RHL: I'll send out a call for inclusion and cut by the end of the week. Don't need to wait for ellipses
Present: RHL, TSA, KTL, Roc, Fergal, REO, Gregory, Suzy, Russ Laher, ACB, Simon, Martin, David Wittman, Jim, Jacek No-one from Penn; RHL will ping. Interesting things. UW: ACB: difference code speedups, probably SSE due to Eigen Implement Alard-Lupton kernels. No spatial modelling yet. Regularise delta function basis functions. Lessons learned: Either Alard/Lupton or regularised Alard/Lupton can be tuned. We don't yet know how to do the tuning. Steve: can regularise lots of ways; e.g. 1st, 2nd, 3rd derivatives. Must worry about boundary conditions. Some are better than others, e.g. diagonal banding in Kernels --- maybe problems in cross-term determination. RHL: please get the spatial stuff working. All code's on #876 Bickerton: get AL working, then fiddle. Simon: Nothing. Get camera description out today. RHL: are you responsible for ISR? RHL: please sign up for trac categories RHL: Put Subaru data through LSST python Fergal: SIP distortion code. Using afw::SourceMatch; and LSQ fitter. Untested. Will chat with Dustin. Steve: Cleaning out all numerical recipes code from the stack... Gave up on GSL integrators as they don't understand functors; they use some C hack (struct with a function pointer and void *data) Gregory: could you post the details to lsst-data? Dustin: Hello. UCDavis. New PSF API proposal We think it's nice, and people will like it. Please read. Meeting Thursday/Friday to sign off if we can't get signoff by email. IPAC: Suzy: Last call today for inputs on DC3b schedule. MONTHLY REPORTS are due Russ Laher: Trying to ramp back up on LSST and escape PTF. We want to do 2.1, WCS errors. RHL: Please work with Fergal --- Tim's the point man for defining what we want. Roc: Started to listen in on MOPS. They've made progress in getting day-mops into framework, with C, openOrb, python. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- GPUs. Meeting with folks at Split, Croatia. Still need to identify individuals in apps. Will conduct a poll on lsst-data. TSA: I have experience! Mario recommends CUDA at this point. Failures. Please ensure that all your products are passing tests. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- KT: Wants input on inputs... What are the pipeline/stage inputs; do they come from a database? E.g. how are the astrometric source lists split up? How are their sky tiles defined? Do we need things that need to be inherently updated in place in a DB? RHL: how is calibration done? KT: astrometric calib adds columns. TSA: photometric calib could work the same way, adding calibrated magnitudes. KT: I don't hear any need for live access to the DB during pipeline operations. TSA: One exception in LSST (not DC3b) is alert production. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Can Roc run meeting next week? Yes.
Present: Gregory, RHL, ACB, Mike Jarvis, Bickerton, Fergal, Jacek, REO, Russ Laher, KT, Meyers, Martin, Wittman, Wozniak, Simon, AJC -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- PSF API Martin: Kernel is changed. #905 Mike Jarvis: I can work with what what we have. Martin: Have we had trouble with std::complex and swig? [Silence] Will post example. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- AFW Outstanding changes are: #905 Change Kernel API #955 Removal of NR (1 hour) #956 DetectionSet -> FootprintSet RHL will cut today or tomorrow. N.b. will break other packages; owners should expect to fix or face the wrath of the BuildBot. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Coadds: Russell: No outlier rejection. TSA: How do we handle moving objects? RHL: Just use masks from diffims? TSA: What about variables? KTL: We don't have diffims before we build templates RHL: Ah. RHL: I'm concerned about the cores of bright stars, but PSF matching helps. RHL: The course of least resistance is to do outlier rejection. KT: Have to worry about memory issues. RHL: How many pixels do we lose at the edge of the CCD? 23 pixels. REO: Will add this to the statement of work. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- State of diffim. ACB: Got spatial stuff up, but tested. No numbers yet. RHL: state of ip_diffim tests? Wozniak: Could compare with standard A&L code. RHL: Please make this happen -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Focal plane description. RHL: Document, not code, from Simon? Simon: Yes. Need more comments (got them from KT and TSA). RHL: on disk? Simon: from FITS RHL: No; I think from config file KTL/TSA: Split per-exposure from configuration from static camera description KT: Probably don't want to use Andy Rasmussen's code as is for now. RHL: Do we need CCD heights etc. for physical models of PSF? Mike: We don't currently use this AJC: Should feed back to Andy Rasmussen RHL: Let's concentrate on focal plane geometry. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- WCS: Wcs should use coordinates relative to the origin of the CCD TSA: Would be nice to do it now... Fergal: OK, OK, I'll do it. Please send email traffic (REO will do) -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Source for DC3 Jacek: Schema for big tables for Source and FaintSource. My FaintSource schema is 4 columns (id, exposure id, flux, sky) TSA: FaintSource and forced photom are 1-to-1. Also Source 5-sigma full-up measuring for at least DC3b Let's take 10am Pacific on Friday 2009-10-9 to further discuss. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Suzy: Can you remind folks to take a look at this Apps schedule and send comments to me? I have already received comments from Russell and Andy B. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- The TCT decided to use radians internally and in APIs. Externally (files, DBs) use degrees Some dissent, which should be aired to the list. Basically, things like ra, degrees, angular tolerances are not [usually] passed to trig functions, so it's much easier to say, "Match within 5arcsec" than "Match within 2.42e-5 radians"
Present: Jon Meyers, David Wittman, RHL, TSA, KTL, Suzy Dodds, Fergal, Deborah, Martin, ACB, Roc, Russ Laher, REO, Steve Bickerton, AJC
RHL: Talking with Monet. RHL will get his centroider into the stack, and then Monet expects to query DB for stars. TSA: Need to get output back into LSST stack. RHL: Will talk to Monet about integrating his analysis code. Fergal: One function away from SIP from list of points --- then debugging. Need to be able to calculate forward matrices (A and B); need Ap and Bp maybe stealable from Dustin. Technical issue: SourceMatch is a std::tr1::tuple and Serge wrote custom typemaps in one direction. Bickerton: Hand-coded (perl) parasoft-like checker. Suzy: state of stellar photometry? Got the aperture mags more-or-less right. Now to see if everything works. RHL: How do we go about running photometry on e.g. all the exposures of some CCD TSA: Install orca etc. and run DC3a framework. TSA has blog/ notes that he could post. UC Davis: Suzy: Multifit looked like 3 serial tasks, but you're part way through 3. Martin: Yes, that's right. Part of this is iterative design Martin: porting non-LSST code into LSST (first an ellipse library, then some models). Then multifit-fitter. Jim: I'm a bit more optimistic. UW: ACB: ip_diffim unit tests and doxy. RHL: What's the chi^2 of the difference image? ACB: We're not there yet. Working with Wozniak Simon [in absentia]: Internal review --- he's a juror this week. No thoughts yet on disk format. Policy files are acceptable AJC: What do we want from design review? (Suzy'll try to clarify) AJC: Starting on 2e4 deg^2 of stellar catalogue data. Ray tracing coming along; new atmosphere, Zeljko's cloud model. Working with Split group on GPUs --- images in a fortnight. TSA: Is Zeljko's cloud model too good --- I'm worried. RHL: Is the seeing bad enough? DGM's says it's too good, so let's check as soon as images are available. REO: I've gotten chi^2 coadds running. I'm not convinced that things make sense. KTL: did you see my email? REO: did you get my reply? The question is, on the psf-matched coadd how do we get a uniform PSF everywhere. RHL: so you don't match to some defined PSF? AJC: I haven't done that; it would be a bit tricky as you'd need isolated stars. We could generate overlapping images that provide at least one amp- sized patch. RHL: What's the scale of Balkanisation? REO: We can always use a single CCD as a reference image. IPAC: Russ: Working on atpy package, writing unit tests. The package makes it possible to convert data from e.g. mysql and put it in binary fits. Russ: I'd like to work with KT/Serge to get USNO-B1 installed on lsst10 RHL: AJC: does your 2e4 deg^2 catalogue include USNO-B1? AJC: No, but we could. Do you want them? RHL: Do we want to use the same catalogues for sims and CFHT data? -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- =-=-=-=- Suzy: Code reviews for Camera Classes --- Simon's on Jury duty. Use Apps phone con on Wednesday. 30 minutes presentation, then 15-25 min discussion. Probably in a fortnight. Suzy: KTL's skeleton production. KT's preparing a stage listing linked off http://dev.lsstcorp.org/trac/wiki/DC3bProcessingFlow. I'm about 1/2 way thorugh; when I'm done individual pipeline owners will have to pay close attention. Check on developer response next week.
Present: REO, RHL, TSA, Martin, Suzy, Russ Laher, ACB, David Wittman, Lee Armus, AJC, Simon Russell: Reading on PSF QA? Needed for coadds RHL: what exactly do you need. State of Healpix (in stack?) RHL will check on status with KTL Answer: Andy Connolly is looking into desired resolution -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- The main topic is a review of Simon's camera geometry proposal at http://dev.lsstcorp.org/trac/wiki/CCB_CCDCoordSystem -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- I'd also like to discuss moving to the new middleware stage code as there seems to have been a misunderstanding between middleware and applications (or at least me). The links are http://dev.lsstcorp.org/trac/wiki/CreateStageImplementation svn+ssh://svn.lsstcorp.org/DMS/pex/harness/tickets/924/examples http://dev.lsstcorp.org/trac/wiki/DC3bProcessingStages -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- (RHL: I didn't do a very good job of capturing this discussion. Sorry) CurvatureModel needs to be default constructable and be built using a factory with a name specified in a policy file. Use opaque types instead of e.g. std::pair<int, int> to index the rafts. Come back to WCS (add boresight, optical model, ...) and Camera::_origin in a future discussion. Camera is constructed from a policy file string name -> string const& name Define getShape in more detail --- is this a free function or parallel hierarchy
TSA, Roc, RHL, Deborah, Ray, Martin, David Wittman, Simon, ACB, KTL, Suzy, Russ Laher, REO, Jim, Steve, Fergal Stages: KT: Almost every stage for apps is designed to run in parallel (multiple copies of stage running at once) Expect only DB access stages to be serial -- e.g. global astrometry. Stages code should be small, although may have much apps code (KT hopes not). 2 pieces of code in addition to apps code: Stage Default dictionary Plumbing (e.g. image keys) without defaults Parameters with defaults Default parameters can be installed with package --- developers please use this KT added adaptors to hide the clipboard from the users. RHL: Why is a clipboard not a dictionary? Ray: because data items can be shared, but it always looks like a dictionary to a user. Martin: could use Adaptors to validate clipboards. Ray: yes, we're trying to do this Martin: I don't like magically mapping fooKey -> foo (All: this is a good idea but a detail) Where do these files go: RHL: where do these stage/detection files go: KT: meas/pipeline, ip/pipeline RHL task to create/reorganise meas/pipeline/policy Policies meas/pipeline/python/lsst/meas/pipeline Stage code Ray/RHL: That's OK. Martin: KT, how far does the skeleton pipeline run? KT: in datarel Towards an LSST Manual: We only have doxygen-level micro comments. TSA: There's lots of good stuff on trac, but it's hard to find. Steve: Manual level should ship with the code. Ray: Start an outline, and put in links when available. RHL: Can this be done with volunteer labour? Ray: My initial outline can be done that way; filling gaps is harder. RHL: Need to hire someone as we ramp up TSA: Maybe 18months from now... Other code reviews: Suzy: Design reviews for key areas, e.g.: Diffim Multifit Interslice Is this a good idea? What areas do we want to review? How about diffim? TSA: Yes. Algorithmic ACB: I'm not ready --- need to run some data PSFs & Models Multifit How did camera classes go? Well. RHL: when will there be an implementation? Simon: ask me next week.
Present: RHL, Russ Laher, David Gehrig, Martin, John Meyers, Schuyler, ACB, David Wittman, KTL, Simon, Suzy, ACB, Deborah, REO, Steve Bickerton, Fergal Agenda David proposed switching this phone con with the DCn phone con on Tuesdays at 11 PST. The basic idea is that there's a major overlap, and at least sometimes DCn generates a question to be answered at Apps. KT: Apps discussions affect DCn more often than vice-versa. Resolution: don't change things now, but if the opportunity comes up let's move one of the phone cons by a couple of days. What can we get running by Jan 4th? Status: Princeton: Steve: worked on a few tickets; done some SDSS work. Was observing last week. RHL: state of photometry codes? Steve: pretty good; aperture mags no longer need a Psf. RHL: Plan for testing, and will it be ready for January? Steve: no SEGVs. Need to test if flux measurements are accurate on simulations RHL: plans for scientific testing? Steve: what do you want? RHL: meet the SRD Suzy: action to figure out where we'll be at the end of the month David: Need to get some sort of CPU estimate by 2010-01-15 KTL: don't need good science, just expensive science RHL: do you owe any stage code? Steve: No RHL: Misc loose ends for new afw release. Writing some stage code For Jan 4: Problems with minuit Suzy: Dave Monet? RHL: Centroids are in Fergal: Astrometry is basically working. RMS of WCS solution c. 0.1 asec Waiting on APIs for astrometry.net to get photometry from KD catalogues Working on Tim's CTIO problems (failing on 50% of frames) RHL: Stage code? Fergal: Yes, but waiting on API. RHL: Could we write the stage code now? Fergal: It's mostly done IPAC: Suzy: Serge has been doing WISE, not LSST. KT, do we need him? KTL: yes, there are stages that match catalogues with billions of entries Suzy: I'll talk with him today Russ: AWOL on PTF, should be back this month. Worked on meeting coding standards. Skeleton stages for WCS verification if we need it, seeing Fergal's work Deborah: Need to work with Tim on SDQA development, maybe just before Christmas? RHL: Do the stages generate SDQA data, or just metadata? Deborah: I thought that we'd use the metadata; let's do that for the first of the year Russ: OK RHL: Are you all happy? Deborah: not really. We thought that Tim wanted us to provide independent checks. RHL: OK, let's do that. U Washington: RHL: We're out of time, what shall we do now? REO: chi^2 coadd staging is wrong. KTL: I permit this to change David: I'll be asking for the big picture next Tuesday RHL: I'll be trying to get it during the rest of this week
Applications WG minutes for 12/9/09. Attendees: Suzy, Schuyler, Tim, Jim, David Gehrig, Dave Wittman, KT, Jon, Fergal, Mike Jarvis, Deborah, Russell, Russ, Steve, Andy This meeting was dedicated to reviewing the status of the skeleton stage implementation. Each developer was asked to provide status on the stages assigned to them (see attachment). Tim mentioned that he thought there were two items being mixed together: 1. the need to assemble the skeleton stages into something that runs 2. producing science output that gives a reasonable estimate of compute resources needed from Teragrid. Will the resource estimation question be answered with running skeleton stages linked together or just based on running individual stages? It’s not clear who is in charge of collecting resource estimation info or how it's being done. Not a lot of work days left. Tim to talk with Jeff about this. [** In a special DM management meeting held 12/10, Jeff clearly stated that item 1 above this the highest priority, and item 2 is a secondary priority. Please refer to Jeff’s email notes of that meeting for specific actions.] Developers are asked to put their stage code in the trunk and to link it to the DC3b processing page. Below are two emails from KT on how to do this. “I have added an example of how to link your completed stage and dictionary into the Trac stage list at http://dev.lsstcorp.org/trac/wiki/DC3bProcessingStages. The example is for the meas.pipeline.MultifitStage. Please do this for all skeleton stages as you complete them. Thanks.” “Information on where various files should live for DC3b has been added to the page on creating a stage implementation: http://dev.lsstcorp.org/trac/wiki/CreateStageImplementation” Suzy's assessment is that most stages are on track for a build by Jan 4. The exceptions to that are association pipeline, multifit, and perhaps diff-im. More than one developer appears to be waiting on a new version of AFW. We need Robert to give his status on that. Also, Robert is asked to review Andy and Simon’s code before they get committed to the trunk. There are still five unassigned stages. We should adhere to the notion that the trunk can run - don't break the trunk. Finally, the group felt using the straight interpretation of the JPL Healpix code was the way to proceed for now.
Present: RHL, Lynne, Russell, Martin, KT, Dustin, Jim, Simon, Fergal, Steve, ACB KT: Trying to put a pipeline together this afternoon RHL: Has anyone got stages that aren't on KT's page? RHL: I do Simon: mine are working on a branch, but the tests fail RHL: Merge ip_isr to the trunk and dummy the tests RHL: OK, over to Martin; his presentation (aspossible updated after this phonecon) is at http://dev.lsstcorp.org/trac/attachment/wiki/MartinDubcovsky/PSF_Multifit_Design_Review.pdf Note: RHL was too busy harassing Martin to take good notes on the presentation. I think it basically went quite well, and I await the resulting code. Two details that came out were: 1. Multifit will get phometrically calibrated Exposures 1. We need a "Heavy" Footprint with pixels included. As a sidebar, there was some thought that Footprints should be able to handle e.g. HEALPix coordinates. We will be having a meeting on 2009-12-23
Present: RHL, Mike Jarvis, Schuyler, Deborah, Jim Bosch, TSA, Steve, ACB, Martin Agenda: Healpix NSF review TSA: Went very well. We expect a report in January. Status: RHL: State of ISR stages? ACB: Mostly done (RHL: but maybe not checked in) Jim: The multifit stages are very dummy... TSA: Any progress on getting me a number? Jim: we're getting close. Martin: tests running by the end of the week. Stuck on Wcs ticket -- need afw::geom::affine transform. Mike Jarvis: In DES I return the Jacobean, and always represent the Wcs as a polynomial Swig by the end of next week. RHL: Where are the Camera classes? ACB: I think that Simon didn't use them in the ISR, so they are bottom of his list TSA: I just hacked up the DC3a stuff RHL: Healpix. Andy? ACB: I don't know TSA: I need to know the size of the sky tiles rather than the ACB: There are PSF-matched template images, and non-PSF matched per-band coadds, and chi^2 images The PSF-matched ones assume isolated images and an ideal PSF (currently a single Gaussian). AOB? Jim: Box code is in ^/DMS/afw/tickets/1095 We will be having a meeting on 2009-12-30
Present: RHL, REO We discussed pressing items of mutual interest. You just had to be there.