Last modified 7 years ago Last modified on 10/12/2012 09:09:01 PM

Design Review for Forced Photometry

NOTE: the review scheduled for Friday, October 12 concerns only the second half of this document, the pipe_tasks section.


  • Add forced photometry for model magnitudes (#2338)
  • Allow forced photometry to be run in more modes ("coadd to coadd", posssibly "ccd to ccd" and "external to ccd", as well as "coadd to ccd"), and be able to run without camera-specific customization or databases (#2350).

Changes in meas_algorithms

Most changes confined to two files; already implemented (would be ready for code review, but testing blocked by #2350):

This adds the applyForced member function to Algorithm class, which takes a reference SourceRecord and an AffineTransform that maps the reference coordinate system to the measurement coordinate system in addition to the arguments to apply. The default implementation just calls apply, and works for any algorithm in which the only information from the reference catalog is the centroid. In addition, Algorithms are informed at construction-time whether they will be used for forced photometry or regular photometry.

This just changes the measurement framework to use the new forced-measurement API for Algorithm. A small amount of refactoring

Design Issues

  • Thought about making ForcedAlgorithm a separate class, but this would have required a lot more infrastructure to deal with the common case where the centroid is the only information from the reference catalog, including another algorithm registry.
  • applyForced could take the reference Wcs object instead of an AffineTransform. I think in most cases algorithms the first thing the algorithm would do with the Wcs object would be to create the AffineTransform, and I can't think of a reason why they'd need the Wcs itself, but the Wcs does contain more information.
  • The reference source object to applyForced could take a SimpleRecord instead of a SourceRecord. This would make slots unavailable on the reference source, but it would make it slightly more natural to use external catalogs as the reference catalog. This change would also require changes in pipe_tasks, because the forced photometry task currently assumes SourceRecord there.
  • There are currently no checks (at least in the base class) to ensure that only applyForced is called when isForced=True at construction time, and vice-versa. I'm not sure if this would be desirable in general, but it probably doesn't matter because in practice only unit tests and the measurement framework currently use the Algorithm APIs.
  • These changes did not introduce the macro system in Algorithm that simulates templated virtual member functions, but they do serve as a reminder of its ugliness. We could get rid of this and clean things up if we were willing to limit ourselves to applying source measurement algorithms only to Exposure<float> and not Exposure<double>. I don't think we use the latter anywhere at present. This would be a separate ticket.

Changes in pipe_tasks

I have not even started to implement this, and I have a number of open questions I'd like to resolve before I begin (in fact, I'd love to not ultimately be the one responsible for implementing much of this, as it isn't my area of expertise and I have other commitments):

  • What are the desired command-line interfaces for for forced photometry in different modes?
  • Which modes are the highest priority, and which can be put off for later?
  • Do any of the command-lines we might want suggest making changes in the pipe_base argument parser?
  • Do we need any new butler/skymap functionality (i.e. spatial queries) to implement any of these in a camera-generic way?
  • Do we need any new butler/mapper functionality to support saving forced photometry outputs?
  • Should "coaddName" actually be a data ID component rather than a mapper prefix?
  • Does this also cover the needed interfaces for a MultiFit task? (I think it does, even though the actual data access may work differently.)

Proposed Command Line Interfaces

In all of the command lines below, the reference data ID argument can only be specified once and may not contain wildcards, while the usual --id can be specified multiple times and used with wildcards.

I'd also be curious to know whether it would be easier to implement different modes as different bin scripts or as a single bin script with a subcommand argument that changes what the remainder of the arguments should be; right now I'm assuming the latter would be difficult to implement so I've demonstrated the former below.

Coadd to Coadd or CCD: <camera> <ref-coadd-name> --ref-id <ref-data-id> --id <meas-data-id> <camera> <ref-coadd-name> <meas-coadd-name> --ref-id <ref-data-id> --id <meas-data-id>

<ref-coadd-name> --- expect reference sources to be in mapper as '<ref-coadd-name>_src'
<meas-coadd-name> -- expect measurement image to be in mapper with this name
<ref-data-id> ------ partial data ID (usually just filter or band) used to fully resolve the reference coadd data ID; tract and patch are automatic
<meas-data-id> ----- full data IDs for images(s) to measure on

Arbitrary Catalog to Coadd or CCD (also allows CCD to be used as input): <camera> <ref-mapper-name> --ref-id <ref-data-id> --id <meas-data-id> <camera> <ref-mapper-name> <meas-coadd-name> --ref-id <ref-data-id> --id <meas-data-id>

<ref-mapper-name> -- name of the mapper entry for the reference catalog
<meas-coadd-name> -- expect measurement image to be in mapper with this name
<ref-data-id> ------ full data ID for reference catalog (no spatial queries done)
<meas-data-id> ----- full data IDs for images(s) to measure on

Design Issues

  • I expect the "arbitrary catalog" versions to be used for CCD-to-CCD forced photometry (probably only for tests) as well as wide-area external catalogs. It'd be nice if we didn't have to add a mapper entry for such a catalog, and could instead just pass a filename, and I could just interpret <ref-mapper-name> as a filename if --ref-id is not passed, but we might also want that kind of implicit-mapper-entry-from-filename functionality in the butler/mapper.
  • Note that while the command lines are essentially the same in the from-coadd and from-cat versions, the versions that use coadds as input do not require the reference data ID to be complete; I assume we can use the skymap to determine the tracts and patches for a particular CCD. (Can we?)
  • Do we need to put anything about the skymap on the command line? Or can we get the skymap just by knowing the coadd name? Is the "what should be" answer to this question different from the "what is" answer?
  • If we make coadd names into data ID components, the "ToCoadd" and "ToCcd" forms can all be merged (at least in terms of command-line interfaces).
  • If we care about all these modes, we have a new explosion of mapper names for the outputs, now potentially with prefixes for both the reference and the measurement images. If we make coadd names into data ID components, this might get better, but it's still problematic because we'd need to merge the reference data ID with the measurement data ID to get something unique. I really have no idea what the best solution to this problem is; I don't think we want to hard-code all the combinatorial possibilities for forced photometry into the mapper or elsewhere, even if we limit ourselves to combinations we think we want in production.

Notes from brainstorm meeting 2012-Oct-12

Present: Jim, Paul, Robert, Russell, Mario, KTL

  • References contain measurements done on other exposures
  • They need to be re-measured (in a restricted sense) on a new exposure
    • Some of the parameters are held fixed
  • Most common case: coadd -> exposure, centroid only thing held fixed
  • Parameters held fixed can be defined in config for algorithms
  • How do we define right set of objects (incl filters)?
  • Discussed open questions
  • Butler needs:
    • Duck-type (all sources are the same -- needed to be able to retrieve)
    • Kind (gives the data id schema)
    • Step that produced dataset (for identification)
  • Have an exposure, find reference objects that fall on that exposure
    • Reference object from coadd is specified by filter, tract, patch
  • Could implement on database
    • Want to be able to run on just files
  • Can skymap find patches based on bounding box? Yes
  • Each coadd has its own skymap
  • [Should chisquared coadds be a filter?
    • No, but coadd-type + filter should look like chisquared]
  • Arbitrary catalog
    • Need to be able to name it
  • May not need CCD-to-CCD
    • CCDs won't line up exactly
    • Could do as single-depth coadd
    • Need some way to select out which CCD is the reference
  • Coadd-to-coadd and Coadd-to-CCD are main modes to support
    • Put off catalog-to-CCD until later
  • How do we save output source table?
    • Needs to have some sense of input
    • Kind of coadd, filter (ref-filter, ref-coadd-kind)
  • Needed for HSC
    • Not for commissioning
    • But for science
  • Jim to develop use cases for butler next week
  • Estimate and prioritize butler changes on Monday, Oct. 22


Add comment