wiki:Winter2013/CoaddPsfReview
Last modified 6 years ago Last modified on 10/15/2012 06:30:27 PM

Design Review for Stackfit Psf Class (Ticket #2316)

October, 2012 -- For Winter 2013

CoaddPsf is a C++ class which is being implemented to make the Psf determination present in the Stackfit algorithm available as a generic class. It is intended to allow a spatially varying Psf model any Coadd, not just for those processed by the Stackfit pipeline.

This review will not concern any other features of the Multifit/Stackfit? implementation for Winter 2013.

Goals:

Review the design of the subclass CoaddPsf, a subclass of lsst::afw::detection::Psf which allows spatially varying Psf kernel estimation on coadded images. This class should have the same functionality as other Psf subclasses, so as to make its use transparent. Its relationship to its associated Kernel class, CoaddPsfKernel should be reviewed. This will make the Stackfit Psf functionality generally available from meas_algorithms.

It is desirable that the coaddition process and the CoaddPsf generation be independent (the two should be able to run in conjunction, but the CoaddPsf should be able to be built later from the products of the Coadd pipeline. To achieve this goal, metadata from calibration (Psf, Wcs, Photocal) and from coaddition (bounding boxes, weights) must be preserved. The current DLS Stackfit pipeline already does this, but in its own peculiar way. This review should establish a best practice method for persisting these products.

Design:

The CoaddPsf class is the class which the user will interact with. It is possible to create a CoaddPsfKernel separately, but it is not clear why that has any utility. The CoaddPsf class accumulates Psf models, Wcs, Photocal, and bounding boxes for each component calexp used to build the coadd. This is a modest amount of information compared to the input images themselves.

The relationship between Kernel, Psf, and KernelPsf is somewhat of a mystery to me. It is also unclear to me why the user of these classes is allowed to manipulate the kernel directly. However, I have followed the design paradigm closely, even though some of the inherited methods do not make sense for CoaddPsf. It would be good to have a discussion of this during the review.

It is our expectation that we can use Russell's coadd pipeline to build the coadd for lensing, and that some form of "standard" coadd will be useful for doing lensing. Some minor changes may be required to the coadd pipeline.

Caveats:

It has been a design issue whether we stack psfs from warped images, or from the original calexps. The former design is the one adopted for the DLS pipeline. For this project, we have instead chosen instead to do our Psf modeling on the original (unwarped) images. This allows reuse of the ImgChar pipeline products as they stand, and makes CoaddPsf fit into the existing pipeline without redundancy.

However, we must be able to transform the existing unwarped Psf kernels to the Coadd coordinate system without loss of information. We do not need to be able to transform the PCA models: but we do need to be able to transform the kernels which those models generate.

We have agreed that we are not going to worry about pixel exclusion right now. This has to do with which pixels in the input images actually contribute to the Psf of the coadd at any given point. To attempt to solve this problem at all would require the CoaddPsf class to access much more information about the individual calexp exposures than is desirable (at least the mask and variance planes). It is also not clear that we can actually estimate the effect of pixel exclusion on the Psf at a point in any meaningful way. Our design therefore largely ignores this problem, concentrating instead on the somewhat crude approach of just finding the calexps whose bounding boxes overlap the given point.

Class design:

On the surface the CoaddPsf class is to act like any other Psf. However, it houses a subclass of Psf Kernel called CoaddPsfKernel. This is the component which responds to the computeImage call by building a convolution Kernel for any given point in the coadd as required.

class CoaddPsf : public lsst::afw::detection::KernelPsf {
public:
    typedef PTR(CoaddPsf) Ptr;
    typedef CONST_PTR(CoaddPsf) ConstPtr;

    /**
     * @brief constructors for a CoadPsf
     *
     * Parameters:
     */
    explicit CoaddPsf();

    explicit CoaddPsf(PTR(lsst::afw::math::Kernel) kernel);

    explicit CoaddPsf(PTR(lsst::meas::algorithms::CoaddPsfKernel) kernel);

    virtual lsst::afw::detection::Psf::Ptr clone() const {
        return boost::make_shared<CoaddPsf>(*this);
    }

    void addPsfComponent(PTR(lsst::afw::detection::Psf)  psf, PTR(lsst::afw::image::Wcs) wcs, lsst::afw::geom::Box2I bbox, double weight);

    PTR(lsst::meas::algorithms::CoaddPsfKernel) getCoaddPsfKernel();
};

The CoaddPsfKernel is a member variable of CoaddPsf which collects information about the calexps which comprise the coadd.

class CoaddPsfKernel : public lsst::afw::math::Kernel {
public:

    explicit CoaddPsfKernel() {
    };

    lsst::afw::math::Kernel::Ptr clone() const;

//  This is the critical piece of code to override.  The image needs to come from
//  the vector of PCA models, not from a spatially varying model itself
    double computeImage(afwImage::Image<double> &image, bool doNormalize, double x=0.0, double y=0.0) const;

protected:
    void addPsfComponent(PTR(lsst::afw::detection::Psf)  psf, PTR(lsst::afw::image::Wcs) wcs, lsst::afw::geom::Box2I bbox, double weight);

The Component structure includes the Wcs, Psf, bounding box, and possibly photometric information about each calexp. One Component is kept by the CoaddPsfKernel? for each input calexp. As mentioned about, the Wcs and bounding box are of the original calexp, not the warped image.

//  Vector of components used to house the information from individual images
private:
    struct Component {
        PTR(lsst::afw::detection::Psf) psf;
        PTR(lsst::afw::image::Wcs) wcs;
        lsst::afw::geom::Box2I bbox;
        double weight;
    };
    typedef std::vector<Component> ComponentVector;
    ComponentVector _components;

Coadd Metadata:

It is an issue how the information required for the Component structure is going to be fetched from the ImgChar pipeline. This is the Psf model and Wcs which are produced by processCcd.

While we could have chosen to just persist the data ids of the calexps which were used, there is not current reliable link between any identifiers and the actual physical calexps. Furthermore, trying to build such a link would make CoaddPsf dependent on database or repository mechanisms across multiple machines.

Since we know that any program which wants to make use of a CoaddPsf will need to have access to the coadd fits file, our design choses to persist this information in the coadd fits file itself. The binding of data and metadata is ensure with not additional mechanisms or effort.

The coadd pipeline does not currently have a mechanism for persisting detailed metadata. We therefore propose the following:

  1. That as part of this proposal we write code in the C++ addToCoadd routines to keep the calexp Wcs and Psf information in memory until the coadd fits file is written.
  1. That this information be save in the coadd fits file.
  1. The plausible options seem to be to save this information either in the fits header itself, or in a fits table attached to the coadd fits file as an extension. The latter is much preferred by the authors, but the pros and cons of different methods should be discussed at this review.

We have chosen to persist WCS information and bounding boxes from the unwarped input images (calexp's) as opposed to those of the warped images. These bounding boxes are needed to know which images from the input set contribute (or partially contribute) to pixels in the coadd. The bounding boxes just require the NAXIS and CRPIX, and Wcs information from the original images. For this information to be available when the coadd is save, it must be stored in the header of the warped images by the warper, then retrieved and store by the coadd process when getCoadd() is called.

The old DLS Stackfit pipeline used the following mechanism in the fits header to store metadata. These are comment lines which are generally ignored by the fits file.

HIERARCH NCOMPONENTS =     112
HIERARCH COMPONENT0 = 'R/2000-11-27/obj514_1.fits 0.000362 0 0 2049 4097'
HIERARCH COMPONENT1 = 'R/2000-11-27/obj514_2.fits 0.000382 0 0 2049 4097'
HIERARCH COMPONENT2 = 'R/2000-11-27/obj514_3.fits 0.000492 4915 1032 7003 5180'
HIERARCH COMPONENT3 = 'R/2000-11-27/obj514_4.fits 0.000471 7041 1054 9092 5193'
HIERARCH COMPONENT4 = 'R/2000-11-27/obj514_5.fits 0.000455 679 5164 2731 9306'

Our current effort will require a more sophisticated mechanism. I suggest an appended fits table extension with the following columns:

A single column for each scalar from NAXIS, CRPIX, CRVAL, the CD matrix, the distortion parameters. A single column for each scalar from Photocal or zeropoint information. A single column for each scalar required to express the PCA Psf model for this image.

I expect this design to produce some controversy. Please discuss!

Image Selection:

This is not a part of the Psf class review.

Other considerations:

If it does not already exist, we need a standard butler/datamap map type for the lensing quality coadd. We need to be sure that calexp information (Wcs, Psf, Photocal, and possibly (shudder) pixel exclusion information.

Notes from review on 2012-Oct-15

Reviewee: Perry

Present: Jim, Russell, Robyn, RHL, Mario, KTL

Capturing code from James Jee

Derives from Psf class so that it is usable normally

CoaddPsf contains CoaddPsfKernel How to build CoaddPsf?

  • CoaddPsf (or at least its Kernel) needs to be updated with new images
  • Building code needs to know it is a CoaddPsf[Kernel]
  • PsfFactory seems to be getting in the way, but it may be obsolete + Need to check to see if PsfFactory needs to be supported + Need to check that CoaddPsf constructor using Kernel can be removed

Psf has dimensions, but computeImage() takes image with dimensions

  • Probably need to try to return image with requested dimensions; RHL verified later

Building PSFs from original unwarped images

  • Requires transformation from PSF kernels to coadd -- separate task

Excluded pixels

  • Add special (non-base-class) interface to exclude certain images

Make sure that CONST_PTR is used where appropriate

CoaddPsfKernel::addPsfComponent() should be private, not protected

Photometric information might be zeropoint but probably is part of weight

Desirable to have ComponentVector be public

  • Construct read-only CoaddPsf using it
  • Copy ComponentVector but not deep-copy things in it

Should coadds persist all metadata for calexps that went into it? Yes

Persistence

  • Should metadata be persisted as FITS binary table or as extension headers?
    • Binary table preferred
    • Headers have minimum size (but not really a problem for thousands)
  • Will we have different types of WCS/PSF in same coadd? Maybe, but not now
  • Need to have all data in one file
    • Can't lose it
  • Use FITS as a packaging technology or something else like ZIP/tar?
    • Perry and Jim to prototype

Exposure

  • Will now persist and retrieve PSF, including CoaddPsf
  • Psf needs to have an interface for FITS persistence

CoaddExposure

  • Contains ComponentVector
  • PSF may be a CoaddPsf or some other kind of Psf (if matched)

ComponentVector

  • Add Exposure identifier to component
  • Add flag to say whether was actually included? No
  • Add Exposure identifier creation and parsing APIs to Mapper