wiki:Winter2014/yusra/measureCcdFailureModes
Last modified 5 years ago Last modified on 02/17/2014 01:55:46 AM

Failure modes for Run3.

The wiki page Run3 compares SDSS mags to all measureCcd measurements that returned: a number, had no flags set in either processCcd or measureCcd, and ended in a normal fit.state. This page will log information on the datapoints that didn't make it onto the plots in Run3: The NaNs, the flagged entries, and the unusual fit.states.

Of all sources measured in this run:

648 047 Returned normally
35 253 %5 returned abnormally

Of these 35,336:

1 012 Returned NaNs
34 241 Returned a number

1012 NaNs

Of the NaN's, 624 were children of blends, 32 are parents of blends.

221 had flags set by processCcd. By "flags set" I mean one of (flagNegative, flagBadMeasCentroid, flagPixEdge, flagPixInterpAny, flagPixInterpCen, flagPixSaturAny, flagPixSaturCen, flagBadPsfFlux, flagBadApFlux , flagBadInstFlux, flagBadCentroid, flagBadShape). The only one I did NOT check is flagBadModelFlux, since we're exploring a new calculation of model flux and I don't care what the model flux algorithm did in processCcd, since we're looking to replace it. Most of the 1012 DID indeed have the flagBadModelFlux set in processCcd.

COUNTS: flagPixInterpAny =218, flagPixInterpCen=48, flagPixSaturAny=102, flagPixSaturCen= 30 with an unsurprising correlation matrix of: http://content.screencast.com/users/YusraAlSayyad/folders/Default/media/aa71f8ba-3fea-45fa-a0cc-037baab1c3cc/nansCorrMatrix.png

314 sources were not in a blend and did not have flags set by processCcd: Thumbnails of this manageable set of 314: http://content.screencast.com/users/YusraAlSayyad/folders/Default/media/7ebdef9a-0c84-4c47-82f0-b51064dcbb46/nans1.jpeg http://content.screencast.com/users/YusraAlSayyad/folders/Default/media/e7540aa2-b82b-49ba-95e4-694591620423/nans2.jpeg

The thumbnail labels map to a dataId lookup table here: http://lsst-web.ncsa.illinois.edu/~yusra/W14/noFlagOrBlends.dat. Calexps may be found in /lsst8/yusra/W14/test_meas/calexp_dir_72/ and modelfits can be found in /lsst8/yusra/W14/test_meas/measureCcdOutputRX2129Deblend

There are a handful of bright, isolated objects that are repeatedly producing nans in multiple observations (e.g. labels 294-309, 198-222)

TODO: Watch the fitting of these sources to see at what step NaNs are being produced.

34241 Real Numbers with Flags

CountFlag
190244640 = 0x1220: STATUS_TR_UNCHANGED, STATUS_STEP_ACCEPTED, FAILED_MAX_OUTER_ITERATIONS,
129111 = 0x0001: CONVERGED_GRADZERO <-- No failures
1202576 = 0x0240:STATUS_STEP_ACCEPTED, FAILED_EXCEPTION
53764 = 0x0040: FAILED_EXCEPTION
3578736 = 0x2220: STATUS_TR_DECREASED, STATUS_STEP_ACCEPTED, FAILED_MAX_OUTER_ITERATIONS
17716928 = 0x4220: STATUS_TR_INCREASED , STATUS_STEP_ACCEPTED, FAILED_MAX_OUTER_ITERATIONS
33128 = 0x0080: FAILED_NAN

The 95% of measurements returned with 8450=0x2102 (STATUS_TR_DECREASED,STATUS_STEP_REJECTED, CONVERGED_TR_SMALL). For reference, state flag definition from include/lsst/meas/multifit/optimizer.h:

 enum StateFlags {
        CONVERGED_GRADZERO = 0x0001,
        CONVERGED_TR_SMALL = 0x0002,
        CONVERGED = CONVERGED_GRADZERO | CONVERGED_TR_SMALL,
        FAILED_MAX_INNER_ITERATIONS = 0x0010,
        FAILED_MAX_OUTER_ITERATIONS = 0x0020,
        FAILED_EXCEPTION = 0x0040,
        FAILED_NAN = 0x0080,
        FAILED = FAILED_MAX_INNER_ITERATIONS | FAILED_MAX_OUTER_ITERATIONS | FAILED_EXCEPTION | FAILED_NAN,
        STATUS_STEP_REJECTED = 0x0100,
        STATUS_STEP_ACCEPTED = 0x0200,
        STATUS_STEP = STATUS_STEP_REJECTED | STATUS_STEP_ACCEPTED,
        STATUS_TR_UNCHANGED = 0x1000,
        STATUS_TR_DECREASED = 0x2000,
        STATUS_TR_INCREASED = 0x4000,
        STATUS_TR = STATUS_TR_UNCHANGED | STATUS_TR_DECREASED | STATUS_TR_INCREASED,
        STATUS = STATUS_STEP | STATUS_TR,
   

TODO: look at sources (which are not blends or w/ processCcd flags), models/residuals and watch descent of optimizer for examples above. Prioritize in order of high counts to low counts.