Ticket #773 (closed defect: fixed)

Opened 10 years ago

Last modified 7 years ago

ip_diffim spatial fit fails

Reported by: RayPlante Owned by: becker
Priority: critical Milestone:
Component: ip_diffim Keywords:
Cc: rhl, ktl, dubcovsky Blocked By:
Blocking: Project: LSST
Version Number:
How to repeat:

/lsst/DC3root/ticketFiles/773 contains the appropriate files, but the difference imaging stage succeeds instead of failing, for an unknown reason.

Description (last modified by ktl) (diff)

Difference imaging fails with "Spatial fit fails" on visit 705197, CCD 0, amplifier 0 of run rlp1151.

Relevant packages:

afw                   3.3.15    	Current Setup
ctrl_dc3pipe          3.3.1     	Current Setup
ip_diffim             3.3.8     	Current Setup
meas_algorithms       3.0.7     	Current Setup

Change History

comment:1 Changed 10 years ago by rowen

  • How to repeat modified (diff)
  • Component changed from ip_isr to ip_diffim

comment:2 Changed 10 years ago by rhl

  • Cc dubcovsky added
  • Status changed from new to closed
  • Resolution set to fixed
  • reviewstatus changed from notReady to selfReviewed

Martin Dubcovsky fixed this typo in r8964, raising a RuntimeError

As there's no how-to-repeat test case I can't easily check that this works, but I'm sure enough to mark it as fixed.

comment:3 Changed 10 years ago by ktl

  • Status changed from closed to new
  • Resolution fixed deleted

The problem is that the exception is being thrown at all, not that it is being thrown incorrectly.

Unfortunately, a reproducible test case has not yet been able to be created.

comment:4 Changed 10 years ago by ktl

  • Cc rhl added
  • Priority changed from normal to critical
  • How to repeat modified (diff)
  • Description modified (diff)
  • Summary changed from ISR stage typo: fails to raise Exceptions.LsstRuntime to ip_diffim spatial fit fails

comment:5 Changed 10 years ago by becker

  • Status changed from new to assigned

comment:6 follow-up: ↓ 10 Changed 10 years ago by becker

Ray - Is the following an irrelevant error from IPSD-rlp1151.log?

  File "/lsst/home/rplante/svn/pex_harness-trunk/python/lsst/pex/harness/Pipeline.py", line 421, in tryPreProcess
    stage.preprocess()
  File "/lsst/home/rplante/svn/ctrl_dc3pipe-trunk/python/lsst/ctrl/dc3pipe/VisitMetadataStage.py", line 12, in preprocess
    visitId = event.get("visitId")
AttributeError: 'NoneType' object has no attribute 'get'

comment:7 Changed 10 years ago by becker

Also Ray - Can you run only this image through the pipeline, and turn Trace up to level 5?

comment:8 follow-up: ↓ 9 Changed 10 years ago by becker

This exception is thrown from :

    fit      = afwMath.minimize(function,
                                pars,
                                stepsize,
                                values,
                                errors,
                                cols,
                                rows,
                                nSigmaSq)
    if not fit.isValid:
        raise RuntimeError('Spatial fit fails')

I am presuming that there are not enough constraints for the requested function to be fit. Meaning not enough objects were found around which to build kernels. In fact, since both the background and kernel vary with zeroth spatial order, this means zero objects were found.

By the way, who changed the kernel order to be zeroth?

comment:9 in reply to: ↑ 8 Changed 10 years ago by ktl

Replying to becker:

By the way, who changed the kernel order to be zeroth?

I guess that would be me, in [8733]. I think I did this because ip_diffim/pipeline/ImageSubtractStageDictionary.paf has kernelSpatialOrder: 0.

comment:10 in reply to: ↑ 6 ; follow-up: ↓ 11 Changed 10 years ago by ktl

Replying to becker:

Ray - Is the following an irrelevant error from IPSD-rlp1151.log? visitId = event.get("visitId") AttributeError: 'NoneType' object has no attribute 'get'

I believe this is irrelevant. It's in the last loop, so it may be a shutdown transient.

comment:11 in reply to: ↑ 10 Changed 10 years ago by RayPlante

Replying to ktl:

Replying to becker:

Ray - Is the following an irrelevant error from IPSD-rlp1151.log? visitId = event.get("visitId") AttributeError: 'NoneType' object has no attribute 'get'

I believe this is irrelevant. It's in the last loop, so it may be a shutdown transient.

That is correct. We have noted that we still are not shutting done pipelines well.

comment:12 follow-up: ↓ 13 Changed 10 years ago by becker

  • I am not getting email updates when this ticket is updated
  • I don't know how to debug this unless I can actually run the pipeline where its failing.

comment:13 in reply to: ↑ 12 Changed 10 years ago by ktl

Replying to becker:

  • I am not getting email updates when this ticket is updated

Is your E-mail address configured properly in the Trac Preferences (link at top of each page)?

  • I don't know how to debug this unless I can actually run the pipeline where it's failing.

I will try a single-visit full-up run to see if I can get it to fail.

comment:14 Changed 10 years ago by ktl

Hmm. Having some trouble with a single visit, single amplifier pipeline run, as WCS determination fails (and difference imaging succeeds).

comment:15 follow-up: ↓ 18 Changed 10 years ago by becker

Perhaps the problem actually is that the Wcs fails, somehow the process moves on to difference imaging anyways, and because the images are misaligned there are no good kernels?

What does difference imaging "succeeds" mean in this context?

comment:16 Changed 10 years ago by rowen

Just for the record: if WCS determination fails then the initial guess is used. I don't know how good the initial guesses are, but if the initial guess is good then subsequent stages should work normally.

comment:17 Changed 10 years ago by rhl

I'd be pretty surprised if the initial guess were good enough. There should probably be QA in the diffim stage on the astrometric solution.

comment:18 in reply to: ↑ 15 Changed 10 years ago by ktl

Replying to becker:

What does difference imaging "succeeds" mean in this context?

"Succeeds" means that it does not issue a "Spatial fit fails" message. I would expect the opposite behavior: if the WCS is bad, then the differencing has problems. Instead, there were no problems in the WCS determination for this exposure during rlp1151 (using astrometry_net_data usnob).

comment:19 Changed 10 years ago by becker

I suspect that the answer lies here. Can you check the Wcs status of those images that issue a "spatial fit fails" message in diffim?

comment:20 Changed 10 years ago by ktl

The WCS determination succeeded for those images.

I'm no DS9 expert, but the sci image (/lsst/DC3root/rlp1151/IPSD/output/sci/v705197-e0/v705197-e0-c000-a00.sci_img.fits) looks pretty lousy, and it appears to be offset by about 8 pixels or so from the template image (/lsst/DC3root/rlp1151/IPSD/output/tmpl/v705197/v705197-c000-a00.tmpl_img.fits).

comment:21 Changed 10 years ago by becker

Indeed, the images' Wcs are not aligned well enough for subtraction. Off by ~8 pixels in x, ~6 in y in the center of the image; the solution is better near the LLC. Are there any Sdqa diagnostics from the Wcs stage that indicate the quality of the solution?

comment:22 Changed 10 years ago by ktl

Note that the misaligned WCS still does not explain why this case fails to difference (with "Spatial fit fails") in the pipeline but succeeds when run standalone. The WCS should be round-tripped correctly via the sci and template FITS files.

comment:23 Changed 10 years ago by becker

  • Status changed from assigned to closed
  • Resolution set to fixed

The issues that prompted this particular ticket (problems with the image variances, and Wcs round-tripping) have been addressed, so I am closing this ticket. Re-open if problem arises again.

comment:24 Changed 7 years ago by robyn

  • Milestone DC3a Completed deleted

Milestone DC3a Completed deleted

Note: See TracTickets for help on using tickets.