Ticket #801 (closed defect: fixed)

Opened 10 years ago

Last modified 10 years ago

Image appears to be outside template

Reported by: ktl Owned by: rowen
Priority: critical Milestone:
Component: ip_diffim Keywords:
Cc: becker, rhl, fergal Blocked By:
Blocking: Project: LSST
Version Number: 3.3.8+svn(uncommitted)
How to repeat:
$ cd /lsst/DC3root/ticketFiles/801
$ python fail.py

Description

In run rlp1173, visit 704897: CCD 2 amp 0, CCD 3 amp 2, CCD 3 amp 3, and CCD 3 amp 6 appear to produce template bounding boxes that do not overlap the template image (T0004_D4_25_i_img.fits).

For example, CCD 3 amp 6 produces these messages:

ip.diffim.TemplateBBoxStage: scienceDimensions=1024, 1153; templateDimensions=19354, 19354

RuntimeError: BBox does not overlap template; desired region (without border) is (31990, 1264) through (133783, 14653)

The ip_diffim used for this run appears to differ from 3.3.8+svn9730 only in having additional logging.

Change History

comment:1 Changed 10 years ago by ktl

  • How to repeat modified (diff)

comment:2 Changed 10 years ago by rhl

My guess is that the astrometry is (still) wrong

comment:3 follow-up: ↓ 5 Changed 10 years ago by fergal

I may be reading the code wrong, but it seems as if fail.py is reading a Wcs directly from the header of T0004_D4_25_i_img.fits, and not solving using WcsDeterminationStage?, so meas::astrom isn't involved here.

comment:4 Changed 10 years ago by rowen

  • Status changed from new to assigned

comment:5 in reply to: ↑ 3 Changed 10 years ago by ktl

Replying to fergal:

it seems as if fail.py is reading a Wcs directly from the header of T0004_D4_25_i_img.fits

True, but the template bounding box is computed using the Wcs of the Exposure, which is generated (earlier) using WcsDeterminationStage and is suspect.

comment:6 follow-up: ↓ 8 Changed 10 years ago by rowen

  • Owner changed from rowen to fergal

I modified fail.py to print more information. The template and science image certainly do not appear to overlap, so I am turning this over to Fergal. The expanded output follows. Note that the details of the RuntimeError? text differ, which surprises me.

Science Exposure Path: v704897-e0-c002-a00.sci
Science WCS:
AP_0_1 = -0.279217
AP_0_2 = -3.59508e-06
AP_0_3 = 1.51294e-09
AP_1_0 = -0.625849
AP_1_1 = 1.10755e-05
AP_1_2 = -6.27363e-09
AP_2_0 = -8.31883e-06
AP_2_1 = 8.66343e-09
AP_3_0 = -3.98427e-09
AP_ORDER = 3
A_0_2 = 0.000528099
A_0_3 = -4.07919e-06
A_1_1 = 0.00129754
A_1_2 = -1.52405e-05
A_2_0 = 0.000794934
A_2_1 = -1.8932e-05
A_3_0 = -7.82141e-06
A_ORDER = 3
BP_0_1 = -0.551086
BP_0_2 = -3.403e-05
BP_0_3 = -5.13862e-11
BP_1_0 = -0.670065
BP_1_1 = 9.5997e-05
BP_1_2 = 1.13403e-09
BP_2_0 = -6.75972e-05
BP_2_1 = -2.92779e-09
BP_3_0 = 2.01796e-09
BP_ORDER = 3
B_0_2 = 0.000951293
B_0_3 = -5.63709e-06
B_1_1 = 0.00237824
B_1_2 = -2.14531e-05
B_2_0 = 0.00148816
B_2_1 = -2.72024e-05
B_3_0 = -1.15109e-05
B_ORDER = 3
CD1_1 = 5.08925e-05
CD1_2 = -1.31986e-07
CD2_1 = 2.08108e-06
CD2_2 = -4.97208e-05
CRPIX1 = 1401.82
CRPIX2 = 2505.19
CRVAL1 = 334.13
CRVAL2 = -17.3527
CTYPE1 = "RA---TAN-SIP"
CTYPE2 = "DEC--TAN-SIP"
CUNIT1 = ""
CUNIT2 = ""
EQUINOX = 2000
NAXIS = 2
RADECSYS = "FK5"

importing: templateWcs lsst.afw.image Wcs
pex.harness.iostage.input: loading T0004_D4_25_i_img.fits as templateWcs
Template's WCS:
CD1_1 = -5.16667e-05
CD1_2 = 0
CD2_1 = 0
CD2_2 = 5.16667e-05
CRPIX1 = 9677.5
CRPIX2 = 9677.5
CRVAL1 = 333.879
CRVAL2 = -17.7322
CTYPE1 = "RA---TAN"
CTYPE2 = "DEC--TAN"
CUNIT1 = "deg"
CUNIT2 = "deg"
EQUINOX = 2000
NAXIS = 2
RADECSYS = "FK5"


Corners of Template:
0.0, 0.0 = 334.40551, -18.23144
19354.0, 0.0 = 333.35277, -18.23144
0.0, 19354.0 = 334.40258, -17.23155
19354.0, 19354.0 = 333.35570, -17.23155

Corners of Science Exposure:
0.0, 0.0 = 350.40610, -39.44368
1024.0, 0.0 = 339.79656, -24.86546
0.0, 1153.0 = 340.44683, -26.07969
1024.0, 1153.0 = 335.45136, -19.11047

ip.diffim.TemplateBBoxStage: TemplateBBoxStage process: _rank 0 stageId 0
ip.diffim.TemplateBBoxStage: scienceDimensions=1024, 1153; templateDimensions=19354, 19354
Traceback (most recent call last):
  File "fail.py", line 55, in <module>
    run("v704897", "c002", "a00")
  File "fail.py", line 39, in run
    clip3 = stage3.run(clip2, 0)
  File "/lsst/DC3/stacks/gcc412/24apr/Linux64/pex_harness/3.3.2/python/lsst/pex/harness/SimpleStageTester.py", line 47, in run
    return self.runWorker(clipboard, sliceID)
  File "/lsst/DC3/stacks/gcc412/24apr/Linux64/pex_harness/3.3.2/python/lsst/pex/harness/SimpleStageTester.py", line 96, in runWorker
    self.stage.process()
  File "/lsst/DC3/stacks/gcc412/24apr/Linux64/ip_diffim/3.3.8/python/lsst/ip/diffim/pipeline/TemplateBBoxStage.py", line 44, in process
    templateDimensions, borderWidth)
  File "/lsst/DC3/stacks/gcc412/24apr/Linux64/ip_diffim/3.3.8/python/lsst/ip/diffim/diffimTools.py", line 359, in computeTemplateBbox
    (llc[0], llc[1], urc[0], urc[1]))
RuntimeError: BBox does not overlap template; desired region (without border) is (-261405, -458808) through (-19176, -17260)

comment:7 Changed 10 years ago by rowen

(I forgot to subtract 1 pixel from the corners reported above; this does not change the results in any significant way, but I did correct fail.py, so future runs should differ slightly).

comment:8 in reply to: ↑ 6 Changed 10 years ago by ktl

Replying to rowen:

Note that the details of the RuntimeError? text differ, which surprises me.

The exception text from fail.py comes from the first failure in CCD 2 amp 0 while the text in the ticket description comes from CCD 3 amp 6.

comment:9 follow-up: ↓ 12 Changed 10 years ago by fergal

  • Owner changed from fergal to rowen

I ran this field through the astrometry solver to create a Wcs. astrometry.net produces a Wcs that differs from the one in the header by about 8" in dec. For the object at 510, 427 (assuming the origin is at the bottom left of v704897-e0-c003-a02.sci), I get

Wcs in header : 22 15 52.10 -17 14 46.6 Wcs from astro.net: 22 15 52.12 -17 21 53.5

I found this object in the template image (T0004_D4_25_i_img.fits), and its coordinates agreed with the astro.net numbers. This field is being correctly solved, and the bug isn't with astro.net

I mapped the diagonal of sub-image (i.e v704...) to pixels in the template image. The astro.net wcs maps the diagonal to xy=(7538,16064)->(8565,17150). The header wcs maps the diagonal onto a region that only partially overlaps the reference image. Is it possible the wrong Wcs is being read from the header?

Anyways, it seems like my small corner of code isn't the problem this time, so I'm re-assigning the ticket.

comment:10 Changed 10 years ago by ktl

Odd. In DS9, looking at c003-a02.sci, I get 22:07:58 / -17:32:25 for the object at 510/427 using the WCS in the exposure's header. Looking at the pre-WCS-determination post-ISR image, which should have the original CFHT WCS estimate, I get 22:15:53 / -17:22:20. Neither of these matches Fergal's results, although the latter is closer.

The WCS in the sci exposure header is supposed to result from astrometry.net, so either there is something different between the pipeline run and Fergal's standalone run or the exposure's WCS is not getting set correctly to the pipeline-computed values.

Note that Fergal appears to have been referring to c003-a02, which is neither the first image processed in fail.py (c002-a00) nor the one in this ticket's description (c003-a06).

FYI, here are the log messages from WCS determination for c003-a02:

Using 248 sources with flux > 20000.0; initial list had 300 sources
Setting number of stars in solver to 45
RA/Dec at initial WCS origin = 333.88458, -17.731583 deg
Predicted image scale = 0.195889813374 arcsec/pixel
Set image scale min=0.178081648521; max=0.215478794711
Set normal parity

comment:11 Changed 10 years ago by ktl

Also, the "E" and "N" vectors in DS9 for the science image using its header look very distorted (nearly 180 deg apart, instead of 90 deg).

If astrometry.net is producing the correct WCS (can you post those values, Fergal?), then either it is getting corrupted as it is inserted into the image by WcsDeterminationStage or it is getting corrupted as it is being written out.

comment:12 in reply to: ↑ 9 ; follow-up: ↓ 13 Changed 10 years ago by ktl

Replying to fergal:

I ran this field through the astrometry solver to create a Wcs.

Which sources did you use for this? Did you run a new detection, or did you take the sources from the database or wcssrc.boost files? Did you use only the sources from this amp, rather than those from the entire CCD as the pipeline does?

comment:13 in reply to: ↑ 12 ; follow-up: ↓ 14 Changed 10 years ago by fergal

Replying to ktl:

Replying to fergal:

I ran this field through the astrometry solver to create a Wcs.

Which sources did you use for this? Did you run a new detection, or did you take the sources from the database or wcssrc.boost files? Did you use only the sources from this amp, rather than those from the entire CCD as the pipeline does?

I used a small script rhl wrote to extract sources from this amplifier alone. I'll repeat the result for the field listed in fail.py.

Where and how are the sources created for this ticket? I'd like to run this source list through astrometry.net and check the result directly.

Here's the Wcs that was returned for v704897-e0-c003-a02.sci:

AP_0_1 = -1.03346e-05
AP_0_2 = 9.08727e-07
AP_0_3 = 4.94676e-09
AP_1_0 = 1.1615e-05
AP_1_1 = 4.19223e-06
AP_1_2 = 9.67772e-09
AP_2_0 = 1.03799e-05
AP_2_1 = 1.18745e-08
AP_3_0 = 9.3781e-09
AP_ORDER = 3
A_0_2 = -8.92566e-07
A_0_3 = -4.87104e-09
A_1_1 = -4.0904e-06
A_1_2 = -9.52739e-09
A_2_0 = -1.03042e-05
A_2_1 = -1.16919e-08
A_3_0 = -9.28881e-09
A_ORDER = 3
BP_0_1 = -2.4855e-06
BP_0_2 = -1.50131e-06
BP_0_3 = 1.17725e-09
BP_1_0 = -5.25171e-06
BP_1_1 = -2.36315e-07
BP_1_2 = 1.91863e-09
BP_2_0 = -2.6301e-06
BP_2_1 = -3.55193e-10
BP_3_0 = -3.10127e-09
BP_ORDER = 3
B_0_2 = 1.49668e-06
B_0_3 = -1.16119e-09
B_1_1 = 2.16297e-07
B_1_2 = -1.91426e-09
B_2_0 = 2.61441e-06
B_2_1 = 3.34919e-10
B_3_0 = 3.08936e-09
B_ORDER = 3
CD1_1 = 5.18052e-05
CD1_2 = 3.82115e-07
CD2_1 = 1.82397e-07
CD2_2 = -5.15978e-05
CRPIX1 = 686.712
CRPIX2 = 2784.62
CRVAL1 = 333.977
CRVAL2 = -17.3674
CTYPE1 = "RA---TAN-SIP"
CTYPE2 = "DEC--TAN-SIP"
CUNIT1 = ""
CUNIT2 = ""
EQUINOX = 2000
NAXIS = 2
RADECSYS = "FK5"

comment:14 in reply to: ↑ 13 Changed 10 years ago by ktl

Replying to fergal:

Where and how are the sources created for this ticket?

They have been sent to Fergal in separate E-mail.

The WCS computed by the pipeline for v704897-e0-c003-a02.sci is very different from the standalone one -- in particular, the SIP distortion terms are orders of magnitude larger:

AP_0_1 = -0.100794
AP_0_2 = 0.000190386
AP_0_3 = 1.17183e-07
AP_1_0 = -0.978472
AP_1_1 = 3.63134e-05
AP_1_2 = 4.53013e-08
AP_2_0 = 1.88449e-06
AP_2_1 = 5.79261e-09
AP_3_0 = 2.45794e-10
AP_ORDER = 3
A_0_2 = 0.000108197
A_0_3 = -3.22209e-06
A_1_1 = 6.80053e-05
A_1_2 = -2.7649e-06
A_2_0 = 6.20987e-06
A_2_1 = -7.02274e-07
A_3_0 = -5.44929e-08
A_ORDER = 3
BP_0_1 = 0.134305
BP_0_2 = -0.000172034
BP_0_3 = -6.2068e-08
BP_1_0 = 0.0850745
BP_1_1 = -2.89303e-05
BP_1_2 = -2.4165e-08
BP_2_0 = -1.21867e-06
BP_2_1 = -3.09361e-09
BP_3_0 = -1.3086e-10
BP_ORDER = 3
B_0_2 = -2.91155e-05
B_0_3 = 3.33277e-07
B_1_1 = -2.18592e-05
B_1_2 = 3.90213e-07
B_2_0 = -4.42987e-06
B_2_1 = 1.74265e-07
B_3_0 = 2.31458e-08
B_ORDER = 3
CD1_1 = 5.16852e-05
CD1_2 = 9.37399e-08
CD2_1 = 2.20873e-07
CD2_2 = -5.0895e-05
CRPIX1 = 856.473
CRPIX2 = -1928.22
CRVAL1 = 333.985
CRVAL2 = -17.2437
CTYPE1 = "RA---TAN-SIP"
CTYPE2 = "DEC--TAN-SIP"
CUNIT1 = ""
CUNIT2 = ""
EQUINOX = 2000
NAXIS = 2
RADECSYS = "FK5"

The initial WCS guess for this image from the post-ISR Exposure, which needs to be offset by the XY0 of (0, 2306), is:

CD1_1 = 5.194e-05
CD1_2 = 0
CD2_1 = 0
CD2_2 = -5.194e-05
CRPIX1 = -1055.5
CRPIX2 = 7347.2
CRVAL1 = 333.885
CRVAL2 = -17.7316
CTYPE1 = "RA---TAN"
CTYPE2 = "DEC--TAN"
CUNIT1 = ""
CUNIT2 = ""
EQUINOX = 2000
NAXIS = 2
RADECSYS = "FK5"

comment:15 Changed 10 years ago by fergal

Unfortunately, it's looking as if the distortion polynomials are the cause of the problem. Looking at the object at 565,583 in image v704897-e0-c003-a02.sci, the template image suggests coords of
22 15 52.83 -17 22 22.4

A wcs created by astro.net from the source list sent to me by KT (derived from the full chip) produces
22 06 02.48 -17 35 28.3
but if I ignore the distortion terms in that wcs, I get
22 15 52.63 -17 22 23.6
which is much closer to the mark.

I'll look into it some more.

comment:16 Changed 10 years ago by fergal

  • Status changed from assigned to closed
  • Resolution set to fixed

Bug was in the way the SIP distortion calculation function was called. One of the parameters is how much scatter in the centroids you expect based on centroiding issues alone. This "jitter" was set to a fixed value that was too small for CFHT fields.

Now jitter is calculated as the sum of the squared of the scatter in the indices and the scatter in the linear wcs solution. This seems to have fixed the problem.

Note: See TracTickets for help on using tickets.