Last modified 9 years ago Last modified on 08/05/2010 03:34:30 PM

DC3b PT1 Processed Exposures

Refer to DC3bPT1ProductionRunStatus for more details regarding the production runs processing the following visits.

Unless otherwise noted, all productions runs use full FOV datasets.


Processing of 104 visits from the "blessed" list that are contained within the dense star coverage area was attempted.

  • Run prod01 (52 visits) = v85408535, v85408556, v85470977, v85470982, v85471005, v85471036, v85471048, v85471150, v85501857, v85501858, v85501865, v85501867, v85501873, v85501875, v85501884, v85501910, v85501919, v85501920, v85501927, v85501933, v85501934, v85501936, v85501945, v85501973, v85502008, v85502068, v85505197, v85505200, v85505306, v85505309, v85505331, v85518116, v85518193, v85518212, v85518215, v85518237, v85518312, v85518316, v85518525, v85545709, v85545800, v85545802, v85545825, v85563078, v85563090, v85563096, v85563138, v85563185, v85563215, v85563219, v85563260, v85563293
  • Run prod02 (26 visits) = v85563399, v85563403, v85563478, v85563553, v85563561, v85563716, v85563814, v85563816, v85563848, v85563915, v85563945, v85597903, v85597921, v85597943, v85615987, v85616014, v85616024, v85616031, v85616069, v85616071, v85616093, v85616095, v85616129, v85616157, v85616160, v85616262
  • Run prod03 (26 visits) = v85656318, v85656337, v85656377, v85656399, v85656455, v85661686, v85661792, v85674959, v85674995, v85675004, v85675024, v85675132, v85678811, v85679351, v85679451, v85683432, v85683541, v85697374, v85697495, v85723734, v85723902, v85755741, v85769988, v85770019, v85770053, v85770079

The following visits on the "blessed" list did not even have processing attempted:

  • v85501926: missing all E001 images on MSS (no images on lsst2)
  • v85518223: missing all images on MSS (later found to be present on lsst2)
  • v85656379: missing all images on MSS, in "incomplete" directory on lsst2 (6419 of 6426 files present)
  • v85656443: missing all images on MSS and lsst2
  • v85656570: missing all images on MSS, in "incomplete" directory on lsst2 (6415 of 6426 files present)

One field (v85679351) was not processable, and 233 additional CCDs failed to be fully processed for one reason or another (see below).

The remaining 19,234 CCDs were processed into calexp, icSrc, psf, and src datasets along with their associated sdqaAmp and sdqaCcd datasets. At least 438 of these may be suspect due to failures in photometric calibration; additional CCDs may be suspect due to poor WCS determination.

On Aug. 5, 11 additional visits were processed, plus v85408535 was processed again.

  • Run prod04 (12 visits) = v85408533, v85408535, v85470739, v85470745, v85470825, v85470826, v85572334, v85572335, v85661428, v85661433, v85770027, v85770084

Of these 2268 CCDs (including 189 from v85408535, which was processed before in run prod01), 507 CCDs failed WCS determination (including 64 from v85408535). An additional 86 failed PSF determination (none from v85408535) and 113 failed photometric calibration (including 3 from v85408535).

Data Issues (not fixable)

  • v85679351: incorrect filenames containing v85641699 (originals not on lsst2)
  • v85501865/E001/R24/S22/C00 missing (checked on MSS, originals not on lsst2)
  • v85563553/E000/R13/S21/C06 is corrupt (checked on MSS and lsst2)
  • v85563553/E000/R43/S20/C16 is corrupt

Data Issues (fixable)

Incorrect filenames containing filter numbers ("_f5_"). All were fixed only locally at SLAC.

  • v85616024, v85656455, v85755741, v85770053

Apps Issues preventing src generation

Fail PSF determination (no images to analyze in PCA), runs prod01-prod03:

  • v85471005/R12/S10, v85501858/R20/S10, v85518312/R22/S02

Also failing PSF determination in run prod04:

  • v85770027 (40 CCDs), v85770084 (46 CCDs)

Ridiculous plate scale for WCS determination (#1387):

  • v85501857/R21/S21, v85518312/R43/S12, v85769988/R21/S20, v85770079/R21/S20

Less than 20 bright sources:

  • v85545825/R30/S12 has 19 bright sources

"All points cleared out" in match with catalog:

  • v85723734/R22/S11, v85723902/R22/S11

Failed WCS determination (initial position verification):

  • prod01: 87 CCDs
    • v85408535-fr (64 CCDs), v85470982-fy (3), v85471005-fy (1), v85471036-fy (1), v85471048-fy (1), v85471150-fy (4), v85501858-fi (2), v85501873-fz (1), v85518525-fz (2), v85545709-fy (2), v85545800-fy (4), v85545825-fy (2)
  • prod02: 90 CCDs
    • v85615987-fy (8 CCDs), v85616014-fy (10), v85616024-fy (3), v85616031-fy (7), v85616069-fy (9), v85616071-fy (9), v85616093-fy (7), v85616095-fy (8), v85616129-fy (12), v85616157-fy (5), v85616160-fy (11), v85616262-fy (1)
  • prod03: 43 CCDs
    • v85674959-fy (7 CCDs), v85674995-fy (12), v85675004-fy (4), v85675024-fy (8), v85675132-fy (12)

Total = 220 CCDs

Also failing WCS determination:

  • prod04: 507 CCDs (443 new)
    • v85408533-fr (55 CCDs), v85408535-fr (64), v85470739-fz (3), v85470745-fz (2), v85470826-fy (1), v85572334-fi (30), v85572335-fi (27), v85661428-fg (100), v85661433-fg (90), v85770027-fu (70), v85770084-fu (65)

Apps Issues not preventing src generation

Failed photometric calibration (either too few data points or flags indicate all matches have bad photometry):

  • prod01: 227 CCDs
  • prod02: 169 CCDs
  • prod03: 42 CCDs

Total = 438 CCDs

Also failing photometric calibration:

  • prod04: 113 CCDs (110 new)

WCS verification source outside CCD boundary:

  • v85501873/R33/S12, v85597903/R20/S02, v85769988/R43/S11

Failed to compute SIP distortions (least squares):

  • v85470977/R34/S22, v85674959/R32/S20, v85674959/R33/S22, v85674959/R42/S01, v85674995/R32/S02, v85674995/R34/S00, v85674995/R34/S01, v85674995/R34/S21, v85674995/R41/S20, v85675024/R41/S20, v85675132/R03/S01, v85675132/R03/S02, v85675132/R14/S21, v85675132/R20/S10, v85675132/R30/S20, v85675132/R31/S22, v85675132/R34/S22, v85675132/R41/S21, v85616014/R03/S02, v85616014/R03/S12, v85616014/R22/S00, v85616024/R10/S20, v85616024/R41/S20, v85616031/R41/S20, v85616069/R10/S20, v85616069/R30/S21, v85616071/R01/S00, v85616071/R03/S02, v85616071/R11/S11, v85616093/R10/S20, v85616095/R01/S00, v85616095/R03/S02, v85616095/R30/S20, v85616129/R01/S00, v85616157/R01/S00, v85616157/R10/S00, v85616160/R01/S00, v85616160/R11/S11, v85616160/R13/S02, v85616160/R30/S20, v85616160/R42/S20


Production: lsst:ktl20100709 D2-fg 118 visits

v729994-fg, v729995-fg, v729996-fg, v729997-fg, v729998-fg, v733329-fg, v733330-fg, v733331-fg, v733332-fg, v733333-fg, v739524-fg, v739525-fg, v739526-fg, v739527-fg, v739528-fg, v741672-fg, v741673-fg, v741674-fg, v741675-fg, v741676-fg, v742005-fg, v742006-fg, v742007-fg, v742008-fg, v742009-fg, v742010-fg, v742344-fg, v742345-fg, v742558-fg, v742559-fg, v742560-fg, v742561-fg, v742562-fg, v744548-fg, v744549-fg, v744550-fg, v744551-fg, v744552-fg, v745239-fg, v745240-fg, v745241-fg, v745243-fg, v745244-fg, v769879-fg, v769880-fg, v769881-fg, v769882-fg, v769997-fg, v770007-fg, v773544-fg, v773545-fg, v773546-fg, v773547-fg, v773548-fg, v774011-fg, v774012-fg, v774013-fg, v774014-fg, v774015-fg, v774642-fg, v774643-fg, v774644-fg, v774645-fg, v774646-fg, v776397-fg, v776398-fg, v777109-fg, v777123-fg, v777124-fg, v777125-fg, v777126-fg, v777127-fg, v777863-fg, v777864-fg, v777866-fg, v777867-fg, v779947-fg, v779948-fg, v779949-fg, v779950-fg, v779951-fg, v780817-fg, v780818-fg, v780819-fg, v780820-fg, v780821-fg, v784470-fg, v784471-fg, v784472-fg, v784473-fg, v784491-fg, v784492-fg, v784493-fg, v784494-fg, v784495-fg, v787784-fg, v787785-fg, v787786-fg, v787787-fg, v787788-fg, v787789-fg, v787790-fg, v787791-fg, v787792-fg, v787793-fg, v787794-fg, v787795-fg, v787796-fg, v788455-fg, v788456-fg, v788457-fg, v788458-fg, v788459-fg, v789038-fg, v789039-fg, v789040-fg, v789041-fg, v789042-fg

Production: lsst:abe_cfht201011_06 D2-fr 1 visit


Production: lsst:abe_cfht201016_01 D2-fr 198 visits

v729990-fr, v729991-fr, v729992-fr, v729993-fr, v732754-fr, v732755-fr, v732757-fr, v732758-fr, v732759-fr, v733224-fr, v733225-fr, v733227-fr, v733228-fr, v733229-fr, v733230-fr, v733231-fr, v733232-fr, v733233-fr, v733234-fr, v733241-fr, v733435-fr, v733436-fr, v733437-fr, v733438-fr, v733439-fr, v739195-fr, v739196-fr, v739197-fr, v739198-fr, v739199-fr, v739518-fr, v739519-fr, v739520-fr, v739521-fr, v739522-fr, v739523-fr, v739789-fr, v739790-fr, v739791-fr, v739792-fr, v739793-fr, v741666-fr, v741667-fr, v741668-fr, v741669-fr, v741670-fr, v741998-fr, v741999-fr, v742000-fr, v742001-fr, v742002-fr, v742003-fr, v742004-fr, v742553-fr, v742554-fr, v742555-fr, v742556-fr, v742557-fr, v744543-fr, v744544-fr, v744545-fr, v744546-fr, v744547-fr, v745234-fr, v745235-fr, v745236-fr, v745237-fr, v745238-fr, v769606-fr, v769607-fr, v769608-fr, v769874-fr, v769875-fr, v769876-fr, v769877-fr, v769878-fr, v773536-fr, v773537-fr, v773538-fr, v773539-fr, v773540-fr, v774004-fr, v774005-fr, v774006-fr, v774007-fr, v774008-fr, v774635-fr, v774636-fr, v774638-fr, v774639-fr, v774640-fr, v774647-fr, v774648-fr, v775097-fr, v775098-fr, v775099-fr, v775100-fr, v775101-fr, v776251-fr, v776252-fr, v776253-fr, v776254-fr, v776255-fr, v776382-fr, v776383-fr, v776384-fr, v776385-fr, v776386-fr, v777136-fr, v777137-fr, v777138-fr, v777139-fr, v777140-fr, v777848-fr, v777849-fr, v777850-fr, v777851-fr, v777852-fr, v779940-fr, v779941-fr, v779942-fr, v779944-fr, v780335-fr, v780336-fr, v780337-fr, v780338-fr, v780339-fr, v780785-fr, v780786-fr, v780787-fr, v780811-fr, v780812-fr, v780813-fr, v780814-fr, v780815-fr, v781356-fr, v781357-fr, v781359-fr, v781360-fr, v781370-fr, v781371-fr, v781372-fr, v781373-fr, v781374-fr, v784233-fr, v784234-fr, v784235-fr, v784237-fr, v784238-fr, v784239-fr, v784240-fr, v784241-fr, v784476-fr, v784477-fr, v784478-fr, v784479-fr, v784480-fr, v787731-fr, v787732-fr, v787733-fr, v787734-fr, v787735-fr, v787737-fr, v787738-fr, v787739-fr, v787777-fr, v787778-fr, v787779-fr, v787780-fr, v787781-fr, v787782-fr, v787783-fr, v788052-fr, v788053-fr, v788054-fr, v788055-fr, v788056-fr, v788057-fr, v788555-fr, v788556-fr, v788557-fr, v788558-fr, v788559-fr, v789043-fr, v789044-fr, v789045-fr, v789046-fr, v789047-fr, v792245-fr, v792246-fr, v792248-fr, v792249-fr, v792250-fr, v793310-fr, v793311-fr, v793312-fr, v793313-fr, v793314-fr

Production: abe_cfht20100711_07 D2-fu 18 visits

v731047-fr, v731048-fr, v731049-fr, v731050-fr, v731051-fr, v731052-fr, v731053-fr, v774957-fr, v774958-fr, v774959-fr, v774960-fr, v774961-fr, v774962-fr, v774963-fr, v774964-fr, v774966-fr, v774967-fr, v774969-fr

Data Issues (not fixable)

For all visits in all productions, CCD3 is bad.

Data Issues (fixable)

Error exceptions, while processing a particular amp or ccd, should probably inhibit all subsequent processing of that particular datum. It appears that the CcdAssembly Exceptions taken due to the bad CCD 3 are ignored. All subsequent pipelines fail due to Missing Data. Such failures should prevent any processing in subsequent pipelines. For example, CcdAssembly has a FATAL error which is then manifested downstream in CrSplit, ImgChar and SFM.

In all productions run on ABE, there were spurious memory exception errors which primarily (I didn't check every single one so I can't say 'all') occurred in the JobDone stage when wrapping up the current pipeline's processing and clearing the clipboard.

Applications Issues


In CcdAssembly, there were 22 separate instances of the message: "unmasked nans/infs found in ccd exposure".

The instances follow. Notice that only 2 visits are impacted; all instances occurred in IsrCcdDefectStage?.

 ccdassembly_14/Slice0.log-LsstException: (1, '1 unmasked nans/infs found in ccd exposure: 8261165, CFHT 27')
 ccdassembly_14/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_18/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_20/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_22/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_26/Slice0.log-LsstException: (1, '1 unmasked nans/infs found in ccd exposure: 8261165, CFHT 27')
 ccdassembly_28/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_32/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_33/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_34/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_36/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_38/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_39/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_4/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_41/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_43/Slice0.log-LsstException: (1, '1 unmasked nans/infs found in ccd exposure: 8261165, CFHT 27')
 ccdassembly_44/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_47/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_6/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_7/Slice0.log-LsstException: (2, '2 unmasked nans/infs found in ccd exposure: 7432193, CFHT 17')
 ccdassembly_8/Slice0.log-LsstException: (1, '1 unmasked nans/infs found in ccd exposure: 8261165, CFHT 27')
 ccdassembly_9/Slice0.log-LsstException: (1, '1 unmasked nans/infs found in ccd exposure: 8261165, CFHT 27')

In ImgChar, there were 2 instances of "Failed to determine PSF"