wiki:DC3bPT1.2ProductionRunStatus
Last modified 8 years ago Last modified on 08/01/2011 09:14:26 AM

DC3b PT1.2 Production Run Status

This document lists runs of the DC3b PT1.2 Release Production pipelines

Refer to DC3bPT12ProcessedVisits for more detailed information on which visits were processed by which run(s).

PT1.2 DC3b 3000 series

RunID Collection # of Visits Started Run On Original Product Directory Comments Execution Time Backup
Grand Total 449 / 449 visits complete A configuraton of 95 workers on 16 nodes was used for all runs
ingestProcessed on all runs is complete
pt12prod_im3001 ImSim 40 (7556/7560 CCDs) 23 July 2011 Lonestar Data on Abe/Lincoln? No reported errors; 4 silent errors; 3:22:48
pt12prod_im3002 ImSim 40 (7541/7560 CCDs) 23 July 2011 Lonestar Data on Abe/Lincoln? 14 errors reported in icPsfDetermination Stage;
details follow in box below
3:22:49
pt12prod_im3005 ImSim 40 (7556/7560 CCDs) 24 July 2011 Lonestar Data on Abe/Lincoln? No reported errors; 4 silent errors; 4:41:59
pt12prod_im3006 ImSim 40 (7552/7560 CCDs) 24 July 2011 Lonestar Data on Abe/Lincoln? 1 reported error; 7 silent errors; 3:34:10
pt12prod_im3007 ImSim 40 (7550/7560 CCDs) 25 July 2011 Lonestar Data on Abe/Lincoln? 4 reported error; 6 silent errors; 3:35:54
pt12prod_im3008 ImSim 40 (7554/7560 CCDs) 25 July 2011 Lonestar Data on Abe/Lincoln? No reported errors; 6 silent errors; 3:46:52
pt12prod_im3009 ImSim 40 (7557/7560 CCDs) 26 July 2011 Lonestar Data on Abe/Lincoln? No reported errors; 3 silent errors; 3:23:38
pt12prod_im3010 ImSim 40 (7556/7560 CCDs) 26 July 2011 Lonestar Data on Abe/Lincoln? No reported errors; 4 silent errors; 3:24:02
pt12prod_im3011 ImSim 40 (7554/7560 CCDs) 27 July 2011 Lonestar Data on Abe/Lincoln? 4 reported errors; 2 silent errors; 3:21:03
pt12prod_im3013 ImSim 40 (7556/7560 CCDs) 28 July 2011 Lonestar Data on Abe/Lincoln? No reported errors; 4 silent errors;
A network outage between TACC and NCSA caused us to lose a few hours in the 3000 timetable, but connectivity is restored.
pt12prod_im3014 ImSim 40 (7552/7560 CCDs) 29 July 2011 Lonestar Data on Abe/Lincoln? No reported errors; 8 silent errors;
pt12prod_im3015 ImSim 9 (1701/1701 CCDs) 29 July 2011 Lonestar Data on Abe/Lincoln? No reported errors; no silent errors;


Source Association 3000 series

221/221 Sky Tiles are complete.
The ingest for Source Association (without reference matching) is complete (database: rplante_PT1_2_u_pt12prod_im3000).
The reference matching calculation for Source Association is done.


im3001 failed sensors

Four pipelines died without exceptions:

worker id log id visit raft sensor failed stage
PT1Pipe_28 6499767 886736181 2,3 1,2 icRemeasure
PT1Pipe_58 6498785 886736181 0,1 2,0 sfmSourceMeasure
PT1Pipe_61 7290678 886946741 1,2 1,2 icRemeasure
PT1Pipe_75 5180058 887676511 1,2 1,0 icRemeasure

im3002 failed sensors

The following 14 sensors caused exceptions in PSF determination or elsewhere (traceback messages below):

worker id log id visit raft sensor message
PT1Pipe_49 574361 887252571 1,4 0,2 0: Message: You only have 2 eigen images (you asked for 3)
PT1Pipe_23 614883 887252571 3,2 2,2 0: Message: You only have 2 eigen images (you asked for 3)
PT1Pipe_49 628943 887252571 3,2 0,0 0: Message: Please provide at least one Image for me to update
PT1Pipe_63 692560 887252571 4,2 2,0 0: Message: Please provide at least one Image for me to update
PT1Pipe_17 699606 887252571 1,2 0,0 0: Message: You only have 2 eigen images (you asked for 3)
PT1Pipe_28 718246 887252571 1,1 2,2 0: Message: You only have 2 eigen images (you asked for 3)
PT1Pipe_76 730400 887252571 4,1 2,1 0: Message: You only have 1 eigen images (you asked for 3)
PT1Pipe_19 731691 887252571 4,1 0,0 0: Message: You only have 2 eigen images (you asked for 3)
PT1Pipe_20 4381258 886891941 2,4 2,0 IndexError: invalid index
PT1Pipe_74 7850813 887252551 3,4 0,2 0: Message: Please provide at least one Image for me to update
PT1Pipe_49 7872127 887252551 0,3 1,2 0: Message: Please provide at least one Image for me to update
PT1Pipe_36 7881017 887252551 0,3 1,1 0: Message: You only have 1 eigen images (you asked for 3)
PT1Pipe_68 8011494 887252551 3,1 2,1 0: Message: You only have 2 eigen images (you asked for 3)
PT1Pipe_30 8396820 885594721 1,2 1,0 0: Message: Please provide at least one Image for me to update

In addition, five pipelines died without exceptions:

worker id log id visit raft sensor failed stage
PT1Pipe_41 5892787 886948051 3,0 1,1 icRemeasure
PT1Pipe_47 5285719 886947111 1,4 0,2 icRemeasure
PT1Pipe_48 8464949 885594721 2,1 1,1 icRemeasure
PT1Pipe_64 3891562 886862911 3,4 0,0 icRemeasure
PT1Pipe_84 6054970 886998681 1,2 2,1 icRemeasure

im3005 failed sensors

Four pipelines died without exceptions:

worker id log id visit raft sensor failed stage
PT1Pipe_15 8522257 886894861 3,0 1,2 icSourceMeasure
PT1Pipe_17 6468552 886469101 0,3 0,2 icRemeasure
PT1Pipe_6 7821599 886735471 4,2 1,0 icRemeasure
PT1Pipe_83 4192557 888085351 3,0 1,1 icRemeasure

im3006 failed sensors

The following sensor caused an exception in PSF determination (traceback message below):

worker id log id visit raft sensor message
PT1Pipe_65 9022137 888082761 1,1 0,0 RuntimeError: No candidate PSF sources

Seven pipelines died without exceptions:

worker id log id visit raft sensor failed stage
PT1Pipe_16 4090733 886238041 4,3 1,1 icRemeasure
PT1Pipe_24 2839534 886294821 1,3 2,2 icRemeasure
PT1Pipe_31 628619 887252591 1,2 1,2 icRemeasure
PT1Pipe_42 4607587 886469601 2,3 1,1 icRemeasure
PT1Pipe_58 2722161 886294821 4,3 2,2 sfmSourceMeasure
PT1Pipe_85 1175015 887676141 1,3 2,2 icRemeasure
PT1Pipe_95 6448269 885450721 1,0 2,0 icRemeasure

im3007 failed sensors

The following sensors caused exceptions in PSF determination (traceback messages below):

worker id log id visit raft sensor message
PT1Pipe_84 6216446 886257211 0,3 1,2 0: Message: You only have 2 eigen images (you asked for 3)
PT1Pipe_37 6920063 886915171 4,1 2,1 RuntimeError: No candidate PSF sources
PT1Pipe_54 7612638 887252941 3,4 1,2 RuntimeError: No candidate PSF sources
PT1Pipe_74 7636473 887252941 4,3 2,1 RuntimeError: No candidate PSF sources

Six pipelines died without exceptions:

worker id log id visit raft sensor failed stage
PT1Pipe_27 4742024 886468841 2,0 0,2 icRemeasure
PT1Pipe_34 2540786 887476271 1,2 2,1 icRemeasure
PT1Pipe_63 3856835 886237731 4,1 2,2 icRemeasure
PT1Pipe_75 5800782 886236101 2,0 1,2 icRemeasure
PT1Pipe_76 2725212 887477741 1,4 0,2 icRemeasure
PT1Pipe_90 4723534 886468841 1,1 2,0 icRemeasure

im3008 failed sensors

Six pipelines died without exceptions:

worker id log id visit raft sensor failed stage
PT1Pipe_15 1448741 888138611 1,0 0,0 icRemeasure
PT1Pipe_3 2921524 886893801 0,3 0,1 icRemeasure
PT1Pipe_33 8578892 887477511 4,3 1,1 icRemeasure
PT1Pipe_47 9391200 885449191 2,2 1,2 icRemeasure
PT1Pipe_5 3643859 887755541 3,3 1,2 icRemeasure
PT1Pipe_57 5174373 886264391 1,3 1,2 icRemeasure

im3009 failed sensors

Three pipelines died without exceptions:

worker id log id visit raft sensor failed stage
PT1Pipe_13 4377442 888117910 2,3 0,2 icRemeasure
PT1Pipe_46 6739456 887000460 3,4 2,1 icRemeasure
PT1Pipe_6 3706930 887282620 2,2 1,0 icRemeasure

im3010 failed sensors

Four pipelines died without exceptions:

worker id log id visit raft sensor failed stage
PT1Pipe_34 7088672 888116880 1,1 1,0 icRemeasure
PT1Pipe_41 7690244 888160970 3,3 2,2 icRemeasure
PT1Pipe_75 769233 887889240 1,1 2,0 icRemeasure
PT1Pipe_95 621987 887889240 4,2 0,1 sfmSourceMeasure

im3011 failed sensors

The following sensors caused exceptions:

worker id log id visit raft sensor message
PT1Pipe_11 363125 888381700 1,3 0,1 0: Message: Please provide at least one Image for me to update
PT1Pipe_88 8081133 887238700 2,4 0,2 0: Message: Please provide at least one Image for me to update
PT1Pipe_76 8105696 887238700 0,3 0,0 0: Message: Please provide at least one Image for me to update
PT1Pipe_81 8211716 887238700 3,2 1,1 0: Message: You only have 2 eigen images (you asked for 3)

Two pipelines died without exceptions:

worker id log id visit raft sensor failed stage
PT1Pipe_16 1895862 887030150 3,0 0,2 icRemeasure
PT1Pipe_6 245446 888380680 3,4 1,0 icRemeasure

Traceback messages

Sample Exception/Errors in  pt12prod_im3002 :

visit 887252571 raft 1,4 sensor 0,2
|  574361 | PT1Pipe_49 | Traceback (most recent call last):
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/Slice.py", line 575, in tryProcess
    stageObject.applyProcess()
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/stage.py", line 353, in applyProcess
    self.process(clipboard)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_pipeline/4.4.0.1/python/lsst/meas/pipeline/psfDeterminationStage.py", line 72, in process
    psf, psfCellSet = self.psfDeterminer.determinePsf(exposure, psfCandidateList, sdqaRatings)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/pcaPsfDeterminer.py", line 352, in determinePsf
    psf, eigenValues, fitChi2 = self._fitPsf(exposure, psfCellSet)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/pcaPsfDeterminer.py", line 63, in _fitPsf
    self._kernelSize, self._nStarPerCell, self._constantWeight)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/algorithmsLib.py", line 975, in createKernelFromPsfCandidates
    return _algorithmsLib.createKernelFromPsfCandidates(*args)
LsstCppException: 0: lsst::pex::exceptions::LengthErrorException thrown at src/image/ImagePca.cc:493 in double lsst::afw::image::<unnamed>::do_updateBadPixels(const lsst::afw::image::detail::MaskedImage_tag&, const typename lsst::afw::image::ImagePca<MaskedImageT>::ImageList&, const std::vector<double, std::allocator<double> >&, const typename lsst::afw::image::ImagePca<MaskedImageT>::ImageList&, long unsigned int, int) [with ImageT = lsst::afw::image::MaskedImage<float, short unsigned int, float>]
0: Message: You only have 2 eigen images (you asked for 3)

 |

visit 887252571 raft 3,2 sensor 0,0
|  628943 | PT1Pipe_49 | Traceback (most recent call last):
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/Slice.py", line 575, in tryProcess
    stageObject.applyProcess()
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/stage.py", line 353, in applyProcess
    self.process(clipboard)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_pipeline/4.4.0.1/python/lsst/meas/pipeline/psfDeterminationStage.py", line 72, in process
    psf, psfCellSet = self.psfDeterminer.determinePsf(exposure, psfCandidateList, sdqaRatings)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/pcaPsfDeterminer.py", line 352, in determinePsf
    psf, eigenValues, fitChi2 = self._fitPsf(exposure, psfCellSet)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/pcaPsfDeterminer.py", line 63, in _fitPsf
    self._kernelSize, self._nStarPerCell, self._constantWeight)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/algorithmsLib.py", line 975, in createKernelFromPsfCandidates
    return _algorithmsLib.createKernelFromPsfCandidates(*args)
LsstCppException: 0: lsst::pex::exceptions::LengthErrorException thrown at src/image/ImagePca.cc:425 in double lsst::afw::image::<unnamed>::do_updateBadPixels(const lsst::afw::image::detail::MaskedImage_tag&, const typename lsst::afw::image::ImagePca<MaskedImageT>::ImageList&, const std::vector<double, std::allocator<double> >&, const typename lsst::afw::image::ImagePca<MaskedImageT>::ImageList&, long unsigned int, int) [with ImageT = lsst::afw::image::MaskedImage<float, short unsigned int, float>]
0: Message: Please provide at least one Image for me to update

 |

visit 887252571 raft 4,1 sensor 2,1
|  730400 | PT1Pipe_76 | Traceback (most recent call last):
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/Slice.py", line 575, in tryProcess
    stageObject.applyProcess()
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/stage.py", line 353, in applyProcess
    self.process(clipboard)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_pipeline/4.4.0.1/python/lsst/meas/pipeline/psfDeterminationStage.py", line 72, in process
    psf, psfCellSet = self.psfDeterminer.determinePsf(exposure, psfCandidateList, sdqaRatings)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/pcaPsfDeterminer.py", line 352, in determinePsf
    psf, eigenValues, fitChi2 = self._fitPsf(exposure, psfCellSet)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/pcaPsfDeterminer.py", line 63, in _fitPsf
    self._kernelSize, self._nStarPerCell, self._constantWeight)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/algorithmsLib.py", line 975, in createKernelFromPsfCandidates
    return _algorithmsLib.createKernelFromPsfCandidates(*args)
LsstCppException: 0: lsst::pex::exceptions::LengthErrorException thrown at src/image/ImagePca.cc:493 in double lsst::afw::image::<unnamed>::do_updateBadPixels(const lsst::afw::image::detail::MaskedImage_tag&, const typename lsst::afw::image::ImagePca<MaskedImageT>::ImageList&, const std::vector<double, std::allocator<double> >&, const typename lsst::afw::image::ImagePca<MaskedImageT>::ImageList&, long unsigned int, int) [with ImageT = lsst::afw::image::MaskedImage<float, short unsigned int, float>]
0: Message: You only have 1 eigen images (you asked for 3)

visit 886891941 raft 2,4 sensor 2,0
| 4381258 | PT1Pipe_20 | Traceback (most recent call last):
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/Slice.py", line 575, in tryProcess
    stageObject.applyProcess()
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/stage.py", line 353, in applyProcess
    self.process(clipboard)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_pipeline/4.4.0.1/python/lsst/meas/pipeline/psfDeterminationStage.py", line 72, in process
    psf, psfCellSet = self.psfDeterminer.determinePsf(exposure, psfCandidateList, sdqaRatings)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/pcaPsfDeterminer.py", line 265, in determinePsf
    rms = residuals[:,k].std()
IndexError: invalid index


Exception/Errors in  pt12prod_im3006 :

| PT1Pipe_65 | 9022137 | Traceback (most recent call last):
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/Slice.py", line 575, in tryProcess
    stageObject.applyProcess()
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/stage.py", line 353, in applyProcess
    self.process(clipboard)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_pipeline/4.4.0.1/python/lsst/meas/pipeline/psfDeterminationStage.py", line 70, in process
    psfCandidateList = self.starSelector.selectStars(exposure, sourceSet)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/secondMomentStarSelector.py", line 96, in selectStars
    clumps = psfHist.getClumps(display=display)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/secondMomentStarSelector.py", line 197, in getClumps
    raise RuntimeError("No candidate PSF sources")
RuntimeError: No candidate PSF sources

Exception/Errors in  pt12prod_im3007 :


| PT1Pipe_37 | 6920063 | Traceback (most recent call last):
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/Slice.py", line 575, in tryProcess
    stageObject.applyProcess()
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/stage.py", line 353, in applyProcess
    self.process(clipboard)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_pipeline/4.4.0.1/python/lsst/meas/pipeline/psfDeterminationStage.py", line 70, in process
    psfCandidateList = self.starSelector.selectStars(exposure, sourceSet)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/secondMomentStarSelector.py", line 96, in selectStars
    clumps = psfHist.getClumps(display=display)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/secondMomentStarSelector.py", line 197, in getClumps
    raise RuntimeError("No candidate PSF sources")
RuntimeError: No candidate PSF sources
 |
| PT1Pipe_54 | 7612638 | Traceback (most recent call last):
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/Slice.py", line 575, in tryProcess
    stageObject.applyProcess()
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/stage.py", line 353, in applyProcess
    self.process(clipboard)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_pipeline/4.4.0.1/python/lsst/meas/pipeline/psfDeterminationStage.py", line 70, in process
    psfCandidateList = self.starSelector.selectStars(exposure, sourceSet)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/secondMomentStarSelector.py", line 96, in selectStars
    clumps = psfHist.getClumps(display=display)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/secondMomentStarSelector.py", line 197, in getClumps
    raise RuntimeError("No candidate PSF sources")
RuntimeError: No candidate PSF sources
 |
| PT1Pipe_74 | 7636473 | Traceback (most recent call last):
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/Slice.py", line 575, in tryProcess
    stageObject.applyProcess()
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/stage.py", line 353, in applyProcess
    self.process(clipboard)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_pipeline/4.4.0.1/python/lsst/meas/pipeline/psfDeterminationStage.py", line 70, in process
    psfCandidateList = self.starSelector.selectStars(exposure, sourceSet)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/secondMomentStarSelector.py", line 96, in selectStars
    clumps = psfHist.getClumps(display=display)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/secondMomentStarSelector.py", line 197, in getClumps
    raise RuntimeError("No candidate PSF sources")
RuntimeError: No candidate PSF sources
 |
| PT1Pipe_84 | 6216446 | Traceback (most recent call last):
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/Slice.py", line 575, in tryProcess
    stageObject.applyProcess()
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.4.0.1/python/lsst/pex/harness/stage.py", line 353, in applyProcess
    self.process(clipboard)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_pipeline/4.4.0.1/python/lsst/meas/pipeline/psfDeterminationStage.py", line 72, in process
    psf, psfCellSet = self.psfDeterminer.determinePsf(exposure, psfCandidateList, sdqaRatings)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/pcaPsfDeterminer.py", line 352, in determinePsf
    psf, eigenValues, fitChi2 = self._fitPsf(exposure, psfCellSet)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/pcaPsfDeterminer.py", line 63, in _fitPsf
    self._kernelSize, self._nStarPerCell, self._constantWeight)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.4.1.1/python/lsst/meas/algorithms/algorithmsLib.py", line 975, in createKernelFromPsfCandidates
    return _algorithmsLib.createKernelFromPsfCandidates(*args)
LsstCppException: 0: lsst::pex::exceptions::LengthErrorException thrown at src/image/ImagePca.cc:493 in double lsst::afw::image::<unnamed>::do_updateBadPixels(const lsst::afw::image::detail::MaskedImage_tag&, const typename lsst::afw::image::ImagePca<MaskedImageT>::ImageList&, const std::vector<double, std::allocator<double> >&, const typename lsst::afw::image::ImagePca<MaskedImageT>::ImageList&, long unsigned int, int) [with ImageT = lsst::afw::image::MaskedImage<float, short unsigned int, float>]
0: Message: You only have 2 eigen images (you asked for 3)

 |



PT1.2 DC3b 2000 series

RunID Collection # of Visits Started Run On Original Product Directory Comments Execution Time Backup
pt12prod_im2005 ImSim 10 (1887/1890 CCDs) 23 June 2011 Lonestar Data on Abe/Lincoln? 95 workers on 16 nodes 5:40:34*
pt12prod_im2006 ImSim 20 (3767/3780 CCDs) 23 June 2011 Lonestar Data on Abe/Lincoln? 95 workers on 16 nodes 1:50:43
pt12prod_im2007 ImSim 10 (1879/1890 CCDs) 23 June 2011 Lincoln Data on Abe/Lincoln? 15 workers on 4 nodes 8:57:08
pt12prod_im2008 ImSim 40 (7542/7560 CCDs) 23 June 2011 Lonestar Data on Abe/Lincoln? 189 workers on 32 nodes 1:51:15
pt12prod_im2009 ImSim 40 (7546/7560 CCDs) 24 June 2011 Lonestar Data on Abe/Lincoln? 189 workers on 32 nodes 1:51:08
pt12prod_im2010 ImSim 40 (7547/7560 CCDs) 24 June 2011 Lonestar Data on Abe/Lincoln? 189 workers on 32 nodes 1:52:08
pt12prod_im2011 ImSim 40 (7550/7560 CCDs) 24 June 2011 Lincoln Data on Abe/Lincoln? 79 workers on 16 nodes 8:07:21
pt12prod_im2014 ImSim 19 (3586/3591 CCDs) 25 June 2011 Lincoln Data on Abe/Lincoln? 79 workers on 16 nodes
one visit (out of 20) was not on disk
4:04:52
pt12prod_im2015 ImSim 20 (3768/3780 CCDs) 25 June 2011 Lonestar Data on Abe/Lincoln? 189 workers on 32 nodes 1:02:54
pt12prod_im2019 ImSim 40 (7552/7560 CCDs) 26 June 2011 Lonestar Data on Abe/Lincoln? 95 workers on 16 nodes 3:26:19
pt12prod_im2022 ImSim 1 (189/189 CCDs) 28 June 2011 Lincoln Data on Abe/Lincoln? The missing visit of pt12prod_im2014 1:00:36
pt12prod_im2024 ImSim 20 (3773/3780 CCDs) 29 June 2011 Lincoln Data on Abe/Lincoln? 79 workers on 16 nodes 4:38:07
pt12prod_im2025 ImSim 20 (3776/3780 CCDs) 30 June 2011 Lincoln Data on Abe/Lincoln? 79 workers on 16 nodes 4:13:54
pt12prod_im2029 ImSim 20 (3771/3780 CCDs) 10 July 2011 Lonestar Data on Abe/Lincoln? 95 workers on 16 nodes 1:44:01
pt12prod_im2030 ImSim 20 (3770/3780 CCDs) 10 July 2011 Lonestar Data on Abe/Lincoln? 95 workers on 16 nodes 1:51:01
pt12prod_im2031 ImSim 20 (3773/3780 CCDs) 11 July 2011 Lonestar Data on Abe/Lincoln? 95 workers on 16 nodes 1:39:47
pt12prod_im2032 ImSim 22 (4149/4158 CCDs) 12 July 2011 Lonestar Data on Abe/Lincoln? 95 workers on 16 nodes 1:56:18
pt12prod_im2033 ImSim 26 ( CCDs) 12 July 2011 Lonestar Data on Abe/Lincoln? 95 workers on 16 nodes 2:23:42

Note: execution time is from the start to the end of the logs and does not include time spent waiting in queues or file transfer time.

  • = run with daf_butlerUtils 4.4.0.0 and thus slow registry access.

Runs pt12prod_im2005 -- pt12prod_im2022 will comprise the first 280 visits in skytile order.

Initial runs prior to starting in skytile order (tail end visits)

pt12prod_im2002 ImSim 12 (2263/2268 CCDs) 22 June 2011 Lonestar Data on Abe/Lincoln? 95 workers on 16 nodes 3:27:27*
pt12prod_im2004 ImSim 9 (1698/1701 CCDs) 22 June 2011 Lonestar Data on Abe/Lincoln? 95 workers on 16 nodes 4:57:53*

Total so far: 64323 CCDs processed out of 64449, 0.2% failures.

Source Association

Source Association and the associated ingest are complete for the 221 skytiles. Results are currently at /cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im2000/SourceAssoc/results on Abe/Lincoln?.

PT1.2 DC3b 1000 series

RunID Collection # of Visits Started Run On Original Product Directory Comments Backup
pt12prod_im1015 ImSim 4 (v886469051-fz v886469580-fz
v886917261-fr v887001130-fr)
3 June 2011 Lonestar /lsst3/DC3/PT1_2/datarel-runs/pt12prod_im1015 47 workers on 8 nodes
pt12prod_im1017 ImSim 4 (v886735530-fz v886736300-fz
v887029100-fr v887030190-fr)
3 June 2011 Lonestar /lsst3/DC3/PT1_2/datarel-runs/pt12prod_im1017 47 workers on 8 nodes
pt12prod_im1020 ImSim 6 5 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1020
47 workers on 8 nodes;
Submitted 8 visits, 6 completed, next visit obtained numerous OverflowError: math range error iStage 280
pt12prod_im1029 ImSim 10 8 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1029
47 workers on 8 nodes;
No Y band visits
No reported errors.
pt12prod_im1031 ImSim 10 8 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1031
47 workers on 8 nodes;
No Y band visits from this point
No reported errors.
pt12prod_im1032 ImSim 10 9 June 2011 Lincoln Data on Abe
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1032
47 workers on 8 nodes;
No reported errors.
pt12prod_im1040 ImSim 5 9 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1040
5 visits completed
production halted during 6th
No errors reported; cause under investigation
pt12prod_im1048 ImSim 13 10 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1048
Submitted 14; 13 were completed.
The last visit did not complete on wall clock;
Three errors reported; details forthcoming
pt12prod_im1050 ImSim 11 11 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1050
47 workers on 8 nodes;
No reported errors.
pt12prod_im1052 ImSim 10 11 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1052
47 workers on 8 nodes;
Three total errors reported
pt12prod_im1053 ImSim 5 11 June 2011 Lincoln Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1053
No errors reported;
pt12prod_im1054 ImSim 10 12 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1054
47 workers on 8 nodes;
No errors reported but one CCD missing/not processed
pt12prod_im1055 ImSim 5 12 June 2011 Lincoln Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1055
No errors reported;
pt12prod_im1057 ImSim 6 12 June 2011 Lincoln Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1057
pt12prod_im1058 ImSim 2 12 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1058
2 visits processed;
run halted after Y band visit mistakenly included
pt12prod_im1060 ImSim 7 12 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1060
7 visits processed;
one CCD missing/not processed
pt12prod_im1061 ImSim 10 13 June 2011 Lonestar Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1061
No reported errors, but three CCDs missing/need to investigate
pt12prod_im1062 ImSim 3 15 June 2011 Lincoln Data on Abe at
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1062
Completed visits ImSim 131


Validation Run

pt12prod_im1028 ImSim 4 (v886469051-fz v886469580-fz
v886917261-fr v887001130-fr)
8 June 2011 Lincoln Data on Abe
/cfs/scratch/users/lsst/shelves/datarel-runs/pt12prod_im1028
47 workers on 8 nodes
Repeats pt12prod_im1015 on Lincoln

Observed Exception in  pt12prod_im1020 :

This appears to be specific to the visit "887305180":


harness.slice.visit.stage: Begin stage loop iteration iStage 280
  LOOPNUM: 26
  RUNID: pt12prod_im1020
  JobId: unknown
  stagename: icApCorrect
  DATE: 2011-06-06T09:35:42.902984
  workerid: PT1Pipe_13
  STAGEID: 280
  TIMESTAMP: 1307352976902984000
  LEVEL: 0
  SLICEID: 0
  PIPELINE: main-ImSim

harness.slice.ApertureCorrectionStage - parallel: Estimating Aperture Correction is in process
  RUNID: pt12prod_im1020
  JobId: unknown
  DATE: 2011-06-06T09:35:42.910593
  workerid: PT1Pipe_13
  TIMESTAMP: 1307352976910593000
  LEVEL: 0
  SLICEID: 0
  PIPELINE: main-ImSim

harness.slice.visit.stage.tryProcess FATAL: Traceback (most recent call last):
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.3.0.0/python/lsst/pex/harness/Slice.py", line 563, in tryProcess
    stageObject.applyProcess()
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/pex_harness/4.3.0.0/python/lsst/pex/harness/stage.py", line 353, in applyProcess
    self.process(clipboard)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_pipeline/4.3.1.2/python/lsst/meas/pipeline/apertureCorrectionStage.py", line 71, in process
    apCorrCtrl, log=self.log)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.3.2.0/python/lsst/meas/algorithms/apertureCorrection.py", line 371, in __init__
    self.fit = PolyFit2D(xList, yList, self.apCorrList, weights, poly)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.3.2.0/python/lsst/meas/algorithms/apertureCorrection.py", line 123, in __init__
    self.coeff, self.resid, self.rank, self.singval = self._fit(z, w, xTerms, yTerms, self.orderPairs)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/meas_algorithms/4.3.2.0/python/lsst/meas/algorithms/apertureCorrection.py", line 173, in _fit
    coeff, resid, rank, singval = linalg.lstsq(terms, w*z)
  File "/share1/projects/tg/lsst/lsst_home_445/Linux64/external/numpy/1.2.1/lib/python/numpy/linalg/linalg.py", line 1289, in lstsq
    nlvl = max( 0, int( math.log( float(min(m, n))/2. ) ) + 1 )
OverflowError: math range error