Last modified 9 years ago Last modified on 06/21/2010 09:24:46 AM

DC3b PT1 Integration Status

This page is meant to be used to track the level of integration of the PT1 stage of DC3b

Please review and update the status on the following. Forward additional issues which should be tracked to completion.

Tickets associated with the "PT1 Production Run Start" milestone may be tracked here:

Pipeline Status

  • Cluster: Pipelines running through SFM; SourceAssoc run by hand #1298 (Ray).
  • Abe:
    • Abe filesystems still fragile.
    • CFHT:
      • SST pipelines, from ISR through SourceAssoc, are running on CFHT data.
      • Current CFHT calibration data doesn't support the entire CCHT raw dataset.
    • ImSim
      • SST pipelines, from ISR through SFM are running with ImSim data. Last SST to test is
      • calibration and some raw data now available.

Apps issues

The following are likely due to visits extending beyond the area of the input star catalog. AndyC is generating a list of acceptable visits (and eventually CCDs).

  • PSF failure on one CCD of 110:
    • Visit 85770111 raft 4,2 sensor 1,2
  • WCS failures on 17 ImSim CCDs of 110:
    • Visit 85656362
      • Raft 2,3 sensor 1,2
      • Raft 2,4 sensors 0,0 1,0 2,0
    • Visit 85712498
      • Raft 4,2 sensor 0,2
    • Visit 85748123
      • Raft 3,3 sensors 1,2 2,0
      • Raft 4,3 sensor 0,2
    • Visit 85751931
      • Raft 3,3 sensor 2,1
    • Visit 85769959
      • Raft 4,3 sensors 0,0 1,0
    • Visit 85770111
      • Raft 4,3 sensor 0,0
    • Visit 85801502
      • Raft 3,1 sensor 2,2
      • Raft 4,2 sensors 0,0 0,1 1,0 1,1
  • Photocal failures on 11 ImSim CCDs of 110:
    • Visit 85656362
      • Raft 2,4 sensor 0,1 (all matches have bad photometry)
      • Raft 2,4 sensor 1,1 (all matches have bad photometry)
      • Raft 2,4 sensor 2,1 (insufficient data points for order)
    • Visit 85712498
      • Raft 4,2 sensor 1,2 (insufficient data points for order)
    • Visit 85748123
      • Raft 3,4 sensor 1,0 (warning: slope of fitting function is not 1)
      • Raft 3,4 sensor 2,0 (insufficient data points for order)
    • Visit 85751931
      • Raft 4,3 sensor 0,1 (insufficient data points for order)
      • Raft 4,3 sensor 0,2 (insufficient data points for order)
    • Visit 85769959
      • Raft 4,2 sensor 1,2 (all matches have bad photometry)
      • Raft 4,3 sensor 0,1 (insufficient data points for order)
    • Visit 85770111
      • Raft 4,3 sensor 1,0 (insufficient data points for order)
  • CrReject parameter tuning for CFHT still a question.
  • Off-chip sources (RHL, Serge) (not a constraint to production start)
    • RHL wonders if solution necessary for PT1
  • Extremely large position error sources (RHL, Serge) (not a constraint to production start)
    • RHL checking this is an actual problem and wonders if solution necessary for PT1

Middleware issues

  • Abe plugins for orca #1012 (SRP).
  • JobOffice/butler integration for SourceAssoc sky tiles #1298 (Ray)
  • SDQA conversion to CSV for ingest. (not a constraint to production start)
  • Exposure table info not available for ingest (KTL) (not a constraint to production start)
  • WCS matches not persisted (KTL) (not a constraint to production start)

Data issues

  • No word on copy of calibration products to Abe project disk (David)
  • No word on copy of CFHT and ImSim raw focal planes to mass store and availability on Abe (Ray)
  • Chopping of CFHT calibration products underway (David, ACB)
  • 1534 CFHT focal planes chopped on lsst10 (u, g, r filters only).
  • 180 ImSim focal planes at least partially transferred to NCSA.
    • One sensor has bad TAI and MJD-OBS times.
    • Several sensors and one entire visit missing from SLAC data.
    • Some focal planes overlap edge of star catalog.
  • Need to build registries for complete sets of CFHT and ImSim raw images (KTL)
    • Registry built for 150 ImSim focal planes, more being added.
    • Registry built for CFHT focal planes.

Production Issues

  • Configure end-to-end production environment on lsst and ABE #1280 (Robyn)


Detailed Status of Individual Pipelines




Simple differencing causes dipoles due to as-yet-unexplained movement of sources between snaps. (ImSim team thinks this may be real, but there is a lot of push-back and further verification is underway.)

We will run background subtraction and cosmic ray rejection on snap 0 only.


WCS verification numbers look bogus. (Inputs to verification look fine. Not a constraint to production.)

meas_astrom uses degrees internally for both catalogs and source lists.



Attribute computation may need to produce inverse variance weighted means rather than unweighted means. Implementation to follow initial investigation of SourceAssoc? performance.


Blackboard / JobOffice

  • How are the various forms of external input data (raw images, calibration files, catalogs, etc) being moved into position on ABE?
    • Calibration data will be kept on spinning disk permanently
    • For PT1, we will have a rudimentary script run before initiating production that will restore raw images and intermediate data from MSS. What is the status of this script?

Orca / Harness

  • Orca's Abe plug-in is still being worked on; however, the mechanical issues have been solved.

data access and other middleware support

IOStages and Formatters

Data Butler

(Longer-term, need to refactor common code out of mappers into butlerUtils.)

Status of PT1 DR-Production as a Whole

For the disk layout expected by PT1 software for the datasets, see:

Another helpful document describing the input/output/trigger of pipelines is at:

Test Datasets

> 1. Test subset of raw ImSim images.

Full focal planes in two snaps:

Removed to make room for other images ~ 21 June 2010

Circa: 21 June 2010

One amp in two snaps:

> 2. ImSim calibration images.

Raw full focal planes in two snaps that need to be processed:

Processed calibration images.

One amp:

> 3. Test subset of raw CFHT images (amplifier-chopped).

Chopped full focal planes:

> 4. CFHT calibration images (amplifier-chopped).

Chopped (limited runs so far):

Some additional notes:

> > 8. Intermediates/outputs computed from ImSim images.
> $AFWDATA_DIR/ImSim/postISR/v85751839-fr/s[01]/R23-S11-C*.fits
> $AFWDATA_DIR/ImSim/postISRCCD/v85751839-fr/s[01]/R23-S11.fits

        These will also be obsoleted soon.

Preparation of data collections


For more detailed information on the datasets, see

New TABLES showing what and where for the various input datasets; please fill-in

Data Set.................Dest loc...............%Lead
Raw Images lsst:/lsst/DC3/data/obstest/CFHTLS/D1/raw DG
....test lsst:/lsst/DC3/data/obstest/CFHTLS/D2/raw 15 images DG
lsst:/lsst/DC3/data/obstest/CFHTLS/D3/raw 3 images DG
lsst:/lsst/DC3/data/obstest/CFHTLS/D4/raw DG
....transient lsst10:/usr/data/mysql2/CFHTDeep/D{1-4}/raw DG
....mass storemss:/UROOT/projects/eiw/lsst/repos/obs/CFHTLS/D1/raw ? DG
mss:/UROOT/projects/eiw/lsst/repos/obs/CFHTLS/D2/raw ? DG
mss:..../D3/raw DG 0
mss:/UROOT/projects/eiw/lsst/repos/obs/CFHTLS/D4/raw ? DG
Calib Fileslsst:/lsst/DC3/data/obstest/CFHTLS/calib/{bias flat} DG
....transient lsst10: /usr/data/mysql1/CFHT/calib/{bias flat } final set AB
....ABE abe:/cfs/projects/lsst/DC3/data/obs/CFHTLS/calib/{bias flat} 20% DG
....mass storemss:/UROOT/projects/eiw/lsst/repos/obs/CFHTLS/calib/{bias flat} 10% DG
Stds Catalogslsst:/lsst/DC3/data/astrometry_net_data/cfhtlsDeep/ 100% FG/TSA
....ABE abe:....../astrometry_net_data/cfhtlsDeep/ 100% RP


Data Set.................Dest loc...............%Lead
Raw Images (1,2) lsst10:/usr/data/mysql2/Imsim/ AC+
lsst:/lsst/DC3/data/afwdata/trunk/ImSim/ 2 img + calib AC+
Calib Files(2) lsst10:/usr/data/mysql2/ImsimCalib/{bias dark flat qe_maps} 100% AC+
Std Catalogslsst:/lsst/DC3/data/astrometry_net_data/imsim 100% FG/TSA

Note 1: Raw Images w full FOV and including PSF & ISR

Note 2: all ImSim data as of April 7 has been moved to: lsst10:

Data Set.................Dest loc...............%Lead
Raw Images DG
Calib Fileslsst10:/usr/data/mysql2/CFHTWide/W3/raw/v845493-fi/s00 AB
lsst10:/usr/data/mysql2/CFHTCalib/[bias, flat, fringe]/v06Am03* AB
Stds Catalog FG/TSA

Generate/Assemble? Raw Image Data Acquire/Create? exposures to process (DS*)

Data Set Comments Responsible Status
PT1 CFHTLS Inventory NCSA copy of CFHTLS; acquire full Deep. 30% of Deep images resident @NCSA, ready for chunking. DG/DS/AB In Work

Note: Chunked CFHT-LS data must be migrated to mass store.

Running production

Data quality analysis / validation

Additional Issues