wiki:DC3bPT1IntegrationStatus
Last modified 9 years ago Last modified on 06/21/2010 09:24:46 AM

DC3b PT1 Integration Status

This page is meant to be used to track the level of integration of the PT1 stage of DC3b

Please review and update the status on the following. Forward additional issues which should be tracked to completion.

Tickets associated with the "PT1 Production Run Start" milestone may be tracked here: http://dev.lsstcorp.org/trac/query?group=status&milestone=PT1+Production+Run+Start

Pipeline Status

  • Cluster: Pipelines running through SFM; SourceAssoc run by hand #1298 (Ray).
  • Abe:
    • Abe filesystems still fragile.
    • CFHT:
      • SST pipelines, from ISR through SourceAssoc, are running on CFHT data.
      • Current CFHT calibration data doesn't support the entire CCHT raw dataset.
    • ImSim
      • SST pipelines, from ISR through SFM are running with ImSim data. Last SST to test is SrcAssoc_ImSim.py.
      • calibration and some raw data now available.

Apps issues

The following are likely due to visits extending beyond the area of the input star catalog. AndyC is generating a list of acceptable visits (and eventually CCDs).

  • PSF failure on one CCD of 110:
    • Visit 85770111 raft 4,2 sensor 1,2
  • WCS failures on 17 ImSim CCDs of 110:
    • Visit 85656362
      • Raft 2,3 sensor 1,2
      • Raft 2,4 sensors 0,0 1,0 2,0
    • Visit 85712498
      • Raft 4,2 sensor 0,2
    • Visit 85748123
      • Raft 3,3 sensors 1,2 2,0
      • Raft 4,3 sensor 0,2
    • Visit 85751931
      • Raft 3,3 sensor 2,1
    • Visit 85769959
      • Raft 4,3 sensors 0,0 1,0
    • Visit 85770111
      • Raft 4,3 sensor 0,0
    • Visit 85801502
      • Raft 3,1 sensor 2,2
      • Raft 4,2 sensors 0,0 0,1 1,0 1,1
  • Photocal failures on 11 ImSim CCDs of 110:
    • Visit 85656362
      • Raft 2,4 sensor 0,1 (all matches have bad photometry)
      • Raft 2,4 sensor 1,1 (all matches have bad photometry)
      • Raft 2,4 sensor 2,1 (insufficient data points for order)
    • Visit 85712498
      • Raft 4,2 sensor 1,2 (insufficient data points for order)
    • Visit 85748123
      • Raft 3,4 sensor 1,0 (warning: slope of fitting function is not 1)
      • Raft 3,4 sensor 2,0 (insufficient data points for order)
    • Visit 85751931
      • Raft 4,3 sensor 0,1 (insufficient data points for order)
      • Raft 4,3 sensor 0,2 (insufficient data points for order)
    • Visit 85769959
      • Raft 4,2 sensor 1,2 (all matches have bad photometry)
      • Raft 4,3 sensor 0,1 (insufficient data points for order)
    • Visit 85770111
      • Raft 4,3 sensor 1,0 (insufficient data points for order)
  • CrReject parameter tuning for CFHT still a question.
  • Off-chip sources (RHL, Serge) (not a constraint to production start)
    • RHL wonders if solution necessary for PT1
  • Extremely large position error sources (RHL, Serge) (not a constraint to production start)
    • RHL checking this is an actual problem and wonders if solution necessary for PT1

Middleware issues

  • Abe plugins for orca #1012 (SRP).
  • JobOffice/butler integration for SourceAssoc sky tiles #1298 (Ray)
  • SDQA conversion to CSV for ingest. (not a constraint to production start)
  • Exposure table info not available for ingest (KTL) (not a constraint to production start)
  • WCS matches not persisted (KTL) (not a constraint to production start)

Data issues

  • No word on copy of calibration products to Abe project disk (David)
  • No word on copy of CFHT and ImSim raw focal planes to mass store and availability on Abe (Ray)
  • Chopping of CFHT calibration products underway (David, ACB)
  • 1534 CFHT focal planes chopped on lsst10 (u, g, r filters only).
  • 180 ImSim focal planes at least partially transferred to NCSA.
    • One sensor has bad TAI and MJD-OBS times.
    • Several sensors and one entire visit missing from SLAC data.
    • Some focal planes overlap edge of star catalog.
  • Need to build registries for complete sets of CFHT and ImSim raw images (KTL)
    • Registry built for 150 ImSim focal planes, more being added.
    • Registry built for CFHT focal planes.

Production Issues

  • Configure end-to-end production environment on lsst and ABE #1280 (Robyn)

Appendix

Detailed Status of Individual Pipelines

ISR

CcdAssembly

CrSplit

Simple differencing causes dipoles due to as-yet-unexplained movement of sources between snaps. (ImSim team thinks this may be real, but there is a lot of push-back and further verification is underway.)

We will run background subtraction and cosmic ray rejection on snap 0 only.

ImageChar

WCS verification numbers look bogus. (Inputs to verification look fine. Not a constraint to production.)

meas_astrom uses degrees internally for both catalogs and source lists.

SFM

SourceAssoc

Attribute computation may need to produce inverse variance weighted means rather than unweighted means. Implementation to follow initial investigation of SourceAssoc? performance.

orchestration

Blackboard / JobOffice

  • How are the various forms of external input data (raw images, calibration files, catalogs, etc) being moved into position on ABE?
    • Calibration data will be kept on spinning disk permanently
    • For PT1, we will have a rudimentary script run before initiating production that will restore raw images and intermediate data from MSS. What is the status of this script?

Orca / Harness

  • Orca's Abe plug-in is still being worked on; however, the mechanical issues have been solved.

data access and other middleware support

IOStages and Formatters

Data Butler

(Longer-term, need to refactor common code out of mappers into butlerUtils.)


Status of PT1 DR-Production as a Whole

For the disk layout expected by PT1 software for the datasets, see: http://dev.lsstcorp.org/trac/wiki/DC3bDataOrganization

Another helpful document describing the input/output/trigger of pipelines is at: http://dev.lsstcorp.org/trac/attachment/wiki/DC3bMiddlewareTasks/DC3bOrchestration/datadependencies.txt

Test Datasets

> 1. Test subset of raw ImSim images.

Full focal planes in two snaps:

Removed to make room for other images ~ 21 June 2010
/lsst/DC3/data/obstest/ImSim/raw/v85470982-fy/E*/R*/S*/imsim_85470982_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85471048-fy/E*/R*/S*/imsim_85471048_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85656362-fy/E*/R*/S*/imsim_85656362_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85801502-fy/E*/R*/S*/imsim_85801502_R*_S*_C*_E*.fits.gz

Circa: 21 June 2010
/lsst/DC3/data/obstest/ImSim/raw/v85501867-fi/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85501910-fz/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85501933-fi/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85501973-fz/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85502008-fi/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85502068-fi/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85505200-fi/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85563138-fz/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85563260-fz/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85563293-fz/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85563478-fz/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85563716-fz/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85563814-fz/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85563848-fz/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz
/lsst/DC3/data/obstest/ImSim/raw/v85563945-fz/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz


One amp in two snaps:
$AFWDATA_DIR/ImSim/imsim_85751839_R23_S11_C00_E00[01].fits.gz

> 2. ImSim calibration images.

Raw full focal planes in two snaps that need to be processed:
/lsst/DC3/data/obstest/ImSim/raw{bias,dark,flat}/v*/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz

Processed calibration images.
/lsst/DC3/data/obstest/ImSim/{bias,dark,flat}/v*/E*/R*/S*/imsim_*_R*_S*_C*_E*.fits.gz

One amp:
$AFWDATA_DIR/ImSim/{bias,dark,flat_r}/imsim_*_R23_S11_C00_E000.fits

> 3. Test subset of raw CFHT images (amplifier-chopped).

Chopped full focal planes:
/lsst/DC3/data/obstest/CFHTLS/D2/raw/
    v787650-fi/s00/c*-a*.fits
    v787796-fg/s00/c*-a*.fits
    v788455-fg/s00/c*-a*.fits
    v788556-fr/s00/c*-a*.fits
    v788559-fr/s00/c*-a*.fits
    v787731-fr/s00/c*-a*.fits
    v788033-fi/s00/c*-a*.fits
    v788548-fi/s00/c*-a*.fits
    v788557-fr/s00/c*-a*.fits
    v789033-fi/s00/c*-a*.fits
    v787784-fg/s00/c*-a*.fits
    v788052-fr/s00/c*-a*.fits
    v788555-fr/s00/c*-a*.fits
    v788558-fr/s00/c*-a*.fits
    v793310-fr/s00/c*-a*.fits
/lsst/DC3/data/obstest/CFHTLS/D3/raw/
    v788965-fr/s00/c*-a*.fits
    v792170-fi/s00/c*-a*.fits
    v792933-fg/s00/c*-a*.fits

> 4. CFHT calibration images (amplifier-chopped).

Chopped (limited runs so far):
/lsst/DC3/data/obstest/CFHTLS/calib/{bias,flat,fringe}/v05Am0[34]*/c*-a*.fits


Some additional notes:

> > 8. Intermediates/outputs computed from ImSim images.
> 
> $AFWDATA_DIR/ImSim/postISR/v85751839-fr/s[01]/R23-S11-C*.fits
> $AFWDATA_DIR/ImSim/postISRCCD/v85751839-fr/s[01]/R23-S11.fits

        These will also be obsoleted soon.

Preparation of data collections

CFHT-LS

For more detailed information on the datasets, see http://dev.lsstcorp.org/trac/wiki/DC3bCFHTLSInputData

New TABLES showing what and where for the various input datasets; please fill-in

CFHT-LS Deep
Data Set.................Dest loc...............%Lead
Raw Images lsst:/lsst/DC3/data/obstest/CFHTLS/D1/raw DG
....test lsst:/lsst/DC3/data/obstest/CFHTLS/D2/raw 15 images DG
lsst:/lsst/DC3/data/obstest/CFHTLS/D3/raw 3 images DG
lsst:/lsst/DC3/data/obstest/CFHTLS/D4/raw DG
....transient lsst10:/usr/data/mysql2/CFHTDeep/D{1-4}/raw DG
....mass storemss:/UROOT/projects/eiw/lsst/repos/obs/CFHTLS/D1/raw ? DG
mss:/UROOT/projects/eiw/lsst/repos/obs/CFHTLS/D2/raw ? DG
mss:..../D3/raw DG 0
mss:/UROOT/projects/eiw/lsst/repos/obs/CFHTLS/D4/raw ? DG
Calib Fileslsst:/lsst/DC3/data/obstest/CFHTLS/calib/{bias flat} DG
....transient lsst10: /usr/data/mysql1/CFHT/calib/{bias flat } final set AB
....ABE abe:/cfs/projects/lsst/DC3/data/obs/CFHTLS/calib/{bias flat} 20% DG
....mass storemss:/UROOT/projects/eiw/lsst/repos/obs/CFHTLS/calib/{bias flat} 10% DG
Stds Catalogslsst:/lsst/DC3/data/astrometry_net_data/cfhtlsDeep/ 100% FG/TSA
....ABE abe:....../astrometry_net_data/cfhtlsDeep/ 100% RP

ImSim

Data Set.................Dest loc...............%Lead
Raw Images (1,2) lsst10:/usr/data/mysql2/Imsim/ AC+
lsst:/lsst/DC3/data/afwdata/trunk/ImSim/ 2 img + calib AC+
Calib Files(2) lsst10:/usr/data/mysql2/ImsimCalib/{bias dark flat qe_maps} 100% AC+
Std Catalogslsst:/lsst/DC3/data/astrometry_net_data/imsim 100% FG/TSA

Note 1: Raw Images w full FOV and including PSF & ISR

Note 2: all ImSim data as of April 7 has been moved to: lsst10:

CFHT-LS Wide
Data Set.................Dest loc...............%Lead
Raw Images DG
Calib Fileslsst10:/usr/data/mysql2/CFHTWide/W3/raw/v845493-fi/s00 AB
lsst10:/usr/data/mysql2/CFHTCalib/[bias, flat, fringe]/v06Am03* AB
Stds Catalog FG/TSA





Generate/Assemble? Raw Image Data Acquire/Create? exposures to process (DS*)

Data Set Comments Responsible Status
PT1 CFHTLS Inventory NCSA copy of CFHTLS; acquire full Deep. 30% of Deep images resident @NCSA, ready for chunking. DG/DS/AB In Work

Note: Chunked CFHT-LS data must be migrated to mass store.

Running production

Data quality analysis / validation

Additional Issues