wiki:DC3bCFHTLSInputData
Last modified 9 years ago Last modified on 04/16/2010 10:22:01 AM

Input Data for DC3b Processing

For DC3b we will process an entire set of CFHT Legacy Survey Deep and Wide survey raw images, which were obtained in five bands: u*, g', r', i', z' . These data became public along with the release of the T0005 reduced data as processed by the TeraPix pipeline.

Raw Images

The following attributes for each field are taken from the CFHT-LS survey description.

Field RA Dec Extent N_Images Notes
Deep 1 (D1) 02:26:00 -04:30:00 1o x 1o 3151 Lies within W1
Deep 2 (D2) 10:00:29 +02:12:21 1o x 1o 2746 On the COSMOS/ACS survey field
Deep 3 (D3) 14:17:54 +52:30:31 1o x 1o 3528 Lies within W3
Deep 4 (D4) 22:15:31 -17:44:05 1o x 1o 2998 Around the quasar LBQS2212-17
Wide 1 (W1) 02:18:00 -07:00:00 8o x 9o 2855 On the XMM LSS field
Wide 2 (W2) 08:54:00 -04:15:00 7o x 7o 1145
Wide 3 (W3) 14:17:54 +54:30:31 7o x 7o 1956 On the Groth Strip
Wide 4 (W4) 22:13:18 +01:19:00 4o x 4o 974 On the VVDS 22h & UKIDSS DXS fields

Each image consists of:

  • 36 CCDs covering the central 0o.96 x 0o.94 of the focal plane, arranged in an array of 4 rows of 9 sensors
  • Each CCD has 2 amplifiers, and 2k x 4K photo-active pixels (2080 x 4644 pixels including virtual overscan)
  • The pixels are 13.5 um, with a spatial scale of 0.187 arcsec/pixel at image center
  • Pixels have linear response (within 1%) over the dynamic range of the A/D converter: 65536 ADU, or ~110,000 e- at the nominal gain. Full-well is ~150,000 e-
  • A full raw image consists of 378 Mpixels, and occupies 680 MB on disk for each full-frame (16 bit int) image

The raw data volume (without filtering on-sky calibration images) works out to 8.4 TB for the Deep fields and 4.7 TB for the Wide fields, uncompressed. The complete catalogs (including data outside the nominal date range for the T0005 release) can be found below:

To understand the meaning of the fields in these catalogs (many of which have multiple parameters), see the guide.

The above logs contain images that were obtained outside the interval of time that defines the T0005 stacks. They also contain a variety of images that were taken for some reason other than forming stacks of the fields, including shorter exposures for astrometric calibration with 2MASS and photometric calibration of brighter standards. A number of other short exposures were apparently taken to pre-survey some of the fields. The images to be excluded are those with the following properties:

  • For all fields:
    • Dates before 2003-May-23
    • Dates after 2008-Feb-16
    • Field_ID (third column in the log file) contains any of the sub-strings: ['astrom', 'pre', 'phot', 'zenith']
  • Wide fields only:
    • Exposure times < 500s
  • Deep fields only:
    • Exposure times < 180s

The above criteria yield give a grand total over all fields of about 15,654 raw images: 4954 Wide (3.4 TB) and 10700 Deep (7.2 TB). Note that the exposure times are not all identical even with these criteria, which may be useful for testing purposes. This list also includes images with poor photometric quality which will test how the pipelines handle them.

Raw Data Description

A fairly thorough description of the raw data may be found on the MegaCam web site. Below is a diagram from that website of the MegaCam focal plane array, where the numbers label the CCD sensors, and the letters (A, B) label the amplifiers. Small gaps between sensors are 70 pixels (13 arcsec), and the large gaps between rows are 425 pixels (80 arcsec). These data are re-formatted prior to input to the highly parallel LSST data processing pipelines. For DC3b, the chunk size for parallel processing will be identical to the size of a single MegaCam CCD amplifier.

A Performance Summary is available on the MegaCam web site.

Calibration Data

Calibration reference files, or "master de-trend" files in the CFHT parlance, can be obtained from a custom CADC web query form. After filtering out all but the latest versions of the de-trend files, there remain a total of 471 unique files. By type, they are:

Type Number Filter
Mask 5 Any
Bias 54 Any
Dark 56 Any
Fringe 94 i, i2, z
Flat 50 u
Flat 54 g
Flat 54 r
Flat 46 i
Flat 4 i2
Flat 54 z

For a total of 262 flats. All of the de-trend files have date ranges over which they are valid, so it will be imperative during the DC3b run to match the raw files to the correct de-trend files.

Calibration Notes

Useful references for CFHT-LS calibration include the following:

Among the key points to be noted for DC3b processing are the following:

  • Generally, the calibration reference files with the largest version number should be used. This is particularly important for the flats, since additional corrections (e.g., pixel scale and field illumination) have been folded in
  • The static bad pixel masks are the exception, in that different version numbers apply at different times. These tracked the cases where problematic video boards caused drop-outs of some CCDs or amplifiers over time.
  • The bad pixel masks as used for Elixir processing follow a different convention than that used for LSST: a value of zero indicates a bad pixel, and a value of 1 indicates a good pixel. For LSST this is reversed. Therefore the bad pixel masks for DC3b processing have been altered to reflect the LSST convention.
  • No information is available to support a cross-talk correction, so we will not make that correction in DC3b
  • The detectors have a linear response to ~0.1%, so no linearity correction is needed
  • The correction for dark current is very small, and was not often applied in the later years of the survey. Also, the dark master de-trend files do not appear to have been conditioned to remove the noise at extremely low levels. Thus, we have elected not to perform a dark correction for DC3b.

Issues to be Resolved

The following is a punch-list of issues to be resolved prior to DC3b processing:

  • Verify that valid calibration files exist for each and every raw input file to be processed

Attachments