Last modified 8 years ago Last modified on 07/21/2011 11:03:24 AM

LSST Data Challenge Handbook

The Data Challenge Handbook v1.1 revision is complete, and documents the Summer 2011 Data Release. Here is the latest galley proof: DCH_v1.1.

The following sections are being retained for now (rather than being reorganized or deleted) for historical reasons.

Development of the LSST Data Challenge 3 User Guide [History]

The following is a straw-man outline for an LSST Data Challenge 3 User Guide (tentatively called the Data Challenge Handbook-DCH), and some thoughts on how it can be brought to fruition. The purpose of a Data Challenge Handbook is to describe the origin, processing, and the format of the archived data products and catalog in enough detail for a scientist-user, who is not necessarily familiar with them, to make informed use of the data and supporting services for scientific analysis. This inaugural version will focus on the Simulation and CFHT-LS survey datasets that are being processed in DC3b to test the LSST Data Management system. The DCH will be loosely modelled on the NOAO Data Handbook ( in style and level of detail in the presentation, although the specific content is of course different. The emphasis will be to describe the data products and what techniques were used to produce them, while only summarizing things like the details of the DM system per se.

Since its primary audience is the Science Collaboration membership, who will be analyzing the output of DC3b and later productions, it will be pitched at scientists who are familiar with using survey data, but who may know little about LSST processing or products. The bulk of the DCH will be divided into parts describing the archive and retrieving data, the input survey data, a description of the data products, and the data processing pipeline. The DCH will not describe in depth any particular application that might be used for analysis, but will likely include some examples of exploring the data with popular applications such as... TBD (e.g., Aladin, TopCat, DS9, etc.). Pointers to on-line resources or software repositories will be especially helpful.

The content of the DCH will be provided by LSST scientists and software developers, either in the form of existing reports or papers (or updates to them), or through original material authored by LSST staff with appropriate expertise. The editor of the DCH will ensure that the needed material is found, updated, or solicited from appropriate authors; the editor may also contribute some of the narrative, tables, and figures, and will compile, edit, and format the final product. The DCH needs to be developed with the help of LSST Science staff, but it would probably benefit from input from some representatives of the intended target audience. The resulting Handbook should be reviewed by various internal experts on the subject matter and representative users; ongoing feedback from the broader user community after publication should be encouraged and incorporated as well in due course.

The DCH should be released in time to benefit the first external (to the LSST project) users, which means April-May 2010. It will take the form of a PDF document, complete with hyperlinks to external sources (such as relevant LSST Project documents and the LSST Science Book), and posted on the LSST website. At present it looks like MS-Word will be the authoring toolkit. This Handbook is anticipated to be one component of user training support materials.


Front Matter

Chapter 1: Introduction to LSST Data Products

1.1 Generic description of the Science Data Product

  • Processing levels
  • Images (see the path hierarchy)
    • These include input (raw) image, and outputs that are expected to be of interest to Science Collaboration members. See below for complete list.
  • Catalog & Tables
    • Source & Object Catalogs
      • [Moving Objects, Tranient]
  • Derivable products
    • light curves, etc.
  • Calibration reference files
    • Static Mask, Bias, Dark, Flat-field (one per filter), Fringe (i- and z-band only)
    • Astrometric & Photometric standards catalog
  • Simulation Database
  • Provenance and metadata, keywords, associations between data products

The following tables are a complete list of data products, only some of which will be described in the User Guide. Items in italic are input calibration reference products. Data products that will be described in the User Guide are marked "Yes" in the column in UG? while those marked "No" will not be described, nor will access by non-expert users be supported through the Science User Interface.

Image Product Availability in UG? DataPackage Brightness Units Size
Raw Image PT1 Yes Int16 ADU Amp (LSST or CFHT)
Post-ISR Image PT1 No Float32 Linearized ADU/s Amp (LSST or CFHT)
Post-ISR CCD Image PT1 No Float32 Linearized ADU/s CCD
Visit Image PT1 No Float32 Linearized ADU/s CCD
Calibrated Exposure PT1 Yes Float32 Linearized ADU/s CCD
PSF Matched Image PT2 No Float32 Linearized ADU/s CCD
Template Co-Addition PT2 Yes Float32 Linearized ADU/s sky tile (??)
Difference Image PT2 Yes Float32 Linearized ADU/s CCD
Masked Exposure PT2? Yes? Float32 electron/s CCD
Deep Detection Co-Addition PT2? Yes Float32 electron/s sky tile (??)
Bias Image PT1 Yes Float32 ADU Amp (LSST or CFHT)
Dark Image PT1 Yes Float32 ADU Amp (LSST or CFHT)
Flat-field Image PT1 Yes Float32 Normalized ADU Amp (LSST or CFHT)
Fringe Image PT1 Yes Float32 Arb. linear units Amp (LSST or CFHT)

Note: Size of Amp image is 1k x 4k for CFHT-LS and 512 x 2k for ImSim, not counting overscan pixels; CCD image is 2k x 4k for CFHT-LS and 4k x 4k for ImSim. All but Raw and calibration reference files are stored as MEFs: one extension each for the Science (Float32), Variance (Float), and Mask (Int16) image arrays.

Non-Image Products Availability in UG? Notes
SatDefects PT1 No List of Defects, at Amp level
Static Bad Pixels PT1 Yes Part of CameraGeom at Amp level
PSF PT2 Yes At CCD level

Catalog Product Availability in UG? Notes
SFM Source List PT1 No ---
Difference Source List PT2 No ---
Associated Moving Source Set PT2 No ---
Deep Detection List PT2? No ---
New Object List PT2? No ---
Forced DIASource List PT2? No ---
Forced Source List PT2? No ---
Final objects: ObjectAssoc PT2? No ---
Final objects: PhotoCal PT2? No ---
Object Catalog PT1 Yes ---
Source Catalog PT1 Yes ---
Astrometric Calibration Catalog PT1 Yes Merged w/Photometric Calib catalog
Photometric Calibration Catalog PT1 Yes Merged w/Astrometric Calib catalog

1.2 References & background material

Chapter 2: Accessing LSST Data

2.1 Authorization & Access

  • Archive Overview/How? to acquire an account
  • Data volumes, storage, computing support [, hardware and software you may need?]
    • Note: no computer resources, other than what's required for catalog searches and retrieving data files, will be provided by LSST/DM

2.2 Browsing, queries & Retrieval

  • Browsing the science catalog (database, supporting schema, etc.)
  • Browsing the data product archive
  • Data retrieval examples
    • It would be helpful to give a sense of the latency to be expected for image retrievals
  • Forming queries (and any interfaces we provide to construct queries)
  • Cross-matching
  • Supported VO services [if any]

2.3 References & background material

Chapter 3: Input Data

3.1 Image Simulation Data

3.2 CFHT Legacy Survey Data

  • Survey characteristics (cadence, filters, depth, focal plane description)
  • See the MegaCam Web pages on the following topics:
  • Known data foibles
  • Transformation to LSST raw formats; virtual focal plane geometry

3.3 References & background material

Chapter 4: Data Processing & Calibration

4.1 Pipeline description

(see DC3b Information on the Science Wiki for a starting point)

  • Processing overview: see processing stage description, and flow chart:
    • how each product was derived from a science point of view
    • the key methods and algorithms used in the processing
    • artifact flagging
    • Approach to calibration
    • Construction of calibration reference files

4.2 Sources of Error

  • Description of the accuracies to be expected
  • limitations of the data, any foibles, and sources of error

4.3 References and further information

Provide key reference to literature and on-line material that describe in detail the methods used and advantages/limitations.

Chapter 5: Data Quality Assessment

5.1 Summary of data quality assessment

5.2 Summary of needed assessments

  • Expectations for feedback from SCs
    • Description of results
    • Details of methodology for supporting analysis
    • Posting of SC-generated source code, output data, etc.

Back Matter