Last modified 10 years ago Last modified on 12/10/2008 04:23:23 PM

Association Pipeline

LSST Database


The planned DC3 association pipeline, including use of inter-slice communication to allow for parallelization of the association pipeline across multiple machines, is described here.


We are planning to start implementing the Association Pipeline for DC2 once we finish partitioning tests, in particular performance tests of prepare and post process phases, described in detailed in DC2 Database Partitioning Tests?. Estimates date: early July 2007.

Main tasks of Association Pipeline [ this list is not complete yet! ]:

  • nightly processing: prepare
    • prepares Catalogs and Databases
    • read in all configuration parameters/policies such as match radius, stripe width, number of partitions, source classification, etc.
  • visit: prepare
    • wait/receive signal about next FOV
    • create or truncate tables for keeping per-visit information
    • load Object Catalog partitions
    • start MOPS prediction code
  • visit: real time process
    • execute cross-matches (against MOPSPred, against Object, maybe more)
    • update Object Catalog
  • visit: post process
    • write all rows marked as 'dirty' from in-memory tables to disk-based tables
  • nightly processing: post process

Design options for integration with the pipeline framework are given in AssocPipelineIntegration.

The following tasks are likely to be handled by other pipeline components, not the Association Pipeline directly.

  • mapping from 3000 files (one per amplifier) to our partitions
  • ingest DIASources produced by Detection Pipeline
  • ingest CSV files produced by Image Pipeline

Information about Catalog Ingest Service (CIS) used in DC1 can be found here.