Version 23 (modified by jbecla, 13 years ago) (diff)


Assigned Action Items

LSST Database

2007-06-20, DataAccWG

  • schema change: flagging provisional objects [Jacek]
  • follow up with transient working group [Lynne, Serge]

2007-06-13, DataAccWG

  • update Source Classification table [Kem]
  • redo schema for Source Classification table, add probability [Jacek]

DBMSStorage [K-T]

  • Create realistic (but not fully real) test cases. Probably Exposure and DIASource for output; maybe DIASource or Object for input.
  • Import code into SVN.
  • Rewrite code to accommodate boost::serialization, if that is chosen.
  • Investigate hacking Coral to allow setting multiple fields in one call.

2007-05-25, DataAccWG

  • propose metadata definition [Tim, by June 1st]

2007-05-23, DataAccWG

  • find why we need historical DIASource for Association Pipeline [Jacek]
  • merge MOPS schema with Precursor Schema [Jacek, Francesco]

2007-05-16, DataAccWG

  • Check with Deborah about QA [Serge]
  • run cross-match tests with Serge's DIASources using SQL Server [Maria]
  • follow up on slow cross-match query with MySQL [Jacek]

DC2 plan

  • coral decision [due May 15, all 4]
  • partitioning tests done [due May 31, Serge]
  • cross matching tests done [due June 15, Serge]
  • dbms prototype, incl bulk insert, bulk read and things we discover during Serge's tests [due June 30, K-T, Serge]
  • catalog ingest service, including integrating with dbmsStorage [due July 20, K-T, Serge]
  • db software installation on ncsa hardware: Coral, MySQL [due June 30, Jacek]
  • performance tests of ncsa hardware [July 15 Serge]
  • deployment of CIS, DBMSStorage [July 31, K-T]
  • db and underlying file system tuning (ncsa hardware) [Aug 30, all]
  • integration of schema with application code and input image data [Aug 30, Jacek]

Schema related changes for DC2 [Jacek]

  • merge Object_photoZ with Object for nightly
  • rework source classif - move to provenance, add probability
  • add column to DIASource to keep track of data source (dc2 only)
  • object table: xyColor should be "NULL", not "NOT NULL"
    • muRA, muDecl and parallax too?
  • flag "dirty" in Object Catalog to mark rows that need to be writen from in-memory table to disk
  • match table
  • need a flag in Object table to mark provisional objects (objects produced at the base camp, not by deep detection)
  • schema for DIASourceIdTonight table

2007-05-11, DataAccWG

  • document in trac issue about binding queries [K-T]
  • rebuild CORAL [K-T/Jacek]
  • document in UML: formatters / thread safety [K-T]
  • get coral approved [Jacek]
  • measure CORAL performance overhead [K-T]
  • check when Maria is back [Ani]

2007-05-09, DataAccWG telecon about metadata

For Jeff Bartels, check on May 23

  • use cases
  • which columns in current schema belong to metadata
  • sizes, in particular
    • average number of metadata elements per Object, Source and DIASource
    • distinct values vs values shared between many Objects/Sources?/DIASources
  • flexibility vs speed
  • interaction between pipelines ad metadata stored in database, in particular during Alert Generation
  • schema design: what to store in metadata, what to store in main "core" tables


  • how to efficiently persist vectors in mysql
  • look into mysql archive engine

2007-05-08, discussion with Tim

  • add flag "data source" to Object catalog - for DC2 only [Jacek]

2007-05-04, DataAccWG telecon

nightly processing

  • understand poor-disk performance during loading chunks from disk to memory [Serge]
  • try doing cross match using extract from Object table (objId, ra, decl, x,y,z) [Serge]
  • check with application people about cross matching against var object [Jacek]
  • try doing cross match first against variable objects first [Serge]
  • document requirements / assumptions [Jacek]
  • investigate why dropping in-memory table slow [Serge]
  • test number of tables per db to find sweet spot [Serge]
  • check with Tim whether proper motion adjustment will be made only during deep detection, or during nightly processing too [Jacek]

mysql related

  • bad performance of composite index on (zoneId, ra) - follow up with mysql [Jacek]
  • MyISAM uses 1K disk blocks (8K in new generation myisam) - investigate why it is not tunable [Jacek]


  • sql stored procedures coding standards [Jeff]
  • check existing standards related to stored procedures [Ani]
  • investigate how easy is to move disks/RAM from lsst-dev03 to lsst-dev02 [Jacek]

2007-05-02, DataAccWG telecon

  • define baseline approach how we are going to do association in DC2 [due May 11, Serge, Ani, K-T, Jacek]
  • ping Ray about cleaning AltFormatter [Jacek]
  • legal issues - Stanford copyrights [Jacek, Jeff]
  • DBMSStorage [K-T]
    • cleanup namespaces

Unassigned Action Items / Todo's

  • schema related (after DC2)
    • add table for keeping multiple matches (at the moment we only keep the best match)