Version 19 (modified by ktl, 13 years ago) (diff)

Formatting fixes

Assigned Action Items

LSST Database

DBMSStorage [K-T]

  • Create realistic (but not fully real) test cases. Probably Exposure and DIASource for output; maybe DIASource or Object for input.
  • Import code into SVN.
  • Rewrite code to accommodate boost::serialization, if that is chosen.
  • Investigate hacking Coral to allow setting multiple fields in one call.

2007-05-25, DataAccWG

  • propose metadata definition [Tim, by June 1st]

2007-05-23, DataAccWG

  • find why we need historical DIASource for Association Pipeline [Jacek]
  • merge MOPS schema with Precursor Schema [Jacek, Francesco]

2007-05-16, DataAccWG

  • Check with Deborah about QA [Serge]
  • run cross-match tests with Serge's DIASources using SQL Server [Maria]
  • follow up on slow cross-match query with MySQL [Jacek]

DC2 plan

  • coral decision [due May 15, all 4]
  • partitioning tests done [due May 31, Serge]
  • cross matching tests done [due June 15, Serge]
  • dbms prototype, incl bulk insert, bulk read and things we discover during Serge's tests [due June 30, K-T, Serge]
  • catalog ingest service, including integrating with dbmsStorage [due July 20, K-T, Serge]
  • db software installation on ncsa hardware: Coral, MySQL [due June 30, Jacek]
  • performance tests of ncsa hardware [July 15 Serge]
  • deployment of CIS, DBMSStorage [July 31, K-T]
  • db and underlying file system tuning (ncsa hardware) [Aug 30, all]
  • integration of schema with application code and input image data [Aug 30, Jacek]

2007-05-11, DataAccWG

  • document in trac issue about binding queries [K-T]
  • rebuild CORAL [K-T/Jacek]
  • document in UML: formatters / thread safety [K-T]
  • get coral approved [Jacek]
  • measure CORAL performance overhead [K-T]
  • check when Maria is back [Ani]

2007-05-09, DataAccWG telecon about metadata

For Jeff Bartels, check on May 23

  • use cases
  • which columns in current schema belong to metadata
  • sizes, in particular
    • average number of metadata elements per Object, Source and DIASource
    • distinct values vs values shared between many Objects/Sources?/DIASources
  • flexibility vs speed
  • interaction between pipelines ad metadata stored in database, in particular during Alert Generation
  • schema design: what to store in metadata, what to store in main "core" tables


  • how to efficiently persist vectors in mysql
  • look into mysql archive engine

2007-05-08, discussion with Tim

  • add flag "data source" to Object catalog - for DC2 only [Jacek]

2007-05-04, DataAccWG telecon

nightly processing

  • understand poor-disk performance during loading chunks from disk to memory [Serge]
  • try doing cross match using extract from Object table (objId, ra, decl, x,y,z) [Serge]
  • check with application people about cross matching against var object [Jacek]
  • try doing cross match first against variable objects first [Serge]
  • document requirements / assumptions [Jacek]
  • investigate why dropping in-memory table slow [Serge]
  • test number of tables per db to find sweet spot [Serge]
  • check with Tim whether proper motion adjustment will be made only during deep detection, or during nightly processing too [Jacek]

mysql related

  • bad performance of composite index on (zoneId, ra) - follow up with mysql [Jacek]
  • MyISAM uses 1K disk blocks (8K in new generation myisam) - investigate why it is not tunable [Jacek]


  • sql stored procedures coding standards [Jeff]
  • check existing standards related to stored procedures [Ani]
  • investigate how easy is to move disks/RAM from lsst-dev03 to lsst-dev02 [Jacek]

2007-05-02, DataAccWG telecon

  • define baseline approach how we are going to do association in DC2 [due May 11, Serge, Ani, K-T, Jacek]
  • ping Ray about cleaning AltFormatter [Jacek]
  • legal issues - Stanford copyrights [Jacek, Jeff]
  • DBMSStorage [K-T]
    • cleanup namespaces

Unassigned Action Items / Todo's

  • schema related (after DC2)
    • add table for keeping multiple matches (at the moment we only keep the best match)