LSST Database

[LSST Data Management]

Welcome! This page is used for managing internal information related to the LSST Database. The team working directly on this project include:

• Jacek Becla (SLAC)
• Kian-Tat Lim (SLAC)
• Andy Hanushevsky (SLAC)
• Serge Monkewitz (Caltech/IPAC)
• Maria Nieto-Santisteban (JHU)
• Ani Thakar (JHU)

This database is expected to reach ~100 petabytes -- don't be surprised if you find that we are considering not-so-typical approaches here and there :-)

Read more about the key LSST Database challenges, and how we plan to address them.

We expect to build a database that will be VO-compliant. Read more...

Tickets

• All open database related tickets grouped by milestone
• All open database related tickets grouped by owner

Schema

• Naming Conventions
• Schema Browser
• DC3:
  • DC3a Database Schema (MySQL-compatible)
  • DC3b Database Schema (MySQL-compatible)

• DC2:
  • DC2 Database Schema Description
  • DC2 Database Schema (MySQL-compatible)
• Baseline:
  • Database Schema Description
  • Database Schema (MySQL-compatible): without comments, with comments

Queries

• Common Queries - data analysis
• Queries in Data Production
• Queries for Performance Tests
• Questions about Data Analysis Queries
• SQL Programming Style
• Queries for Validating Schema

Simulated data

• Simulated data (500 million stars in Milky Way)

Database architecture related

• Scalable Architecture for Query Access
• Evaluation of existing solutions
• Testing Scalability
• Science Data Quality Analysis (SDQA)
• Metadata
• FlexibleSchema
• Provenance
• Neighbors
• Synchronizing Object Ids
• Views in MySQL
• Compression
• Object Split
• Transactions
• Spatial Support
• Computation in database vs. computation in application
• PixelProcessing
• ImageMetadataProposal
• Cutouts
• Persisting Time
• SourceReduction
• UniqueIds
• DC3
  • Database setup for DC3a
  • Some discussions on Database setup for DC3 (old)
  • Provenance in DC3
  • MySQL Permissions in DC3

• DC2
  • Database and Nightly Processing in DC2
  • DC2 Database Partitioning Tests
  • DC2 AP in application tests
  • New, simplified DC2 AP-in-application
  • Building DC2 Object Catalog - closest neighbor search

Software Layered On Database

• CORAL - Relational Abstract Layer
• DBMSStorage Prototype
• Association Pipeline

Related publications of interest

• A machine Learning Classification Broker for Petascale Mining of Large-scale Astronomy Sky Survey Databases, Kirk Borne
• The WFCAM Science
• Organizing the Extremely Large LSST Database for Real-Time Astronomical Processing, ADASS, London, UK, 2007: paper, talk
• Designing a Multi-petabyte Database for LSST, SPIE, Orlando, FL, 2006: paper, talk

• External Links
  • External Links

• Related talks
  • LSST Database Architecture, presented at LSST Conceptual Design Review, Tucson, AZ, September, 2007

• Some other notes
  • Stored Procedures: cross-dbms compatibility
  • Amdahl Numbers for LSST

Internal laundry lists

• Action Items
• Issues to Discuss
• MySQL Related
• Database Related questions for Application group
• Pipeline Database Setup for DC3+