LSST Database

[LSST Data Management]

Welcome! This page is used for managing internal information related to the LSST Database. The team working directly on this project include:

•  Jacek Becla (SLAC)
• Kian-Tat Lim (SLAC)
•  Daniel Wang (SLAC)
•  Andy Hanushevsky (SLAC)
• Serge Monkewitz (Caltech/IPAC)

Students:

• Bipin Suresh (Stanford)

Past collaborators:

• Maria Nieto-Santisteban (JHU)
• Ani Thakar (JHU)

This database is expected to reach ~100 petabytes -- don't be surprised if you find that we are considering not-so-typical approaches here and there :-)

Read more about the key LSST Database challenges, and how we plan to address them.

We expect to build a database that will be VO-compliant. Read more...

Tickets

• All open database related tickets grouped by milestone
• All open database related tickets grouped by owner

Schema

• Naming Conventions
•  Schema Browser
• DC3:
  • DC3a Database Schema (MySQL-compatible)
  • DC3b Database Schema (MySQL-compatible)
  • DC3bDataProducts
  • DbDC3bSchemaUpdates
  • DbDC3bSimSchema
  • DbSchemaUpdateApr09
• Post-DC3b Schema Changes

• DC2:
  •  DC2 Database Schema Description
  • DC2 Database Schema (MySQL-compatible)
• Baseline (pretty old, see DC3a):
  •  Database Schema Description
  • Database Schema (MySQL-compatible): without comments , with comments

Queries

• Common Queries - data analysis
• Queries in Data Production
• Queries for Performance Tests
• Questions about Data Analysis Queries
• SQL Programming Style
• Queries for Validating Schema

Query Service (qserv)

• qserv (see top of this document for the links to several related sub-documents)

qserv Alternatives

• Experiments with Hive Implementation and testing of sample data with Hive

Simulated data

•  Simulated data (500 million stars in Milky Way)

Hardware Sizing

• Model for how database hardware capabilities will evolve

Databases in Data Challenges

• DC3
  • Database Backup and Replication in DC3b
  • Database Scalability in DC3b
  • Database Ingest in DC3b
  • Database setup for DC3a
  • Storing Reference Catalog
  • Some discussions on Database setup for DC3 (old)
  • Provenance in DC3
  • MySQL Permissions in DC3
  • Duplicating the DC3a databases
  • Synthetic data
  • Calibrated sources

• DC2
  • Database and Nightly Processing in DC2
  • DC2 Database Partitioning Tests
  • DC2 AP in application tests
  • New, simplified DC2 AP-in-application
  • Building DC2 Object Catalog - closest neighbor search

All others

• Tests with Solid state disks
• MOPS Partitioning
• Science Data Quality Analysis (SDQA)
• Association Pipeline
• Databases and GPUs
• Metadata
• FlexibleSchema
• Provenance
• Neighbors
• Synchronizing Object Ids
• Views in MySQL
• Compression
• Object Split
• Transactions
• Spatial Support
• Computation in database vs. computation in application
• PixelProcessing
• Image Access
• ImageMetadataProposal
• Cutouts
• Persisting Time
• UniqueIds
• Query Results
• Quick mysql-related guide to random things

Related publications of interest

•  A machine Learning Classification Broker for Petascale Mining of Large-scale Astronomy Sky Survey Databases, Kirk Borne
•  The WFCAM Science
• Organizing the Extremely Large LSST Database for Real-Time Astronomical Processing, ADASS, London, UK, 2007:  paper,  talk
• Designing a Multi-petabyte Database for LSST, SPIE, Orlando, FL, 2006:  paper,  talk

• External Links
  • External Links

• Related talks
  •  LSST Database Architecture, presented at LSST Conceptual Design Review, Tucson, AZ, September, 2007

• Some other notes
  • Stored Procedures: cross-dbms compatibility
  • Amdahl Numbers for LSST

Internal laundry lists

• Action Items
• Issues to Discuss
• MySQL Related
• Database Related questions for Application group
• Pipeline Database Setup for DC3+

Obsolete

• CORAL - Relational Abstract Layer
• DBMSStorage Prototype
• SourceReduction