wiki:db/Ingest
Last modified 6 years ago Last modified on 09/25/2013 09:48:50 AM

Database Ingest

Architecture:

  • Custom loader that takes input data from pipelines and loads to qserv
  • Input data: pipelines will be delivering data in chunks as produced by individual pipeline processes. Format: most likely fits tables.
  • The loader loads data atomically, e.g. entire chunk either loaded successfully, or not loaded at all.
  • Loader dependencies: afw (because of fits tables, and qserv loader API)