LSST Unstructured Data Management

Scope

Files play a major role in LSST. A holistic approach is needed to manage files during their lifecycle including (but not limited to) creation, transfer, modification, rest, distribution and archival. The tools, protocols and file-formats used may be different (or remain the same) during the different processes in the lifecycle of a LSST file.

Objective

To experiment and provide a continued assessment of:

• different middleware and software tools
• protocols (TCP/IP, non-TCP)
• file formats (which format is good for each process in the lifecycle of a LSST file)

Given the large scale usage of files in LSST,

• we will evaluate which attributes or attributes have to be measured and analyzed during this effort.