wiki:DC3bDataLocations
Last modified 9 years ago Last modified on 10/27/2010 12:29:32 PM

STATUS OF THIS PAGE: DRAFT

This page describes where DC3b data is.

Related Pages

States

All PT1.1 output data falls into one of three states: New, Candidate, and Released.

New Data is data that has just been generated by a production run. It resides only on the platform that performed the run, and is usually on scratch space.

Candidate Data is data that is under DM review.

Released Data is data that has been vetted by DM, and has been released to external collaborators.

State Transitions

Scripts will be created and used by administrators for handling the state transitions.

"ScriptA" (for the New->Candidate transition)

Current Scope:

  1. Tarring up files using appropriate tarring strategy
  2. Copy tar files to mss:$ROOT/datarel-runs (see below for $ROOT location for mss)
  3. Create metadata file for REDDnet

Future Scope:

  1. Ingest REDDnet metadata into LStore
  2. Initiate (async) preload of data into REDDnet depots
  3. Copy data to NFS server for DM developers (Note: This is being done now manually)
  4. (maybe) delete data from source location (i.e. abe scratch)

"ScriptB" (for the Candidate->Released transition)

  1. (move to the "unqualified" datarel structure?)
  2. TBD

File References

Files can be accessed by concatenating a Root Location with a Logical Filename.

Root Locations

LSST dev servers /lsst2/datarel-runs (PT1.1 output)
/lsst3/transfer/pt1_1 (imsim)
/lsst/DC3/data/
abe /cfs/projects/lsst/DC3/data/datarel-runs (PT1.1 output)
/cfs/projects/lsst/DC3/data/
mss root /UROOT/projects/eiw/lsst/repos/
web server root http://lsst1.ncsa.uiuc.edu/lsstdata/dc3product/
REDDnet root TBD

Logical Filenames

Here are a few selected examples of logical filenames. These are names that users will specify when accessing a particular files, regardless of the underlying storage infrastructure.

  • obs/CFHTLS/D1/raw/v777028-fr/s00/c00-a0.fits
  • obs/ImSim/raw/v85501968-fz/E000/R14/S10/imsim_85501968_R14_S10_C10_E000.fits
  • datarel-runs/daues-im0033/...

Tar files in Mass Storage

ImSim

Tar up files at the exposure level.

For example, for the file

obs/ImSim/raw/v85501968-fz/E000/R14/S10/imsim_85501968_R14_S10_C10_E000.fits

the corresponding containing tar file is

obs/ImSim/raw/v85501968-fz/E000.tar

Output Files

Output files will be tarred up as follows:

datarel-runs/$RUNID/work/* datarel-runs/$RUNID/work.tar.gz
datarel-runs/$RUNID/update/calexp/$VISITID datarel-runs/$RUNID/update/calexp/$VISITID.tar.gz
datarel-runs/$RUNID/update/icSrc/* datarel-runs/$RUNID/update/icSrc.tar.gz
datarel-runs/$RUNID/update/psf/* datarel-runs/$RUNID/update/psf.tar.gz
datarel-runs/$RUNID/update/sdqaAmp/* datarel-runs/$RUNID/update/sdqaAmp.tar.gz
datarel-runs/$RUNID/update/sdqaCcd/* datarel-runs/$RUNID/update/sdqaCcd.tar.gz
datarel-runs/$RUNID/update/src/* datarel-runs/$RUNID/update/src.tar.gz

Note: This tarring strategy may change without notice. Further, such changes are transparent to the user for all cases except when the user is shelling directly into mss.

REDDnet Metadata

filename=Unique/path/to/file,tarball=unique/path/to/tarball,tarball_size=99999

Comma separated: key=value where:

filename (Unique) relative path to an individual file (that really lives in a tarball on MSS)
tarball Relative path to tarball containing said file.
tarball_size Size in bytes of tarball.

The pathnames will be relative paths, and are the same as the logical names defined above.

Attachments