wiki:DC3bHardwareRequirements
Last modified 9 years ago Last modified on 04/16/2010 12:16:39 PM

The following are the cumulative hardware requirements for DC3b. These resources will be needed on an ongoing basis until at least the end of calendar year 2011.

PT1 PT2 PT3 Notes
Mar-Apr 2010 Jul-Sep 2010 Oct-Dec 2010
Requirements
Service Units 346K 489K 1,260K
Spinning Disk 5-10TB 10-30TB 10-100TB
Databases
-- ImSim Database 5TB 5TB 5TB
-- PT Database 3TB 4TB 6TB
Tape
Total Tape 162TB 196-356TB 354-794TB
* Reliable Tape (Dual Copy) 12TB 36TB 94TB
-- Input Data (ImSim) 6TB 18TB 47TB
* Unreliable Tape (Single Copy) 150TB 180-320TB 260-700TB
-- Input Data (CFHTLS) 10TB 10TB 10TB
-- Intermediate & Output Data 140TB 170-310TB 250-690TB
Notes CFHT plus 12%
of stage3
35% of stage3

[KTL] I have modified these numbers to reflect the following:

  • Addition of a crude estimate of CFHT Deep and Wide processing to all performance tests.
  • Cumulative totals instead of per-stage totals, reflecting that later performance tests include all of the same images as earlier ones.
  • Possibly not persisting intermediates for stage2 and stage3, but assuming that stage1 intermediates will be persisted through the entire process.
  • Possible use of spinning disk only as a small cache even in later performance tests.
  • Explicitly called out the database disk. See dbDC3bHardware?

[KTL] Note the very large uncertainties in the later numbers. These uncertainties are unlikely to be reduced until after stage1 is well underway. The primary reason for the uncertainty is that it is unknown whether intermediates have to be persisted for analysis or whether they can be re-created from provenance (or not needed at all).

[KTL] The database storage should ideally be attached to lsst10 or another large, permanently-LSST-owned database server with at least 4 gigabit/sec available bandwidth (300 megabytes/sec practical bandwidth) by the time of PT3. Having it be close in network terms to abe/lincoln is desirable but not absolutely required.