wiki:DC2DataTransfer
Last modified 12 years ago Last modified on 06/26/2007 08:48:29 AM

DC2 Data Transfer

One of the tasks included in Data Challenge 2 is a set of tests of high-throughput data moving configurations. That is, we're trying to learn the most effective way to move the many large files which will have to go from La Serena to NCSA on a daily basis.

We will therefore as part of DC2 perform a series of tests of parallel file systems and data movers to determine which pairing is the most effective at moving LSST data. For each method, we will conduct at least three trial runs, recording the amount of data moved, reliability of the connection, and elapsed time.

Parallel File Systems to be Tested

Open Source

Note: Lustre is open source but not without license. Chris is currently pursuing what that means in the context of DC2.

Commercial

This list is preliminary; other possibilities may be included.

Data Movers to be Tested

Open Source

This list is preliminary; other possibilities may be included.

Test Sites

Teragrid Test

A series of tests will be performed transfering data from one Teragrid site to another. The Teragrid sites are TBD.

These tests will include:

  • GridFTP test using a parallel client configuration
  • Sector test using one client and one server

Other Test Sites

Other tests will be performed using the NCSA cluster 'globular' as server and these remote sites:

  • LSST site at La Serena, Chile

Tests will be performed using two configurations:

  • Multiple servers from NCSA's cluster 'globular' to single clients at the remote site
  • Single server from NCSA to single clients at the remote site