wiki:Winter2014/Design/Middleware
Last modified 5 years ago Last modified on 11/19/2013 03:59:01 AM

Winter 2014 Process Control Middleware Activities

DRP Simulator

Goal: Build a simulation of the Data Release Production that is realistic in terms of dataflow and the number of files created. This includes iterating through the sky, handling overlaps between sky regions, and handling different-sized regions for different tasks.

Stretch Goal: Have the sizes of files and I/O latencies (including tape) be realistic.

In order to enable the simulations to run quickly, the lengths and CPU consumption of tasks will not be simulated.

The simulation will be implemented in the Pegasus workflow tool and, simultaneously, in another system. We will try to use Astro-WISE, but if code for that proves to be unavailable, we will try OWL. We will also attempt to compare with an estimate of the difficulty of producing a custom workflow system.

Pegasus tasks for SRP:

  1. Build initial dataflow in Pegasus - completed
  2. Write up assessment of execution, usability - One Day
  3. Improve data flow with realistic file creation - One Week
  4. Write up assessment of execution, usability - Two Days
  5. Improve data flow with patches, overlap, and ordering - Three Weeks
  6. Write up assessment of execution, usability - Two Days

Astro-WISE/OWL tasks for GD:

  1. Evaluate Astro-WISE in greater detail - Two weeks
  2. Implement DRP using Astro-WISE if software is deployable locally (dependency may be Oracle, availability of source, etc) or OWL - Three Weeks.
  3. Write up assessment of execution, usability of Astro-WISE/OWL implementation - Two Days

AP Prototype

Goal: Build an advanced prototype of the Alert Production that uses DM Events instead of the OCS interface but otherwise moves data and executes jobs in the same way as the baseline design. If this works well, it could be refined into the actual Alert Production framework.

Tasks for SRP and GD:

  1. Build AP prototype framework for components (including event sender, receiver, and API for replicator batch job/data copier, redirector, heartbeat monitoring). - Two weeks
  2. Implement and add batch job submitter - One week
  3. Implement and add replicator batch job/data copier - One week
  4. Implement and add redirector (Archive DMCS component) - One week
  5. Implement and add data preload and puller worker batch job - One week
  6. Write up of implementation including assessment of execution and usability - One Week