NOAO Pipeline Framework

Introduction

NOAO has developed an advanced and flexible  pipeline framework. This framework is being used to support both the Mosaic and NEWFIRM pipelines. Both pipelines as well as the framework itself are described in detail in a number of papers (available at the same  web site).

An evaluation copy of NHPPS 1.0 can be downloaded from  here ( readme and  example pipeline description). One thing worth noticing is the small footprint of the software: less than 300 KB compressed. NHPPS is written in Python, is self contained and has no external dependencies.

Pipeline Data Model

Overview

• System is a collection of Pipelines (potentially running across machines/CPUs etc.).
• Pipeline is collection of Modules.
• Module is an structured set of atomic code (Actions/Conditional Actions) and rules (Triggers).
• Actions are either global (pipeline-wide) or local (Module-wide). They are divided into PreProcessing and Processing Actions.
• Conditional Actions are Actions whose Commands depend on the exit code of the Module Processing Action. They can be global (pipeline-wide) or local (Module-wide). PostProcessing Actions are Conditional Actions. They can provide a catch all clause as well.
• Actions can have any number or Triggers. They are implicitly ORed.
• Triggers can have any number of Requirements (i.e. Trigger conditions) and they are tied together using the Trigger.logicalOp logical operator. It is therefore possible to define a Trigger condition that depends, for instance, on the state of different pipeline stages (e.g. execute module 3 of pipeline A at the same time as module 5 of pipeline B but only when module 2 of pipeline C fails).

Detail on Requirement and Command

• Native Command is simply Python code that the framework can execute without resorting to a system call.
• Requirements are of three types:
  • Time Requirements are alarms and/or timers.
  • File Requirements wait for a file with a given name pattern to appear in a given directory.
  • Status Requirements wait until a given Module (from a given Pipeline and System) status satisfies the given logical condition.

Module execution flow chart

1. Modules are activated when at least one of their Triggers evaluates to True.
2. Once the Module is activated, its Pipeline (i.e. global) PreProcessing Action is executed.
3. If that Action exits cleanly (exit code = 0), then the Module PreProcessing Action is executed.
4. If that Action exits cleanly (exit code = 0), then the Module Processing Action is executed.
5. Depending on the exit code of the Module Processing action, a Module PostProcessing Action is selected and executed.
6. Depending on the exit code of the Module Processing action, a Pipeline (i.e. global) PostProcessing Action is selected and executed.
7. The Module quits.

Modules can be forced to execute their Processing Action regardless of the exit code of any (local or global) PreProcessing Action (by setting Module.force=True).

Notes

The pipeline execution framework keeps track of the instantaneous state of all the pipeline systems and their modules (via the implementation of the  BlackBoard pattern).