wiki:PolicyFormats
Last modified 8 years ago Last modified on 10/04/2010 02:04:04 PM

Policy File Formats

from: DC2 Management -> Other Components -> Policies

Related Documents:

Introduction

The Policy design calls allows for policy files to be encoded in at least three formats: XML, JSON (http://www.json.org/), and the LSST-specific format Policy Authoring Format (PAF). Of these three, only JSON and PAF have been implemented, and of these PAF is expected to be most convenient for creating policy files by hand. From a programmatic point of view, it doesn't matter which format is used; the code remains the same at the application level. This page describes how Policies are encoded in each of these formats.

For each format, we look at Policy files and Dictionary files, which use different data models. A policy file is essentially a list or tree of parameter names and their values. The values are restricted to be one of the following types:

  • boolean
  • integer
  • double
  • string
  • Policy (a subtree/sublist of parameters accessible under one name)
  • an array of any of the above.

A parameter cannot have a set of values of mixed type. The first time the type for a parameter is established within a policy, it is an error to assign values of a different type to it.

A Dictionary is essentially a list of parameter definitions, in Policy format. Each definition has a name representing the parameter being defined; its value is a Policy which includes name-value pairs that indicate the parameter's type, semantic definition, default value, and usage restrictions.

The mwi package includes some annotated examples the examples/policies directory. (Remember, however, that the XML format is not currently supported!)

Policy Authoring Format (PAF)

Policy Files

The format is very similar to property files commonly used in Java and UNIX applications. examples/policies/EventTranmitter_policy.paf is a simple example. A simpler example might look like this:

#<?cfg paf policy ?>
#
# this is a comment
#
standalone:  true          # a boolean parameter (value is true or false)
filter.threshold: 32.5     # a double parameter
filter.maxIterations: 13   # an integer parameter
label:  Special Filter     # a string value

Leading Line

It is recommended, but not required, that the first line of the policy file contain a content identifier: #<?cfg paf policy ?> This line signals to the Policy framework that the file contains policy parameter names and values in PAF format, so that it can choose the proper parser. If this line does not appear, the framework can usually recognize the format either from the file extension (.paf) or from the first parameter definition.

Spacing

One can be fairly liberal with the use of spacing. The following is legal:

standalone              :  true          
    filter.threshold    :  32.5     
    filter.maxIterations:  13   
     label              :  Special Filter     # a string value

It is illegal to insert spaces within a parameter name.

Parameter Names

Everything left of the first colon (:) is the parameter name which can either be a so-called "top-level" parameter name--one that contains no periods (.)--or a hierarchical name. Each field delimited by dots must begin with a letter and as a whole contain only letters or numbers. The following are legal names:

width
Width
WIDTH
image2jpeg.cmd
convolve.gauss1.width
convolve.gauss2.width

Floating Point Values

The usual IEEE formats for floating point numbers are supported:

convolve.gauss1.width:  1.2
convolve.gauss2.width:  0.9
convolve.gauss1.width:  .22
convolve.gauss2.width:  1.457e-02
convolve.gauss3.width:  1e-3

Boolean Values

The strings true and false are recognized as boolean values:

standalone: true
verbose:  false

If you want a string value set to either "true" or "false", enclose the value in quotes (see next section):

# These are string values, NOT booleans
standalone: "true"
verbose:  "false"

Variations with different letter cases are not recognized as booleans; instead, they will be interpreted as strings:

# These are NOT booleans; they are loaded as strings
standalone: True
verbose:  FALSE

String Values

A string value can be indicated by enclosing the value in double or single quotes (" or '). Single quotes can be used if the value is to explicitly include a double quote, and vice versa. The quotes can be omitted if the value...

  • does not start with "true" or "false",
  • does not start with a number,
  • fits on one line, and
  • is intended to be a scalar value.

The following are legal string-valued parameters:

label1:  select a function
label2:  width
label3:  'target image'
label4:  "the center's position"
choices: "gaussian" 'box' "airy"   # a 3-element array of string values
help:   "A long explanation can span across multiple lines
         as long as the value is enclosed in quotes.  When multi-line
         values are parsed, each new-line character and its surrounding 
         spaces will be replaced with a single space."

Array values

Array valued parameters can be formatted in one of two ways. They can either be placed one line, delimited by spaces:

standalone:           true true false           # boolean array
filter.threshold:     32.5 0.9 .22 1.457e-02    # double array
filter.maxIterations: 13 21 27 50               # integer array
label:                "joe" "fred" "evelyn"     # a string value

or they can be broken up across several lines:

filter.threshold:     32.5  0.9 
filter.threshold:     .22   
filter.threshold:     1.457e-02  

In both examples above, "filter.threshold" will contain a 4-element array.

It is illegal to mix types within an array. It is also an error to separate array values with characters other than spaces (e.g. commas):

# ILLEGAL!
filter.threshold:  3  4.7    # mixed types!
graph.minimum:  5
graph.minimum:  3.2          # changed types!

filter.threshold:  3, 4.7    # don't use commas!
graph.labels:  "x", "y"      # don't use commas!

Policy Values

Hierarchical parameters can be indicated in two ways. The short hand version shown thus far captures the hierarchy in the parameter name with the use of dots (.):

filter.threshold: 32.5     # a double parameter
filter.maxIterations: 13   # an integer parameter

In this example the name "filter" implicitly resolves to a Policy value--a set of named parameters--containing 2 parameters, "threshold" and "maxIterations". This can alternatively be written explicitly:

filter: {
   threshold:      32.5
   maxIterations:  13
}

In this latter form, the "sub-parameters" must each be on a separate line. The can however share a line with the opening or closing brace:

filter: {     threshold:  32.5
          maxIterations:  13    }

Note that the opening brace must be on the same line as the Policy name:

# ILLEGAL!
filter: 
{
   threshold:      32.5
   maxIterations:  13
}

Note also that once you establish that a name is at the root of a hierachy (i.e. that it is of type Policy), it is illegal to assign another type to it:

filter.threshold:  32.5    
filter:            "Airy"    # ERROR!  Type for filter has changed!

Arrays of Policies

It is possible to have an array of sub-Policies accessible via a single name. The way to accomplish this in the policy file is to repeat the use of the parameter name, giving each a different set of values:

filter: {     threshold:  2.5
          maxIterations:  150   }
filter: {     threshold:  32.5
          maxIterations:  13    }
filter: {     threshold:  35.0
          maxIterations:  13    }

Including Other Policy Files

If a parameters value starts with an at sign (@), the value type is assumed to be a Policy and the string following the @ is interpreted to be a file path. When parsed via (createParser()), an attempt will be made to open that file and read the contents as Policy. Each of the parameters given there will be assumed to be sub-parameters under the name given in the first policy file. For example, suppose a policy file contains:

filter:  @filter_policy.paf

and the file filter_policy.paf contains:

threshold:      32.5
maxIterations:  13

This would be equivalent to the following defined in a single file:

filter.threshold: 32.5     
filter.maxIterations: 13   

By default, the createPolicy(policyFile) function will look for the referenced Policy files under the same directory as the original Policy file. Alternatively, you can pass an additional pathname (also called a repository in Policy parlance) to the function that will be assumed to contain the policy files to be included, e.g.:

   p = Policy.createPolicy("policy.paf", "examples/policies")

URN References

As of pex_policy version 3.5, the PAF format also supports references to policy files in other [EUPS] packages via URNs of the form urn:eupspkg:<package>[:<repository>]:path.

For example:

filter: @urn:eupspkg:ctrl_dc3pipe:pipeline/platform:abecluster.paf

Within PAF files, urn:eupspkg: may also be abbreviated with a second @:

filter: @@ctrl_dc3pipe:pipeline/platform:abecluster.paf

A repository, an optional element of a URN, acts as a current working directory for the policy, with respect to internal file references. For example, these three URNs refer to the same file, but with different reference semantics.

# this file's internal references will be relative to ${ctrl_dc3pipe_home}/pipeline/platform
# a reference "@filter_policy.paf" would resolve to ${ctrl_dc3pipe_home}/pipeline/platform/filter_policy.paf
filter: @@ctrl_dc3pipe:pipeline/platform:abecluster.paf

# and this one's will be relative to ${ctrl_dc3pipe_home}/pipeline
# a reference "@filter_policy.paf" would resolve to ${ctrl_dc3pipe_home}/pipeline/filter_policy.paf
filter: @@ctrl_dc3pipe:pipeline:platform/abecluster.paf

# and this one's will be relative to ${ctrl_dc3pipe_home}
# a reference "@filter_policy.paf" would resolve to ${ctrl_dc3pipe_home}/filter_policy.paf
filter: @@ctrl_dc3pipe:pipeline/platform/abecluster.paf

Dictionaries

A Dictionary is intended to provide a definitions for a set of policy parameters. A dictionary file uses the same underlying format as its policy counterpart. The difference is that a defined set of parameter names are used to define the policy parameters. A simple example of a Dictionary in PAF format can be found as examples/EventTransmitter_dict.paf. The example below defines a single parameter called "verbosity":

#<?cfg paf dictionary ?>
#
# this is a comment
#
target: Log

definitions: {
   verbosity: {
      type:         "string"
      description:  "the threshold message 'loudness' that a log message must 
                     exceed in order for it to be recorded to the log, given 
                     as a logical name."
      default:      "WARN"
      minOccurs:    0        # means parameter is optional
      maxOccurs:    1        # parameter may occur only once in policy file
      allowed:  {
         value:        "DEBUG"
         description:  "include messages intended for recording while debugging"
      }
      allowed:  {
         value:        "INFO"
         description:  "include informational messages reporting on normal behavior"
      }
      allowed:  {
         value:        "WARN"
         description:  "include messages warning about abnormal but non-fatal behavior"
      }
      allowed:  { 
         value:        "FATAL"
         descripiton:  "include messages reporting severe or fatal behavior"
      }
   }
}

In this example, the verbosity takes a string value that is restricted to a defined set. The parameter is optional, but it should not appear more than once. If not provided, it's default values is "WARN".

Leading Line

Just as in a policy file, it is recommended, but required, to start with a comment line containing the content identifier, <?cfg paf dictionary ?>. This indicates that the file is a policy dictionary in PAF format. If this identifier is not present, the file can be recognized as a dictionary either from its file extension or by the presence of a top-level parameter named "definitions".

Top-level Parameters

As can be seen from the example above, a dictionary takes advantage of the hierarchical formatting capabilities of the policy format. At the top level of the hierarchy, two parameters can appear:

Name Required? Type Defintion
target recommended string The name of the class, object, or system that this dictionary is intended for. Currently, this is only used for documentation purposes.
definitions required Policy The definitions of each term, where each parameter name is a Policy parameter being defined.

The definitions Parameter

All the policy parameters defined by the dictionary is stored under the definitions parameter. Each of the names that appear under the definitions "policy" is the name of a policy parameter being defined. The name cannot be hierarchical (that is, it must not contain any dots, '.'). Its value represents the definition of the parameter, and it must be of Policy type; that is, its value will be a set of parameters that provide the formal definition.

Definition Parameters

The parameters that may appear within a definition include:

Name Required?TypeDefintion
type recommended string the type of the value expected, one of "int", "bool", "double", "string", and "policy". If not provided, any type ("undefined") should be assumed. If the type is Policy, a dictionary for its terms can be provided via "dictionary"/"dictionaryFile".
description recommended string The semantic meaning of the term or explanation of how it will be used.
minOccurs optional int The minimun number of values expected. 0 means that the parameter is optional, > 0 means that it is required, > 1 means that a vector is required. If not specified or < 0, 0 should be assumed.
maxOccurs optional int The maximun number of values expected. 0 means that the parameter is forbidden to occur, 1 means that the value must be a scalar, and > 1 means that an array value is allowed. If not specified or < 0, any number of values is allowed; the user should assume a vector value.
default optional * A value that will be assumed if none is provided.
dictionaryFileoptional string A file path to the definition of the sub-policy parameters. This is ignored unless "type" equals "Policy".
dictionary optional Policy the dictionary policy object that defines sub-policy parameters using this above, top-level schema. This is ignored unless "type" equals "Policy".
allowed optional Policy A description of the allowed values.

*the type must be that specified by the type parameter.

When the policy parameter being defined has the type Policy, the dictionary parameter provides a means to define the "sub-Policy parameters" supported below the parent parameter. The contents of the dictionary parameter is the same as the top level of a dictionary file: it contains the names target and definitions, the latter containing the definitions of the "sub-Policy" terms. For example, a dictionary like this:

definitions: {
   function: {
      type:        Policy
      dictionary:  {
         target: Function
         definitions: {
            name:  {
               type:  string
               ...
            }
            width: {
               type:  double
               ...
            }
         } 
      }
   }
}

implies a policy file that can look like this:

function.name:   Airy
function.width:  5.7

Often when the policy parameter is defined to be of type Policy, the definition will include a dictionaryFile item instead of a dictionary. The path provided as the value is the path to another dictionary file containing the definitions of the "sub-Policy" paramters. Normally, this path should _not_ be absolute; that is, the application should look for the file relative to current working directory (as set by the application).

The allowed term let's you describe the allowed values the polic parameter may have. If no allowed parameter is provided, then now restrictions on the the value will be assumed. This parameter can include the following "sub-Policy" parameters:

Name Required? Type Defintion
value optional * One allowed value. This should not be an an array of values.
description optional * a description of what this value indicates.
min optional * The minimum allowed value, used for int and double typed parameters.
max optional * The maximum allowed value, used for int and double typed parameters.

*the type must be that specified by the type parameter.

JSON Format

Policy Files

Dictionaries

FAQ

General

On PAF Format

1. How do I specify a string that contains "{" or "}"?
2. How do I specify a string that contains ''both'' double and single quotes?
3. How do I specify a string containing an actual newline?
4. How do I specify non-ASCII UTF-8 characters in strings?
5. How do I specify an array of Policies?
1. How do I specify a string that contains "{" or "}"?
Be sure to enclose the string value containing the brace in quotes.

2. How do I specify a string that contains both double and single quotes?
At this time, it is not possible to include both kinds of quotes in a single scalar string value (though it would not be difficult to add).

3. How do I specify a string containing an actual newline?
At this time, it is not possible to include an actual newline in a string value (though it would not be difficult to add).

4. How do I specify non-ASCII UTF-8 characters in strings?
I don't rightly know, which probably means it is not yet supported.

5. How do I specify an array of Policies?
Use multiple policy entries, each with the same name. See Arrays of Policies above.