wiki:PolicyHowto
Last modified 8 years ago Last modified on 09/15/2010 08:34:15 AM

How to use Policies and Dictionaries

This page provides a "How-To" approach to adding support for Policy-based configuration to a class in both C++ and Python, especially for Stage implementations.

More about Policies:

Use Cases

Configure an object with a policy file

You have a class which you would like to configure with parameters stored in a policy file. In either C++ or Python:

  • Update your class to take a Policy object, either via a constructor or a set method
  • Extract configuration parameters from the Policy, within your class

http://ditaa.org/ditaa/render

After you have added support to your class:

  • Load a Policy object from a file
  • Pass the Policy object to an instance of your class

Fill in a policy with default values

For simplicity, you might want to populate a policy file with only those values that differ from predictable defaults. DefaultPolicyFile makes it easy to load defaults from a package, to fill in the rest of the policy.

Configure a Stage with a Policy plus defaults

For example, when a Stage is instantiated by the harness, it has the option of accepting a Policy for configuration. You can override setup() to configure the Stage, with a few simple steps:

  1. Load defaults using DefaultPolicyFile
  2. Merge defaults into self.policy (which was provided by the harness) to fill in missing values. Note that the defaults file may be either a Policy or a Dictionary file. A Dictionary file is preferred because it will validate the policy received from the harness as well as any future values added to that policy.
  3. Extract parts of the Policy (sub-Policies) to configure your Stage's components.
import lsst.pex.policy as pexPolicy

class MyStage(harnessStage.ParallelProcessing):
    def setup(self):

        # 1. load defaults using DefaultPolicyFile
        policyFile = pexPolicy.DefaultPolicyFile("my_package", "MyDefaults.paf", "pipelines")
        defaults = pexPolicy.Policy.createPolicy(policyFile, policyFile.getRespositoryPath(), True)

        # 2. merge defaults into self.policy
        if self.policy is None:
             self.policy = Policy()
        if defaults.canValidate():
             self.policy.mergeDefaults(defaults.getDictionary())
        else:
             self.policy.mergeDefaults(defaults)

        # 3. Extract sub-policies to configure your stage's components
        subPolicy = self.policy.getPolicy("subtask")
        doSubtask(subPolicy)
        ...
Loading a Policy with a URN

New in pex_policy 3.5

As an alternative to DefaultPolicyFile, you may use URNs to load policies:

        policyFile = pexPolicy.UrnPolicyFile("urn:eupspkg:my_package:pipelines:MyDefaults.paf")
        defaults = pexPolicy.Policy.createPolicy(policyFile)

or simply

        defaults = pexPolicy.Policy.createPolicyFromUrn("urn:eupspkg:my_package:pipelines:MyDefaults.paf")

URNs may also be embedded as internal references in Policy files.

Loading a Policy file into memory

It's helpful to start with how to create a Policy object in memory as you will need to do this at least in the course of debugging your code. The preferred way to create a Policy instance is via the static createPolicy() factory function. This factory function takes a file path as input which can either be a policy file or a dictionary file. In either case, a Policy with all of the parameters from that file will be loaded into it. If a path to a dictionary is provided, the parameters that are loaded are the ones defined in the dictionary and the values will be their defined defaults.

In C++, this factory function returns a raw pointer to a Policy. It's up to you to handle safely as you wish and deleting it when you are done. The typical practice will be to immediately wrap the Policy object into a shared_ptr:

   #include "lsst/pex/policy.h"
   using namespace lsst::pex::policy;

   // load Policy from file, and wrap in a Boost shared_ptr
   Policy::Ptr policy(Policy::createPolicy("examples/policies/EventTransmitter_policy.paf"));

   bool doall = policy->getBool("doall", false);

(The Policy constructors that take an input file assume that the contents are in policy format. Reading a dictionary via a Policy constructor is not supported.)

In Python, create the instance and go (you don't have to worry about pointers):

   from lsst.pex.policy import Policy

   policy = Policy.createPolicy("examples/policies/EventTransmitter_policy.paf")

   doall = policy.get("doall", False)

Adding Policy support to a Class

Accepting a Policy Object

At this point in the development of the LSST software framework, a typical way to pass a Policy object to an object to be configured would be via a special constructor:

class MyClass : private Citizen {
public:

    /**
     * create an instance and configure it with the given policy
     * @param policy    the Policy object that configuration parameters will be drawn
     *                     from.  See policies/MyClass_dict.paf for a description of 
     *                     the expected parameters.
     */
    MyClass(const Policy& policy);

    ...
};

You may instead or in addition accept a Policy object via a set method:

     ...
     /**
      * set the configuration for this instance.  This will override any previously 
      * set configuration parameters.
      * @param policy    the Policy object that configuration parameters will be drawn
      *                     from.  See policies/MyClass_dict.paf for a description of 
      *                     the expected parameters.
      */
     void setPolicy(const Policy& policy);
     ...

Whether you accept a Policy object via a constructor or a set method depends on the answers to the following questions:

  • Can the class function without a being passed a Policy? If not, then you probably should only accept via a constructor, and that constructor should be the only public constructor available.
  • Can you allow configuration parameters to be set after other class functions have been called? If so, then a set method may appropriate. A set method provides more flexibility; for example, the caller that configures the object need not be the one that constructed it.
  • Can you allow policy parameters to be set twice? Either way, be sure to document what happens when the setPolicy() function is called more than once.

If you want to explicitly share an instance of a Policy with the caller, then you should accept a Policy::Ptr; e.g:

    MyClass(const Policy::Ptr& policy);

Configuring Your Class

Inside the implementation of the Policy-accepting constructor or function, you have two choices: either extract and apply all of the parameters right away, or save the Policy object and extract parameters as you need them. If you choose to hold onto the Policy object, here are a few tips to keep in mind.

First, if you copy the Policy object by assignment, a shallow copy is made. Consider:

class MyClass : private Citizen {
private:
    Policy _policy;

public:

    MyClass(const Policy& policy) : Citizen(typeid(this)), _policy(policy) {
        ...
    }

    void setPolicy(const Policy& policy) {
        _policy = policy;    // does a shallow copy
        ...
    }

    ...
};

In both the case of the constructor and the set function, a shallow copy of policy will be loaded into _policy. This means that copies will be made only of the values of the top-level, non-Policy-typed parameters. policy and _policy, however, will still share instances (via an internal shared_ptr) of any "sub-Policy" values. If you prefer a deep copy, use Policy's special copy constructor:

class MyClass : private Citizen {
private:
    Policy _policy;

public:

    MyClass(const Policy& policy) : Citizen(typeid(this)), _policy(policy, true) {
        ...
    }

    void setPolicy(const Policy& policy) {
        _policy = Policy(policy, true);    // does a deep copy
        ...
    }

    ...
};

If you are sharing an instance, then you don't want any copying:

class MyClass : private Citizen {
private:
    Policy::Ptr _policy;

public:

    MyClass(const Policy::Ptr& policy) : Citizen(typeid(this)), _policy(policy) {
        ...
    }

    void setPolicy(const Policy::Ptr& policy) {
        _policy = policy;    // no copying is done; instance shared with caller
        ...
    }

    ...
};

Extracting Parameters

Extracting Parameters in C++

Now you will want to use the Policy object to configure your class. This is done via Policy's get member functions.

We later describe how to document the parameters your class requires by creating a Dictionary file. However, you might consider creating this file while you are adding the retrieval code to your class.

You will write the parameter retrieval code expecting parameters to have specific types. This is manifested in the choice of the get function you use.

    bool getBool(const string& name, bool defval) const;
    int getInt(const string& name, int defval) const;
    double getDouble(const string& name, double defval) const;
    const string& getString(const string& name, const string& defval) const;

These get methods allow you to specify a default value to return if the value is not set:

    int total = _policy.getInt("total", 10);
    string name = _policy.getString("name", "");
    double width = _policy.getDouble("width", 5.0)
    bool doall = _policy.getBool("bool", false);

Any of these methods will throw a lsst::mwi::policy::TypeError exception if a parameter exists with the given name but has a value of a different type than what the method returns.

Note that there are versions of the get functions that do not accept a default value. These versions will throw a lsst::mwi::policy::NameNotFound exception if the parameter is not currently set.

Occasionally you will want to retrieve an array of values; in this case, use one of these methods:

    const BoolArray& getBoolArray(const string& name) const;
    const IntArray& getIntArray(const string& name) const;
    const DoubleArray& getDoubleArray(const string& name) const;
    const StringPtrArray& getStringArray(const string& name) const;

The returned values are STL vector objects:

    typedef vector<bool> BoolArray;
    typedef vector<int> IntArray;
    typedef vector<double> DoubleArray;
    typedef vector<StringPtr> StringPtrArray;
    typedef shared_ptr<string> StringPtr;

Thus you can use STL iteration techniques to access the values:

    const Policy::DoubleArray offsets = p.getDoubleArray("offsets")
    Policy::DoubleArray::const_iterator it = offsets.begin();
    while (it != offsets.end()) {
        double off = *it;
        ++it;
    }

Of course, you can pull out parameters deep within the policy hierarchy:

   string verb = _policy.getString("transmitter.logVerbosity", "INFO")

How you group your parameters into a hierarchy is completely up to you. You can keep all the parameters flat and at the top level, or you can group them into logical groups of related parameters. The advantage to this is that the entire set of related parameters can be extracted at once as a so-called "sub-policy". In fact, this is expected to be the typical way to configure sub-components of a class:

   Policy::Ptr transmtrPolicy = _policy.getPolicy("transmitter");
   Transmitter transmtr(*transmtrPolicy);

In this example, the Transmitter class has be implemented to be configured by a Policy object.

This is

Access Policy Parameters in Python

You may find yourself accessing policy data from Python. In Python, access is a little simpler. For one, you can just use the generic get() method to retrieve values:

>>> from lsst.mwi.policy import Policy
>>> policy = Policy.createPolicy("examples/policies/EventTransmitter_policy.paf")
>>> doall = policy.get("doall", False)
>>> doall
False
>>> name = policy.get("name", "")
>>> name
"fred"

Array accesss is similarly simple:

>>> offsets = policy.getArray("offsets")
>>> offsets
[12.3 143.1 44.54 12.0]

Access to parameter names is similar:

>>> names = policy.names(True)   # true input returns top-level names only
>>> names
["offsets" "receiver" "standalone" "threshold" "transmitter"]

Documenting the Expected Parameters

As you add the code to your class that extracts parameters from a policy, you should document the names you look for in a separate policy dictionary file. I recommend that this file be stored in a directory called policies (directly under trunk). The format of a dictionary file is explained in PolicyFormats.