wiki:RaysSetupUseCases
Last modified 12 years ago Last modified on 05/07/2008 11:02:33 PM

Use Cases for Setting Up the Software Environment for Execution and Development

These use cases reflect not only our needs as I have experienced them and my own opinions regarding best practices.

Production Execution

1. Setup for Application Execution

In this case, the user wants to run programs (e.g. pipelines) for what they actually do or produce rather than for development. In this case, there is a canonical installation of the software stack on the platform where the application will be run. If this is a shared platform, the installation will typically be shared as well. The user issues a simple command indicating the name of the package that contains the application(s) to be run that will setup the environment (including PATH, PYTHONPATH, LD_LIBRARY_PATH, etc. as needed). This environment will transparently support all libraries and applications that the package of interest depends on. The version of each package that is loaded into the environment is first one that is first supportable by the packages that depend on it and is second marked as being the blessed/preferred version to be used out of all of those installed in the stack. If a blessed version that satisfies the first constraint is not available, the latest version that does is set up.

How this was supported in DC2: The user has eups in his/her path and a default EUPS_PATH is set pointing to the installed software stack; this is accomplished by sourcing an loadlsst.*sh file. The LSST_DEVEL environment variable should not be set prior to sourcing this file; doing so puts the private stack at the front of the EUPS_PATH, giving it preference over the shared stack. To run the pipelines, the user issues the command "setup dc2pipe". Blessed versions of packages are called "current" in EUPS parlance.

Comments: It is inconvenient that LSST_DEVEL must be set prior to sourcing the loadlsst.*sh file. It would be better if one could easily turn on and off use of the private stack.

2. Setup for executing a particular version of an application

In this use case, the user does not want to run the currently accepted version of an application but either an older version (e.g. to reproduce old results) or a newer version (e.g. to take advantage of a beta feature). In this use case, the package is installed into the canonical (possibly shared) stack. The user issues a command indicating both the package containing the desired application(s) and the package version. The environment is set to use that version as well as the versions of other packages it depends on. Like (1) above, the version of each package (other than the one explicit mentioned by the user) that is loaded into the environment is first one that is first supportable by the other packages that depend on it and is second marked as being the blessed/preferred version to be used out of all of those installed in the stack.

How this was supported in DC2: the basic setup is the same as (1) above except that the user issues the command "setup dc2pipe 1.2+svn4455"--i.e. specifying the version of the package.

3. Setup for executing an application using a particular version of a dependent package

In this use case, the user wants to run some version (either blessed or otherwise) of an application but to use a specific version of a dependent package (Package X). This may be an older version (e.g. that lacks a known bug) or a newer version that has a new feature. Either way, the user expects that the version is compatible with the desired version of the package that directly contains the application, regardless of what dependency database (i.e. EUPS table files) say. The user issues one or a few commands to load the needed packages--including the specific version of Package X--into the environment. For each package that Package X depends on, the version chosen will be according to the rules stated in (1). For each package that depends on Package X, the system should prefer packages that are first consistent as best the system knows and secondly blessed. The user should be made aware in some way if the system believes (from its dependency database) that setup versions are incompatible.

An obvious variation on this is that there multiple dependent packages the the user wants to explicitly set the version for.

How this was supported in DC2: The basic setup starts the same as (1) or (2) above. After the initial setup, the user issues another setup command to override the default with the desired version of a dependent package: "setup mwi 1.3+svn4455". Care must taken with the order that the override commands are issue, as any one of them can can cancel a previous override. That is, the override commands must be done in dependency order. EUPS will issue a warning message if a previously setup version of a package is swapped out for a different one due to dependency issues (but see comment below).

Comments: I experienced this use case many times during the end of DC3. Note that the dc2pipe package contained only scripts so it was straight-forward to swap its dependent packages in and out without worry.

The sentence about package incompatibilities can be handled either as a warning on a command or by forcing the user to use a special command line switch. The latter is preferred; see below.

Despite the warning messages from EUPS, it was not always clear what version of a package might get loaded. Sometimes this message would be issued multiple times for the same package as different versions got swapped in and out. It's also easy to either miss these message in a standard informational output or misunderstand them.

It would be best if the user did not have to understand the dependency relationships in order to issue the commands in the right order. Also selecting specific packages can be complex. It would be best if there was a command line option that prevents previous versions from being automatically unset.

Installation

4. Installing a tagged release into the stack

This use case covers the situation when a tagged release is installed into the canonical (possibly shared) stack from SVN. The user checks out the package from a tagged branch. The user issues a command to load the environment necessary to build the package and to test it. After a successful build, the user can run not only that packages own tests but also applications or tests of other packages that can depend on the new release. (The latter testing presumably involves more environment setup.) The package is then installed into the canonical software stack and labeled automatically with the right version number for that release. The user may or may not mark this new version as blessed, indicating to other users that this new version is now recommended as the default.

How this was supported in DC2: The has eups all setup (as described in 1. above). The user issues the an "svn co" command to check out the code into a private directory. The user "cd"s into this directory and types "setup -r .". Building commences by typing "scons"; tests are automatically run. If the user wants to test applications from other packages that depend on the new release, it may be necessary to issue additional setup commands according to 3. above. The package is installed by typing "scons install current" which installs the package into $LSST_PKGS/package/version, where version is the name of the tagged release (e.g. "1.3"). If I didn't want this new version to be current, I would type "scons install declare".

5. Installing a beta version of a package

In this use case, the user wants to make available a non-tagged version of the package in order to make a new feature available. Because of its beta nature, it will not usually be marked as blessed, but sometimes it may. The user checks the package out of the trunk or perhaps from another non-tagged branch in SVN. Building and testing proceeds as in 4. above. The package is then installed into the canonical software stack and labeled with a version indicating the SVN revision number. It should be obvious that this version is not a tagged release. If the package was check out from the trunk, default choice for the version should work well with the dependency system. If it was checked out from some other branch, the user may need to set it explicitly to something sensible. The user may or may not mark this version as blessed.

How this was supported in DC2: After check out, this use case was handled just as 4. was. The version by default was set to svn#### where #### was the SVN revision number. In this case, I would explicitly set the version number to tagged-version+svn#### (e.g. "1.3+svn4921").

6. Reinstalling a version of a package

Occasionally it is necessary to reinstall a package. Doing so should complete eliminate the previous installation having the same version.

How this was supported in DC2: To be safe, it was necessary to explicitly remove the previously installed version before installing the new one due to a now-corrected bug. I would do this by "rm -rf $LSST_PKGS/mwi/1.3+svn3522; scons install". (The declare target was not necessary since I would not bother to undeclare the previous installation.)

7. Removing a package

Occasionally it is helpful to remove packages from an installed software stack (e.g. if it is known to be defective).

How this was supported in DC2: The lsstpkg handles this: lsstpkg remove mwi 1.3+svn3122.

Comments: EUPS had a bug that caused lsstpkg to sometimes remove the wrong version.

Development

8. Adding and testing new features into a package

In this use case, the developer wishes to add a new feature to a package via a ticketed branch and test it out. The developer checks the version out from the branch and issues a command to load the environment necessary to build and test the package. After making code changes, the user can rebuild the package (without installing it) and run tests. Changes are checked back in as prudent.

How this was supported in DC2: The user cd's into the directory containing the checked out code and issues a "setup -r ." Typing "scons" will build the package as needed.

9. Adding and testing new features into several packages simultaneously

In this use case, it is assumed that a new package requires changes to multiple packages simultaneously. This often will take several days (sessions). Typically during initial development, the user can work on one package at a time and does so to ensure a package is self-consistent. During the debugging stage, changes may be necessary to multiple packages simultaneously. Because the work is spread over several sessions, it would be helpful if the setup of the packages under development could be "remembered" and easily re-instantiated.

There are other variations in which the developer may wish to continually re-use a private version of a package across many sessions. There should be a mechanism that prevents/discourages private versions from being installed into shared stacks.

How this was supported in DC2: Private versions of packages can be installed into the canonical stack if the stack is not shared with other users. If it is, the user should install private versions into his/her own LSST_DEVEL stack. This must be explicitly set up using mksandbox. The LSST_DEVEL must be set up prior to to sourcing the loadlsst.*sh file.

During the initial development stage, the first packages to be worked on is checked out and "setup -r ." is issued in the usual way. Once that initial work on that package is done and tested, the package can be installed into the LSST_DEVEL area by typing "scons install declare" or "scons install current". The next package can be checked out and developed in the same way. During debugging, the various packages can be rebuilt and reinstalled as necessary. Usually no new setup commands are necessary. Alternatively, a "setup -r ." in each package directory under development, in which case, the private (re-)install steps are not necessary. However, private installs allow the system to "remember" proper setups across sessions.

RO adds: I personally used a simpler technique. I checked out each package and declared it current, but never installed it. Then one single "setup" command for the highest-level package would set them all up and I could get to work. Because nothing was installed I never had to remember to "scons install" and also avoided the install bug. I found this very convenient and would be sorry to have the DC3 "preferred technique" be any more complex.

Comments: The previously mentioned bug required that the user remove an old package before re-installing an updated version. This is done via "rm -rf $LSST_DEVEL/Linux/package/version; scons install". No re-declares or re-setups were necessary.

The comment from 1. regarding the inconvenience of setting the LSST_DEVEL applies here as well.

I had originally envisioned that we would prevent users from installing private versions into a shared stack by using filesystem write permissions to restrict installations to administration logins. However, we later decided that we needed the flexibility to allow anyone to install a package to share.