Last modified 5 years ago Last modified on 07/11/2014 02:30:24 PM

OBSOLETE Don't even think about using these tools on the new stack 7/2014

Refer to the new tools: LSST/DMS/devenv/lsstsw.git LSST/DMS/devenv/lsst_build.git 7/2014


Releasing a New Version of an LSST Product

A Developers can create a new versions of an LSST product by creating a git tag with a name matching the version (e.g. "") in accordance with the project's version numbering conventions. Once done, other users can export from git and "build by hand" that version of the product. The version of the product is considered released when the version has been packaged up and is available on the LSST product distribution server for automated download and installation using eups distrib install. This documentation describes tools that allow a developer to release a product to the server. This can be done using a tool called submitRelease.

It's worth noting that when we tag a product in git with a version number (e.g. ""), we are giving an official name to the set of files that make up that version. When we release it on the server, we make two additional statements:

  • We establish an exact list of versions of dependencies that our product should be built with; a modified version number, e.g. "", is given to this combination.
  • We can optionally label the release with an eups server tag, such as "beta" or "stable", that recommends the release for some use.


The LSST distribution server, where users and developers get official releases of LSST products, is managed for stability and adherence to LSST policy. Similarly, the development cluster features a default installed software stack (LSST_HOME=/lsst/DC3/stacks/default) where the latest official releases are installed which is also maintained for stability. To help ensure this, both resources are setup as generally read-only (owned by the "lsstsw" user). Nevertheless, developers have the ability to release new versions of products to the server and get them installed on the cluster once they have been git-tagged in the repository. The submitRelease tool has been installed on the cluster to write and install new packages as the lsstsw user. Naturally, a developer's ability to write to these resources is limited; if you have special needs or problems not covered by this tool you can request assistance by issuing a Trac ticket or sending email to the RayPlante, the software management czar.

Consequently, releasing an LSST product must be done from one of the cluster compute servers. The submitRelease tool is made available via the devenv_servertools product which, on the cluster, is setup automatically as a dependency of lsst.

Note that it is assumed that once you have tagged in a product in git, you have fully tested the product and ensured that its unit tests all pass. (See Testing Your Version below for more details.)

In order to release a version of your package, you essentially need to create an eups manifest file which sets the exact versions of your products dependents that you want your product to be built with. There are several ways that you can do this, depending on the needs of your product:

  • You can let submitRelease create a manifest for you. In this case, it will select stable versions of the dependencies. Alternatively, you can tell it to use beta versions if desired.
  • You can create your own manifest using the script and possibly modify it by hand. This selects dependencies just as the submitRelease script does.
  • If you need to more carefully control the versions of its dependencies, you can create it yourself using eups distrib create.

Each of these variations are described below.

In all cases, when you submit the manifest, it will be tested to ensure that it produces a successfully built product. This includes checking that all the tests pass. If any failure occurs, release of the product is cancelled. You can tell submitRelease to ignore failed tests via a -i option and release the product anyway, but its use is discouraged.

When a product version is released with a manifest, it is released with a "modified" version that adds a +N build number; the first release is always "+1" (as in ""). In addition to this release, submitRelease will re-release all of the products that depend on your product, incrementing their build numbers. The so-called "up-reving" of your product's dependents can be disabled with the -D option.

In the examples we show below, we imagine that we are releasing version of the utils product.

Summary of Command-line Syntax

The command line syntax is listed below.

Usage: submitRelease [-j NUM -t TAGS -iCnVh] product version [manifest]

  -t TAG[,TAG]  when creating a manifest on the fly (i.e. manifest is not
                provided) prefer dependencies with these (server-assign) tags.
  -R          re-release already released version: without this, the release
                will not be proceed if the version has already been released.
  -i          ignore failed tests: don't let failed tests prevent release
  -j NUM      use NUM threads when building (default is 4)
  -D          do not uprev the product's dependents
  -C          prep and test the release but do not commit it
  -n          do not clean up the work area before exiting
  -V          force validation: check that the tests pass even if the tests
                were run implicitly when a default manifest was created.
  -h          print this help and exit

See the section "About Other Option" below for more information.

Testing Your Version

You may have carried out all of your testing on your product's master branch (and made sure all tests pass) just before you tagged the version in git. Sometimes, though, it's good to try testing directly on the tagged source, particularly if something goes wrong with submitRelease. To extract the tagged version of the source, use the git archive command:

git archive --format=tar --prefix=utils- --remote=$LSST_DMS/utils.git | tar xf -

This will unpack the package into a directory call utils- To test, you can change into that directory, setup with EUPS, and run scons as you usually do. For a quick tutorial, follow the the first 3 steps of Use Case 3 below.

Running submitRelease

The next three subsections show how to use submitRelease to release a new product version under the three scenarios summarized above in the Overview.

Use Case 1: Let submitRelease create the manifest for you

Letting the tool create the manifest file (i.e. the specific dependencies) is the simplest and probably the most common use case, and it is certainly most appropriate for releases that are aimed at eventually being tagged as "stable". The simplest call to release version "" of product "utils" would be:

submitRelease utils

This will:

  • check out the of utils from the git repository
  • do a test build and run all the tests
  • create a default manifest
  • deploy the tar-ball and manifest to the NCSA distribution server
  • create re-releases of the product's dependents (i.e. products that depend on utils) and release those to the server.
  • install all new products into the default stack on the cluster.

The product will be released with a modified version that adds the build number. In our example, where we presume that this is the first time this version is released, the version would be "". If there is already a release on the distribution server (either actually released or simply staged), then the build number might be higher.

By default, submitRelease chooses selects dependencies for the product according to the constraints in the product's table file, prefering versions tagged "stable". Often you may find you need your product to depend on the beta version (or some other tag, e.g. "Winter2012"); this can be done simply by adding -t option:

submitRelease -t beta utils

Typically if something goes wrong, the tool will prevent the product and its new dependents from being released. See the section on Debugging below for tips on handle those failure.

About Other Options

Here are some other options you might find handy:

If you just want to see if the package passes all its tests but not actually release it, use this option.
If you don't see a need to make your fellow developers rebuild a large number of dependent products after your release, you prevent the "up-reving" of the dependents with this option.
While use of this option is discouraged, occasionally we need to release a product even though not all of its configured tests pass; this option enables this.
Sometimes you need to re-release a product with a different manifest (because you want a different set of dependencies). By default, submitRelease will detect if the version of the product has already been released and will not re-release it. With the -R option, you can tell submitRelease that you know it's already been released and that a new one is needed. ("Fixing" an already released version is not supported.)
-j NUM
If no one else is using the cluster node you are running on submitRelease on, you can get some extra oomph (or less) with this option. NUM is passed to scons, setting the number of build threads it will use. Values higher than 8 usually usually show diminishing returns.
When submitRelease does its work, it creates a work area for building and testing which usually gets cleaned up when the script exits. This option prevents that clean up which can sometimes be helpful for troubleshooting failures (see below). If you use this, you will need to run clearSubmitRelease afterward to clean up.

Use Case 2: Create a model manifest with

This is perhaps the least typical use case, but it provides a straight-forward way of creating and testing a manifest. In fact, the tool is exactly how submitRelease creates one on the fly. Creating one explicitly does give you the opportunity to tweak the manifest "by hand" before handing it over to submitRelease.

To create a manifest, you run -M utils

This will:

  • check out the of utils from the git repository
  • do a test build and run all the tests
  • create a default manifest called utils- in your current directory also accepts the -t option for preferring dependencies with certain tags (behaving the same as submitRelease). Run -h to see other available options.

Once you have created (and possibly tweaked) the manifest, you can pass it to submitRelease to release it:

submitRelease utils utils-

This will:

  • do a final check of the manifest to ensure it can install the product
  • deploy the tar-ball and manifest to the NCSA distribution server
  • create re-releases of the product's dependents (i.e. products that depend on utils) and release those to the server.
  • install all new products into the default stack on the cluster.

Note that you can still specify the -t option in this situation: while it won't effect the product's manifest, it can affect the manifests of the re-released manifests. (Thus, -t has no effect when -D is used.)

Use Case 3: Give submitRelease a custom manifest

This scenario covers the case when you need full control over determining the versions of the dependencies used to establish a release. In summary, the steps in this case would be:

  1. extract the source code from the master branch of the product's git repository
  2. use EUPS to setup the exact set of dependencies you want
  3. build, test, and install your product into a sandbox
  4. run eups distrib create to create the manifest
  5. run submitRelease to submit your manifest and release your product.

As you go through these steps, keep in mind the following:

  • You must only setup dependencies that are installed in the default stack on the cluster. (It's okay if you have tagged them with personal user tags.) That is, the dependency versions must be those that are available to all users through the LSST distribution server. You cannot depend on product versions that have not been released, yet.
  • Don't skip step 3 where you do a test build and install. Make sure all tests pass.

Step 1: extract the source

Get a clean copy of the tagged source code:

git archive --format=tar --prefix=utils- --remote=$LSST_DMS/utils.git | tar xf -

Step 2: set the dependencies

The usual setup-for-building can establish a baseline that satisfies the table file:

cd utils-
setup -r .

Alternatively, you can use the -t or -T options to prefer or require certain tagged versions. (-t is generally used for server tags, and -T for user tags.) If your package depends on "beta" versions, for example, you can:

setup -t beta -r .

Note: The exact behavior of -t changes with eups 1.2.31. Prior to this version, -t forces the selection of dependencies tagged with the given name. With 1.2.31, the tagged version will only be setup if it satisfies the constraints placed on it by the table file; otherwise (in the absence of other switches or special configurations), you will typically get the latest version that satisfies the table file. The new option -T forces the select of a tagged dependency, regardless of what the table file constraints are.

After the initial setup, you can over-ride certain product setups by running setup with -j (which setups the named product but not its dependencies) or -k (which setups dependencies only if versions are not already setup). You can use eups list to see what you have setup:

eups list --setup

Step 3: Do a test build and install

To install a product, you need a place to install it. I recommend setting up a private sandbox to do this. A sandbox is a private directory tree where you can install products and later setup up for private use. This directory tree requires a special organization. To establish a sandbox, use mksandox:

cd ..   # if you are still in the product source directory
mksandbox mysandbox
EUPS_PATH=$PWD/mysandbox:$EUPS_PATH   # assuming bash as the shell
cd utils-

This creates and initializes a directory called mysandbox. We then tell EUPS to use it by updating the EUPS_PATH environment.

Now you are ready to build and install. Do the usual build and look for any failed tests:

scons opt=3 -j 4

Usually when using -j to speed things up, it's hard to see from the messages which tests have failed. An easy way to check this then is to look at the contents of the tests/.tests directory which contains unit tests results. A failed test will result in a file called testname.failed; the contents give the output from the test.

When you've determined all tests have passed you can install it (assuming you updated your EUPS_PATH); specify the version you want it to have on the server:

scons opt=3 version= install declare

Don't worry too much about the build number (the "+1" part) as it will be updated as needed by submitRelease according what's already on the server.

If the install is successful, you are ready to create the manifest.

Step 4: create the manifest

The eups distrib create command is used to deploy a product to a distribution server. To use this command, you need to create a personal distribution directory:

mkdistserver mydistserver

This creates a distribution directory (using the "lssteups" conventions) called mydistserver.

To create the manifest for your just installed product, your EUPS_PATH still needs to include your sandbox. Then you can:

eups distrib create -j -f generic -s mydistserver -d lsstbuild utils

Step 5: pass manifest to submitRelease

The manifest will be written into mydistserver/utils/ You can now pass this manifest onto submitRelease:

submitRelease utils mydistserver/utils/

Assigning Server Tags (e.g. "beta")

If you intend that your newly released product should eventually be considered the stable version, recommended for production use and development, then it should first be declared as beta. This can be done after releasing the product with submitRelease by assigning the EUPS server tag "beta" to the product using tagRelease:

tagRelease -t beta utils

This will assign the server tag "beta" to utils In addition, it will also tag all the product's dependents. Assuming you just did "up-reved" the dependents, tagging them beta too is usually appropriate; however, you can use the -D option to prevent this.

When Things Go Wrong

The testing that submitRelease does is the final testing a product undergoes before release. Ideally, build and integration problems have already been addressed ahead of time. Nevertheless, this testing can uncover general integration issues that might be hidden when working in your personal environment.

Computing the wrong +N

If a package +10 already exists then submitRelease will try to re-release that package as +10, which will naturally fail. One workaround is to make a new git tag for the package, then submitRelease that. See ticket #2035.

User Collision (Clearing locks)

submitRelease prevents multiple users from releasing the same product. If you try to release a product simultaneously with another user, you should see a message like this:

Release of utils is already in progress by user rplante@lsst6 (pid=21033) 2012-01-25 11:27

Before proceeding further, you should coordinate with the other user named in the message, particularly if the date is recent. (Note that the PID may refer to a user on a different cluster machine.) It may be that the "lock" is on because the user used -n and is still examining the results. If the date is old though, it may be that the "lock" is simply "stale" and can be safely removed.

Anyone (not just the creator) can clear the lock with clearSubmitRelease:

clearSubmitRelease utils

Build, Test, or Install Failures

When submitRelease builds the product or does a test install, you may see errors reported and then one of these messages:

submitRelease: Product build failed; ...
submitRelease: Failed to create manifest
submitRelease: test-install failed
2 tests failed: ...

If the displayed messages are not revealing enough about why things failed, you can rerun submitRelease with the -n option. When you do, you'll see something like this at the end of the output:

Work space (test builds, etc.) not removed:
run clearSubmitRelease to clean it up

You can change into that directory and poke around to see what went wrong. You can find the test build under the utils- subdirectory. If the test-install failed, look in the test/EupsBuildDir subdirectory.

When you are done examining the results, clean them up with clearSubmitRelease:

clearSubmitRelease utils

The best way to debug such failures is to go back and try manually building the product from the tagged version of the code (see Testing Your Version above).

If tests are failing and you understand why, you may decide that the tests are not easily fixable but are otherwise not consequential to the use of the product. In this case, you can tell submitRelease to ignore the test failures and release the product anyway; do this by re-running with the -i option. Of course, use of this option is discouraged, and note that failed tests would prevent it from being tagged as "stable".

Up-rev Install Failures

By default, submitRelease will automatically re-release all of the products that depend on your product (we call this up-reving the dependents). It is possible that some of these products will need changes of their own in order to build and use your new version; this fact may cause the installation of the up-reved products to fail.

When a dependent product fails to build against your newly released product, submitRelease will cancel the release of the dependent product. This is not an error per se, and so you don't need to do anything apart perhaps from informing the dependent product's guru that changes are needed. To note which dependent products failed, look for messages like this:

The following dependent products did build successfully:
Their release have been cancelled.