wiki:EupsTips
Last modified 7 years ago Last modified on 08/29/2012 04:37:03 PM

From Eups

This is an archive of tips for using the Extended/Evil? Unix Product System (EUPS).

Philosophy

The "new" version of eups (1.2.x) includes a system to track the exact dependencies used to build a particular product. This has always been eups' intention, but version 1.1.x failed to achieve it. There are two reasons for this:

  • We can exactly recover a version used for some specific processing (e.g. pipette 4.6.1.2), including all of its dependencies (e.g. afw 4.6.1.0) to reproduce a bug or a result
  • If you're using a specified version, you know that nothing will change in your environment until you ask it to

Without this stability, a user is vunerable to the frequent ABI changes that occur in LSST C++ development and the resultant "broken stack" that occurs.

When you install something, its table file is "expanded" to include a full list of the exact dependencies it was built with, with recursive dependencies flattened. This protects against setting up a binary package with a dependency it isn't compatible with. This means that installing a package that wasn't built against other installed packages (or packages that have otherwise had their table expanded) is probably a bad idea. It's definitely a bad idea to install a package that was built against a LOCAL (-r) setup, and eups will print a warning as part of the install.

By `is compatible with', we mean, "Was built against and successfully installed" --- so it at least passed its unit-tests.

If you look at the installed table file (typically something like /path/to/product/ups/product.table) you'll see the setups in an if type == exact block. That's what the expansion does. There's no need to craft this block yourself when writing table files (and indeed you should never be editing table files by hand)

The "exact block" is used whenever you refer to a product by name and version, or eups resolves a product to a version and name (this is actually configurable using the "version resolution order" (VRO), described later); to see this in action type something like

$ setup afw -v
setup -v afw
Setting up: afw                             Flavor: Linux64    Version: 4.7.2.0+3
Setting up: |daf_base                       Flavor: Linux64    Version: 4.7.1.0+2
Setting up: |utils                          Flavor: Linux64    Version: 4.7.1.0+2
Setting up: |base                           Flavor: Linux64    Version: 4.7.1.0+2
Setting up: |boost                          Flavor: Linux64    Version: 1.47.0+3
...

note that the products such as daf_base are all indented just one space --- that is, they are directly resolved from afw's table file.

Tags

You are likely already familiar with packages being marked "current", but may not realize that this is [now] a special case of a general mechanism called a "tag". Tags are a powerful and useful way of simplifying your workflow when using multiple packages.

Before you use a tag, you need to declare it to eups (this requirement is intended to help avoid typos). Put the following in your ~/.eups/startup.py file:

hooks.config.Eups.userTags += ["ticket1234", "git"]

(I.e. two tags, ticket1234 and git). As a special case there's no need to declare your own user name as a tag before using it.

The only user tags you will see are your own, plus those you ask to see from other users. To see another user's tags, add an entry (tuple of tags, username); for example:

hooks.config.Eups.userTags += [(("ticket1234", "git"), "price")]

You can use wildcards for tag names when importing tags. The following is a real-world example that includes all my 'data challenge' tags (e.g., HSC-DC2, price-DC2, etc):

hooks.config.Eups.userTags += [(("git", "trunk", "*DC*",), "price")]

To see which tags you have defined, say eups tags. You'll see "global tags" as plain strings and "user tags" as prefixed by "user:". Global tags are tags shared across the stack (e.g., "current", "stable") and are managed centrally.

Once your user tags are defined, you can tag packages:

$ eups declare afw 4.7.3.2 -t ticket1234

The definition of the tag is stored within your ~/.eups directory. You can also tag uninstalled versions:

$ eups declare -t ticket1234 -r ~/LSST/afw

(a version tag:ticket1234 will be declared, also solely within your ~/.eups directory.

You can also install packages (generally into your sandbox or a "public" stack) with a tag:

$ scons opt=3 install declare --tag=ticket1234

although if it is a user tag (e.g. rhl), it's probably clearer to do this in two stages; first install a version in the usual way, and then apply your tag to it.

Then, to get packages marked with that tag when setting up another package, do:

$ setup -t ticket1234 pipette 4.7.1.1

That will get pipette 4.7.1.1 and its dependencies, except that when it gets to afw (or any other package with the ticket1234 tag), it will use that.

You can have multiple "-t <tag>"s on the same setup, and they are treated as a priority order. So I can say:

setup -t ticket1234 -t git foo 5.6.7.8

and I'll get foo 5.6.7.8 and for each of its dependencies I'll get the one that's tagged "ticket1234", or if that doesn't exist I'll get the one that's tagged "git", or if that doesn't exist I'll get the one that's tagged "current" (that's implicit). Now, note that because tags are effectively getting around the "exact" block mentioned earlier, you need to keep ABI compatibility in mind when you use them like this; eups is allowing you to shoot yourself in the foot, but you're the one that does the shooting.

If you're in doubt as to why you're getting a particular version of a package, check the "version resolution order" (VRO):

$ eups vro -t ticket1234 -t git
type:exact commandLine ticket1234 git version versionExpr current

These version descriptors are tried from left to right:

  • type:exact (I think) refers to when I say "setup --exact".
  • commandLine is a version named explicitly on the command-line
  • tags listed on the command-line, in order
  • version (I think) refers to the version recorded in an installed table file
  • versionExpr (I think) refers to a version expression such as ">=1.2.3.4"
  • current is the last resort

If in doubt, try adding a few "-v"s to your setup command and you'll see what version it's getting, and from which element of the VRO.

You can see that the tags make it easy to override groups of packages. For example, if you're working on a problem in meas_algorithms that also uses a local build of afw, but you're using an older version of pipette to test with, you can tag both your afw and meas_algorithms repositories with the same tag, and then setup pipette while specifying the tag --- you'll get pipette's and all its dependencies except for your local afw and meas_algorithms.

You can also provide a filename as the argument to the --tag option. The format is the output from eups list (or a set of setupRequired lines as in a table file) and every product listed is treated as if it's had a tag applied. For example,

eups list -s > foo.tag
...
setup -t foo.tag afw

should exactly reproduce your original setup of the specified package (afw in the example) and its dependencies. To re-setup everything just as it was, try this loop (-j prevents setting up dependencies):

while read pkg ver; do setup -j $pkg $ver; done < foo.tag

Of course, you can edit foo.tag to choose only interesting packages. Using a file as a tag is like writing a shell file containing lots of setup commands except that it's probably more convenient, probably faster, and certainly more flexible. For example,

setup -t rhl -t foo.tag afw

sets up that original state, except that anything tagged rhl takes precedence.

Default tags

To get a set of default tags for each setup command, put something like the following in your ~/.eups/startup.py:

def cmdHook(Eups, cmd, opts, args):
   if Eups and cmd == "setup":
       if not opts.tag:
           opts.tag = ["rhl", "Winter2012c", "beta"]

           if opts.verbose >= 0:
               import utils
               print >> utils.stdinfo, "Adding default tags: %s" % (", ".join(opts.tag))

eups.commandCallbacks.add(cmdHook)

Suggested Workflows

Working on a single package foo

  • Clone/checkout/pull foo's source
  • setup -r .
  • Edit
  • git commit -am "Message"
  • scons opt=3
  • Repeat until tests pass
  • scons opt=3 install declare --tag={something}
  • (Optional) Run integrated test using {something} tag

RHL: I don't like this. Is something a user tag or a global tag? If a user tag, there's no need to install it unless you want to expand the tablefile; if a global tag you should git tag it first.

KTL: {something} is supposed to be a user tag. I think it's useful to install the package so that you get a well-defined copy that you can depend on for builds of dependent packages without any risk of it changing underfoot.

Note that the setup -r . will setup the working directory and the current versions of all packages listed in the table file. If you'd rather use foo x.y.z's dependencies, but use your own copy of the source code, say

setup -r . foo x.y.z

Working on two or more interrelated packages

Personally, I [Paul] like to tag the git repositories I care about as 'git' (eups declare afw -t git -r /path/to/afw), and then setup -t git pipette (there's no need to invent a version name if you're declaring a tag). This is using a tag to mix-and-match between versions that you're actually working on. If other people might want to use your tags, please use a name such as price-git rather than git; eups user tags don't (currently) support namespaces to differentiate my git tag from RHL's.

It's probably a good idea to say

$ eups list -t price

from time-to-time and check that only packages that you really need private copies of are tagged price.

If you want to work on afw and mess_algorithms simultaneously, the recommended workflow is:

$ git clone git@git.lsstcorp.org:LSST/DMS/afw
$ git clone git@git.lsstcorp.org:LSST/DMS/meas_algorithms
$ cd afw
$ setup -r .
# or setup -r . afw 4.7.1.0+1 to get that version's exact dependencies
# edit; git commit; repeat
$ scons opt=3
# passes tests
$ eups declare -r . -t {username} # or {username}-git as above
$ cd ../meas_algorithms
$ setup -t {username} -r .
# or setup -t {username} -r .  meas_algorithms 4.7.1.0+1
# edit; git commit; repeat
$ scons opt=3
# oops, problem that needs to be fixed in afw
$ cd ../afw
# edit
$ scons opt=3; cd ../meas_algorithms; scons opt=3
# to rebuild both; afw working directory is declared and still setup

Note that a "git checkout" or "git pull" in the declared afw directory above (e.g. to work on a different ticket) may cause problems, since the setup of meas_algorithms still uses it. See the next subsection for ways to deal with this.

Working on multiple versions of a package

There are multiple solutions, each with their merits. They divide into whether you use git or eups for the switching.

  • Clone multiple versions of the relevant repo using "git clone", and declare them as different versions using "eups declare -r /path/to/product package version", or set them up explicitly with "setup -r /path/to/product". This is using eups to do the switching.
    • Clone the entire history and all the branches in the usual way.
    • If you're worried about the extra overhead in the download, you could clone your local "main" repository. That means that when you're done you have to push to your main repo and then push from the main repo to the LSST central repo.
    • Clone only the single branch of interest; see http://stackoverflow.com/questions/1778088/how-to-clone-a-single-branch-in-git
  • Single clone of the relevant repo. Declare it once only using "eups declare -r /path/to/product package version". But then you have to use git to switch between the versions rather than eups.

Lock Problems

The new EUPS has a locking system to prevent contention from corrupting the database. Unfortunately it sometimes results in commands needlessly failing. Workarounds include:

  • Add the argument --nolock to your setup or eups command.
  • Clear locks using the command eups admin clearLocks; this is especially useful if the locks are clearly stale.
  • Disable locks on a repository by putting hooks.config.site.lockDirectoryBase = None in a startup.py file: either in ~/.eups/ or globally for a site. Use eups -v startup to list possible startup files (without -v you only see non-empty startup files that already exist). If you are using a sandbox with a shared stack and having frequent lock errors then this is definitely the way to go!

Distribution

Creation

A distribution (also known as a package server) allows one to clone packages between systems. Of course, the LSST package server is the official place to get packages, but there may be times when you want to do something unofficial --- like reproducing your setups on a different machine run by a different user so they can debug a particular problem. Here's how to do that:

Setup

Here is some stuff to put in your ~/.eups/startup.py or $EUPS_PATH/site/startup.py. The purpose is:

  • Make the command-line shorter by specifying some good defaults (otherwise, the command-line can be pretty long and complicated)
  • Specify how to build LSST products (the "build files" in the LSST products use a macro to enhance configurability and reduce boilerplate; this defines the macro)
# Make the command-line shorter by specifying some good defaults
def cmdHook(Eups, cmd, opts, args):
    if cmd.split()[0] == "distrib":
        subcmd = cmd.split()[1]
        if subcmd == "create":
            opts.distribTypeName = "builder"
            opts.allowIncomplete = True
            opts.useFlavor = "generic"
            opts.serverDir = "/path/to/packages" # Or wherever you want to put the package server; or leave it off
eups.commandCallbacks.add(cmdHook)

# How to build LSST products (pull from git)
#
# LSST products' build files should contain:
# @LSST BUILD@
# build_lsst @PRODUCT@ @VERSION@ @REPOVERSION@ [repository name]
hooks.config.distrib["builder"]["variables"]["LSST BUILD"] = """
build_lsst() {
    if [ -z "$1" -o -z "$2" -o -z "$3" ]; then
        echo "build_lsst requires at least three arguments"
        exit 1
    fi
    productname=$1
    versionname=$2
    repoversion=$3
    reponame=$4
    if [ -z "$reponame" ]; then
        reponame=$productname
    fi
    builddir=${productname}-${versionname}
    if [ -d $builddir ]; then
        rm -rf $builddir
    fi
    mkdir $builddir &&
    git archive --format=tar --remote=git@git.lsstcorp.org:LSST/DMS/${reponame}.git ${repoversion} | tar -x -C $builddir &&
    cd $builddir &&
    setup -r . &&
    scons opt=3 install version=$versionname
}
"""

General use

Install the product you want to distribute, and all its dependencies. When you build+install, make sure you're using the correct dependencies, because those dependencies that you use will flow into the distribution. Then, it should be a simple matter of:

eups distrib create PRODUCT VERSION

If you didn't specify a server directory in your startup.py, then you also need to add --server-dir=/path/to/packages. If you have some packages in your dependencies that don't have available build files, try adding -S buildFilePath=:/path/to/buildFiles (note the colon; it's important that there be an empty element of that path).

Then you should be able to point your friends to the /path/to/packages and have them put it in their EUPS_PKGROOT. There are a number of ways to specify this. If the directory is visible on the web, you can use http://my.machine.org/path/to/packages. If your friend has ssh access, use scp:my.machine.org:/path/to/packages. Finally, if your friend is on the same system, you can just specify /path/to/packages.

Rebuilds

The distribution mechanism can also be used to generate rebuilds of dependent packages. It does this by creating rebuild packages (e.g., version "1.2.3+1" out of version "1.2.3") with the correct dependencies. These packages don't exist in the eups system, but you can use eups distrib install to get them. Because the "plus versions" generated by the default build versioning system can appear to be official packages generated by LSST, there is some additional setup (in startup.py required to use this feature:

global defaultRepoVersioner, defaultVersionIncrementer
defaultRepoVersioner = hooks.config.Eups.repoVersioner
defaultVersionIncrementer = hooks.config.Eups.versionIncrementer
global hscSuffix
hscSuffix = '_yourNameHere' # Put your name/initials here

# Get the "repository version" from a build version
def hscRepoVersioner(product, version):
    if version[-len(hscSuffix):] == hscSuffix:
        import re
        return re.sub(r"^([\w.+-]*[0-9.])[a-z]+" + hscSuffix + "$", r"\1", version)
    return defaultRepoVersioner(product, version)

# Increment a build version using "letter versions"
def hscVersionIncrementer(product, version):
    import re
    match = re.search(r"^([\w.+-]*[0-9.])([a-z]+)" + hscSuffix + "$", version)
    if not match:
        return version + "a" + hscSuffix

    repoVersion = match.group(1)
    letters = match.group(2)

    letterVersionNumber = 0
    for l in letters:
        if not l >= 'a' and l <= 'z':
            raise RuntimeError("Version %s contains an illegal character %s" % (iversion, l))
        letterVersionNumber = 26*letterVersionNumber + (ord(l) - ord('a'))

    letterVersionNumber += 1

    letters = ""
    while letterVersionNumber:
        letters += chr(letterVersionNumber%26 + ord('a'))
        letterVersionNumber //= 26

    letters = ''.join(reversed(letters))

    return repoVersion + letters + hscSuffix

hooks.config.Eups.repoVersioner = hscRepoVersioner
hooks.config.Eups.versionIncrementer = hscVersionIncrementer

The functions are prepended with "hsc" because these are what we use for HSC development. (Looking at them, they probably don't work as written in the presence of multiple sources of rebuilds.)

Now, to create a package that will rebuild all dependencies of pipette 4.7.1.2 that are dependent upon afw and make them dependent now on afw 4.7.2.1, use:

$ eups distrib create pipette 4.7.1.2 --rebuild afw:4.7.2.1

This will tell you that it's creating a new package (not pipette 4.7.1.2, but something like pipette 4.7.1.2a_hsc). Then you can

$ eups distrib install pipette 4.7.1.2a_hsc

manifest.remap

The manifest.remap file in $EUPS_PATH/site/ and/or $HOME/.eups/ provides a set of mappings for either creating or installing a distribution. The format is:

Product[:serverVersion]        [product:]myVersion   [flavor]

Anything after a # is treated as a comment. This remaps the Product (and a specific serverVersion if specified) to myVersion. If myVersion is None, then the package will be ignored (which can cause problems if it's a dependency, so be careful!).

One thing this comes in useful for is using your own system's version of various packages. For example, this is how I got things building on my MacBook? Pro (10.7):

$ mkdir -p $EUPS_PATH/DarwinX86/python/system/ups
$ cp $HOME/LSST/devenv/buildFiles/python/python.cfg $EUPS_PATH/DarwinX86/python/system/ups
$ touch $EUPS_PATH/DarwinX86/python/system/ups/python.table
$ eups declare python system -r $EUPS_PATH/DarwinX86/python/system
$ touch gcc.table
$ eups declare gcc system -M gcc.table -r none
$ cat $EUPS_PATH/site/manifest.remap
python  system
gmp     None
mpfr    None
mpc     None
gcc     system

Then you can eups distrib install as normal, and it'll use the system python and system gcc to satisfy EUPS dependencies, and not try to download and install different versions.

Note that you can not use the system gcc to actually build the stack -- only clang is known to work on Macs.

Dream server

The "dream server" is a new feature (since version 1.2.28). The intent is to use general build files (e.g., those in LSST/DMS/devenv/buildFiles) that reference @PRODUCT@ and @VERSION@ to create a package that doesn't exist anywhere --- not even on another server. We dream about the existence of a package, and it appears. This approach has its limitations (we cannot use it to install a tree of products, since we don't know what version to use for each product; and it assumes that a product's build system never changes, whereas we know that, e.g., configure flags are often added as features are added, and some may be required for a proper installation), but it is also quite useful.

To use it, add at the end of your EUPS_PKGROOT environment the string |dream:/path/to/buildFiles where the pipe (|) is the usual delimiter for EUPS_PKGROOT (colons are often used in URLs), and /path/to/buildFiles is the path to your local copy of LSST/DMS/devenv/buildFiles. Then, you should be able to say:

$ eups distrib list ds9
No matching products available from primary server (http://dev.lsstcorp.org/pkgs/std/w12)
No matching products available from secondary server (scp:ipmu:/data/packages)
No matching products available from secondary server (dream:/home/price/LSST/devenv/buildFiles)
$ eups distrib install ds9 6.2
Required product xpa 2.1.13+1 is already installed; use --force to reinstall
Installing ds9 6.2 for Linux64...
Package ds9 6.2 installed successfully
$ eups list ds9
   6.2        	current

And now I have ds9. Note that I must have the xpa package installed already or this will fail, as it's a dependency.

If an install fails, have a look at the build log for clues. Most likely, you typed the version name wrong, or the build system for a particular product has changed. You're welcome to update the build files in LSST/DMS/devenv/buildFiles, but please do not check in customizations to specific machines, or changes required to build older versions of a particular product (we generally want the buildFiles to track the latest stable version of a product).

It's possible this functionality interferes with eups distrib install PRODUCT VERSION --just --force, but it's not yet clear. If it does, just remove the "dream" part of the EUPS_PKGROOT. It really only needs to be there all by itself for those situations when you want to use the dream server.