wiki:TCTPackageProposal
Last modified 11 years ago Last modified on 01/08/2009 05:17:22 PM

Package Proposal

Introduction

I believe that the package management system, based on eups, that we are using is working reasonably well. We have, however, found a series of problems and infelicities exposed by the complexity of the LSST's product layering and some enhancements are needed.

This document draws on work done by KTL and REO, as well as the author, RHL.

There are three aspects to package management:

  • Using a set of packages (including some that are checked out of svn)
  • Defining a set of packages that work together
  • Installing defined sets of packages on a local machine

Justification for Yet Another Package Manager

There are so many package managers out there that we need to justify the choice of using our own. The primary reason is that we need to be able to mix-and-match versions at a very fine level, while being able to reconstruct (and use!) old versions. To the best of my knowledge this is not provided by any of the standard distributions (e.g. yum; ports; fink) --- they can install a consistent set of packages, but only one version is installed at any given time.

One might hope to use e.g. yum as an installer, and maintain information about the set of packages in use within eups. This is in theory possible, but would still require a complex wrapper around e.g. yum to install in the correct places; we instead use a set of light-weight "build" files to do the installation --- and there is no reason why they shouldn't use e.g. yum as the "transport layer".

Using a set of products

A product is in use when it is, "setup". There have been some objections to this name, but I propose keeping it. See section wiki:TCTPackageProposal#SyntacticSugar on a way to define aliases for setup/unsetup when configuring eups.

Historically we have used a vague concept, "current" to mean the default version of a package.

What is current?

Types of Tag

This document envisages:

  • newest: The most recent installed version
  • stable: The version that the collaboration currently supports; project scope
  • alpha: A version that's being tested; local scope
  • beta: A version that's being tested; project scope

XXX Is alpha/beta right? We need a project scope tag in addition to stable Tags with project scope are attached to versions centrally; locally scoped versions may be defined by users or sites.

Note that tag types must be defined before they can be used, typically in in a file in \$EUPS_STARTUP, probably in source:lssteups/python/lsst/eupsStartup.py. E.g.

    import eups
    eups.defineValidTags("beta")

The tags "current" and "stable" are defined by eups (although we would be free to delete them if the need arose. Users are free to add their own tags in their own startup files; e.g. I use --tag=rhl to define sets of versions that I want setup, followed by setup --tag=rhl.

If we wish, we can define per-user tags; one way would be via the lssteups's lsstEupsStartup.py file; we need to define a tag and add it to the beginning of all setup commands:

import eups, os

user = pwd.getpwuid(os.geteuid())[0]		# user name, e.g. "rhl"

eups.defineValidTag(user, ["beta", "current"])

def cmdHook(cmd, argv):
    """Always setup --tag=rhl if available"""

    if cmd == "setup":
        argv.insert(1, "--tag=%s" % user)

eups.commandCallbacks.add(cmdHook)

Note that this allows other users to setup my versions if they so desire, via setup --tag=rhl ...

Syntactic Sugar

You can define aliases for commands via the eupsCmdHook function defined in the file pointed at by your (optional) $EUPS_STARTUP variable. For example, K-T proposed:

  • eups fetch: eups distrib install
  • eups publish: eups distrib create

If you want to provide aliases for "setup" and/or "unsetup" you can use a configure option, e.g.

./configure --with-setup-alias=activate:deactivate

A shell script such as lsstpkg may be provided, but only if it adds significant value. N.b. it may be too late to apply this criterion.

Specifying Package Dependencies

SpecifyingPackageDependencies?

We currently specify a product's dependencies as a list in the SConstruct file, e.g.

env = scons.makeEnv(
    "afw",
    r"$HeadURL: svn+ssh://svn.lsstcorp.org/DMS/afw/trunk/SConstruct $",
    [
        ["boost", "boost/version.hpp", "boost_system:C++"],
        ["boost", "boost/serialization/base_object.hpp", "boost_serialization:C++"],
        ["cfitsio", "fitsio.h", "m cfitsio", "ffopen"],
        ["wcslib", "wcslib/wcs.h", "m wcs"],
...
    ],
)

env.libs["afw"] += env.getlibs("boost wcslib cfitsio ...")

This is clearly not a good design. A reasonably-standard way of handling a related problem is via pkg-config which allows me to ask any package what compiler and linker flags are required to build it. We should adopt something similar.

I propose that each package specify its required list of libraries in the form of a python list identical to that now passed to makeEnv; let us call this a requirements list. The product would only include its direct dependencies in its requirements. makeEnv would then recursively lookup all the product's dependent products, extract their requirements list, merge them, and proceed as at present.

It is not yet clear to me if we really need two types of requirements list; those needed to build the package, and those needed to use the package. The need for such a distinction is at least in part dependent on when we resolve shared library symbols. If we assume that we don't support os/x <= 10.4, then shared libraries can be handled the same way as Linux; see Serge's notes on wiki:LinkingDarwin.