wiki:Winter2012/sconsUtilsDocumentation
Last modified 7 years ago Last modified on 01/11/2012 02:27:50 PM

Documentation for the Winter2012 Upgrades to sconsUtils

Note: sconsUtils now has Doxygen-generated documentation. If you don't find the answer you're looking for here, please look there (especially under the "Modules" tab).

What's Changed

Dependency Handling and Configuration

  • Rather than have each package specify how to configure all its dependencies, a package now declares how to configure itself, using a new Python-language <package>.cfg file in its ups directory. These specify the names of all dependencies, and the .cfg files for those dependencies are examined recursively.
  • autoconf tests are now disabled by default; you can use scons --checkDependencies to run them, but this will only be necessary when debugging build problems.
  • The CheckSwig function has been removed. SWIG itself is set up when you declare it as a dependency in a .cfg file, and packages that provide a SWIG interface now automatically update the SWIGPATH environment variable when they are configured.
  • env.getLibs(categories) can be used to extract the libraries the current package should be linked with. Multiple categories may be specified at once; a SWIG module is usually linked with LIBS=env.getLibs("main python"), while a C++ unit test is usually linked with LIBS=env.getLibs("main test"). Several categories are available:
    • "self" will return a list of most libraries provided by dependencies (i.e. the libraries usually need to build the packages own library).
    • "main" will return all the libraries in "self", plus the package's own library.
    • "test" will return a list of libraries to be linked only with tests (the boost_test package, for instance, puts the boost_unit_test_framework library in this category).
    • "python" will return a list of libraries to be linked only with Python modules (the python package puts all its libraries in this category).
  • External (third-party) packages will now put their include directories in the (sconsUtils-specific) XCPPFLAGS variable instead of CPPPATH; this sort of thing is encouraged by SCons users as an optimization. This means SCons will not consider external headers as dependencies, and will not waste time checking them for changes when determining what to rebuild.

Doxygen Builds

  • Doxygen config files are now generated by sconsUtils; inputs and files to exclude are now generated automatically by SCons.
  • Packages can define Doxygen config include files to be used by dependent packages (specified in the .cfg file). Essentially all of the Doxygen configuration now lives in lsst/base, and is automatically included by packages that depend on it.
  • Doxygen TAG files (http://www.stack.nl/~dimitri/doxygen/external.html) are used to link to documentation in other packages. This doesn't provide a completely cross-linked set of documentation, because the links only go one-way (from dependent to dependency), but it's a step in the right direction, and it causes Doxygen to generate many of the same warnings that appear when Doxygen is run on multiple packages at once to generate a complete set of cross-linked documentation.
  • The "doc" and "examples" targets are no longer built by default when you simply run "scons" with no arguments. Running "scons doc" now correctly checks for dependencies, and will rebuild or not rebuild based on whether changes have been made.
  • Doxygen html output now goes in "doc/html", not "doc/htmlDir", and is installed in the same (relative) location.
  • Warning: Doxygen section names must now be unique across packages. Please prefix section names with the package name, e.g. in afw secIntro is now afwSecIntro.

SConsUtils Usage

  • The actual name in Python has been changed from lsst.SConsUtils to lsst.sconsUtils to match LSST naming conventions, and it's now a full Python package with several submodules rather than a single large module. The name change is also an easy way to catch any SCons files using the old version that escape the package update script.
  • Reflecting the fact that the old MakeEnv did many things that should only be run once, the special sconsUtils environment object is now available as a module variable: lsst.sconsUtils.env. This also allows much of the setup of the environment to be done when sconsUtils is first imported (and never accidentally repeated), and it allows the boilerplate-replacement scripts discussed below to work without being passed the environment explicitly.
  • The lsst.sconsUtils.scripts module now contains high-level functions that can be used to do all or most of the work of most SConstruct and SConscript files. The typical SConstruct file will contain only a call to lsst.sconsUtils.scripts.BasicSConstruct, and SConscript files can siimilarly use one of lsst.sconsUtils.scripts.BasicSConscript.{lib,python,tests,doc,examples}. See the sconsUtils documentation for more information on their usage; the python and tests SConscripts in particular will often require some customization. Reading the code for these functions should give you an idea of what's going on under the hood.

lsstimport

We're now using the method described here to make multiple directories appear as a single Python package. This involves adding a non-empty __init__.py file to python/lsst/.

version.py

With the git transition that happened around the same time, we lost the ability to embed revision numbers directly into our Python code. Instead, sconsUtils will by default generate a "version.py" file for every package. This will generally require even pure-Python packages to be "built" with SCons before they can be used.

How to Convert an LSST Package

I have provided a Python script to do much of the conversion automatically. You'll need to download it, put it in your path, and run it from an SVN working copy of the package you want to convert, with the EUPS name of the package (e.g. "meas_algorithms", not "meas/algorithms") as the only argument.

It should only be run on a clean SVN working copy, as it modifies svn:ignore properties and will add and remove some files, and doesn't make backups (that's what version control systems are for). It also overwites some files that contain logic that should be preserved; I find this very easy to do with an svn-aware graphical merge tool (I use one called "meld"). In detail:

  • It adds/updates the __init__.py files.
  • It removes the doxygen.conf file and adds an empty doxygen.conf.in file (which usually won't need to be modified).
  • It overwrites the SConstruct file and and SConscript files it finds with simpler ones.
    • You'll probably want to diff the tests/SConscript file to make sure you preserve any logic about which tests to run when, but it should automatically be pretty intelligent about making SWIG modules and not trying to run them as tests (as long as you don't have additional sources that go into the module besides the .i file, but there are ways to deal with that - see the sconsUtils Doxygen documentation.
    • You may need to modify the python/*/SConscript file to keep it from trying to build included .i files as separate modules.
    • If you have SConscript files in include or someplace else unusual, you'll have to modify those yourself. Note that instead of Import("env") you should do from lsst.sconsUtils import env. You may also want to update lsst.sconsUtils.targets; again, see the sconsUtils Doxygen documentation.
  • It adds a basic ups/<package>.cfg file that usually will need to be modified.
    • Generally, you'll want to look at the original version of the SConstruct file (or etc/dependencies.dat, if applicable) to see what to put in the dependencies dict. Remember that dependencies are automatically recursive, so you should only include packages whose interfaces you actually use.
    • If this is a pure-Python package, change the keyword arguments to Configuration appropriately: set headers=[], libs=[], hasSwigFiles=False. The defaults will be correct for most packages that have C++ components.

It's also a fairly simple script, so you can just look through it to see what it does. And, of course, the best way to learn may just be to look at the packages in the Winter2012a branch that have already been converted.