Ticket #334 (closed defect)

Opened 11 years ago

Last modified 7 years ago

Create new templates for package files

Reported by: ktl Owned by: ktl
Priority: normal Milestone:
Component: sconsUtils Keywords:
Cc: smm, rhl, rowen, robyn, RayPlante Blocked By:
Blocking: Project: LSST
Version Number:
How to repeat:

not applicable


Certain files are common to (or similar across) all packages. Many of these have been generated in an ad hoc manner and there is little documentation on their proper use and customization in new packages. New, cleaned-up templates should be created.

These files include at least:

  • SConstruct
  • doc/{SConscript,doxygen.conf}
  • lib/SConscript
  • python/lsst/$package/{SConscript,__init__.py,${package}Lib.i}
  • tests/SConscript
  • ups/*

Change History

comment:1 Changed 11 years ago by ktl

  • Status changed from new to assigned
  • Component changed from unknown to scons

comment:2 Changed 11 years ago by rowen

It would to put certain common operations in utility scripts. For example a script that would find all files in a nested hierarchy of directories would allow so to:

  • simplify lib/SConscript by searching for all .cc files in src
  • simplify SConstruct by searching for all SConscript files

RO already has a findFiles function bit it's a lot to bring in for just this.

comment:3 Changed 11 years ago by ktl

  • Status changed from assigned to inTicketWork

I think these work better than what we have. Can someone check that these work elsewhere?


# -*- python -*-
# Setup our environment
import glob, os.path, re
import lsst.SConsUtils as scons

dependencies = ["boost", "python", "utils", "pex_exceptions"]

env = scons.makeEnv("daf_base",
                    r"$HeadURL: svn+ssh://svn.lsstcorp.org/DMS/daf/base/trunk/SConstruct $",
                    [["boost", "boost/shared_ptr.hpp"],
                     ["python", "Python.h"],
                     ["utils", "lsst/utils/Utils.h", "utils:C++"],
                     ["pex_exceptions", "lsst/pex/exceptions/Runtime.h", "pex_exceptions:C++"]
LSST Data Access Framework base package

# Boilerplate below here.  Can we make this automatic somehow?

pkg = env["eups_product"]
env.libs[pkg] += env.getlibs(" ".join(dependencies))

# Build/install things
for d in Split("lib python/lsst/" + re.sub(r'_', "/", pkg) + " examples tests doc"):
    SConscript(os.path.join(d, "SConscript"))

env['IgnoreFiles'] = r"(~$|\.pyc$|^\.svn$|\.o$)"

Alias("install", [env.Install(env['prefix'], "python"),
                  env.Install(env['prefix'], "include"),
                  env.InstallAs(os.path.join(env['prefix'], "doc", "doxygen"),
                                os.path.join("doc", "htmlDir")),
                  env.InstallEups(env['prefix'] + "/ups", glob.glob("ups/*.table"))])

scons.CleanTree(r"*~ core *.so *.os *.o")

# Build TAGS files
files = scons.filesToTag()
if files:
    env.Command("TAGS", files, "etags -o $TARGET $SOURCES")



# -*- python -*-
import glob

env.Program(glob.glob("*.cc"), LIBS=env.getlibs(env["eups_product"]))

# Tests
import lsst.tests
tests = lsst.tests.Control(env, verbose=True)

if tests.runExamples:


# -*- python -*-
import glob

pkg = env["eups_product"]
# Replace by "../src/*/*.cc" for afw and others as needed.
env.SharedLibrary(pkg, glob.glob("../src/*.cc"),
    LIBS=filter(lambda x: x != pkg, env.getlibs(pkg)))


# -*- python -*-
import glob

env.Program(glob.glob("*.cc"), LIBS=env.getlibs(env["eups_product"]))

# Tests
import lsst.tests

tests = lsst.tests.Control(env, verbose = True)

# This one seems a bit harder to make generic.
for target in tests.run("*.py"):
        "../python/lsst/daf/base/_baseLib%s" % (env['LDMODULESUFFIX']))


# -*- python -*-


Still looking at the SWIG stuff...

comment:4 Changed 11 years ago by ktl

[SwigFAQ] should help with the SWIG side of this.

After discussion with RHL, it appears that we cannot (yet) get away from listing all the recursive dependencies of packages that this one needs, even though it seemed that deprecating MacOS 10.4 would solve this issue. If package A needs B, but B has an include file that includes a header from C, then A will need to list C in its SConstruct on all platforms. There is a longer-term idea of how to get around this, but for now, we'll have to list dependencies of dependencies.

comment:5 Changed 11 years ago by ktl

SwigFAQ, that is...

comment:6 Changed 11 years ago by smm

I pointed Russ at this ticket and he implemented and tested the above build file changes for sdqa with no issues. He suggests adding *.pyc to the CleanTree() call.

comment:7 Changed 11 years ago by ktl

I think there's actually a problem with the examples and tests SConscripts in that they try to build a single program using all *.cc files instead of building individual programs out of each .cc file.

On the other hand, given our use of capable test harnesses, it's not exactly clear to me when a new test should be placed in a separate .cc file and when it should just be added to an existing one.

comment:8 Changed 11 years ago by smm

In the DC2 AP I produced a single executable from all the test .cc files. The drawback is that sometimes you end up having to run through a bunch of tests to get to one that is failing (in my case I couldn't just reorder/turn off tests due to use of a single random number sequence). Another consequence is that a seg fault in an early test means the rest of the tests won't get run.

I'm now of the opinion that my DC2 strategy was a bad idea. For DC3 I'd like to both partition things into more executables and make sure that each test obtains (and records) a distinct random number seed.

So, I'd say one executable per test .cc file is a resonable default. If we want the ability to combine a bunch of .cc files into a single test executable and have multiple test executables, then the tests/SConscript may have to become package specific. An alternative would be to introduce a subdirectory per C++ test executable. All .cc files in a subdirectory would be combined into one executable (and any top-level tests/*.cc files could be linked in). The generic tests/SConscript would also test whether a subdirectory contains its own SConscript before proceeding - this would allow more complicated testcases to have localized custom build rules.

comment:9 Changed 11 years ago by ktl

OK, so if we have multiple .cc files in examples and tests, the corresponding SConscript lines should change from:

env.Program(glob.glob("*.cc"), LIBS=env.getlibs(env["eups_product"]))


for target in glob.glob("*.cc"):
    env.Program(target, LIBS=env.getlibs(env["eups_product"]))

comment:10 Changed 11 years ago by rhl

scons doesn't do a great job on reporting errors importing SConscript files, so the boiler plate should probably read:

for d in Split("lib python/lsst/" + re.sub(r'_', "/", pkg) + " examples tests doc"):
        SConscript(os.path.join(d, "SConscript"))
    except Exception, e:
        print >> sys.stderr, "%s: %s" % (os.path.join(d, "SConscript"), e)

I print the error rather than raising an error as this code runs before options such as --help are processed (as help uses information that's gathered why building the Environment)

comment:11 Changed 11 years ago by rhl

The InstallEups line can be simplified to

env.InstallEups(env['prefix'] + "/ups")

as table and build files are automatically installed.

comment:12 Changed 11 years ago by ktl

The new templates will be checked into package DMS/devenv/templates.

comment:13 Changed 10 years ago by robyn

KT, Have you completed your initial pass on the devenv template directory? If so, please document its use on twiki (if not already done) and then close the ancient ticket.

comment:14 Changed 8 years ago by ktl

  • Status changed from inTicketWork to closed

Some templates were generated; new versions should probably be created as part of the Winter2012 rework.

comment:15 Changed 7 years ago by robyn

  • Milestone DC3 Infrastructure Prep deleted

Milestone DC3 Infrastructure Prep deleted

Note: See TracTickets for help on using tickets.