wiki:PT1_2SoftwareInfrastructure
Last modified 8 years ago Last modified on 02/02/2011 10:56:09 AM

PT 1.2 Environment and Build Tools

This page is for prioritization of requested software-related changes for DC3 PT1.2.

Goals for 1/25:

  • A prioritization of these changes
  • For the things that need to go through TCT, a proposal for requested changes.

Third Party Components

g++

Decide on version.

Bosch: One component of the multifit package doesn't compile on gcc 4.5; this is a known bug in gcc and a regression from gcc 4.4. I can work around it, but it would be easier to go with gcc 4.4.

boost

Upgrade to boost 1.45?

KTL: I'm having some issues building afw with 1.45, particularly having to do with serialization of measurement classes. 1.46 (trunk) seems at first to be better, but I may be able to get 1.45 to work with some more effort.

python

Suggested upgrade is to python 2.7.

Ray has made python 2.7 available through the stack, and will try a stack build to see what 2.7 coughs up.

swig

Suggested change is to swig 2.0. RHL: "This has Serge's shared_ptr fixes, and will require boilerplate changes to *.i files and adding more include files."

Bosch: Would very much like to see this upgrade. There are lingering "I don't know why this works" problems in many .i files anyway that need to be addressed at some point, and hopefully wrapping new code will be much less painful.

Jim will do some simple investigation of what's involved with moving to swig 2.0.

eups, Scons, SconsUtils

General agreement on an upgrade to Scons 2.0 and trunk SconsUtils. Upgrade eups to trunk. RHL: "Supports exact setups (i.e. setup foo 1.2.3 guarantees the versions of everything that were used to install foo 1.2.3); Tags (user defined or in files); Remapping products in eups distrib (e.g. use my system python [2.6.1] not 2.5.1 from NCSA)."

mpich2

RHL: "We're running an ancient version."

levmar: Not for PT 1.2

Bosch: multifit would like to make use of the constrained Levenberg-Marquardt implementation found here:

http://www.ics.forth.gr/~lourakis/levmar/

Note that LAPACK is a dependency (see more below).

TMV/LAPACK: Not for PT 1.2

Bosch: According to Mike Jarvis, his TMV (template matrix vector) library provides an easy way to configure and set up BLAS/LAPACK to work with C++, and it gives us another matrix template library to compare Eigen's performance against.

http://sourceforge.net/projects/tmv-cpp/

Coding Practices

Refactor packages

Fuller discussion of refactoring the package tree for PT1.2 is at PT1_2CodeRefactor

RHL: "Do we need as many products as we have, all in separate svn trunk directories? Would a dramatically flattened tree be better? Put _everything_ in one build tree and uprev the whole thing when anything changes (i.e. just one version name for all of LSST?)."

Bosch: Can we reorganize/split up afw?

KTL: As Jim's comment indicates and as Ray has said in an E-mail, we should probably have a few more packages in places and maybe a few less in others ({meas,ip}_utils) but should not dramatically flatten the tree.

Update Code Standards

Review code standards, determine what's value added and what's not, and adjust standards accordingly.

pImpl

Coder pass through code converting to pImpl whenever appropriate. In particular, use macros PTR and CONST_PTR not class typedefs Ptr and ConstPtr along with forward declarations of classes should reduce dependencies without any refactoring.

Refactor "lsst" product

RHL: "Make it possible to say

setup lsst
eups distrib install datarel

and never need the extra "lsstpkg" layer."

Refactor package dependencies on mpich2

RHL: "I'd like to look at the dependencies that force the mpi stuff even for a vanilla afw install. As we look forward to outside users of the stack, this becomes ever more important. I realise that boost-mpi is the reason for the current dependencies."

KTL: This is currently just for serialization of PropertySet, which is handled by the formatter in daf_persistence. Splitting formatters into per-Storage classes would enable the MPI-dependent code to move into a separate package.

Documentation Practices and Tools

Doxygen upgrade

Suggestion: upgrade to latest doxygen.

Doxygen warning cleanup

Coder pass through code to significantly reduce doxygen warnings.

Replace doxygen for python?

Consider changing from doxygen to something Sphinx-based for python code.

Redo schema handling

Master version of schema moves to svn instead of EA. Tools needed to extract comments for schema browser and document stored procedures/functions/views.

Platform Support

OS/X

Address build issues on OS/X. RHL: "One thing that's coming up is clang++, which Craig Loomis tells me can build almost all the stack but isn't publicly available [it'll be in OS/X 10.7 this summer]."

External Technologies

buildbot

Suggestion: run a buildbot with icc on linux.

Requirement: run a daues-bot to test pipelines, not just products. (a.k.a. Tuesday mini-production)

RAA: Current Varients

Hardware: VM of Red Hat 64 bit machine; gcc

  • Nightly:
    • Full Trunk Vs Trunk: trunk package vs all trunk dependencies; updates web-based crosslinked doxygen docs
    • Full Release Pipeline: all current packages of Current Release; local installation directory cleared on start; updates web-based crosslinked doxygen docs
    • Release Incremental: all current packages of Current Release; local installation directory not cleared on start.
  • SVN commit Triggered (in debug):
    • Trunk Vs Current: trunk package vs all 'current' dependencies
    • Trunk Vs Minimal: trunk package vs all 'minimal required' dependencies
  • Note:
    • error emails now provide direct web locator to log file (in contrast to 2 hop pointer);
    • in addition to the package gurus, the last modifier also receives the error email;
    • automatic Ticket generation is not done but is an potential option.

RAA: Possible Varients

  • test pipette versions of algorithm constructs w/ benchmark dataset
  • test sst versions of constructs of algorithm stages w/ benchmark dataset
  • test orca version of pipeline construct w/ benchmark dataset
    • this is different from Daues-bot which will not be comparing output against benchmark data
  • User-Triggered Ticket Vs Current: All packages with specified Ticket# branch are tested against 'current' dependencies
  • User-Triggered Ticket Vs Trunk: All packages with specified Ticket# branch are tested against 'trunk' dependencies
  • Other supported compiler choice (icc?)
  • Other supported platform and specifications (via HW or VM)
  • automatic Ticket generation for some buildbot types (issuing Tickets on trunk failures might be too draconian during R&D development; but should be done for failures on Release products).
  • SVN commit triggered standards checking on Trunk packages
  • SVN commit triggered coverage analysis on Trunk packages
  • lsst doxygen web-access improved
  • ....

Moving from svn to hg?

Consider some form of distributed version control.

lsst-data mailing list

RHL: " It's getting worse as the traffic grows; let's just do it [split it]."

KTL: We have already been setting up side lists, but they don't yet seem to be siphoning off the majority of lsst-data traffic. Is the major missing list an apps list?

JB: How about introducing discussion forum? FudForum?, IPBoard, phpbb, we recently did a comprehensive review of what is available, I can compile this info.