Ticket #1782 (closed enhancement: fixed)

Opened 8 years ago

Last modified 7 years ago

Upgrade LSST code to new third-party packages (not including SWIG 2)

Reported by: jbosch Owned by: jbosch
Priority: normal Milestone:
Component: build process Keywords:
Cc: rhl, price Blocked By:
Blocking: Project: LSST
Version Number:
How to repeat:

not applicable


All LSST packages need to be upgraded to work with new versions of third-party packages. Most changes will be related to Boost and Eigen 3 (see also #1752).

The SWIG 2 upgrade will be separate.

The implementation of this ticket may take place at the same time as #1780 and #1752. Ideally, this transition will be synchronized with the transition to a new version control system.

Change History

comment:1 Changed 8 years ago by DefaultCC Plugin

  • Cc rhl added

comment:2 Changed 7 years ago by jbosch

Work for this ticket is proceeding on branches/Winter2012a, along with #1780 (upgrade sconsUtils), #1752 (upgrade to Eigen 3), and the first part of #1793 (remove daf_data, security).

Upgrading to GSL 1.15 is currently blocked by #1782 (extrapolation in backgrounds).

comment:3 Changed 7 years ago by jbosch

Instructions for testing out the Winter2012a branch on your own stack

Bootstrap a new LSST stack with my test distribution server:

mkdir -p Winter2012a/home
cd Winter2012a/home
wget http://dev.lsstcorp.org/pkgs/test/jbosch/newinstall.sh
bash newinstall.sh
source loadLSST.[cz].sh

This should install the new eups, lsst, lssteups, tcltk, python, scons, doxygen, and sconsUtils packages. Note that "scons" is the third-party package formerly known as "sconsDistrib".

Install additional third-party packages (sorry, no metapackage yet). The following should be enough to get you to afw:

lsstpkg install boost
lsstpkg install swig
lsstpkg install cfitsio
lsstpkg install wcslib
lsstpkg install mysqlclient
lsstpkg install fftw
lsstpkg install numpy
lsstpkg install xpa
lsstpkg install minuit2
lsstpkg install pyfits
lsstpkg install eigen
lsstpkg install gsl

The server provides two versions for both Eigen and GSL, but the branches I've made currently work with the new Eigen and the old GSL.

You'll have to build the other LSST-controlled packages manually from their Winter2012a branches; there are no packages on the distribution server for these. Packages I've already converted (and the order they should be built for convenient reference) are:

base pex_exceptions utils daf_base pex_logging pex_policy daf_persistence ndarray afw cat* ctrl_events* pex_harness* sdqa meas_algorithms meas_astrom meas_multifit meas_utils meas_pipeline

  • = packages that don't depend on afw, but afw doesn't depend on them either.

You can also setup and build all these using my altbuildbot scripts:

svn co $LSST_SVN/contrib/altbuildbot/trunk altbuildbot
cd altbuildbot
mkdir packages
echo "default=branches/Winter2012a" >> package-versions

Now, update the "package-order" file to list only the packages you want, in order; comment things out with "#".

source metasetups.sh

(sorry, csh users, I don't have one for you; I copied this all from eups/bin/setups.sh, so you can probably make your own metasetups by looking at that). Now you can do:

./manage.py -r packages checkout
metasetup -r packages
./manage.py -r packages build --scons-args="-j2"

This will checkout all the packages from SVN, set them up (using setup -k; not declaring), and build them (but not their docs or examples with the above invocation).

comment:4 Changed 7 years ago by jbosch

Instructions for testing out the Winter2012a branch on the LSST cluster

I have made an LSST stack others can use on lsst9.ncsa.illinois.edu; you should be able to use it with any of the RH6 machines.

Just do:

export LSST_HOME=/home/jbosch/Winter2012a/home
export LSST_DEVEL=<wherever>
source $LSST_HOME/loadLSST.[cz].sh

All the new LSST-controlled packages I've converted are tagged and installed as version (or they will be very soon after this is posted).

comment:5 Changed 7 years ago by jbosch

Instructions Update

LSST packages are now available via a secondary EUPS distrib server.

First, make sure your stack's lsst and sconsUtils are up-to-date, either by bootstrapping a new one (see above) or by doing:

lsstpkg install lsst

Then, install my alternate server package:

lsstpkg install pkgs_jfb 

This will add a new URL to your EUPS_PKGROOT, and you'll see many more packages available with lsstpkg list or eups distrib list (you can use either lsstpkg or eups distrib to deal with the packages on the secondary server. None of those are marked current, but you should be able to install them by passing the explicit version to lsstpkg or eups distrib. Dependencies will be handled automatically.

In general, third-party and LSST devenv packages are on the primary (lssteups/ncsa-style) server, and all other LSST packages are on the secondary (builder/princeton-style) server.

comment:6 Changed 7 years ago by price

  • Cc price added

I can't update using eups distrib install. What's the correct EUPS_PKGROOT?

price@neverland:~ $ EUPS_PKGROOT="http://dev.lsstcorp.org/pkgs/test/jbosch" eups distrib install lsst
Invalid config parameter EXTERNAL_DIST_URL ignored
Invalid config parameter EXTERNAL_TABLE_URL ignored
Installing tcltk 8.5.9+1 for generic...
Failed to install tcltk 8.5.9+1: Failed to open URL http://dev.lsstcorp.org/pkgs/test/jbosch/tcltk/8.5.9/tcltk.bld
eups distrib: Failed to open URL http://dev.lsstcorp.org/pkgs/test/jbosch/tcltk/8.5.9/tcltk.bld

comment:7 Changed 7 years ago by price

Needed the latest lssteups:

EUPS_PKGROOT="http://dev.lsstcorp.org/pkgs/test/jbosch/bootstrap" eups distrib install lssteups
setup lssteups
EUPS_PKGROOT="http://dev.lsstcorp.org/pkgs/test/jbosch" eups distrib install lsst

(Note the presence of .../bootstrap in the first EUPS_PKGROOT --- there's a mini package server hosting the lssteups product.

comment:8 Changed 7 years ago by jbosch

  • Status changed from new to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.