wiki:LSSTCalibSimComputing
Last modified 10 years ago Last modified on 04/21/2009 06:12:18 PM

It is our intention to develop code and catalogs on the LSST cluster at NCSA and make it openly available to the LSST community. This page is meant to describe how to use that cluster for this work.

Getting started with LSST computing

Machines

The machines that are available for the calibration simulation are these:

  • lsst4.ncsa.uiuc.edu
  • lsst5.ncsa.uiuc.edu
  • lsst6.ncsa.uiuc.edu
  • lsst7.ncsa.uiuc.edu
  • lsst8.ncsa.uiuc.edu
  • lsst9.ncsa.uiuc.edu hosts the /lsst and /share/lsst NFS filesystems
  • lsst10.ncsa.uiuc.edu hosts the MySQL server
  • dev.lsstcorp.org
  • svn.lsstcorp.org hosts the Subversion repository

We should use lsst{4,5,6,7,8} for interactive work and development (since lsst9 hosts an NFS server and lsst10 hosts a database server, we might want to stay away from them).

Setting up the environment

Taken straight from the getting started guide, I made a setup that you can source when you begin your session:

source /lsst/images/sim/cal/setcalsim

After you've done that, you should be able to do this without errors:

[haggerty@lsst10 ~]$ python
Python 2.5 (r25:51908, Sep  6 2007, 12:35:07) 
[GCC 3.4.6 20060404 (Red Hat 3.4.6-8)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy

Note that since the home directories are not common node-to-node, if you want to put this in your .bashrc (/home/yourname/.bashrc), if you log onto a different machine, you may find your environment is not set up correctly.

Subversion repository

The easiest way to browse the code we have so far in the Subversion repository is to use the browser: http://lsstdev.ncsa.uiuc.edu/trac/browser/sims/cal. If you haven't used code management tools before, you may need an introduction to them, and there are links to some of them here. The basic idea of the workflow is that you "check out" a local copy of the code from the repository like this:

[haggerty@lsst10 foo]$ echo $LSST_SVN
svn+ssh://svn.lsstcorp.org
[haggerty@lsst10 foo]$ svn checkout $LSST_SVN/sims/cal/1 dev
haggerty@svn.lsstcorp.org's password: 
haggerty@svn.lsstcorp.org's password: 
A  dev/trunk
A  dev/trunk/MySQL
A  dev/trunk/MySQL/create_DIASource.sql
A  dev/trunk/MySQL/create_observations.sql
A  dev/trunk/MySQL/drop_all_tables.sql

which then creates a directory tree as you see above in the working directory where you ran the svn command. You can go make changes and then "commit" the changes back to the repository. Two handy references are: http://dev.lsstcorp.org/pkgs/GettingStarted.html#svnsurvival and http://meg.icepp.s.u-tokyo.ac.jp/subprojects/software/subversion/working.html. Here's an example where I added a dummy file in the MySQL directory:

[haggerty@lsst10 trunk]$ svn status MySQL
?      MySQL/foo

Other commands to know are svn add, svn commit, svn diff. A couple of quirks to be wary of:

  • after an svn add or svn delete, the change doesn't take effect until you do an svn commit
  • be careful with abbreviations, particularly if you're a cvs user--co is commit in cvs, but checkout in svn
  • you can do an svn commit -m "inline comment" myfile for inline comments or set an environment variable SVN_EDITOR that will pop your favorite text editor to write your comments on the changes you commit; it's a really good idea to write meaningful comments
  • getting the directory levels right can be confusing; this example will import a whole directory as a directory:
svn import pointing $LSST_SVN/sims/cal/1/trunk/pointing

Most of the browsing functions are easier from Trac: http://lsstdev.ncsa.uiuc.edu/trac/browser/sims/cal/1/trunk. I checked out a couple of copies of code into common areas: /lsst/images/sim/cal/software/pro and /lsst/images/sim/cal/software/dev. The idea is that the pro area has copy of what's in the repository at any moment and the dev area is a work in progress.

Database access

Here is an example of database access:

[haggerty@lsst4 ~]$ mysql -h lsst10.ncsa.uiuc.edu calsim
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 152660 to server version: 5.0.45-community

Type 'help;' or 'h' for help. Type 'c' to clear the buffer.

mysql> show tables;
+------------------+
| Tables_in_calsim |
+------------------+
| atmosphere       |
| brightstars      |
| filter           |
| instrument       |
| pointing         |
| pointing_meta    |
| sed              |
+------------------+
7 rows in set (0.00 sec)

mysql> describe sed;
+---------------+----------+------+-----+---------+-------+
| Field         | Type     | Null | Key | Default | Extra |
+---------------+----------+------+-----+---------+-------+
| spectral_type | char(10) | NO   | PRI |         |       |
| model         | char(80) | YES  |     | NULL    |       |
| gravity       | char(10) | YES  |     | NULL    |       |
| flux          | float    | YES  |     | NULL    |       |
| sed           | blob     | YES  |     | NULL    |       |
+---------------+----------+------+-----+---------+-------+
5 rows in set (0.00 sec)

mysql> 

I create and maintain the tables with Makefile shown above in the MySQL directory of the Subversion repository. The "pointing_meta" table is a copy of the Cronos92 OpSim? database. For convenience, I set MYSQL_HOST to lsst10.ncsa.uiuc.edu so you can type "mysql calsim" and there you are.

Filesystems

Note that the cluster machines do not share home directories, so if you log onto one node, you won't see files you made in your home directory on another. I can't see that they're cross-mounted, either, so we'll use /lsst/images/sim/cal to share files across cluster nodes.

lsst9.ncsa.uiuc.edu:/lsst/
                      6.9T  6.8T   80G  99% /lsst

Customization

I found I needed a few things that weren't there with the official package management system eups (type 'eups help" or eups list" to see what you can get). So I could work, I built and installed software in /lsst/images/sim/cal/pub. There isn't too much; I installed Python modules for matplotlib (0.91.2), MySQL (1.2.2), and biggles (1.6.5); biggles needed plotutils, so I built that in/lsst/images/sim/cal/pub, too.