RobynAllsman – LSST Trac – Trac

explicitly, I would think you should pull it in explicitly, even if it is already being pulled in by another package you depend on. This makes dependencies explicit and locally-maintainable. Having to know the dependency hierarchy of other packages is a negative, not a positive.

Comment by robyn on Wed 02 Apr 2008 03:52:02 PM CDT

From: Kian-Tat Lim <ktl@…>

You can get an RSS feed of the Timeline and look at it in an RSS

feed reader, including Apple Mail. You can configure the Timeline to limit it to just Wiki changes, but it won't be limited to a specific page. You can also get RSS feeds of the revision log of something specific in the Subversion tree.

Comment by robyn on Wed 02 Apr 2008 03:54:20 PM CDT

From: Kian-Tat Lim <ktl@…>

OK, while writing up the dependency issue, I stumbled upon a solution
that I think requires minimal change.

It turns out that pex/logging does not actually use pex/exceptions,
although one might claim it should.  That helps a bit.  daf/data is then
the main hub that causes problems.

Here is my proposal to break the circularity:

We move Citizen, DataProperty, and Persistable into a new daf/base
package.  LsstBase, LsstData, SupportFactory, and DataPropertyFormatter
remain in daf/data.  DataProperty and trace/logging are only allowed to
throw standard, non-LSST exceptions (the current situation).
DataProperty's use of the trace facilities must be removed.  This is in
line with the desire to simplify DataProperty and make it more like a
dictionary.

* daf/base ->
* pex/logging ->     daf/base
* pex/exceptions ->  daf/base, pex/logging
* pex/policy ->      daf/base, pex/logging, pex/exceptions
* daf/persistence -> daf/base, pex/logging, pex/exceptions, pex/policy
* daf/data       ->  daf/base, pex/logging, pex/exceptions, pex/policy,
                     daf/persistence

I think this is simple enough that we don't need to take it to the whole
list.  What say you

Comment by robyn on Wed 02 Apr 2008 03:55:18 PM CDT

From: "Serge Monkewitz" <smm@…>

If we haven't already, we need to check that utils, daf/base, and pex/logging either specify alternate C++->Python exception handling policies, don't throw exceptions, or are OK with an exception causing process termination...

Comment by robyn on Wed 02 Apr 2008 03:56:28 PM CDT

From: Russell E Owen <rowen@…> Date: Tue, 1 Apr 2008 15:18:58 -0700

I have edited daf_persistence. It didn't need much work (much less than daf_data and afw) and I may have missed something but it's ready for debug.

Interesting tip: scons --clean turned up some errors in my SCons* files (that would otherwise not have shown up until all the dependencies were ready and I could try building the package).

Also I commend the lib/SConscript file to you. It avoids the use of lambda and finds all the source files automatically -- helpful in reorgs like this.

Comment by robyn on Wed 02 Apr 2008 03:57:45 PM CDT

Date: Tue, 1 Apr 2008 17:20:57 -0500 (CDT) From: Ray Plante <rplante@…>

>>On Tue, 1 Apr 2008, Russell E Owen wrote:
>>>The error is "ImportError: No module named policy"
>>
>>I've duplicated this error.

I've figured this one out. It stems from a limitation of our custom module loader (which is part of the base package). I tested a fairly simple fix that removes this limitation; however, after some thought, I think it would be better not to put it in but rather move the policy.i file one level as Russell has recommended.

Currently our loader requires that all python files associated with a package be in its own directory within the python directory tree. In other words, there cannot be *files* that appear directly under lsst/pex. Furthermore, there can only be one such directory that contains files (apart from empty init.py files) throughout the PYTHONPATH. This guarantees that no matter how the python file (across may PYTHONPATH directories or just one) get installed, they remain nicely segregated.

Our loader finds modules by looking first for the directory that contains the module, and then looks for the module below it. Russell saw his failure because he had pex_logging in head of pex_policy in his PYTHONPATH. The loader found lsst/pex from the pex_logging package, but then failed to find the policy module under at which point the ImportError? was raised.

Comment by robyn on Wed 02 Apr 2008 03:58:19 PM CDT

Date: Tue, 1 Apr 2008 18:39:18 -0700 From: Kian-Tat Lim <ktl@…>

Excuse me for a minute while I vent some frustration...

Our SConscript and *.i files are in general ugly messes because they've been copied from poor templates and hacked on until they worked, with little understanding about why things were going into them. While we need to get DMS up and running fast, we should make fixing these a high priority, probably by coming up with much better templates for people to use and customize that include a lot more documentation on what various items do.

Comment by robyn on Wed 02 Apr 2008 03:59:52 PM CDT

Date: Tue, 1 Apr 2008 20:17:25 -0700 From: Kian-Tat Lim <ktl@…>

Serge wrote:

This is because we are still using relative install_names on MacOSX.

Oh, yeah :-(. Now I remember. I need to drill into my head

that Linux is not the whole world...

Anyway, it's unclear to me whether SConsUtils will ever be changed to use absolute paths in install_name on OSX or if we are just going to drop support for 10.4 and move to 10.5 with @rpath based install_names (as suggested by Francesco).

For the record, I think the latter would be preferable.

Comment by robyn on Wed 02 Apr 2008 04:01:00 PM CDT

Date: Tue, 1 Apr 2008 20:26:58 -0700 From: Kian-Tat Lim <ktl@…>

Russell wrote:

I agree. But look at daf_data which uses darn near every include file available from daf_persistence. (That's from memory, but I was just looking at it a few hours ago). The SConstruct file is going to be pretty ugly if we have to name every include file.

Do we have to? I thought the key was to ensure the package was

available, and that any include file in the package was good enough. We shouldn't have to name every include file in the SConstruct; that's an unmaintainable nightmare, since they're used all over the place. (Even having to maintain the list of packages in SConstruct has bitten me a few times during this reorg.)

Perhaps these [headers not for general use] could simply not be included by the main include file?

That leads to having to define a threshold for "general use",

which seems more complicated. But I can live with that.

Though including an extra include file doesn't add dependencies (other than the file must exist) does it?

Given that we're talking inter-package dependencies, which

either all change or none change, I guess you're right.

Comment by robyn on Wed 02 Apr 2008 04:01:35 PM CDT

Date: Tue, 1 Apr 2008 21:32:44 -0700 From: Robyn Allsman <robyn@…>

As we move through each package we should ensure that both the eups table dependencies and the initial SConstruct dependency checks are adequately detailed.

Comment by robyn on Wed 02 Apr 2008 04:02:26 PM CDT

Date: Wed, 2 Apr 2008 07:38:41 -0700 From: Kian-Tat Lim <ktl@…>

Ray wrote:

Furthermore, there can only be one such directory that contains files (apart from empty init.py files) throughout the PYTHONPATH.

So this is why the lsst/tests.py conflict was occurring. Is the

workaround we came up with (lsst.utils.tests) acceptable as a permanent solution?

Comment by robyn on Wed 02 Apr 2008 04:03:17 PM CDT

Date: Wed, 2 Apr 2008 07:43:11 -0700 From: Kian-Tat Lim <ktl@…>

* "scons tests" often builds the C++ tests but doesn't run them until executed a second time. This has been a longstanding issue.

Comment by robyn on Wed 02 Apr 2008 04:03:46 PM CDT

Date: Wed, 2 Apr 2008 09:48:23 -0500 (CDT) From: Ray Plante <rplante@…>

All pex packages must have empty init.py files in them. This means that an init.py file will always be loaded regardless what other packages are in the PYTHONPATH. They must be empty so that it does not matter which copy gets loaded.

Comment by robyn on Wed 02 Apr 2008 04:04:07 PM CDT

Date: Wed, 2 Apr 2008 08:02:20 -0700 From: Kian-Tat Lim <ktl@…>

I've been going through and setting svn:ignore properties on things to clean up the "svn status" output. Do we want to have things like *.os, *.so, and *.pyc explicit in svn:ignore properties or do we want to make them implicit by having people set their global-ignores config option?

Comment by robyn on Wed 02 Apr 2008 04:04:37 PM CDT

Date: Wed, 2 Apr 2008 08:22:21 -0700 From: Kian-Tat Lim <ktl@…>

Robyn wrote:

My slight preference is for the user to set their own preferences via the global-ignores option. Does the LSST SVN FAQ include the info required to make it easy for them to set?

I've just added this.

Or is it in the "Installing LSST Software" where the users's LSST environment is discussed?

It could go in section 6 of

<http://dev.lsstcorp.org/pkgs/GettingStarted.html>. Where is the original source of this document? Or perhaps Ray could take on adding this.

Comment by robyn on Wed 02 Apr 2008 04:06:59 PM CDT

Date: Wed, 2 Apr 2008 11:49:11 -0500 (CDT) From: Ray Plante <rplante@…>

On Wed, 2 Apr 2008, Russell E Owen wrote:

I have given up entirely on scons install as a result of it.

This is my formula if I know the version is already installed

  rm -rf $LSST_PKGS/<pkg_vers_path>
  scons install

It is not necessary to either undeclare or re-declare the package if you do these one right after the other.

I always just "eups declare" my source directory and thus have it permanently "installed".

A number of folks have tried this on the NCSA cluster only to have annoying but ultimately harmless error message appear for other people. This happens when other people cannot see your personal directory.

This is one of the things that the LSST_DEVEL environment can do for you, even when you don't actually install into it. To review how it works, you set things up (in the DC2 regime)...

   1.  setup build; mksandbox $HOME/sandbox
   2.  setenv LSST_DEVEL $HOME/sandbox
   3.  setenv EUPS_PATH ${LSST_DEVEL}:$EUPS_PATH

Now setup your environment so that LSST_DEVEL is set as above before you source loadLSST.csh; this will set the EUPS_PATH automatically as above.

Now even if you eups declare your own source directory, this fact will be recorded in your sandbox's eups database and not in the one under LSST_HOME.

Comment by robyn on Wed 02 Apr 2008 04:07:45 PM CDT

Date: Wed, 2 Apr 2008 11:05:13 -0700 From: Kian-Tat Lim <ktl@…>

Russell,

If you decide to remove daf_persistance's persistence.h

I'm thinking that you may be right and that a package-wide

include can be generated without implementation files. It probably also means that DbTsvStorage?, and maybe even other Storages, should move to the pimpl pattern, like DbStorage?, so that other packages don't start pulling in headers like the MySQL client.

Even if DataPropertyFormatter? moves to daf/persistence, the

persistence.h will still be useful for afw Formatters.

Comment by robyn on Wed 02 Apr 2008 04:08:58 PM CDT

Date: Wed, 2 Apr 2008 11:38:13 -0700 From: Russell E Owen <rowen@…>

Yes. All my svn working copies are installed into LSST_DEVEL. That way I can muck about with my working copies without worrying about hurting anyone else.

Personally I would recommend NOT using "eups install" of svn working copies into LSST_HOME on a shared system. Occasionally it can be useful (when the trunk has fixes everyone needs) but mostly it can just cause trouble.

Note that on a shared system is a pain in the ass to install stuff into $LSST_HOME. One must undelete $LSST_DEVEL *before* running the lsst setup script or else lsst_pkg fetch fails (this is an outstanding PR). Perhaps there should be a special lsst setup script that intentionally ignores/deletes LSST_DEVEL.

Comment by robyn on Wed 02 Apr 2008 04:09:29 PM CDT

Date: Wed, 2 Apr 2008 12:17:59 -0700 From: Kian-Tat Lim <ktl@…>

I've managed to get pex/policy to work without it, somehow

(after "setup -r ."), but I concur that there should be one for all pex and daf packages.

Comment by robyn on Wed 02 Apr 2008 04:10:18 PM CDT

Date: Wed, 2 Apr 2008 15:06:46 -0500 (CDT) From: Ray Plante <rplante@…>

On Wed, 2 Apr 2008, Russell E Owen wrote:

I further assume that a blank init.py is not needed in the python/lsst/ directory but only in the python/lsst/<package>/ directory.

More importantly, packages must *not* have an init.py under python/lsst (other than base and sconsUtils).

The one in base is the one that imports our special importer hook. (sconsUtils attempts to load the one in base.) If python finds another init.py at this level our importer will not get loaded.

Comment by robyn on Wed 02 Apr 2008 04:11:51 PM CDT

Date: Wed, 2 Apr 2008 13:54:57 -0700 From: Russell E Owen <rowen@…>

P.S. this is a *lot* of packages to keep track of. I think this means we'll have to be *much* more aggressive about pushing out updates in a form that can be downloaded via lsstpkg fetch than we were for DC2. Otherwise there are a *lot* of people who will have to keep track of a host of svn working copies, making a huge net waste of time for the project.

===================================================================

optics?
activemqcpp?
afw? (tickets)
association? (tickets)
base? (tickets)
boost? (tickets)
process? (process tickets)
buildBot? (tickets)
calsim?
standards? (standards tickets)
ctrl_events? (tickets)
daf_base? (tickets)
daf_data? (tickets)
daf_persistence? (tickets)
database? (tickets)
dbSchema? (tickets)
detection? (tickets)
doxygen? (tickets): Doxygen is a source code documentation generation tool used by developers to document classes, methods, and implementation strategy.
eups? (tickets): EUPS manages software dependencies of entities enabling the LSST configuration system to build self-consistent products.
hypernews?
imageProcess? (tickets): Image processing, especially difference image analysis
infrastructure? (tickets)
ip_diffim? (tickets)
ip_isr? (tickets)
jaula? (tickets)
mops? (tickets)
pacman? (tickets)
pex_exceptions? (tickets)
pex_harness? (tickets)
pex_logging? (tickets)
pex_policy? (tickets)
scons? (tickets)
sdqa? (tickets)
security?
svn? (tickets): Anything to do with how we use or manage SVN

TCT? (tickets)
trac? (tickets): Anything to do with trac itself
UML/EA? (tickets)
unknown? (tickets)
utils? (tickets)
vw? (tickets)
wcslib? (tickets)

====================================================================

Download in other formats:

Plain Text