Last modified 10 years ago Last modified on 09/03/2009 11:16:06 PM

DC3b Task: Complete support for validated policy files

Project Plan Status


The output of this task is a plan for addressing all other EUPS-related tasks (or at least the major ones), including:

  • support for read-only stacks
  • support for user and project tags.
  • better support for user stacks


Pickled Product Cache

I need to understand the product cache to address the following problems:

  • shared stacks may be read-only to user, but writes to the cache file are not guaranteed to be only on writes
  • the cache file does not appear to be "multi-user-safe":
    • I've seen odd behavior: eups list does not always show all products; the in-memory cache gets corrupted somehow.
    • the code apparently does not guarantee that all users in-memory view of the products are consistent

How it works:

  • the eups instance keeps a self.versions which is a hierarchical dictionary for looking up products. The dimensions are (in order outer to inner):
    • product database (i.e. a stack), given by the path to the parent directory of ups.db
    • flavor
      • one of the flavor names is "version" whose value is a version string which must match Eups.cacheVersion
    • product name
    • version name

the value at the bottom of the hierarchy is an instance of the Product class, one for each project-version

  • The contents of self.versions[db][flavor] is what is stored in ups.db/flavor.pickleDB
  • eups.buildCache() builds the contents of self.versions
    • calling getProduct() for each product-version list under the ups.db directory causes an entry to be added to self.versions
    • getProduct() will only add to self.versions if an entry is not already there.
    • buildCache() ends by writing out the built self.versions
  • Who calls buildCache()? readDB() when file is detected as missing or not up to date

Conclusion: writing of the cache can happen during read-only operations.

Preliminary Ideas:

  • encapsulate the product lookup and file caching into a class to make code easier to track and know when the file cache is being updated
  • restrict normal file updates to operations that actually write to stack (installation; tagging?)
  • if stack detected to be out of date and stack is read-only, place file cache in user's .eups directory
  • when reading cache for read-only stack, eups can take newest of what's in ups.db and user's .eups directory
  • must be backward compatible
  • ideas may need to change in light of user tag support

Tags and current

existing support:

  • Eups has a currentType which contains the name of the tag that should be considered "current"; could be "stable", "beta",
  • currentType is actually an object of class _Current. An instance of this class can return the filename where its association is (normally) stored.
  • some functions that take a version will accept an instance of _Current which it will internally resolve to a version (like getProduct()).
  • _Current instances are created via Current()
    • so that one and only one instance is created
    • Current() returns the default tag ("current")
    • allowing one to say, for instance, versionName == Current()
  • some assume that a _Current instance is equal Current()
    • ex: getProduct():
        if isSpecialVersion(versionName, setup=False):
            foundCurrent = True
            eupsPathDir, versionName, vinfo, ctype = self.findCurrentVersion(productName)
  • isSpecialVersion(version) returns true if version is of instance _Class
    • extra bool arg, setup can treat Setup() like a regular version if true
  • use of isSpecialVersion(versionName, setup=True) inconsistent with use:
    • see implementation: doesn't match documentation, Setup() will be determined non-special only if setup=True
      def isSpecialVersion(versionName, setup=True):
          """Is versionName special? If setup's False, treat Setup() as normal"""
          if versionName in _Current._tags.values():
              if setup and versionName == Setup():
                  return False
              return True
              return False
    • see in def intern()"
        if isSpecialVersion(product.version):
            return                      # don't intern e.g. Current or Setup
    • other use appear opposite, too.

Preliminary ideas:

  • Eups instance needs to have a "preferred tag" for choosing between multiple satisfactory versions
  • Otherwise, functions like getProduct() should allow caller to get any requested tag in lieu of an explicit version
  • To make code easier to understand, I would like to replace use of Current() with, say, Tag("stable") or Tag("beta").
    • replace isSpecialVersion() with isTag = lambda x: instanceof(x, _Tag)
  • Fix handling of Setup() as version.

Handling Tags in the course of setup

Use Cases:

  1. setup pex_harness
    1. the preferred tag to use when choosing products is obtained from the cache of user's preferences; if not there it defaults to "current" (?, for backward compatibility for existing installations?)
    2. the tag is resolved to a version of pex_harness
    3. as dependencies are processed, whenever multiple versions satisfy the criteria, one is chosen by these rules, in order:
      1. the version tagged with the preferred tag, if it exists
      2. the version tagged with user tag, "default"
      3. the newest version
  1. setup pex_harness stable
    1. stable is recognized as a tag name and is now taken as the "preferred" tag.
    2. proceeds as in Use Case 1. above with step 2.
  1. setup --tag=stable pex_harness beta
    1. stable is taken as the preferred tag
    2. beta is recognized as a tag name, and is resolved to a version of pex_harness
    3. dependencies are process as done Use Case 1, step 3.
  1. setup --tag=stable pex_harness
    • this is identical to Use Case 1., setup pex_harness stable; that is stable versions are preferred for the target product and all its dependencies
  1. setup --tag=stable pex_harness 3.3.1
    • identical to Use Case 3., except that version 3.3.1 is used instead of the version tagged beta
  1. setup pex_harness 3.3.1
    • identical to Use Case 1. except that 3.3.1 is used for pex_harness instead of the version tagged with the preferred tag.

Experiments running eups:

  • Use cases 2 and 3 are not intended to work.
  • --tag failed when one package was expected to have the tag but didn't; other products without the tag proceeded fine
  • without --tag, setup made a number of inexplicable re-setups, switching versions when it was not necessary.


Main work:

  • Create Tags class
    • replaces use of Current() and _Current
    • Tags recognizes registered tag names
    • Eups controls setting of preferred tags and where/when registered tag names are cached.
  • Create products sub module:
    • ProductStack? replacing Eups.versions[db]
    • new Product container class (handling of tags)
    • Eups will cache to user directory for stacks that are read-only
  • create db sub module: interface for finding products in an EUPS db
    • VersionFile?, ChainFile? moved into this module
    • Eups find methods implemented using the db interface
    • Eups caches user tags separately from system ("global") tags
  • Eups gets assignTag(), unassignTag()
    • eups declare only allows update of user tags
    • eups distrib will set system tags


  • UML diagrams for new classes
  • will move modules to a python dir in install dir, some separation of modules will be carried out.
  • Eups methods that seem intended to be public and are still relevant will be preserved, but perhaps marked deprecated.
  • bottom-up unit tests will be built for new code


Things that would be nice to do "while I'm add it" if there's time:

  • move all .py files to a python/eups directory
  • separate more classes out into separate files or modules
  • unit tests!!
  • use standard OptParse
  • apply standard naming conventions to "[Ee][Uu][Pp][Ss]":
    • module: eups (maybe import as EUPS)
    • class: Eups
    • instance: eups or theEups or myEups