Ticket #398 (closed defect: fixed)

Opened 11 years ago

Last modified 7 years ago

Add Feature to Buildbot to list packages&versions.

Reported by: robyn Owned by: bbaker
Priority: minor Milestone:
Component: buildBot Keywords: builds
Cc: RayPlante Blocked By:
Blocking: Project: LSST
Version Number:
How to repeat:

not applicable

Description

I suggest adding a new function to Buildbot to display the list of packages and their versions of the current Builds.

This should probably also include the dependencies on system tools, too.

Change History

comment:1 Changed 11 years ago by bbaker

  • Status changed from new to assigned

comment:2 Changed 11 years ago by bbaker

What do you mean by system tools? Can you give an example?

comment:3 Changed 11 years ago by robyn

The purpose of this requirement is to support the rebuild of previous Software Releases. To satisfy that requirement, we need to know the generic hardware and specific software on which it previously worked. Each software module doesn't need to be explicitly maintained in a table but the required software should be deducible given the clues provided.

So the build should characterize the platform and the state of software as of the build date.

First item recorded should be something like: 'uname -a' to provide the hardware and OS release. Next item: 'gcc -v' to get the gnu C++ version used. Most of the software required should be available from an 'eups list --setup'.

Some of the missing software would be the libraries explicitly mentioned in the SConscript files and those would be available by reviewing the setup packages' SVN contents. However, a complete list would be available after an executable's build via 'ldd -v <executable>'.

The other missing info would be add-in python packages. I don't know if they are explicitly noted as packages in the 'eups list'. And even if they weren't 'setup' for a build, they would be picked up by virtue of their inclusion in the Python search path. To subvert these potentially hidden dependencies, a short python program which only invokes 'modules' provides a list of all python modules available for use from the current python setup.

I can't address the java dependencies since I'm not rated for java. I assume it would have something like python to list all packages available in the specified java environment.

I think this should characterize the hardware and software environment for the build. It would be 'nice' to have this info extracted into a separate logfile to make review easier and not clutter up the build logfile.

comment:4 follow-up: ↓ 5 Changed 11 years ago by bbaker

My notes from a discussion with Ray:

When we run a pipeline, we're going to want to know all this, and keep it with the data as part of provenance. Ray suggests that we put all this in a file in the eups dir, in a standardized format, for each package, as part of the build. (Nightly builds can capture that file and archive it.) So that means we should:

  • Explicitly list all the items we want to keep track of (for example, see Robyn's list in the previous comment)
  • Choose a format for the file that is machine-readable and extensible
  • Include construction of the file as part of the build process
  • (in the future) When a pipeline is run, sweep up all those files from all the packages used in the pipeline, and keep them as part of provenance metadata. Would probably be done by the orchestration layer

I think we can get started on this by adding a step to the build process that creates a simple version of this file. Then we can refine that file until we're happy with it, and in the meantime start archiving it during nightly builds, for practice.

comment:5 in reply to: ↑ 4 Changed 11 years ago by robyn

Replying to bbaker:

My notes from a discussion with Ray:

When we run a pipeline, we're going to want to know all this, and keep it with the data as part of provenance. Ray suggests that we put all this in a file in the eups dir, in a standardized format, for each package, as part of the build. (Nightly builds can capture that file and archive it.) So that means we should:

  • Explicitly list all the items we want to keep track of (for example, see Robyn's list in the previous comment)
  • Choose a format for the file that is machine-readable and extensible
  • Include construction of the file as part of the build process
  • (in the future) When a pipeline is run, sweep up all those files from all the packages used in the pipeline, and keep them as part of provenance metadata. Would probably be done by the orchestration layer

I think we can get started on this by adding a step to the build process that creates a simple version of this file. Then we can refine that file until we're happy with it, and in the meantime start archiving it during nightly builds, for practice.

If you mean <SVNROOT>/DMS/<pkg>/eups/ directory under SVN: I don't think we want to litter the source directories with time dependant files, i.e. every nightly build.

However, if you are archiving under a Release-stamped root directory, then archiving in <build_root>/<pkg>/eups/configuration is reasonable.

comment:6 Changed 10 years ago by bbaker

  • Status changed from assigned to closed
  • Resolution set to fixed

All the automatic builds -- nightly release, nightly trunk, and incremental trunk -- will include eups list and gcc -v, as of tonight's run.

comment:7 Changed 10 years ago by ktl

Sorry I missed the discussion in this ticket before. While Bill's fix deals with the request in this ticket, it does not deal with the long-term provenance problem, which cannot be solved by a nightly build process. Rather than knowing the software that went into a given Buildbot run, we need to know the software that went into a specific installation of the LSST stack. This has to be preserved during the building of that particular stack, likely in its installed ups directories (and definitely not in SVN, which would make no sense). This will have to be part of sconsUtils in the newly-revamped post-DC3a build system.

comment:8 Changed 7 years ago by robyn

  • Milestone Post DC3 deleted

Milestone Post DC3 deleted

Note: See TracTickets for help on using tickets.