Ticket #279 (closed new functionality: fixed)

Opened 12 years ago

Last modified 7 years ago

Pipeline harness must log stage execution and exceptions

Reported by: ktl Owned by: daues
Priority: normal Milestone:
Component: dps Keywords:
Cc: RayPlante, rowen, Tim Axelrod, smm, ktl Blocked By:
Blocking: Project: LSST
Version Number: 3807
How to repeat:

not applicable

Description

The pipeline harness currently logs progress information to a local file via self.LOGFILE.write(). Most of these should become invocations of Trace(). In addition, progress of the pipeline needs to be logged via the mwi logging framework. Code might look like the following in Pipeline.py:

try:

self.logger.log(Log.INFO, "Start preprocess for stage %s" % str(stage.class)) stage.preprocess() self.logger.log(Log.INFO, "End preprocess for stage %s" % str(stage.class))

except Exception, e:

self.logger.log(Log.FATAL, "Exception "%s" in preprocess for stage %s" % (str(e), str(stage.class))) raise e

or something similar in Slice.py with the addition of self._rank to the messages.

Change History

comment:1 Changed 12 years ago by ktl

  • Version Number set to 3807
  • Component changed from unknown to dps

comment:2 Changed 12 years ago by ktl

Greg is already planning to do something more advanced with exceptions: keep the pipeline moving with no data processing in the failed slice until the next iteration/exposure/triggering event. We must still log the exceptions, of course.

comment:3 Changed 12 years ago by daues

  • Status changed from new to assigned

Accept ticket

comment:4 Changed 12 years ago by daues

Logging with the official Logger has been integrated by Ray.

Some important Exception handling has been implemented, namely, a try-catch construct for the execution of process by Slice. As advertised above, this does keep the full Pipeline running in the case that a Slice incurs an exception. A Slice that captures an exception logs information at FATAL level and then goes throught the motions of the remaining Stage loop without actually executing any Stages.

Some additional exception handling, for example, around preprocess /postprocess would be good, so I will leave this ticket open, unless we feel that atomic task is best served by a new ticket.

comment:5 Changed 11 years ago by robyn

  • Milestone set to DC3 MW Pipeline Harness Extensions

comment:6 Changed 10 years ago by daues

  • Status changed from assigned to closed
  • Resolution set to fixed

As of the production runs of DC3a, the harness fulfills the goals of this ticket:

1) try/except constructs are used with each Stage preprocess(), process(), and postprocess()

2) Logging in Pipeline/Slice? has used the latest pex_logging throughout DC3a and has been aligned with the conventions of the Event Monitor within ticket branch pex_harness 717 . The level of the log is configurable in Pipeline policy.

3) With localLogMode of true, log messages for a given Slice will be written to a particular log file, and log messages from within a Stage process() will also appear in that Log file.

4) Pipeline continues the behavior where the complement of Slices keeps executing with the occurence of an exception.

The current status warrants resolving this general ticket; if there are nuances of exception handling (LsstCppException specific?), then those should be captured in fresh tickets.

comment:7 Changed 7 years ago by robyn

  • Milestone DC3b MW Pipeline Harness Extensions deleted

Milestone DC3b MW Pipeline Harness Extensions deleted

Note: See TracTickets for help on using tickets.