Context Navigation

DRAFT last revised 1 May 2009

LSST Software Quality Assurance Guidelines (LSE-13)

This document is maintained under configuration control in the LSST document repository as LSE-13. It is currently edited in Trac, but the authoritative, project-approved version is always the one to be found in Docushare. The "live" Trac version (SDP/QualityAssurance) must always be treated as potentially unstable - it may, for instance, represent a proposed modification to the controlled document.

This document is a subordinate of LSST Software Development Plan (LSE-16) (current unstable version: Trac:SDP/LsstSoftwareDevelopmentPlan).

1. Introduction

The LSST system contains an extremely large amount of custom software. The quality management plan for software must emphasize defect avoidance and earliest possible defect detection and removal. Studies indicate that the relative cost of correcting errors in the software maintenance phase is roughly 100 times that of the cost if the errors are corrected or avoided in the requirements phase of development (Boehm 1981).

A strategy to avoid costly software errors is a formal software Quality Control process with consistent metrics documented through the LSST construction phase and in the operations and action plans. This plan documents the Quality Assurance process ensuring that Quality Control is being carried out correctly and consistently.

1.1 Purpose of Document

The purpose of this document is to describe the process and procedures used by LSST personnel to ensure compliance with a Baseline Software Project Management Plan.

Software Quality Assurance (SQA) is a planned review procedure to provide adequate confidence that the item or product produced conforms to established technical requirements.

In particular, SQA checks: ^{[ 1 ]}

project is properly organised, with an appropriate life cycle;
roles and responsibilities are documented;
documentation and coding standards are followed;
standards, practices and conventions are adhered to;
reviews take place and are properly conducted;
tests are specified and carried out;
problems are recorded and tracked;
projects use appropriate tools, techniques and methods;
software is stored safely and securely;
proper records are kept;
risks to the project are minimised.

1.2 Scope

This document describes the overall review policies and procedures required to accomplish LSST Software Quality Assurance.

Each LSST software subsystem will execute, for each Baseline encompassing a complete lifecycle sequence, a derivative Software Quality Assurance Plan.

1.3 Definitions, Acronyms, Abbreviations

See also Software Development Plan: Definitions, Acronyms, and Abbreviations.

1.4 Management

1.4.1 Tasks

The following tasks outline the sequence of SQA activity during each phase of the Software Development Process.

Key:

M: Baseline Manager
SQA: SQA Reviewer

Tasks in lifecycle phase sequenced order:

On completion of the Baseline's Science Requirements Definition phase, the following should be done

M Check Baseline's Derived Science Requirements are
* documented in the Functional Requirements Specification document
SQA Check Baseline's Project Management Plan exists and contains
* lifecycle description
* process model description
SQA Check Baseline's Configuration Management Plan exists and contains
* documentation configuration management procedures
SQA Check Baseline's Verification and Validation Plan exists and contains
* a plan for verifying the outputs of the Software Requirements phase
* an acceptance test plan

On completion of the Baseline's Derived Software Requirements Definition Phase, the following should be done

M Check Baseline's Software Requirements are documented in the Software Requirements document
SQA Check Baseline's Project Management Plan contains
* conforms to the LSST Project standard for a system engineering method support by SysML.
SQA Check Baseline's Configuration Management Plan contains
* architectural design configuration management procedures
SQA Check Baseline's Verification and Validation Plan contains
* a plan for verifying the outputs of the Application Design phase
* a System Test Plan

On completion of the Baseline's Architectural Design Phase, the following should be done

M Check the Baseline's Architectural Design is documented
SQA Check Baseline's Project Management Plan contains
* component-based WBS
* criteria exist for deciding the transfer readiness of:
.... a component for integration
.... the software for system testing
.... the software for delivery
SQA Check Baseline's Configuration Management Plan contains
* package configuration management procedures
SQA Check Baseline's Verification and Validation Plan contains
* a plan for verifying the outputs of the Detailed Design phase
* an Integration Test Plan

On completion of the Baseline's Detailed Design and Development Phase, the following should be done

M Check the Baseline's Detailed Design is documented
SQA Check coding standards were followed
SQA Check unit testing procedures were followed
SQA Check integration test procedures were followed
SQA Check system test procedures were followed
SQA Check Software Release Project Plan contains
* Baseline's Software Transfer Plan and is complete
SQA Check Baseline's Configuration Management Plan contains
* Product Release configuration management procedures
SQA Check Baseline's Verification and Validation Plan contains
* a test design for each relevant Baseline's Science Requirement

On completion of the Baseline's Transfer phase, the following should be done

SQA Check build and installation procedures
* have been defined in the Baseline's Software Transfer document and
* tested
SQA Check acceptance tests have been successfully run
SQA Check Transfer Risk Assessment procedure completed
SQA Check Transfer Readiness report authorizes transfer.

Refer to DM Tasks for DM specific content.

2. Documentation

Over the course of the Baseline's software development, the following documents will be available for every lifecycle which includes a Transfer Release phase:

Software Release Plans
- Software Project Management Plan
  - Software Transfer Plan
- Software Configuration Management Plan
- Software Verification and Validation Plan
  - Acceptance Test Plan
  - System Test Plan
  - Integration Test Plan
  - Unit Test Plan
- Software Quality Assurance Plan
Baseline Science Requirements document generated from the EA SysML;
Baseline Functional Requirements document generated from the EA SysML;

The requirements documents will be generated as needed using the EA report generator.

If the characteristics of a particular Software Release Plan do not change for the next lifecycle, that particular Plan may be cited as the current plan. In particular, the Configuration Management and Software Quality Assurance Plans might remain stable over many lifecycle iterations.

SQA Team will check the availability of all the documents prior to Integration Testing.

Refer to DM Documentation for DM specific content.

3. Standards, Practices, Conventions and Metrics

3.1 Documentation Standards

The standards, practices, and conventions that will be used to produce documents include the following documents within the LSST Trac instance:

Refer to DM Documentation Standards.

3.2 Design Standards

All LSST Science and Derived Functional Requirement specifications will occur within the Architect tool using the SysML methodology.

Refer to DM Design Standards for DM specific content.

3.3 Coding Standards

Coding standards, practices, and conventions that will be used to write code are defined in the following documents within the LSST Trac instance:

and in this external document:

Sun Microsystems' Java Coding Standards (http://java.sun.com/docs/codeconv/)

Refer to DM Coding Standards for DM specific content.

3.4 Commentary Standards

Commentary standards, practices and conventions that will be used to comment code are defined in the following document within the LSST Trac instance:

Standards for Software Documentation.

3.5 Testing Standards and Practices

Testing standards, practices and conventions that will be used to test the software are defined in the Software Validation and Verification Guidelines document.

3.6 Selected SQA Metrics

The metrics to be used to measure the quality of the software will be defined in each Subsystem's specializations.

3.7 Compliance Monitoring

In general compliance reviews which verify that procedures are being followed correctly, will occur at milestone boundaries. The review process may involve random selection of an artifact or procedure for review.

4. Verification

4.1 Reviews

The technical reviews required to be performed on the LSST software design and implementation are discussed in Software Verification and Validation Guidelines: 3.1 Reviews.

4.2 Testing

Compliance reviews, which verify procedures are being followed correctly, will occur at milestone boundaries; for example, on completion of a testing phase. The review process will involve random selection of test artifacts and/or test procedures for review.

Refer to DM Testing for DM specific content.

4.3 Problem Reporting and Corrective Action

Compliance reviews of the problem reporting procedures described in the Configuration Management Guidelines will be monitored on a regular basis. An Anomaly Status Report will be forwarded to the Subgroup Project Manager categorizing the completed and outstanding issues.

Refer to DM Problem Reporting and Corrective Action for DM specific content.

4.4 Code and Document Control

The procedures used to maintain, archive and document software are defined in the Configuration Management Guidelines.

Refer to DM Code and Document Control for DM specific content.

4.5 Tools, Techniques and Methods

Refer to Verification and Validation Guidelines: 1.5 Tools, Techniques, and Methods and Software Development Plan: 1.4 Tools for the tools, techniques and methods used to develop the software.

4.6 Archival Control

The procedures used to maintain, store, secure and document archival versions of the repositories managed by the tools mentioned in Section 4.5 are in Configuration Management Guidelines: 8. Archival CM. This section describes how adherence to the procedures will be monitored.

The SQA Archival Control review should confirm the media backups by reviewing the report generated by the CM fail-safe recovery procedure.

4.7 Records collection, maintenance and retention

This section identifies the procedures kept by the project for recording activities such as meetings and reviews. It should describe where the records are kept, and for how long. This section also describes how adherence to the record-keeping procedures will be monitored.

Refer to Records collection, maintenance and retention for DM specific content.

5. Training

Each LSST Subsystem will prepare a New Developer's Guide which will

provide information to the new user on how to acquire login access permission to shared facilities; and also
introduce the new Developer to the infrastructure supporting the subgroup's software development.

Refer to DM Training for DM specific content.
Refer to Telescope & Site Training for Telescope & Site specific content.
Refer to Camera Training for Camera specific content.

6. Risk Management

Each LSST Subsystem will perform a risk analysis when a Validated Product is considered ready for Software Transfer according to the LSST Software Development Guidelines. At that point, a risk analysis will consider the advantages of delivering the current Product against the negatives of any outstanding Anomaly Reports.

The decision to advance will be taken by the Project Manager of the Subsystem under review.

7. LSST Subsystem SQA Specifics

The following sections include the subsystem specific policies and procedures for SQA. The headings will reflect the layout of the overarching document. Include only information which deviates or adds to the SQA Guidelines.

7.1 Data Management

The following sections in the SQA Guidelines are supplemented for DM by the provided information.

7-1 DM Introduction

As another strategy to avoid costly software errors, DM adopted the ICONIX Process as the basis for the Software Development Process.

7.1-1.4.1.1 DM Roles and Responsibilities

This section identifies the DM SQA tasks for which each organizational role is responsible. Also refer to Data Management Organization Presentation (Document-4688) for information on the Data Management role responsibilities.

7.1-1.4.1.2 DM Tasks

This section defines the DM SQA tasks and their sequencing which will be carried out.

Key:

SM : DM System Manager
SQA L : DM SQA Lead
App L : DM Applications SQA Lead
MW L : DM Middleware SQA Lead

Replace the corresponding generic Subgroup task with this DM-specific task

* On completion of the Baseline's Derived Software Requirements Definition Phase, the following should be done

M Check Baseline's Software Requirements are documented in the SysML DM Requirements model and the UML Requirements model ( although set marks may not reach Production levels)
SQA Check Project Management Plan
* conforms to the LSST Project standard for a system engineering method support by SysML.
SQA Check Configuration Management Plan contains
* architectural design configuration management procedures
SQA Check Verification and Validation Plan contains
* a plan for verifying the outputs of the Application Design phase
* a System Test Plan

On completion of the Baseline's Architectural Design Phase, the following should be done

M Check the Baseline's Architectural Design is documented in the EA UML Logical model
SQA Check Project Management Plan contains
* component-based WBS
* criteria exist for deciding the transfer readiness of:
.... a component for integration
.... the software for system testing
.... the software for delivery
SQA Check Configuration Management Plan contains
* package configuration management procedures
SQA Check Verification and Validation Plan contains
* a plan for verifying the outputs of the Detailed Design phase
* an Integration Test Plan

On completion of the Baseline's Detailed Design and Implementation Phase, the following should be done

SM Check the Baseline's Detailed Design is complete and documented in the EA UML Logical model
App L and MW L Check coding standards were followed
App L and MW L Check unit testing procedures were followed
SQA L Check integration test procedures were followed
SQA L Check Project Management Plan contains
* Software Transfer plan and is complete
SQA L Check Configuration Management Plan contains
* Product Release configuration management procedures
SQA L Check Verification and Validation Plan contains
* a test design for each user requirement

7.1-2. DM Documentation

The following additional documents will be generated for DM baselined projects

Baseline's Architectural Design document generated from the EA UML Robustness Diagram;
Baseline's Detailed Design document generated from the EA UML Static Domain;

The requirements and design documents will be generated as needed using the EA report generator. DM's baseline Software Release Plans will be made available on the LSST DM Trac system.

SQA will also randomly sample the Requirements to check traceability from Requirements Specification through all stages of the software development cycle. This traceability will involve access of each design document and each test document. A summary will be included in the Transfer Readiness report.

7.1-3.1 DM Documentation Standards

Also refer to the following documents within the LSST Trac instance:

DM Database Common Vocabulary.
DMAcknowledgements:: Acknowledgements by LSST Data Management

7.1-3.2 DM Design Standards

DM adopted the ICONIX Process as the basis for the Software Development Process. All Requirement and Use Case design will occur within the Enterprise Architect tool using the ICONIX Software Process methodology.

In order to maintain traceability from Science Requirements through Coding and Validation Testing, design development should occur using the EA CASE tool. Refer to DM UML Modeling Conventions (Document-469).

Additional information is available at:

7.1-3.3 DM Coding Standards

The DM Documentation index leads to the following documents within the LSST Trac instance:

DM Coding Standards, Policies, and Conventions
Software Development Environment
- VirtualMachines:: Using a Virtual Machine for development
- Installing:: Installing our build system and products
- ToolSets:: Tools that we are using
- DM Prototyping Environment (historic) (Document-1786)

7.1-3.6 DM Selected SQA Metrics

DM will select appropriate metrics quantifying inplementation quality. Examples of such metrics include:

Defects per 1000 lines of source code,
Defects per severity level,
Defects per status,
Defects per test case,
Defects fixed per development time,
Defects found per usage, and
Defects per test hour.

7.1-4.1 DM Reviews

The DM SQA team is responsible for

verifying that every anomaly report (i.e. ticket) is eventually resolved;
verifying that every code modification brought about by an anamoly report is reflected in the baseline static domain model through either forward or reverse engineering;
verifying that a modified module has been successfully tested prior to release.

7.1-4.2 DM Testing

The DM SQA team is responsible for managing the execution of the Integration, System and Acceptance Tests for new baseline releases.

The DM Workflow process (maintained in Trac) describes the process of implementing any change to the baseline and incorporates the steps where SQA oversight is required.

7.1-4.3 DM Problem Reporting and Corrective Action

The SQA team will review the completion status of Anomaly Reports to ensure that problems are being processed in a timely manner.

7.1-4.4 DM Code and Document Control

The SQA team will review the Ticket Workflow procedure to ensure the Coding and Documentation Standards verification is being performed and major deviation issues are being rectified.

Additionally, DM SQA should verify that explicitly disallowed Boost libraries are not being used in a build product.

7.1-4.7 DM Records collection, maintenance and retention

The subgroup Technical Control Team (TCT) decisions and all reviews of major prototyping efforts and baselines should be archived to either the LSST Document Archive or the DM Trac Document archive. The retention period should be the duration of the LSST Facility.

7.1-5 DM Training

DM New User information is available in the following documents in the LSST Trac instance:

7.2 Telescope & Site

7.2-5 Telescope & Site Training

Yet to be specified.

7.3 Camera

7.3-5 Camera Training