© 2014 IBM Corporation

Deployment of a test management solution for a defence project using an integrated tool chain

Code 1183

Dr. Einar Karlsen (IBM, Co-Speaker)

Thomas Schöning (Airbus DS)

1

Introduction

Dr. Einar Karlsen (IBM)-Technical Solution Architect

Thomas Schöning (Airbus Defence and Space)- Integration and Test Manager

Thomas Südbröcker (IBM)- Technical Sales and Solutions + Community of Practices Leader Quality Management (Germany)

2

• Subcontractor in a NATO Project with requirement based development, test and formal verification of a complex system entity

• Find a proper tooling solution for test and verification process using the IBM Rational tooling chain (Jazz Platform)

– Support for relevant standards ISO/IEE 15288, ISO/IEEE 12207, ISO 9001, ISO/IEC 9126

– Support for relevant process: Waterfall (V-Model) with incremental SW Development

– Integration with IBM Rational DOORS to accommodate for requirement traceability

– Automatic document generation for requirements, design and test artefacts (plans, procedures, reports)

– Insight into status and trend of the testing efforts (metrics)

Setting the Scene – Situation

3

• Challenge

– Unknown effort to adequately install, integrate and customize the whole tooling chain to perform modelling, implementation, and test management

– Appropriate test management support (planning, execution, reporting, document generation, etc) according to required process?

• Improvements

– Decrease of costs, risks, inconsistencies and time-to-market through automatic generation of project deliverables

– Collection of metrics to measure test design progress, requirements coverage

– Online/Offline determination of test results while performing the test

Setting the Scene – Risk and Opportunity

4

• Profound orientation for configuration and customization of IBM Rational Tools for a tool-aided test management solution (planning, execution, reporting, documentation) for similar test activities

1. Start early to implement test management prototyping to better estimate the effort to adapt the tools to the project needs

2. Invite technical professionals to have guidance and avoid pitfalls

3. Perform team training

4. Look at the deliverables before customizing your tool chain

Setting the Scene – Position and Benefits

5

• Introduction

– Programme Overview

– Test and Evaluation Strategy

• Test management

– Solution Overview

– Solution Architecture

– Document Generation

– Metrics and Dashboards

• Summary

Agenda

6

• Confidential, multinational NATO C4ISR Project (C4ISR=Command, Control, Communications, Computers, Intelligence, Surveillance, and Reconnaissance)

– Unmanned Air, Command and Control Entities

– Transportable and Mobile Ground Stations for data exception, processing, and archiving facilities

– Communication links to other NATO and interoperable national ground stations, and to NATO and national Command and Control systems

– Exchange of standardized data via different interfaces and communication lines

• Airbus DS responsible for development, test and formal verification of a system entity consisting of several Subsystems

Programme Overview

Test and Evaluation Strategy

Verification

Development

Testing

Entity

Subsystems

CSCIsHWCIs

ServicesComponents

Development

Testing

Development

TestingTest

Test

Test

Verification

Integration

8

• Requirements of System Entity, Subsystems, Computer Software Configuration Items (CSCIs), and Hardware Configuration Items (HWCIs) are kept in DOORS database with traceable entries

• Testing/Verification is classified either to be

– Informal Tests

• conduct informal tests throughout the system development

– Subcontractor Development Tests

• For CSCI, HWCI, HW/CSCI groupings, Subsystem level, Entities and Integration testing

– Formal Verification

• Verification of allocated customer requirements based on proof of derived requirements (Entity, Subsystem, HW/CSCI groupings)

Test and Evaluation Strategy

Test Management – Solution Overview

9

10

Test Management – Solution Overview

• Test Plan

– consists of a number of Test Cases used to proof requirements

– is linked with one or more Requirements Modules (Test Plan views) � coverage and traceability (verification level)

• Test Case

– is linked to one or more particular requirements belonging to the requirements view(s) proofing requirements on different levels

• Test Script

– keeps the Test Procedure with single steps for test execution

• Test Case Execution Record/Test Case Result

– container to keep the Test Case results depending on iteration (integration, dry-run, acceptance) and test environment

• PTR (=Problem Trouble Report)

– keeps the defects occurred during Test Case execution

Test Management – Solution Architecture

11

IBM Rational DOORS

IBM Rational Insight

IBM Rational Publishing Engine

REST, ETL REST, ETL

IBM Rational Quality Manager

IBM Rational Team Concert

REST, ETL

Dashboards, Status and trend reports

Test Plans, Procedures and Reports(Project Deliverables)

RESTREST REST

OSLClinks

OSLClinks

12

Test Management – Test Document Generation

Test Document Generation – Master Test List

Test Document Generation – Test Case Details

15

• Metrics: What you can measure that can be done!

• Metric types: Project, Product, Process, and Personnel

• Definition of metrics for monitoring and control test planning, design, development and execution

– test planning

• Percentage of requirements having a test case

– test design (analysis and concept)

• Percentage of test case with different status (draft, ready for review, approved)

– test development

• Percentage of test scripts with different status (draft, ready for review, approved)

– test execution

• Percentage of planned test cases being executed (passed, passed with condition, failed, blocked, error)

Test Management – Metrics and Dashboards

16

Metrics and Dashboards – Test Case Design

17

Metrics and Dashboards – Test Case Execution

18

• Tooling Solution (Jazz platform) for test management process with combined DT&E strategy (Informal, Development Tests, Formal Verification) applied for system entity (NATO Project)

• Test Management Solution (Overview and Architecture)

– Test Plans linked with requirements view

– Test Cases linked with requirements

– Test Execution Records keeping the results for each Iteration

– Definition of timelines, iterations to support test planning andexecution

• Test Document Generation

– Tooling adaption for artefact templates (Test Plan, Test Case)

– Examples (Master Test List, Test Case Details)

• Metrics and Dashboards

– metrics for monitoring and control test planning, design, development and execution

Summary

Articles on developerWorks

• RQM Template authoring articles on developer Works.

– Using IBM Rational Publishing Engine to generate compliance

documents: Report generation for Rational Quality Manager,

introducing REST service

URLs, importing schemas, and

basic template authoring

– Using IBM Rational Publishing

Engine to generate compliance

documents: Part 2. Report generation for Rational Quality

Manager, showing how to

extract information from grid sections and report on test

results and related defects

Thank You!

Your Feedback is Important!

Access the Innovate agenda tool to complete your session surveys from your smartphone, laptop or

conference kiosk.

21

• Test Plan:

– Artifact Templates (keep Test Cases and TCERs only)

– Artifact Categories (manufacturer, part number, type) to distinguish between different plans (system, software,etc)

• Test Case:

– Artifact Template has been customized to the documentation format needs (Test Plan, Test Procedure)

– Artifact Categories (test area, test method, test event, test type) and RQM section (description, test objective, requirement links,estimation) are used to appropriately keep Test Case data

• Timelines (Development and Production Phase)

• Iteration Types (Milestone, Software Test, System Test)

• Iteration

– directly corresponds with incremental software development (integration tests, dry run, acceptance test)

– support generation of TCER

Test Management – Tooling adaption

22

• Administration of project requirements and test cases as well as the creation of bi-directional dependencies between them

• Planning test effort and milestones

• Defining test goals, acceptance and quality criteria

• Deriving test cases derived from requirement specifications

• Estimation of the effort of the test activities

• Definition of a time plan for each test iteration with finish dates of key activities

• Collection of different test environments and the generation of test configurations

• Creation and administration of test cases (manual and automated tests)

• Recording test results, creating defect and evaluating reports

• Support review and release process for the test plan and individual test cases.

IBM Rational Tools – Usage