Tool Qualification v12.02

Debug and Test ToolsDebug and Test Tools

Tool Qualification

Minimize the risk of systematic faults in the developed product

due to malfunctions of the software tool (introduce or fail to detect errors)


ISO 26262 Part 8 Supported Processes, Chapter 11

Safety-Lifecycle Process

― Big portions of this procoss is covered by software tools today

This works only if

― The risk of systematic fault in the developed product due to malfunctions of the software tool (introduce or fail to detect errors) is minimized

― The software tool works accurate and correct within such a process

Detailed and Detailed and up front analysis up front analysis / risk assessment of the software tools used / risk assessment of the software tools used within a safety project within a safety project


Software Tools

Software Development Tools: May introduce an error in the final product

Software Verification Tools: May fail to detect an error in the final product

Other Software Tools― Depending on the size of a company we are talking of a big number of

tools (including in-house tools, Excel, …)


Part 1 – Overview The Blue Box and ISO26262 Use Cases


ISO 26262-5, 10Hardware

Reference V ISO 26262-6


iSYSTEM Tools within a Safety Project I

Software Development (ISO 26262-6, 9)― Not so critical

― Assembler (interrupt handler, time-critical algorithms, …)

Software Test (ISO 26262-6, 9)Software Test (ISO 26262-6, 9)― Unit TestingUnit Testing

― Integration TestingIntegration Testing

Hardware-Software Integration and Testing (ISO 26262-5, 10/-4,8)Hardware-Software Integration and Testing (ISO 26262-5, 10/-4,8)

System Integration Testing (ISO 26262-4,8)System Integration Testing (ISO 26262-4,8)

ISO 26262-6 mentions: „…The test environment … shall correspond as closely as possible to the target environment …“


iSYSTEM Tools within a Safety Project II - Unit Testing

isystem.connectisystem.test


isystem.connectisystem.test +Trace & Object Code Level Coverage



isystem.connectisystem.test



isystem.connect isystem.test+ Trace & Profiler

isystem.connect isystem.test+ Trace & I/O Module

iSYSTEM Tools within a Safety Project II - Hardware-Software Integration Testing


Part 2 – How to get confidence in the use of software tools

Detailed and Detailed and up front analysis up front analysis / risk assessment of the software tools / risk assessment of the software tools used within a safety project used within a safety project



Confidence in the use of software tools ISO 26262-8-11

How high is the risk that the malfunction of a tool leads to an error in the final product or errors are not detected (Tool Impact, TI)?

How much do you trust your own methods and procedures (your processes) in order to uncover such malfunction of a tool (Tool Error Detection (TD)?

high average lowTrust in process

minimal riskhigh risk

Beside the tool itself also the process plays a major role in getting Beside the tool itself also the process plays a major role in getting confidenceconfidence

TCL2, TCL3: Tool Qualification appliesTCL2, TCL3: Tool Qualification appliesW

hat is th

e info

rmatio

n base o

n which you assess ri

sk and confidence?


How iSYSTEM helps you getting/creating confidence?

Process ― SPICE, CMMI, … iSYSTEM: ISO9001:2008― Output: Process documentation, external audits

Transparency― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) ― Output: Test documentation, release notes (major new features), item (bugs, changes, new features, …)

tracking lists

Documentation ― Description of features/functions (to be used in a safety project or to be qualified)― Explanation how these features/functions should (not) be used― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing

Guidelines, ...― System Requirements in general or specific to a micro controller― Description of well-known workarounds dedicated to a specific micro controller― …

Pre-Qualification Environment: fitIDEA― Regression test suite, reference hardware, reference application, reference test cases

Organisa

tional M

easures


Result of Step 1 Determine a „Tool Confidence Level“ could be

minimal risk, high trust =

no tool qualification needed

or you see a high risk and an average trust in your processes and a tool qualification would

apply

high average lowTrust in process

minimal riskhigh risk

How to q

ualify a softw

are to

ol?


+ recommended++ highly recommended

Methods

ISO 26262-8-11 Tool Qualification Methods

A-D: Measure of the safety relevance of a malfunction (Automotive Safety Integrity Level (ASIL))

TCL2 A B C D

1a Increased confidence from use ++ ++ ++ +

1b Evaluation of the tool dev process ++ ++ ++ +

1c Validation of the software tool + + + ++

1d Dev. in accordance with a safety standard + + + ++

TCL3 A B C D

1a Increased confidence from use ++ ++ + +

1b Evaluation of the tool dev process ++ ++ + +

1c Validation of the software tool + + ++ ++

1d Dev. in accordance with a safety standard + + ++ ++

Qualifying a software tool means checking/discussing all 4 methodsQualifying a software tool means checking/discussing all 4 methods


What are we talking about?

The company iSYSTEM Development and test tools

Standard debugging and IDE functions (memory read, write, step, memory dump, download, flash programming, etc.)

Advanced debugging with trace and profiling (especially time measurements)

Software test with code coverage and unit test


TCL2 A B C D





TCL3 A B C D






Increased Confidence From Use?

How is a software tool used today?― Dedicated version and configuration

― For one or more specific micro controllers

― In conjunction with other software tools (e.g., compilers)

New project: ― The tool setup may definitely differ from the previous project

― Very new controllers will be used

Dynamic release policy of the software tool manufacturer― Official and regression tested release

― Hotfixes (customer wishes)

― Fairly good tested software versionsShould w

e consider t

his meth

od at all?


YES! – Because … In the market since 1986

Thousends of users, word-wide

• Similar projects successfully done in the past

Great cooperation between you and iSYSTEM― Transparency

― Openness

― Pragmatism

ISO 26262





tracking lists




Incr

eased confidence in

use =

iSYSTEM!


TCL2 A B C D





TCL3 A B C D






Evaluation of the tool dev process

Most of the software tool manufacturers do have defined and documented their processes (and lives those, hopefully), ISO9001 or similar

We guess that non of them have a real process model in place (such as CMMI, SPICE, …)

Audit/Assessment of these processes is possible (quality manual, process descriptions, external audit, …)

Zertificates?!

It may be a big challenge to evaluate a dev process, again and againIt may be a big challenge to evaluate a dev process, again and again Too many software tools are already in use, also Excel and so forthToo many software tools are already in use, also Excel and so forth





tracking lists

Documentation ― Description of features/functions (to be used in a safety project or to be qualified)― Explanation how these features/functions should (not) be used― Manuals: User Manual, Hardware Reference Manual, Getting Started Guides, PCB Desing Guidelines, ...― System Requirements in general or specific to a micro controller― Description of well-known workarounds dedicated to a specific micro controller― …


It is p

ossible to

evaluate

iSYSTEMs d

ev and test p

rocesses!


TCL2 A B C D





TCL3 A B C D






Validation of the software tool

It is also the responsibility of the tool manufacturer to provide means and methods to enable a customer to perform a validation

Ideal: Availability of the manufacturer’s test tool suite for a customer. Documentation, Test suite with reference hardware, reference applications and appropriate test cases.

isystem.connect

isystem.connect test

iSYSTEM Development Environment & Debugger winIDEA

isystem.connect for Python wrapper

testIDE

A

Reference Target

Reference Application





tracking lists




iSYSTEM p

rovid

es fitID

EA for t

his exact purp

ose!


isystem.connect

isystem.connect test

iSYSTEM Development Environment & Debugger winIDEA

isystem.connect for Python wrapper

testIDE

A

Reference Target

Reference Application

Consulting & Integration

Services

Test CasesTest Reports

Safety Manual & Documentation

Test GUI

fitIDEA - Tool Pre-Qualification Environment


Tool Pre-Qualification Environment

• If a specific function of an iSYSTEM tool has to get verified in the customer’s use case context

• Consists of reference hardware plus test cases to verify several functions of an on-chip debug and trace tool such as:― Standard debugging and IDE functions, e.g. memory read, write, step, memory

dump, download, flash programming, etc.

― Advanced debugging with trace and profiling (especially time measurements)

― Software test with code coverage and unit test

― …

• Is an extract of iSYSTEM’s internally used regression test tool suite

• May be ported/adapted to a customer’s target system


TCL2 A B C D





TCL3 A B C D






Development in accordance with a saftey standard

View and adapt recommendations of such standards

Transfer know-how to software developers (how others have to do it)




Transparency― Release and test process (http://www.isystem.com/downloads/winidea/release-policy) ― Output: Test documentation, release notes (major new features), item (bugs, changes, new

features, …) tracking lists




iSYSTEM is

willi

ng to le

arn!


ISO 26262

Summary

Detailed and up front analysis / risk assessment of the software tools used within a safety project

Beside the tool itself also the process plays a major role in getting confidence

It is also the responsibility of the tool manufacturer to provide means and methods to enable a customer to perform a validation

Tool manufacturers will improve their development and test processes

Transparency first, qualification second

Technology

Tool Qualification v12.02