Geoffrey Bessin Product Manager â€“ Test RealTime

Testing Safety Critical ApplicationsTesting Safety Critical ApplicationsGeoffrey Bessin

Product Manager – Test RealTimeGeoffrey Bessin

Product Manager – Test RealTime

Safety Critical ApplicationsSafety Critical Applications

If the automobile had followed the same development cycle as the computer, a Rolls-Royce would today cost $100, get a million miles per gallon, and explode once a year,

killing everyone inside.- Robert X. Cringely

If the automobile had followed the same development cycle as the computer, a Rolls-Royce would today cost $100, get a million miles per gallon, and explode once a year,

killing everyone inside.- Robert X. Cringely

AgendaAgendaDefining Our TermsDesign ResponsibilitiesTest ResponsibilitiesConclusion

Defining Our TermsDesign ResponsibilitiesTest ResponsibilitiesConclusion

What is a Safety Critical Application?What is a Safety Critical Application?Safety Critical Application:

An application where human safety is dependent upon the correct operation of the system

ExamplesAircraft fly-by-wire systemRailway signaling systemsMedical devicesTraffic lightCAD Tools

Safety Critical Application:An application where human safety is dependent upon the

correct operation of the systemExamples

Aircraft fly-by-wire systemRailway signaling systemsMedical devicesTraffic lightCAD Tools

Minimization of RiskMinimization of RiskRisk = magnitude of danger * probability of exposureRisk = magnitude of danger * probability of exposureIEC61508 Safety Integrity Levels

Level 4 – PFD 1:10,000 to 1:100,000Catastrophic Community ImpactLevel 3 – PFD 1:1000 to 1:10,000Employee/Community ImpactLevel 2 – PFD 1:100 to 1:1000Major Property/Production Protection – Possible Employee InjuryLevel 1 – PFD 1:10 to 1:100Minor Property/Production Protection

IEC61508 Safety Integrity LevelsLevel 4 – PFD 1:10,000 to 1:100,000Catastrophic Community ImpactLevel 3 – PFD 1:1000 to 1:10,000Employee/Community ImpactLevel 2 – PFD 1:100 to 1:1000Major Property/Production Protection – Possible Employee InjuryLevel 1 – PFD 1:10 to 1:100Minor Property/Production Protection

Obligatory Safety Critical Example Obligatory Safety Critical Example Ariane 5, 1996Exploded 40 secondsafter launchHad reused Ariane 4 codeCode was felt to beadequateAriane 5 was too fast!64-bit number was stuffed into 16-bit variable – overflow error

Ariane 5, 1996Exploded 40 secondsafter launchHad reused Ariane 4 codeCode was felt to beadequateAriane 5 was too fast!64-bit number was stuffed into 16-bit variable – overflow error

Safety Critical StandardsSafety Critical StandardsISO9001 – Recommended minimum standard of qualityIEC1508 – General standardEN50128 – Railway IndustryIEC880 – Nuclear IndustryRTCA/DO178B – Avionics and Airborne SystemsMISRA – Motor IndustryDefence Standard 00-55/00-56

ISO9001 – Recommended minimum standard of qualityIEC1508 – General standardEN50128 – Railway IndustryIEC880 – Nuclear IndustryRTCA/DO178B – Avionics and Airborne SystemsMISRA – Motor IndustryDefence Standard 00-55/00-56

Strategies for Avoiding Critical Software FailureStrategies for Avoiding Critical Software FailureDesign DiversityFault AvoidanceExtensive Testing

Design DiversityFault AvoidanceExtensive Testing



Software DesignSoftware Design

One way is to make it so simple that there are obviously no deficiencies and the other is to make it so complicated that there are no obvious deficiencies.

C.A.R. Hoare

One way is to make it so simple that there are obviously no deficiencies and the other is to make it so complicated that there are no obvious deficiencies.

C.A.R. Hoare

SimplicitySimplicity

Preventing Bugs – Not Just TestingPreventing Bugs – Not Just TestingMature Development Processes

Capability Maturity ModelInspection Methods

Walkthroughs, formal inspections, code readingDesign Styles

TestabilityClarity

LanguagesStrong typingRuntime constraint checkingParameter checking

Mature Development ProcessesCapability Maturity Model

Inspection MethodsWalkthroughs, formal inspections, code reading

Design StylesTestabilityClarity

LanguagesStrong typingRuntime constraint checkingParameter checking

Designing for TestabilityDesigning for Testability

Testability

The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to

determine whether those criteria have been met.

The effort required to apply a given testing strategy to a system.

The ease with which faults in a system can be made to reveal themselves during testing.

Testability

The degree to which a system or component facilitates the establishment of test criteria and the performance of tests to

determine whether those criteria have been met.

The effort required to apply a given testing strategy to a system.

The ease with which faults in a system can be made to reveal themselves during testing.

Since You Can’t Test Safety Into An Application….Since You Can’t Test Safety Into An Application….

OPTION ONESpecify, design and build the perfect system.

Document everythingReview everythingTest everything

OPTION TWOAim for Option One, but accept that man is flawed.

Include error detection and recovery capabilitiesIterate

OPTION ONESpecify, design and build the perfect system.

Document everythingReview everythingTest everything

OPTION TWOAim for Option One, but accept that man is flawed.

Include error detection and recovery capabilitiesIterate



FactsFactsOn average, there are 3 bugs per 100 statementsTesting typically consumes at least 40% of software development labor50% of software projects are behind schedule25% of software projects are abandoned

On average, there are 3 bugs per 100 statementsTesting typically consumes at least 40% of software development labor50% of software projects are behind schedule25% of software projects are abandoned

Hardware’s LessonsHardware’s LessonsCode ownershipTest responsibilityField execution

Code ownershipTest responsibilityField execution

Developer Responsibility

Code Quality

REDUCED RISKREDUCED RISK

Software TestingSoftware TestingTest First

encourages explicit definition of implementation scopehelps separate logical design from physical design from implementationgrows confidence in the correct functioning of the system as the system growssimplifies your designs

Pair ProgramTwo-heads ARE better than one

Use SimulationUsing the intended processor and I/O ports (if applicable)

Static MetricsComplexity metricsCode adherence to formally defined conventions

Test Firstencourages explicit definition of implementation scopehelps separate logical design from physical design from implementationgrows confidence in the correct functioning of the system as the system growssimplifies your designs

Pair ProgramTwo-heads ARE better than one

Use SimulationUsing the intended processor and I/O ports (if applicable)

Static MetricsComplexity metricsCode adherence to formally defined conventions

Fundamental ResposibilitiesFundamental ResposibilitiesUnit TestingCode Coverage AnalysisRequirements Traceability

Unit TestingCode Coverage AnalysisRequirements Traceability

Unit TestingUnit TestingTest harness, stub, driver creationClear data definition

Equivalence classesBoundary conditions

Batch execution for regression testingEasily traceable error reportingAmenable to clear documentationVersionability

Test harness, stub, driver creationClear data definition

Equivalence classesBoundary conditions

Batch execution for regression testingEasily traceable error reportingAmenable to clear documentationVersionability

Test Class/Data Definition – Regression TestingTest Class/Data Definition – Regression Testing

Test ReportingTest Reporting

Code CoverageCode CoverageCoverage Levels

Procedure Entries and ExitsCallsStatements, Decisions, LoopsBasic, Forced and MC/DC Conditions

Clear linkage to test casesEliminate test redundency

Support safety critical coverage levelsEasily documented

Coverage LevelsProcedure Entries and ExitsCallsStatements, Decisions, LoopsBasic, Forced and MC/DC Conditions

Clear linkage to test casesEliminate test redundency

Support safety critical coverage levelsEasily documented

Multi-Level Code CoverageMulti-Level Code Coverage

Requirements TraceabilityRequirements TraceabilityCornerstone to safety-critical development standardsProperly implemented traceability

Ensures all requirements have testsEnsures tests are updated when requirements change

Enables independent verificationBe Careful!

Requirements coverage is necessary but not sufficientDoes all code come from a requirement?

Cornerstone to safety-critical development standardsProperly implemented traceability

Ensures all requirements have testsEnsures tests are updated when requirements change

Enables independent verificationBe Careful!

Requirements coverage is necessary but not sufficientDoes all code come from a requirement?

Test ManagementTest Management

Automation OptionsAutomation OptionsAutomatic, target-independent test deploymentLinkage between test results and source codeRuntime Analysis

Memory leak detectionPerformance ProfilingRuntime Tracing

Automatic, target-independent test deploymentLinkage between test results and source codeRuntime Analysis

Memory leak detectionPerformance ProfilingRuntime Tracing

Test

AnalyzeResolve



The Old DaysThe Old Days

Testing Safety Critical Applications - ConclusionTesting Safety Critical Applications - ConclusionPresent

Responsibility lies with developerDesign and test are equally crucialDevelopers need proper test training

FutureFormal requirements defintion is enabled by automated toolsetsFormal requirements are translated into UML-like modelsModels serve to generate both tests and codeTraceability is “built” into the model

PresentResponsibility lies with developerDesign and test are equally crucialDevelopers need proper test training

FutureFormal requirements defintion is enabled by automated toolsetsFormal requirements are translated into UML-like modelsModels serve to generate both tests and codeTraceability is “built” into the model

Documents

Geoffrey Bessin Product Manager â€“ Test RealTime