Embed Size (px)
Citation preview
How to find defects early and increase
the reliability of software systems
Rakesh Rana
PhD student ([email protected])
Computer Science and Engineering
Chalmers | University of Gothenburg
The work presented in this presentation has been done in collaboration with Asoc. Prof. Miroslaw Staron, Dr. Christian Berger, and Prof. Jörgen Hansson; Chalmers |
University of Gothenburg; Fredrik Törner and Martin Nilsson from Volvo Car Group
The work is partially sponsored by Vinnova FFI program as VISEE project Dnr. 2011-04438
Using Fault Bypass Modeling to improve rapid prototyping and combining
fault injection with mutation testing for early identification of safety defects
This Car Runs on Code
“It takes dozens of mircroprocessors running 100 million lines of
code to get a premium car out of the driveway, and this software is
only going to get more complex” -ieee spectrumRef: http://spectrum.ieee.org/green-tech/advanced-cars/this-car-runs-on-code
Reliability
Fault Injection
• Fault injection is an important and widely used technique for
experimental dependability evaluation of computer systems.
• These techniques has been traditionally used for testing
dependability of the both hardware and software systems.
*Reliability and dependability are very important features of any computer system.
*So how can we enhance reliability in automotive software?
ISO 26262 recommendation for using
fault injection techniques
ISO/DIS 26262 Chapter Reference to recommendation
4
Hardware-software
integration and testing
•Table 5 — Correct implementation of technical safety requirements at the hardware-software
level.
•Table 8 — Effectiveness of a safety mechanism’s diagnostic coverage at the hardware-software
level.
System integration and
testing
•Table 10a — Correct implementation of functional safety and technical safety requirements at
the system level
•Table 13b — Effectiveness of a safety mechanism's failure coverage at the system level
Vehicle integration and
testing
•Table 15 — Correct implementation of the functional safety requirements at the vehicle level
•Table 18 — Effectiveness of a safety mechanism's failure coverage at the vehicle level
5 Hardware integration and
testing
•Table 11 — Hardware integration tests to verify the completeness and correctness of the safety
mechanisms implementation with respect to the hardware safety requirements
6 Software unit testing •Table 10 — Methods for software unit testing
Software integration and
testing
•Table 13 — Methods for software integration testing
Fault Injection
Types of fault injection techniques
• Hardware-Based
• Software-Based
• Simulation-Based
• Hybrid Models
CASE: ABS (Anti-Lock Braking) System
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 150
10
20
30
40
50
60
70
Time in sec
Speed in R
PM
Vehicle and wheel speed with & without ABS
Vehicle Speed without ABS
Vehicle Speed with ABS
Wheel Speed without ABS
Wheel Speed with ABS
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 150
20
40
60
80
100
120
140
160
180
200
Time in sec
Dis
tance in m
Stopping distance with & without ABS
Without ABS
With ABS
CASE: ABS (Anti-Lock Braking) System
0 1 2 3 4 5 6 7 8 9 100
20
40
60
80
100
120
140
Time in sec
Speed in R
PM
Vehicle and wheel speed with fault injection
Vehicle Speed
Wheel Speed
Fig: ABS system-environment model representation in
Simulink with fault injector setup.
CASE: ABS (Anti-Lock Braking) System
The problem do not exist in open loop model configuration.
Scripts are used to provide recorded data as input, while the output is saved as data file
and compared to reference/expected output.
The major limitation with such testing is that it’s limited by the availability of
recorded sensors data as well as need to have the correct output for reference
purposes.
Thus it cannot test systems under conditions where the input and output data is not
available
Or if a new functionality is developed or existing system configuration changed such
that the input/output data do not match to previous instance, this type of testing is
unfeasible.
Closed loop continuous models do not suffer from these limitations.
Why ABS model breaks under Fault Injection
setup?
0 1 2 3 4 5 6 7 8 9 100
20
40
60
80
100
120
140
Time in sec
Speed in R
PM
Vehicle and wheel speed with fault injection
Vehicle Speed
Wheel Speed
Why ABS model breaks under Fault Injection
setup?
FBM Modeling
FBM principle is described as following:
• “If a signal injected with faults or its derivative is used to calculate/control any natural environment parameter(s), the part of signal or its derivative which is used to calculate/control the environment parameter(s) should be made fault free to break the unrealistic feedback loop”
Natural Environment Parameter here refers to such a parameter which is not a property of system but needs correct value from system to define its correct state/value.
ABS: modeling using FBM principle
0 1 2 3 4 5 6 7 8 9 10 11 12 13 140
10
20
30
40
50
60
Time in sec
Speed in R
PM
Vehicle and wheel speed with fault injection (FBM)
Vehicle Speed
Wheel Speed
0 1 2 3 4 5 6 7 8 9 10 11 12 13 140
20
40
60
80
100
120
140
160
Time in sec
Dis
tance in m
Stopping distance with & without fault injection (FBM)
ABS, without FI
ABS, with FI
FBM Advantages
Simple to implement.
Allows closed loop testing with continuous signals
for complex functions with interdependence with
environment.
The functional implementation can be tested very
early (design phase) for wide range of simulated
cases and dependability characteristics also could
be studies and analysed early.
Helps in developing robust functions/software.
Reduces the product development time and late
defects.
FBM: Initial Validation
Fig: ABS system-environment model representation in Simulink
with fault injector setup.
Fault Bypass Modeling
• Can be useful
• Could help with what-if scenarios
analysis
• Fault scenarios currently not looked
into much (at least not at modeling
level)
• Current focus is on correct case
working
• Sensor failure/behaviour models
currently not well advanced/non-
existing
• Need efficient plant models
FBM applied to ABS case
*8th International Joint Conference on Software Technologies - ICSOFT-EA, Reykjavik, Iceland, July 2013
• Objective:
– How models can be used more effectively for early verification and validation?
• Method: Descriptive-Qualitative case study based on empirical observations to
propose a framework which combines methods of fault injection and mutation
testing to be used at the model level that can be used for increasing the efficiency
of ISO-26262 compliance.
Increasing Efficiency of ISO-26262 Verification and Validation by Combining
Fault Injection and Mutation Testing with Model Based Development*
Increasing Efficiency of ISO-26262 Verification and Validation by Combining
Fault Injection and Mutation Testing with Model Based Development*
*8th International Joint Conference on Software Technologies - ICSOFT-EA, Reykjavik, Iceland, 2013
a) Assign TSRs corresponding
to FSRs to Z-outputs
b) Inject Faults (simulating
common defects) to X-inputs
c) Identify critical fault scenarios;
Study fault propagation properties;
Build fault tolerance
e) Repeat Steps (b) & (c) to
test, correct & validate
the function for its “d”
dependencies
d) Cause Mutation to “n” blocks of
function & assess effectiveness of
given test suit using M.Testing
f) Examine Mutation not killed; Update
test cases or build new to detect such
failure scenarios/defects
Improving Fault Injection in Automotive Model Based Development
using Fault Bypass Modelling*
*2nd Workshop on Software-Based Methods for Robust Embedded Systems, INFORMATIK, Germany,2 013
Environment Model
SW system Model
Out_1
Output
Inp_2
Inp_1
Out_2
Natural/State
parameter(s)
• Objective:
– How simulations of functional models be used effectively for early verification
and validation?
• Method: Descriptive-Qualitative case study based on experiment, we propose
and provide proof-of-concept for “fault bypass modelling”, a simple yet effective
framework for correct analysis of simulation in closed loop mode.
*2nd Workshop on Software-Based Methods for Robust Embedded Systems, INFORMATIK, Germany,2 013
0 1 2 3 4 5 6 7 8 9 10 11 12 13 140
10
20
30
40
50
60
Time in sec
Sp
ee
d in
RP
M
Vehicle and wheel speed with fault injection (FBM)
Vehicle Speed
Wheel Speed
0 1 2 3 4 5 6 7 8 9 100
20
40
60
80
100
120
140
Time in sec
Sp
ee
d in
RP
M
Vehicle and wheel speed with fault injection
Vehicle Speed
Wheel Speed
Vehicle Speed
Control Signal
Wheel Speed
Vehicle Speed
Wheel Speed
Relative Slip
ABS Model
Environment Model
Improving Fault Injection in Automotive Model Based Development
using Fault Bypass Modelling*
Conclusions
Prevention: Propose and evaluate methods that can potentially
increase the reliability of software in the automotive domain.
• A framework to combine fault injection and mutation testing approach
applied at behavioural models is introduced.
• Fault Bypass Modelling is introduced that helps to develop robust
software, a proof-of-concept is provided.
Conclusions
Why predict and prevent software defects in automotive domain?
• Predicting defects inflow help us manage defects and testing resources
effectively.
• Effective defect management and defect prevention increases reliability
of software in cars and thus cars themselves.
– It also translates to lower development costs and time to market.
• We can combine approaches such as fault injection and mutation testing
to effectively test for safety related defects, and
• We can also use behavioural models early in the development cycle to
build robust software right from the start.
Thank You
SRGMs to help with
- Optimal allocation of test resources, and
- Assessment of release readiness
For more details
Contact: Rakesh Rana
Back Up Slides
Source: Healing with Art, community on Facebook, https://www.facebook.com/photo.php?fbid=10151903164088141&set=a.378605758140.163024.14524668140&type=1&theater
