Risk base effective testing and quality management in the project

MOTOROLA SOLUTIONS

09 Oct 15

Risk-based Testing & Quality management in a program\project.

© 2015 Motorola Solutions, Inc.

ARTUR GORSKI

Agenda 1. Problem description.

2. Risk 1 - # of undiscovered defects.

3. Risk 2 - # of failing customer scenarios.

4. Risk 3 - # of non-detectable defects.

5. Summary.






5. Summary.



IDEAL IMAGE

1. We know the number of undiscovered defects in each module of the product.

2. We know distribution of Critical, Not-Critical and Cosmetic defects of

undetected faults.

3. We know tests automation robustness for regression testing.

4. We know the number of defects that won`t be detected without e.g. SIT.






5. Summary.


Risk #1 - # of undiscovered defects


Due to the fact that development introduces defects, there is a possibility of finding them in a late phases,

what can interfere with quality gates and delay release.

Let`s use historical data related to our project/product.

This is example data


And pick those that had the same testing process And where estimates related to implementation tasks

were reliable and available.



Gather the data and calculate actual injection rate (# of defects injected per 1 ideal day of implementation).

Release Dev [ID] # of sourced

defects Real Inj.

Rate Improvement

Project C 500 250 0.5 Baseline

Project E 800 320 0.4 20%

Actual project 0.32 20% (planned)



Compare parameters values of a static model, which will help you to judge whether the conditions of

particular modules are better or worse.



Compare parameters values of a static model, which will help you to judge whether the conditions of

particular modules are better or worse.



Now… what are the main factors that influence number of injected defects ?

Technical complexity

Avg. injection rate

Estimate

Static model


Technical Complexity Assessment From 1 to 5 (1 – the easiest)

Tester Developer Architect

3 2 4 5 4 4



Technical Complexity Assessment From 1 to 5 (1 – the easiest)

Count average difference



Average Injection Rate is taken from static model



Estimates are exported from Version1 and compared with burn down chart.



So we are having technical complexity, avg. inj. Rate and estimates.

Let`s simulate # of defects that were injected.



What is Level Of Trust ?

It`s a level of confidence in the received data. From 1 to 6 (1 - complete lack of trust)

LoT should be high (5/6)

LoT should be moderate



in a case of technical complexity

LoT should be high (5/6)

LoT should be low

Count average difference

LoT should be moderate

is moderate



in a case of avg. injection rate think about border values and count disorder

LoT = 5

LoT = 3

[(Max – Avg)/ Avg] * 100%



In practice…

The lower the value the more disorders given input AND

the higher probability of choosing disordered value during simulation.

LoT Value Disorder in %

6 0%

5 20%

4 40%

3 60%

2 80%

1 100%


Example

Estimate = 10

- 60% disorder + 60% disorder

Given value

- 20% disorder + 20% disorder

LoT_1 = 3

LoT_2 = 5


When the parameters are set let`s run simulation.

When the parameters are set, let`s run the simulation. # of simulations

After each simulation each total number of defects is stored.




Monte Carlo

Each result is stored

When simulation is finished a histogram of outputs is generated.

The lowest

The highest

The most probable




Then cumulative representation is constructed

Monitor and control found vs. injected defects in a module. This is example data


Injection and found curves for P1\F1


I ‘know’ how many defects are left, but ‘which’ ones ?





Now we can assess the risk #1





Probability and Impact are used to assess the overall risk Summary of Risk #1 for each module:

P1/F1 P2/F2 P3/F3 Etc.



What can we do with the information ?

Summary of Risk #1 for each module: P1/F1 P2/F2 P3/F3 Etc.

Prioritize execution of test cases Deliver information to key stakeholders about actual product quality Add/Remove testing effort etc.



How do we check whether the prediction is correct ? Wait for feedback from SIT, FAT, UAT, Client usage etc.

Use Software Reliability Engineering e.g. Weibull distributions

CDF: F(t) = 1-e-(t/c)^m

PDF: f(t) = (2/t)(t/c)me -(t/c)^m

Metrics and Models in Software Quality Engineering, 2nd Edition [Chapter 7] Wikipedia.org



How do we check whether the prediction is correct ?

SRE








5. Summary.


Divide your requirements/specifications Requirements/Specifications

# of Automatable # of Non automatable

Module automatability

# of Automated # of Not Automated

Automation status of a module

# of Non automatable + # of Not Automated Risk impact

Risk probability

Risk level


Example data

Automatability

Automation status


P2/F2 Automatability

P2/F2 Automation Status P3/F3 Automation Status



Example data



P2/F2 Automation Status P3/F3 Automation Status



Risk overview Risk #2:

Due to the fact that defects can be injected into already tested areas during on-going development or during defects fixing there is a possibility of delivering a feature with failing functionalities covered by requirements what can decrease customer satisfaction. Probability: Automatability level and Automation Status e.g. 98% and 77% Impact: Number of not automated specifications e.g. 15



Use probability and impact to assess the risk Summary of Risk #2 for each module:

P2/F2 P3/F3 etc.



Summary of Risk #2 for each module: P2/F2 P3/F3 etc.

Change automation strategy to increase automatability of a module Invest more and increase automation level Plan proper manual regression etc.






5. Summary.


Why some of defects are non-detectable during your testing phase ?

Some will be injected after your`s testing phase; Some require software-hardware integration; Some require integration with other systems; Some require clients` environment; Some require specific configuration; Some require specific perspective and higher independency; Some require real devices not simulators; etc.


S

Assess defects that escaped from previous release

Around 61% were Impossible or Hard to find during our testing phase



Risk overview Risk #3:

Due to the fact that some of the defects are related to hardware, software and hardware integration, real user environment, systems integration and are impossible to be detected by simulators there is a possibility of detecting critical defects (that can mask other defects) during e.g. SIT testing phase what can interfere with quality gates and delay release of a program. Probability: Activities and its results done by other testing teams, during demos, dependency to hardware and other system etc. Impact: 61% of undiscovered defects



Use probability and impact to assess the risk




Inform stakeholders about risk Plan early system integration Built new skills in your team etc.






5. Summary.



Plan risks reactions and manage the risks.

Feature Risk #1 Risk #2 Risk #3 Risk reactions

P2/F2 accept

P3/F3

1. Invest in exploratory testing

2. Plan proper manual regression

3. Plan ESI with SIT team

P1/F1 …




Any questions ? ARTUR GORSKI pl.linkedin.com/in/gorskiartur [email protected]

Leadership & Management

Risk base effective testing and quality management in the project