#PNSQC2020 1

Quality-Focused Software Testing in Critical Infrastructure

© SEL 2020

Zoë OensSchweitzer Engineering Laboratories, Inc.

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 2

Scope

Quality

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 3

Better

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 4

Scope

5

4

3

2

1

1 2 3 4 5 6 7 8 9 10

Defects

Scope

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 5

Ranking Features By Criticality

Decision factors Factors weighedby importance

Featureevaluation

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 6

Kim and Kang Method

Give them weight

Rank each

feature

Calculate testing rank

Assign testing

based on rank

Create decision factors

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 7

Example of Critical Infrastructure

Safety (1.0)If this feature fails, how dangerous would this be?

Reach (0.8)If this feature fails, how many end users will this affect?

Usability (0.7)If this feature fails, will end users lose trust in the product?

Complexity (0.7)How complex is this feature?

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 8

Example of Critical Infrastructure

Decision factorPredetermined

weightRank(0–5)

Weighted rank (weight • rank)

Safety 1.0 5 5

Reach 0.8 3 2.4

Usability 0.7 3 2.1

Complexity 0.7 1 0.7

Feature risk 10.2

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 9

Example of Critical Infrastructure

Tiered testing– Preset quality rank tiers– Test to calculated tier

Percent coverage– Determine quality rank percentage– Test to meet total coverage

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 10

Time

Test processLessons

learned from manufacturing

Automation

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 11

Lessons Learned from Manufacturing

Well-defined processes

Fast issue evaluation

Experienced feedback

Cross training

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 12

Test Process

Specification

Code

Test process Test result

Inputs

Output

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 13

Suppliers and End Users

Trust Understanding Ownership

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 14

Test Process

Write Setup Execute Results creation

Specification

Code

Results

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 15

Test Write Improvements

Automation Better communication

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 16

Test Write for Repeated Software Elements

Machine-readable specification

Consistent testing

Assured quality

Button 1

Button 2

Button 3

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 17

Test Setup Improvements

Prebuilt setups Organization

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 18

Test Execute Improvements

Automation

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 19

Test Results Creation

Consistency Automatic creation

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 20

Hidden Costs of Automation

Training Maintenance Documentation

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 21

Cost of failure Fewer defects Metrics

Cost

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 22

Preemptivemetric

Estimate of improvement

Business Value Delivered

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 23

Test Satisfaction

Use real-timeuser feedback

Survey users after receiving device

or service

Give surveystrackable numbers

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 24

Change Failure Rate

Can only becompleted after

release

Is calculated percentage of

release to failures

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 25

Communication

#PNSQC2020 26

Quality-Focused Software Testing in Critical Infrastructure

© SEL 2020

Zoë OensSchweitzer Engineering Laboratories, Inc.

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens

#PNSQC2020 27

Questions?

Quality Focused Software Testing in Critical Infrastructure, Zoë Oens