©2015 InfoStretch Corporation. All rights reserved.

Liana Gevorgyan | May 6, 2015

Measuring Quality

QA Metrics and Trends in PracticeUS - UK - India

©2015 InfoStretch Corporation. All rights reserved.

SECTION 1: TECHNOLOGY IN LIFEBugs Are Costly

1999

Mars Climate Orbiter CrashInstead of using the provided metric system for navigation, the contractor carried out measurements using empirical units and the space craft crashed into Mars.

COST

$135 Million

1996

ARIANE FailureAriane 5 rocket exploded 36.7 seconds after take off. The engine of this satellite was much faster than that of the previous models, but it had a software bug that went unnoticed.

COST

>$370 Million

2003

EDS Fails Child SupportEDS created an IT system for a Child Support Agency in the UK that had many software incompatibility errors.

COST

$1.1 Billion

2013

NASDAQ Trading ShutdownAugust 22, 2013 NASDAQ Stock Market Shut down trading for three hours because of a computer error.

COST

$2 Billion

1985-1987

Therac-25 Medical AcceleratorA software failure caused wrong dosages of x-rays. These dosages were hundreds or thousands of times greater than normal, resulting in death or serious injury.

COST

5 Human Lives

Technology In Our Daily LifeAverage usage of electronic systems in developed countries:

One PC or desktop in each home. 80% of people are using mobile phones 40% of people are driving cars with various electronic systems People are traveling via train, plane on an average once a

year Dozens of other embedded systems in our homes Dozens of software programs in our work place, service

systemsQuality of all mentioned systems are equal to the Quality of life!

SECTION 2: DEFINING THE “WHAT”Known QA Metrics & Trends

10

Defining “What” Metrics and Trends

Measure to Understand

Understand to Control

Control to Improve

11

Several Known QA Metrics and Trends Manual & automation time ratio

during regression cycle Scripts maintenance time during

delivery iteration Daily test cases manual execution Automation effectiveness for issues

identification Issues found per area during

regression Areas impacted after new features

integration Issues identification behavior based

on major refactoring. Software process timetable metrics Delivery process productivity metric Software system availability metrics

Test cases coverage Automation coverage Defined issues based on gap

analysis Ambiguities per requirement Identified issues by criticality Identified issues by area separation Issues resolution turnaround time Backlog growth speed Release patching tendency and

costs Customer escalations by

Blocker/Critical/Major issues per release

QA engineer performance Continuous integration efficiency

12

Metrics Classification

PRODUCT METRICS

PROCESS METRICS

QA METRICS

13

Metrics Examples by Classification

Delivery process productivity metrics Continuous integration efficiency Release patching tendency and costs Backlog growth speed QA engineer performance Software process timetable metrics

Software system stability metrics Identified issues by criticality Identified issues by area separation Customer escalations by

Blocker/Critical/Major issues per release

Ambiguities per requirement Backlog growth speed

PRODUCT METRICS PROCESS METRICS

14

Sample Metrics Visual

55%20%

15%7% 3%

Automated UI and BEAutomated UIIn ProgressPending AutomationNot Feasible

AUTOMATION COVERAGE BUGS BY SEVERITY

3 % 5 %12 %

34 %

45 %

Blocker Critical High Medium Low

15

Visual Depiction Of Sample Trends

Jan Feb March April05

10152025303540

BlockerHigh

Low

BlockerCriticalHighMediumLow

00.5

11.5

22.5

33.5

44.5

5%

ISSUE ESCALATIONS BY CRITICALITY - MONTHLY TREND

1 2 3 4 5 6WEEKS

REJECTED BUGS % PER WEEK

16

ExpectationsSmooth releasesPredefined risks with mitigation plansNice feedback and appreciationTop notch and innovative products

17

Real Life Delivery not always ideal

We are familiar what is patching the release

Lack of process tracking data for analysis

Experimental delivery models not exactly the Best practice models

SECTION 3: DELIVERY PROCESSES & METRICS

Waterfall & Agile

Waterfall ProcessREQUIREMENT

SVALIDATION

ARCHITECTUREVERIFICATION

MODULE DESIGNVERIFICATION

IMPLEMENTATIONSYSTEM TEST

OPERATIONS AND

MAINTENANCEREVALIDATION

20

Agile Process

TESTING/VALIDATION/VERIFICATION

Product Backlog Client prioritized

product features

Sprint Backlog Features assigned

to Sprint Estimated by team Team Commitment

Working Code Ready For

DEPLOYMENT

Time-Boxed Test/Develop

PRODUCT BACKLOG BACKLOG TASKS

Scrum Meetings Every 24 hours

21

Agile Process MetricsSCRUM TEAM SPRINT METRICS

Scrum Team’s Understanding of Sprint Scope and Goal

Scrum Team’s Adherence to Scrum Rules & Engineering

Practices

Scrum Team’s Communication

Retrospective Process Improvement

Team Enthusiasm

Quality Delivered to Customer

Team Velocity

Technical Debt Management

Actual Stories Completed vs. Planned

22

Processes Are Not Always Best Practices Unique way of Agile Transition from Waterfall to Agile Transition from Agile to Kanban

23

Metrics Set Definition for Your Project Process Technology Iterations Project/Team Size Goal

SECTION 4: WEIGHT BASED ANALYSIS FOR QA METRICS & MEASUREMENTS

Mapping With Graph Theory

25

Metric and Trends for your Project

You are watching Metrics/Trends set, are they the right ones?

Trends are in an acceptable range, but the products quality is not improving?

Trying to improve one metric and another is going down?

How do you analyze and fix it?

26

Mapping QA Metrics Info Graph Theory

Process Metrics > A= Metric 1, B=Metric 2…

Actions/Data set that takes effect in Metrics > A1, A2…Metric dependencies of specific action

Product Metrics > C= Metric 3, D=Metric 4…

27

Preconditions & Definitions For Metrics & Actions mapped model Node’s initial weight is predefined and has value from 1-10

Edge’s weight is predefined and has value from 1-10

Connections between Nodes is defined based on dependencies of Metrics from each other and from Actions

All Actions have fixed 1 weight

28

Initial Metrics Model & DependenciesAssumeCurrent Metric set is: 2 Process Metrics -> M1, M2

2 Product Metrics -> M3, M4

Where : M1 has dependency on M3M1 has dependency on M4M2 has dependency on M3

There are 3 Actions or Data sets that have effect on some of the Metrics. Those are A1, A2, A3

Where : M1 has dependency on A1 and A2M4 has dependency on A3

Initial Priority

Initial Priority based on Best Practices

W(M1) = 5W(M2) = 4W(M3) = 3W(M4) = 2

29

Metrics Visualization via Graph

M2

M3

M14

3

M4

5

2

A2

A1

A3Process Metrics > A= Metric 1, B=Metric 2…

Actions/Data set that takes effect in Metrics > A1, A2…Metric dependencies of specific action

Product Metrics > C= Metric 3, D=Metric 4…

30

Weight Assignment On Undirected Graph

M2

M3

M14

3

M4

5

2

A2

A1

A3

23

5

1

1

6

1

1 1Process Metrics > A= Metric 1, B=Metric 2…

Actions/Data set that takes effect in Metrics > A1, A2…Metric dependencies of specific action

Product Metrics > C= Metric 3, D=Metric 4…

31

Calculation Formula for Metrics New Priority Priority of the node is calculated the following way:

where

- initial priority of the node

- node weight assigned by user

- cumulative weight of each node's edges

𝑾(𝑨)

32

New Priority Calculations For One Metric

M2

M32

4

3A2

A11

11

1

Initial PriorityM2 = W(M2)=4

New PriorityM2 = W(M2) * (W(M2-A1) + W(M2-A2) + W(M2-M3))M2 = 4 * (1+1+2) = 4*4 = 16

33

New Priority Calculations For Graph

M1 M2 M3 M4 A1 A2 A3W 5 4 3 2 1 1 1

M1

5 3 5 40

M2

4 2 1 1 16

M3

3 3 2 6 33

M4

2 5 10

C A L C U L AT I O N S

Metrics New

Priority

M1M3M2M4

Metrics Initial

Priority

M1M2M3M4

34

Metrics Priorities: Current Vs. Calculated

Initial Priority Based on Best Practices

M1

M2

M3

M4

Project Dependent Calculated Priority

M1

M3

M2

M4

INITIAL PRIORITY NEW PRIORITY

SECTION 5: METRICS WEIGHT BASED ANALYSIS IN PRACTICE

Defining “How”

36

Metrics Definition For Test Project Process – Agile with Area ownership Technology – SAAS Based Enterprise Web & Mobile App Iteration – 2 weeks Project Size – 5 Scrum Teams Goal – Customer Satisfaction, No Blocker, Critical Issues

Escalation by Customer

37

Key Metrics and DependenciesMetricsM1 - Customer Escalations per defect severity – Product MetricM2 – Opened Valid Defects per Area – Product MetricM3 – Rejected Defects – Process MetricM4 - Test cases Coverage – Process MetricM5 - Automation Coverage – Process MetricM6 - Defect fixes per Criticality – Product Metric

Actions and Data SetsA1 – Customer types per investment and escalations per severityA2 – Most Buggy areas

38

Metrics Initial PriorityWeight assignment and dependency analysis

Metric Name Predefined Node Weight

Metrics By Initial Priority

M1 - Customer Escalations per defect severity

8 M1

M2 – Opened Valid Defects per Area

5 M6

M3 – Rejected Defects 4 M2

M4 - Test cases Coverage 3 M3

M5 - Automation Coverage 2 M4

M6 - Defect fixes per Criticality per Team

6 M5

Node Weight

M1 M2 M3 M4 M5 M6 A1 A2

M1 = 8 2 6 4 5 2

M2 = 5 2 1 3

M3 = 4 1 3

M4 = 3 6 3 3 2

M5 = 2 2

M6 = 6 4 2

39

Graph Creation

3

M1

M3

M2

6

8

4

M4

5

A2

A1 2

51

1

3

M6M5 26

4

31

22

2

40

Calculations and Metrics PrioritizationNode Weight M1 M2 M3 M4 M5 M6 A1 A2 Calculated

PriorityM1 = 8 2 6 4 5 2 152

M2 = 5 2 1 3 30

M3 = 4 1 3 16

M4 = 3 6 3 3 2 42

M5 = 2 2 4

M6 = 6 4 2 36

Initial Priority

M1

M6

M2

M3

M4

M5

Calculated Priority

M1

M3

M6

M2

M4

M5

41

Key Metric Changes & Improvement PlansMetrics by Calculated Priority

M1 - Customer Escalations per defect severity M3 – Rejected Defects

M6 - Defect fixes per Criticality per TeamM2 – Opened Valid Defects per Area M4 - Test cases Coverage

M5 - Automation Coverage

Group Defect by Severity and per Customer investment to understand real picture. 1000 Minor issues can cost more than 1 High severity issue.

Proceed Trainings to low defect rejection, so developers will not spend more time on analysis of invalid issues

Make sure Defect fixes are going in Parallel with new feature development for each sprint

Continuously update Test case after each new issue, to make sure you have good coverage

Automate as much as possible to cut the costs and increase the coverage

42

Monitoring of Trend Based Priority MetricsBased on Process Changes

Jan Feb March April0

10

20

30

40

50

60

70

80

90

60

70 6875

2025

2934

70

80 8278

M1 M3 M6

43

Let the Challenge Begin… & Have FUN

Thank You

Global Footprint

About UsA leading provider of next-gen mobile application lifecycle services ranging from design and development to testing and sustenance.

LocationsCorporate HQ: Silicon ValleyOffices: Conshohocken (PA), Ahmedabad (India), Pune (India), London (UK)

InfoStretch Corporation

References Narsingh Deo, Graph Theory with Applications to Engineering and Computer Science, Prentice Hall

1974. A.A. Shariff K, M.A. Hussain, and S. Kumar, Leveraging un- structured data into intelligent

information – analysis and evaluation, Int. Conf. Information and Network Technology, IPCSIT, vol. 4, IACSIT press, Singapore, pp. 153-157, 2011.

http://en.wikipedia.org/wiki/List_of_software_bugs http://www.starshipmodeler.com/real/vh_ari52.htm http://news.nationalgeographic.com/news/2011/11/pictures/111123-mars-nasa-rover-curiosity-russ

ia-phobos-lost-curse-space-pictures/ http

://www.bloomberg.com/news/articles/2013-08-22/nasdaq-shuts-trading-for-three-hours-in-latest-computer-error

©2015 InfoStretch Corporation. All rights reserved. 46

Q & ALiana Gevorgyan

Sr. QA ManagerInfoStretch Corporation Inc.

liana.gevorgyan@infostretch.comwww.linkedin.com/in/lianag/en

Info@infostretch.comwww.infostretch.com