Automated Detection of Performance Regressions Using Statistical Process Control Techniques

1

Automated Detection of Performance Regressions Using Statistical Process Control

Techniques

Thanh Nguyen, Bram Adams, ZhenMing Jiang, Ahmed E. HassanQueen’s University, Kingston, Canada

Mohamed Nasser, Parminder FloraResearch in Motion, Waterloo, Canada

2

Performance Regression

3

What is a performance regression?

Version 1 Version 1.1

Baseline Target

4

How to detect performance regression?

Applying load

Version 1.1

Version 1 CPU %, Memory usage

CPU %, Memory usage

Detect regression

Challenge in Performance Regression Testing

5

Layer 1 – Agent 1

Layer 1 – Agent 2

Layer 2 – Agent 1

Layer 2 – Agent 2

Layer 2 – Agent 3

Layer 2 – Agent 4

Layer 3 – Agent 1

Layer 4 – Agent 1

56 counters x 8 agents = 448 counters

56 counters x 2 agents = 112 counters

Layer 1 Layer 2

Lots of data

6

Data mining

7

Data mining -> Reduce and Relate

Reduce Relate

8

Proposed approach to use control charts to find performance regression

BaselinePerformance counters

TargetPerformance counters

Determine the LCL, CL, UCL

0 5 10 15 20 25710

720

730

740

750

760

770

780

Performance counter

9

Using control charts to verify load test results

BaselinePerformance counters

TargetPerformance counters

Determine the LCL, CL, UCL

0 5 10 15 20 25710

720

730

740

750

760

770

780

Performance counter

Violation ratio

Reduce

10

BaselinePerformance counters

TargetPerformance counters

TargetPerformance counters

0 5 10 15 20 25710720730740750760770780

Performance counter

BaselinePerformance counters

0 5 10 15 20 25680

700

720

740

760

780

Performance counter

Low violation ratio

High violation ratio

We can use violation ratio to detect regression Relate

11

Is there performance regression?

12

Obstacles #1: Inputs are unstable

1 2 3 4 5 60

5

10

15

20

25

30

35

40

45

Version 1.0Version 1.1

Time

CPU

%

Is there a performance regression?

13

It is very difficult to maintain stable input across test runs

Applying load

Version 1.1

Version 1 CPU %, Memory usage

CPU %, Memory usage

Detect regression

Randomization CacheWarm up

Background tasks

14

Solution #1: Scale the counter according to the input

• Step 1: Determine α and β

• Step 2:CPU% Request/s

15

Solution #1: Example of the effectiveness of scaling

16

Obstacles #2: Multiple inputs

10 20 30 40 50 60 70 80 90 1000

5

10

15

20

25

30

35

Density plot of two test runs

CPU Usage

Dens

ity %

IF … THEN…ELSE…

17

10 20 30 40 50 60 70 80 90 1000

5

10

15

20

25

30

35

Density plot of two test runs

CPU Usage

Dens

ity %

Solution #2: Isolating the counters

Local minima

18

Scale and filter

Applying load

Version 1.1

Version 1 CPU %, Memory usage

CPU %, Memory usage

Detect regression

Scale

Scale

Filter

Filter

19

Case study 1

20

Experiment set up

BaselinePerformance counters

TargetPerformance counters

TargetPerformance counters

Average violation ratio should be low

21

BaselinePerformance counters

TargetPerformance counters

TargetPerformance counters

Average violation ratio should be high

Experiment set up

22

Bad Query

Limit

Extra p

rint

Extra c

onnection

Missing k

ey index

Missing t

ext index

0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%

100.00%

NormalProblem

Series3

Average violation ratio

23

Case study 2

24

Experiment set up

V.S.

Precision is highRecall should be high

25

0

10

20

30

40

50

60

70

80

90

100

PrecisionRecallF

Threshold

%

26