Performance Regression Analysis

Cor-Paul Bezemer

Accomplishments and Challenges

A burnt child dreads fire

Regression testing makes sure we do not make the same mistake twice

Write code

Execute regressiontest

Deploy or fix changes

Regression testing also helps us to make sure we are not re-introducing more bugs

Performance regression testing helps ensure we are not breaking performance

Write code

Execute performance test

Deploy if performance is OK or fix changes

Performance regression testing can/should be done on several metrics

Performance regression testing can/should be done on several metrics

Performance regression testing can/should be done on several metrics

Performance regression testing can/should be done on several metrics

Directions in research on performance regression analysis

Directions in performance regression analysis research

1. Execution profile comparison

2. Source code analysis

3. Visualization

Execution profile comparison

Execution profile comparison

Disadvantages of execution profile comparison

- Actually needs performance tests

- Need to run performance tests several times because of variation

- Small changes may be difficult to interpret

Source code analysis

Identify risky commits and recommend those for testing (Huang et al. @ ICSE 2014)

Disadvantages of source code analysis

- Can not replace 'real' performance measurements (yet)

Visualization of performance regression analysis results

Control charts
(Nguyen et al. @ MSR 2014)

Differential flame graphs
(Bezemer et al. @ SANER 2015)

Deciding on the baseline performance

Deciding on the acceptable performance deviation

Reporting results to the developer

Availability of performance tests

Execution time of tests

Performance regression testing helps ensure we are not breaking performance

Write code

Execute performance test

Deploy if performance is OK or fix changes

Directions in performance regression analysis research

1. Execution profile comparison

2. Source code analysis

3. Visualization

Availability of performance tests

Execution time of tests