Performance-Influence Models: Prediction, Optimization, Debugging · Binary options Numeric options e. Performance-Influence Models for Highly Configurable Systems Vision: Performance-Influence

Performance-Influence Models for Highly Configurable Systems

Performance-Influence Models: Prediction, Optimization, Debugging

Norbert Siegmund

Sven Apel

Alexander Grebhahn

Christian Kästner


Even Domain Experts Struggle with Performance!

2

HPC Experts Meeting at Dagstuhl


Their Problem: Finding the Optimal Configuration for a Given Hardware Platform?

3

cgs

preS

preS

preS postS

postS

postS

restriction à

prol

onga

tion à

Stencil code


What is the Influence of Configuration Options on Performance?

4

Binary options Numeric options

Per

form

ance


Vision: Performance-Influence Models

Determine the influence of configuration options and their interactions

– Understanding

– Debugging

– Prediction and optimization 5

14.3 + 113.3 - 4.5 - 73.9 + 171.7 = 219.9

and use it for:


Learning Procedure

6

Sampling Learning Performance-Influence Model


Sampling Binary and Numeric Options

Binary Options Numeric Options

Structured sampling approaches for the different kinds of options

7

Exponential number!


Heuristics for Binary-Option Sampling

• Random?

– Unlikely to select a valid configuration

– Locally clustered solutions using SAT

• Heuristics

– Option-Wise (OW): { }, { }, { }, { }, {}

– Negative Option-Wise (nOW):

{ }, { }, { }, { }, { }

– Pair-Wise (PW) : { }, { }, { }, { }, { },

{ } 8

Siegmund et al., ICSE‘12


Numeric-Option Sampling (Experimental Designs)

• Fractional factorial designs

• Optimal designs

• Pre-study:

Plackett-Burman Design as best design

9

cgs

preS

preS

preS postS

postS

postS

restriction à

prol

onga

tion à

Multi-grid solver as subject systems


Plackett-Burman Design (PBD)

• Minimizes the variance of the estimates of the independent variables (numeric options)

• …while using a limited number of measurements

• Design specifies seeds depending on the number of experiments to be conducted (i.e., configurations to be measured)

10

Numeric options

Co

nfi

gura

tio

ns

Min Center Max

Value range of a numeric option


Learning Procedure

11



0

800

0

1600

640

0

1600

20

50

100

50

20

100

100

0

16

32

32

32

0

16

400

2500

10000

2500

400

10000

10000

0

0

0

1

1

0

1

0

0

0

0

1

0

1

0

1

1

0

1

0

1

0

1

0

1

1

0

1

Regression Learning

12

1

0

1

1

1

0

1

N/A

1.2

0.9

1.5

1.5

N/A

1.2

2 Log( ) Perf.

833

411

290

799

753

514

416

=

β1 β2 β3 β4 β5 β6 β7 β8 β9 β10

Individual Options Interactions Functions

102.4 84.3 54.1 5.4 1.3

132.3 81.3 56.6 3.5 1.9 -14.1 -5.4 2.4

130.3 83.5 54.2 0.01 0 -14.1 -5.4 1.4 2.1 8.8

Error:

41%

25%

7.4%

Co

nfi

gura

tio

ns

* * * * * * * * * *

Exponential number!

Unlimited candidates!

Performance-Influence Models for Highly Configurable Systems 13

Multiple Regression with Feature Subset Selection

Compression Encryption CacheSize PageSize

Candidates: Models: Errors: Winner:

1 β0+ * β1 50%

125%

72%

35%

2 2

log( )

29%

34%

β0 + * β1 2

2

…

* β2 5%

…

12%

18%

27%

… …

2

log( )

β0 + * β1 + 2

* β2 β0 + * β1 + 2

* β2 β0 + * β1 + 2

* β2 β0 + * β1 + 2

* β2 β0 + * β1 +log( ) 2

β0+ * β1

β0+ * β1

β0+ * β1

β0+ * β1

β0+ log( ) * β1

β0 + * β1 + * β2

Extend model in a stepwise manner

Probe different candidates

• Individual influences, interactions, functions

Siegmund et al., FSE‘15


Learning Procedure

14



Experimental Evaluation

15

Synthetic Prerequisite 1. Exp

Accuracy

Understanding

2. Exp

3. Exp à use cases 4. Exp à domain knowledge

Real world


1. Experiment: Finding the Actual Influences

RQ: Do we find the actually existing influences and interactions?

• Design:

16

Real systems Performance model containing only binary options

Performance model with synthetic

numeric options

Sampling Learning Re-learned model

Ground truth


1. Experiment: Finding the Actual Influences

RQ: Do we find the actually existing influences and interactions?

• Design:

17

Real systems Performance model containing only binary options

Performance model with synthetic

numeric options

Sampling Learning Re-learned model

Ground truth We identified the most relevant influences Average prediction accuracy: 98.5 % !


2. Experiment: Performance Prediction

18

RQ: Which combination of sampling approaches achieve the highest prediction accuracy?

System Domain # Binary Opt. # Numeric Opt. # Constraints # Configs

Dune MGS Multi-Grid Solver 8 3 20 2 304

HIPAcc Image Processing 31 2 416 13 485

HSMGP Multi-Grid Solver 11 3 45 3 456

JavaGC Runtime Env. 12 23 4 1031

SaC Compiler 53 7 10 1023

x264 Video Encoder 8 13 0 1027

Sampling Learning Perf.-influence model

Random/whole population

= Predicting


2. Experiment: Performance Prediction

19

RQ: Which combination of sampling approaches achieve the highest prediction accuracy?

System Domain # Binary Opt. # Numeric Opt. # Constraints # Configs

Dune MGS Multi-Grid Solver 8 3 20 2 304

HIPAcc Image Processing 31 2 416 13 485

HSMGP Multi-Grid Solver 11 3 45 3 456

JavaGC Runtime Env. 12 23 4 1031

SaC Compiler 53 7 10 1023

x264 Video Encoder 8 13 0 1027

Sampling Learning Perf.-influence model

Random/whole population

= Predicting

Average prediction accuracy (PW+PBD): 86.8 % ! Experimental designs are better than random sampling


3. Experiment: Accuracy vs. Complexity

• RQ: Is it necessary to learn accurate but complex models?

20

Learning whole population

Domain Experts

It depends on the use case. Simple models help to identify the most relevant influences.

Collaboration with:

Sergiy

Kolesnikov


4. Experiment: Validation of Domain Knowledge

• RQ: Are we able to validate domain knowledge using a model?

21

Sampling Learning

Domain Expert Perf.-influence model =

Theoretical Knowledge

Validate domain knowledge

Collaboration with:

Carmen

Rodrigo Francisco

Gaspar


Findings and Future Work

22

https://github.com/nsiegmun/SPLConqueror http://fosd.de/SPLConqueror/

Thank You!

Documents

Performance-Influence Models: Prediction, Optimization, Debugging · Binary options Numeric options e. Performance-Influence Models for Highly Configurable Systems Vision: Performance-Influence