Upload
others
View
10
Download
0
Embed Size (px)
Citation preview
Finding Near-Optimal Configurations in Product Lines by Random Sampling
Jeho Oh
1
Don Batory Margaret Myers Norbert Siegmund
Quickly find SPL configurations with
near-optimal performance for a given workload
• Configuration space is often huge: 𝑛 features ≤ 2𝑛 configurations
( 273 optional features: 1082 products, one for every atom in universe )
• Searching for the optimal configuration is daunting,
as benchmarking all configurations is infeasible
• Find a way to get good enough configurations with practical effort
2
I want a hybrid car with laser headlight, but cheap and light as possible.
Now 273/275 options left to decide…
Big Picture
3
randomly sample on configuration space for
near-optimal configurations
featuremodel
user-imposedfeature constraints
near-optimalperforming
configuration
learnperformance
model
useoptimizer
sampleconfigurations
Performance Model Approach
Our Approach
Contributions
• Allow true random sampling of configurations
• Provide statistical bounds on searching by sampling
• Directly search the space for any given workload
4
Search by Random Samplingwhich we describe next…
• To randomly sample configurations
from uniform distribution:
– Identify valid configuration space
– Select a random number in [ 1, total # of configs ]
– Return the configuration with matching number
• Binary Decision Diagram (BDD):
– Compile prop. formula into graph structure
– Derive all possible solutions (configs)
– Allows efficient sampling from traversing BDD
Random Sampling with BDD
5
(𝑟𝑜𝑜𝑡 ↔ 𝑡𝑟𝑢𝑒)∧ 𝑟𝑜𝑜𝑡 ↔ 𝐴∧ (𝐵 → 𝑟𝑜𝑜𝑡)
∧ 𝐶 ↔ 𝐴 ∧ ¬𝐷
∧ 𝐷 ↔ 𝐴 ∧ ¬𝐶
∧ (𝐷 → 𝐵)
𝑟𝑜𝑜𝑡
𝐴 𝐵
𝐶 𝐷
𝐷 implies 𝐵
Feature Model Prop. Formula
BDD
𝑟𝑜𝑜𝑡
𝐴
0 1
𝐵
𝐶
𝐷 𝐷
0 1
2
2
0 1
1
0
03
0 3
Randomly sample from space of all valid
configurations, not space of all features
• 𝑛 random numbers over unit range [0,1], 𝑥 as the number closest to 1
Analyze the distance between 𝑥 and 1, (1 − 𝑥)
• C.D.F. of 𝑥 over 𝑛: 𝑝𝑛 𝑋 ≤ 𝑥 = 0𝑥𝑛 ⋅ 𝑥𝑛−1 ⋅ 𝑑𝑥 = 𝑥𝑛
• Average distance from 𝑥 to 1: 𝐸𝑛 = 011 − 𝑥 ⋅ 𝑛 ⋅ 𝑥𝑛−1 ⋅ 𝑑𝑥 =
𝟏
𝒏+𝟏
(Equations for sampling over discrete space on Section 3.3 of the paper)
Statistics of Random Sampling (1)
6
0 10 1
𝑥 = 0.5
𝐸1 = 0.5
0 1
𝑥 = 0.67
𝐸2 = 0.33
0 1
𝑥 = 0.75
𝐸3 = 0.25
0 1
𝐸4 = 0.20
𝑥 = 0.80
Statistical regularity: 𝐸𝑛 =1
𝑛+1with bounds 𝜎𝑛 ≈
1
𝑛+1
• Correspondences:
– Selection of numbers in [ 0,1 ] ► Selection in [ 1, total # of configs ]
– Closeness to 1 ► Closeness to the best configuration, Ω
Statistics of Random Sampling (2)
7
On average, sample with the best performance has
top 𝟏𝟎𝟎
𝒏+𝟏% performance among all configurations
200
210
220
230
240
250
260
270
0 300 600 900
LLVM (Compiler infrastructure, 11 features, 1024 configs)
𝛀𝟎
𝐸1 =Ω − 𝑥
1024= 0.50𝐸3 =
Ω − 𝑥
1024= 0.25𝐸7 =
Ω − 𝑥
1024= 0.125
Per
form
an
ce
Sorted configurations
Sorted config space
Best performing config
Monotonic graph:
Closer to Ω (x-axis),
better perf. (y-axis)
Statistical Recursive Searching (SRS)
• Can search better than 99 samples for 1% by random sampling
• Feature selection:
– Makes some configs perform better
– Constrict config space
• Search within smaller and better
config space by feature selection
• Find most influential features by:
– Performance difference
– Welch’s t-Test
8
Use samples to recursively constrict the search space
200
210
220
230
240
250
260
270
0 200 400 600 800 1000
LLVM
Per
form
an
ce
Sorted configurations
Statistical Recursive Searching (SRS)
• Can search better than 99 samples for 1% by random sampling
• Feature selection:
– Makes some configs perform better
– Constrict config space
• Search within smaller and better
config space by feature selection
• Find most influential features by:
– Performance difference
– Welch’s t-Test
9
Use samples to recursively constrict the search space
200
210
220
230
240
250
260
270
0 200 400 600 800 1000
LLVM
Per
form
an
ce
Sorted configurations
Statistical Recursive Searching (SRS)
• Can search better than 99 samples for 1% by random sampling
• Feature selection:
– Makes some configs perform better
– Constrict config space
• Search within smaller and better
config space by feature selection
• Find most influential features by:
– Performance difference
– Welch’s t-Test
10
Use samples to recursively constrict the search space
200
210
220
230
240
250
260
270
0 200 400 600 800 1000
LLVM
Per
form
an
ce
Sorted configurations
Statistical Recursive Searching (SRS)
• Can search better than 99 samples for 1% by random sampling
• Feature selection:
– Makes some configs perform better
– Constrict config space
• Search within smaller and better
config space by feature selection
• Find most influential features by:
– Performance difference
– Welch’s t-Test
11
Use samples to recursively constrict the search space
200
210
220
230
240
250
260
270
0 200 400 600 800 1000
LLVM
Per
form
an
ce
Sorted configurations
Statistical Recursive Searching (SRS)
• Can search better than 99 samples for 1% by random sampling
• Feature selection:
– Makes some configs perform better
– Constrict config space
• Search within smaller and better
config space by feature selection
• Find most influential features by:
– Performance difference
– Welch’s t-Test
12
Use samples to recursively constrict the search space
200
210
220
230
240
250
260
270
0 200 400 600 800 1000
LLVM
Per
form
an
ce
Sorted configurations
4 features excluded:
- 1/16 of entire space,
- Better overall performance
EVALUATION
13
ICSE 2012
Evaluation Method
• Use ground truth data from Siegmund et al. (http://fosd.de/SPLConqueror)
• Measured accuracy of search:
– 𝜹𝒙 : % of configurations better than the best config found so far
– 𝜹𝒚 : % performance difference to Ω
• Averaged from 100 searches per different conditions
• Full result is available in Section 5 of the paper
14
SPL Type # features # configs Performance
LLVM Compiler infrastructure 11 1024 Test suite compilation time
BerkeleyDBC Database system 18 2560 Benchmark response time
X264 Video encoder 16 1152 Video encoding time
Apache Web server 9 192 Maximum server load
𝜹𝒙: Theory vs. Actual• 𝛿𝑥 from randomly sampling different # of samples
vs. theoretical value derived using same # of samples
15
Theory matches observations
0%
1%
2%
3%
4%
5%
6%
7%
8%
9%
10%
10 20 30 40 50 60 70 80 90 100
LLVM
X264
BerkeleyDBC
Apache
Theoretical
Apache_Theoretical
Dis
tan
ce t
o 𝛀
(𝜹𝒙)
# of random samples (𝒏)
𝑬𝒏 (Eq. (9))
𝑬𝟏𝟗𝟐,𝒏 (Eq. (7), Apache)
𝜹𝒙 : SRS vs. Random Sampling (non-recursive)
• 𝛿𝑥 of SRS over different # of samples per recursion
vs. theoretical 𝛿𝑥 of random sampling with same total # of samples (𝑁)
16
0%
2%
4%
6%
8%
10%
12%
5 15 25 35
LLVM
0%
2%
4%
6%
8%
10%
12%
14%
16%
5 15 25 35
BerkeleyDBC
0%
2%
4%
6%
8%
10%
12%
14%
5 15 25 35
X264
𝛿𝑥 (Measured)
Dis
tan
ce t
o 𝛀
(𝜹𝒙)
# of samples per recursion (𝒏)
Dis
tan
ce t
o 𝛀
(𝜹𝒙)
# of samples per recursion (𝒏)
Dis
tan
ce t
o 𝛀
(𝜹𝒙)
# of samples per recursion (𝒏)
SRS is more efficient than random sampling alone
𝐸𝑁 (Theoretical)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
0 20 40 60 80 100 120 140 160 180
BerkeleyDBC
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
0 20 40 60 80 100
LLVM
𝜹𝒚: SRS vs. Performance Models (1)
• 𝛿𝑦 over total # of samples in SRS vs.
𝛿𝑦 of perf. models assuming an ideal optimizer (always finds best predicted config)
17
SRSSarkar2015 Siegmund2012
SRS needs many fewer samples for same accuracy, and
yields much better accuracy for same number of samples
Per
form
an
ce d
iffe
ren
ce t
o 𝛀
(𝜹𝒚)
Total # of samples (𝑵)P
erfo
rman
ce d
iffe
ren
ce t
o 𝛀
(𝜹𝒚)
Total # of samples (𝑵)
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
0 20 40 60 80 100
160%
162%
164%
X264
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
0 20 40 60 80 100 120 140 160 180
BerkeleyDBC
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
0 20 40 60 80 100
LLVM
𝜹𝒚: SRS vs. Performance Models (2)
• 𝛿𝑦 over total # of samples in SRS vs.
𝛿𝑦 of performance models assuming an ideal optimizer
18
SRSSarkar2015 Siegmund2012
In SRS, more samples yields better accuracy
Per
form
an
ce d
iffe
ren
ce t
o 𝛀
(𝜹𝒚)
Total # of samples (𝑵)
Per
form
an
ce d
iffe
ren
ce t
o 𝛀
(𝜹𝒚)
Total # of samples (𝑵)
Per
form
an
ce d
iffe
ren
ce t
o 𝛀
(𝜹𝒚)
Total # of samples (𝑵)
𝜹𝒙: Scalability of Searching• Combine SPLs to simulate
larger configuration spaces
• Measure 𝛿𝑥 for SRS and
non-recursive searching
19
𝐸𝑁
0%
2%
4%
6%
8%
10%
12%
5 15 25 35
Apache×X264×LLVM×BerkeleyDBC
Dis
tan
ce t
o 𝛀
(𝜹𝒙)
# of samples per recursion (𝒏)
𝛿𝑥
Accuracy is independent of the size of the configuration space
Combined Systems # of Features # of Confgs.
Apache × LLVM × BerkeleyDBC 38 503,316,480
Apache × X264 × BerkeleyDBC 51 566,231,040
LLVM × Apache × X264 × BerkeleyDBC 62 579,820,584,960
# of random samples
Dis
tan
ce t
o 𝛀
(𝜹𝒙)
0.0%
0.5%
1.0%
1.5%
2.0%
2.5%
3.0%
3.5%
30 50 70 90 110
Theory vs. Actual
ApacheH264
BerkeleyDBC
ApacheLLVM
BerkeleyDBC
E_n (Eqn. (9))𝑬𝒏 (Eqn. (9))
0%
2%
4%
6%
8%
10%
12%
5 15 25 35
Apache × LLVM × BerkeleyDBC
0%
2%
4%
6%
8%
10%
12%
5 15 25 35
Apache × X264 × BerkeleyDBC
Dis
tan
ce t
o 𝛀
(𝜹𝒙)
# of samples per recursion (𝒏)
Dis
tan
ce t
o 𝛀
(𝜹𝒙)
# of samples per recursion (𝒏)
Conclusions and Contributions
1. True random sampling of configuration spaces
2. Guaranteed tight statistical bounds on finding good configurations
3. Can recursively search through configuration space for more efficient searching
4. Scalable search method - accuracy independent of the configuration space size
20
Thank You !Finding Near-Optimal Configurations in Product Lines by Random Sampling
Jeho Oh, Don Batory, Margaret Myers, Norbert Siegmund
21
Supplemental Slides from Now On
22
Statistics of Random Sampling
• 𝑛 random numbers over unit range [0,1], 𝑥 is the number closest to 1Analyze the distance between 𝑥 and 1, (1 − 𝑥)
• C.D.F. of 𝑥 over 𝑛:
𝑝𝑛 𝑋 ≤ 𝑥 = 0𝑥𝑛 ⋅ 𝑥𝑛−1 ⋅ 𝑑𝑥 = 𝑥𝑛
• Average distance from 𝑥 to 1:
𝐸𝑛 = 011 − 𝑥 ⋅ 𝑛 ⋅ 𝑥𝑛−1 ⋅ 𝑑𝑥 =
𝟏
𝒏+𝟏
• Standard deviation:ത𝐸𝑛 = 0
11 − 𝑥 2 ⋅ 𝑛 ⋅ 𝑥𝑛−1 ⋅ 𝑑𝑥 =
1
(𝑛+1)(𝑛+2)
𝜎𝑛 = ത𝐸𝑛 − 𝐸𝑛2 =
𝟏
(𝒏+𝟏)(𝒏+𝟐)−
𝟏
𝒏+𝟏 𝟐
23
0%
2%
4%
6%
8%
10%
10 20 30 40 50 60 70 80 90 100
En
sn
𝐸𝑛
𝜎𝑛
Dis
tan
ce t
o 𝟏
# of random numbers
PCS Graphs of Real Systems
• All configuration measurements sorted by performance (descending order)
24
0.8K
1.3K
1.8K
2.3K
0 50 100 150
Apache (9, 192)
0
10
20
30
40
50
0 500 1000 1500 2000 2500
BerkeleyDBC (18, 2560)
200
400
600
800
0 200 400 600 800 1000
X264 (16, 1152)
200
220
240
260
0 200 400 600 800 1000
LLVM (11, 1024)
Sorted configurations
Per
form
ance
Sorted configurations
Per
form
ance
Sorted configurations
Per
form
ance
Sorted configurations
Per
form
ance
* System Name (# of features, # of configs)
Statistical Recursive Searching(SRS)
25
Use samples to recursively reduce the config space to focus
0
5
10
15
20
25
30
35
40
45
0 500 1000 1500 2000 2500
BerkeleyDBC
Configurations sorted by performance
Per
form
ance
PS32K, HAVE_HASHPS32K, ¬HAVE_HASH
PS16K, HAVE_HASHPS16K, ¬HAVE_HASH
PS8K, ¬ HAVE_HASHPS8K, HAVE_HASH
PS4K, ¬ HAVE_HASHPS4K, HAVE_HASH
PS1K, ¬ HAVE_HASHPS1K, HAVE_HASH
¬ HAVE_CRYPTOHAVE_CRYPTO
1. Random sample from config space
2. From best 2 configs, get common decisions 𝑑
3. For each common decision 𝑑, measure:
𝛿𝑑 = Avg. perf. of samples with 𝑑
𝛿¬𝑑 = Avg. perf. of samples without 𝑑
Δ𝑑 = 𝛿𝑑 − 𝛿¬𝑑
4. If a Δ𝑑 indicates performance improvement,
perform Welch’s T-test to evaluate its certainty
5. Use features certain to improve performance
to constrain the configuration space to recurse
Finding Features to Recurse
26
Decisions to recurse: 𝑓1
𝜙 = 𝐹𝑀
𝑐2 𝑐1Ω0
𝑐2 = {𝑓1, 𝑓2, ¬𝑓3, 𝑓4, ¬𝑓5, 𝑓7, ¬𝑓8}
𝑐1 = {𝑓1, 𝑓2, 𝑓3, ¬𝑓4, 𝑓5, 𝑓7, ¬𝑓8}
𝑑 = {𝑓1, 𝑓2, 𝑓7, ¬𝑓8}
Common Decisions 𝑓1 𝑓2 𝑓7 ¬𝑓8
Δ𝑑 improves perf. Yes No No Yes
𝜙 = 𝐹𝑀 ∧ 𝑓1Ω0
Welch’s T-test Pass - - Fail