Embed Size (px)
Citation preview
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 107/13/2018
Towards Better Understanding of Black-box Auto-Tuning: A Comparative
Analysis for Storage Systems
2018 USENIX Annual Technical Conference
Zhen Cao1, Vasily Tarasov2, Sachin Tiwari1, and Erez Zadok1
1Stony Brook University; 2IBM Research – Almaden;
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 207/13/2018
Outlinel Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 307/13/2018
Motivationl Why tuning storage systems?
uSlow storage impacts I/O bound workloadsuDefault settings are sub-optimaluTuning can provide significant gains
§ 9× [FAST’10]
l Manual tuning is intractablel Auto-tuning storage systems
uBlack-box optimization is promisinguLack of comparison of techniquesuLack of understanding
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 407/13/2018
Contributionsl First comparative study on auto-tuning storage
systemsu5 techniques
l Various aspectsuCumulative & instantaneous throughputu Impacts of hyper-parameters
l Explanations on evaluation resultsuFrom storage perspective
l Future Goal: complete solution to tune storage systems
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 507/13/2018
Outlinel Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 607/13/2018
Conceptsl Storage system
u File system, underlying storage hardware and any layers between them
l Parametersu Configurable optionsu E.g., file-system block size
l Parameter valuesu E.g., 1K, 2K, 4K (Ext4 block size)
l Configurationu Combination of parameter values u E.g., [Ext4, 4K, data=ordered]
l Parameter spaceu All possible configurations
l Hyper-parameter
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 707/13/2018
Challengesl Vast parameter space
uExt4: 59 parameters, 1037 configsuDevices, LayersuDistributed, large-scale
l Discrete and non-numericuLinux I/O scheduler: noop, cfq, deadline
l Non-linearity
l Sensitivity to environmentuHardware & workloads
Manual TuningInefficient
GradientUnavailable
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 807/13/2018
Inapplicable Methodsl Control Theory
uUnstable in controlling non-linear systems
l Supervised Machine LearninguLong training phase
uHigh-quality training data
l Inapplicable or inefficient to serve as the core auto-tuning algorithmuCould be helpful as a supplement
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 907/13/2018
Black-box Optimizationl Successfully applied in auto-tuning system
configurations
l ExamplesuGenetic Algorithms (GA)
uSimulated Annealing (SA)
uBayesian Optimization (BO)
l Obliviousness to system’s internals
Configuration EvaluationResults
evaluate
select
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1007/13/2018
Key Factors
l Fitness: optimization objective(s)
uThroughput, latency, energy, …
uComplex cost functions
l ExplorationuSearch the unvisited area (e.g., randomly)
l ExploitationuUtilize neighborhood or history
l HistoryuHow much past data is kept and used for
exploration/exploitation
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1107/13/2018
Applied Methodsl Simulated Annealing (SA)
l Genetic Algorithms (GA)
l Deep Q-Network (DQN)
l Bayesian Optimization (BO)
l Random Search (RS)uRandom selection without replacement
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1207/13/2018
Genetic Algorithmsl Inspired by natural evolutionl Concepts
uGene: file system, block size, …uAllele: Ext4, XFS, Btrfs, …uChromosome: configurationuPopulation: set of configurations
l Selectionl Genetic operators
uCrossoveruMutation
History
Exploitation vs. Exploration
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1307/13/2018
Outlinel Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1407/13/2018
Experimental Setupl Hardware
uM1: 2 Intel Xeon single-core 2.8GHz CPU, 2G RAM, 73GB Seagate SCSI drive
uM2: 1 Intel Xeon quad-core 2.4GHz CPU, 24G RAM, 4 drives (SAS-HDD 500GB, SAS-HDD 146GB, 1 SATA-HDD, SSD)
l FilebenchuMacro-workloads: fileserver, mailserver, webserver,
dbserveruDefault working set size
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1507/13/2018
Experiment Setup (cont.)l Search spaces
uStorage V1§ File system, inode size, block size, block group,
journal options, mount options, special optionsuStorage V2
§ V1 + I/O scheduler§ 6,222 configurations
l MethodologyuExhaustive Search
§ Storage V2: 4 workloads × 4 devices§ 3+ runs for each configuration§ Collected over 2+ years
uSimulate auto-tuning algorithms
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1607/13/2018
Outline
l Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1707/13/2018
16
17
18
0 1 2 3 4 5
Bes
tT
hro
ugh
pu
t (k
op
s/s)
Time (hrs)
15.2
18.7
M2-Mailserver-HDD3
GASABO
DQNRS
Best Throughput
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1807/13/2018
20%
40%
60%
80%
100%
0 1 2 3 4 5
Per
centa
ge
of
Runs
Time (hrs)
M2-Fileserver-HDD3GASABO
DQNRS
Success rate for finding near-optimal configurations
Near-optimal configuration: one with throughput higher than 99% of the global optimal value.
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 1907/13/2018
Instant Throughput
51015
RS
M2-Mailserver-HDD3
51015
SA
M2-Mailserver-HDD3
51015
Thro
ughput
(kops/
s)
GA
M2-Mailserver-HDD3
51015
DQN
M2-Mailserver-HDD351015
0 1 2 3 4 5
BO
Time (hrs)
M2-Mailserver-HDD3
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2007/13/2018
Genetic Algorithm (GA) Diversity
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2107/13/2018
l Correlation analysisuOrdinary Least Squares (OLS)uExample: block size and journal option are the
most correlated Ext4 parameter (Fileserver, SSD)
l Explanations on evaluation resultsuGA and BO can identify important parameters
through “history”uSA keeps no “history”; thus performs poorlyuDQN spends too much time on exploration uRandom Search
§ Near-optimal configurations take up 4.5% of the whole search space (M2, Mailserver, HDD).
Correlation Analysis
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2207/13/2018
Outline
l Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2307/13/2018
Related Workl Auto-tuning storage
uStorage system design (bin-packing heuristics) [Alvarez et al.]
uData recovery scheduling (GA) [Keeton et al.]uHDF5 optimization (GA) [Behzad et al.]uLustre optimization (DQN) [Li et al.]
l Auto-tuning other systemsuDatabase [Alipourfard et al.]uCloud VMs [Aken et al.]
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2407/13/2018
Outline
l Introductionl Backgroundl Experiment Settingsl Evaluationsl Related Workl Conclusions & Future Work
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2507/13/2018
Conclusions & Contributionsl First comparative analysis on 5 techniques on
auto-tuning storage systemsuEfficiency on finding near-optimal configurationsu Instant throughput
l Provide insights from storage perspectiveu Importance of parameters
§ E.g., impact of mutation rates on convergence
l Valuable datasetsuWill release to public
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2607/13/2018
Future Work
l More complex workloads and search
spaces
l Hyper-parameter tuning
l More sophisticated auto-tuning
u E.g., penalty functions to cope with costly parameter
changes
Towards Better Understanding of Black-box Auto-Tuning (ATC’18) 2707/13/2018
Towards Better Understanding of Black-box Auto-Tuning: A Comparative
Analysis for Storage SystemsZhen Cao, Vasily Tarasov, Sachin Tiwari, and Erez Zadok
