Upload
yuto-hayamizu
View
1.283
Download
1
Embed Size (px)
Citation preview
The Art of Performance Evaluation
Yuto Hayamizu (The University of Tokyo)
2015/06/19 1
PGCon 2015 @ University of Ottawa
Q: What is the role of IT?
2015/06/19 2
The Role of Information Technology
• “Transforming” computing power into business/social values
3
Tabulating Machine (Herman Hollerith, 1980)
Performance• The key criterion of “transformation”
efficiency
4
Tabulating Machine (Herman Hollerith, 1980)
National Population Census in U.S1880: 8 years (by humans)1890: 13 years (by humans, expected) → 18 months
And Today
5Data Is The Source of Values
✓ Big Data✓ Internet of Things✓ Cyber Physical System
Database System• Foundation of “transformation”• The value of an engineer
== Knowledge of how “transformation” works
6Hardware
App App App App
Database System
Performance of Database Systems
• End-to-end study– From physics of hardware to logics of applications
• Learning every component technology is NOT enough– A component technology changes rapidly– An architecture could change drastically
• Acquire the way of learning how “transformation” works
2015/06/19 7
Performance Evaluation• The key process to understand
performance of systems– Identify performance goals which contribute
to user value– Design and execute an evaluation procedure
2015/06/19 8
Common Misconception
2015/06/19 9
The Heart of Performance Evaluation Is Not Individual Skills
2015/06/19 10
Kernel probing and tracingHardware simulation
Statistical analysis
Workload characterizationWorkload generation
Factorial experiments design
Analytical modelingMetrics selection
Application monitoring
Middleware monitoringOS monitoringDB benchmarking
CPU benchmarking
Storage benchmarking
Memory benchmarking
Designing network
Designing storage
Integrating cloud environments
Kernel tuningDB tuning
Network monitoring
But A Philosophy of Orchestrating Individual Skills
2015/06/19 11
Kernel probing and tracing
Hardware simulation
Statistical analysis
Workload characterizationWorkload generation
Factorial experiments design
Analytical modelingMetrics selection
Application monitoring
Middleware monitoring
OS monitoring
DB benchmarkingCPU benchmarking
Storage benchmarking
Memory benchmarking Designing networkDesigning storage
Kernel tuningDB tuning Network monitoring
Example Story• “Find the best price/performance SSD for our
PostgreSQL database”
2015/06/19 12
List allpossible SSDs
Build test systemsfor all SSDs
Run pg_bench onall test systems
“XX tps/$ with SSD Z!”
Smarter Way• “Find the best price/performance SSD for our
PostgreSQL system”
2015/06/19 13Characterize
workload
Model performancewith some metrics
Estimate performancewith datasheets
Validate estimationw/ pg_bench on several SSDs
Narrow downcandidate SSDs
Run pg_bench on candidatesand get the result
Nature of Performance Evaluation
• A goal could change during evaluation– Progress of evaluation deepens insight into
target systems
• A philosophy will guide you
2015/06/19 14
2015/06/19 15
Contrary to common belief, performance evaluation is an art. Like a work of art, successful evaluation cannot be produced mechanically. Every evaluation requires an intimate knowledge of the system being modeled and a careful selection of the methodology, workload, and tools.
— Raj Jain
How can one develop his/her own philosophy?
2015/06/19 16
Learn principles and individual techniques
Put into practice andget experiences
The First Step to Performance Evaluation
• Fundamentals of Performance Evaluation– Principles – Basic Techniques
• Modeling• Measurement• Simulation
... with my practices and experiences
2015/06/19 17
Principles• Define a goal first– Understand the system and identify the
problem to be solved• The most important, and usually the most difficult
part
• Stay goal-oriented– Do not start with a tool or a technique– Evaluation could change a goal itself
• Redesign the process for a new goal
2015/06/19 18
Principles are so simple :)
Basic Techniques
2015/06/19 19
Modeling
MeasurementSimulation
Modeling
2015/06/19 20
Modeling
MeasurementSimulation
Modeling• Building a model: approximation– Understand how it works– Select variables which effect performance– Ignore unnecessary details– Formulate performance
• The core of scientific approach
2015/06/19 21
Example: Throughput of HDD
• Hard disk drive– Platters rotates at constant speed– A cylinder is selected by moving
heads• Important variables– Rotating speed: R [rpm]– Radius of a cylinder: r [m]– Density of a cylinder: D [byte/m]
• Performance model
2015/06/19 22
2πr × D × R / 60 [byte/sec]
Seq. Read Throughput of HDD
2015/06/19 23
Outer platter Inner platter
Example: Latency of HDD• Important variables– Rotating speed: R [rpm]– Avg. rotational latency: Δt = 60/R [sec]– Max seek time: ts [sec]
• Latency model:
2015/06/19 24
ts < (latency) < ts+Δt [sec]
I/O Latency of HDD• a
2015/06/19 25
~ 6ms (rotational latency of 10k)
Queueing Model• Simple queueing model– Avg. arrival rate of customer: λ– Avg. service latency: µ
• Modeling computer system performance– CPU : customer = CPU instructions, server =
execution units in CPU– Storage: customer = I/O requests, server = disks
and controllers– Database system: customer = queries, server =
query executors– ...
2015/06/19 26
Performance Modeling with Queueing Model
• M/M/1 : simple but useful model– 1 server and 1 infinite queue– Customer arrives at rate λ according to
Poission process
2015/06/19 27
Average response timeλ
μ
Waiting time
Response time (sojourn time)
Modeling OLTP Performance with Various CPU Frequencies
•
28
Theoretical response time Measured response time
Modeling Deepens Understanding
• A good model provides good view– Performance values– Scaling trends– Highlight bugs
• A bad model is also informative– Missing key variables– Complexity of systems
2015/06/19 29
Measurement
2015/06/19 30
Modeling
MeasurementSimulation
Measurement• Measurement without modeling is
pointless– Modeling validates measurement– Measurement validates modeling
2015/06/19 31
Model
Guide For Fruitful Measurement
• Selecting workload– Does it characterize your application well?
• Selecting performance metrics– Which metrics define your goal?
• Modeling performance• Selecting environments
– Is it designed for measuring your metrics?• Selecting measurement methods and tools• Conduct measurements• Validating results with models
2015/06/19 32
Goalden rule: Decide methods and tools based on your goal
Measurement != just picking up a benchmark tool and run it
Example story: Storage for OLTP System
• Goal– Build a storage system for OLTP system which
can attain XXXX tps
• Workload– Random I/O from many clients
• Metric– IOPS of storage system (random access)
2015/06/19 33
Example story: Storage for OLTP System
• Modeling performance– What affects your metrics?– IOPS (random access)
• I/O block size: 8KB ~ 16KB• I/O concurrency: 1 ~ 1000• The length of I/O queue in:
– I/O scheduler of OS– HBA driver– Storage controller– Storage device
2015/06/19 34
Example story: Storage for OLTP System
• Measurement environment– Carefully design the flow of resources– Your metrics: IOPS of the storage system
• Ensure whether I/O requests really reach the storage system
2015/06/19 35
Server Storage
Proc
esso
rs
Memo
ry
Stor
age C
ontro
ller
(with
cach
e)
Example story: Storage for OLTP System
• Measurement methods and tools– Select an appropriate tool or make it yourself
• What should be measured?• What can be measured?
– How measured value represent your metric?• Ex) iostat value != IOPS of storage devices
2015/06/19 36
Server Storage
Proc
esso
rs
Memo
ry
Stor
age C
ontro
ller
(with
cach
e)
Example story: Storage for OLTP System
• Conduct measurement– Plan your experiments with minimal steps
• Exhaustive parameter combinations ...?• A good model tells you where should be measured
2015/06/19 37
Next measurement should be ...
Example story: Storage for OLTP System
• Validate measurement results with your model– Results match the model
• Right understanding of system performance
– Results do not match the model• Missing something• Improve your model or
measurement method
2015/06/19 38
Simulation
2015/06/19 39
Modeling
MeasurementSimulation
Simulation• Simulating unavailable systems– Coming hardware devices– Unimplemented software components
• Useful technique for– More detailed behaviour analysis than
modeling
2015/06/19 40
I/O Replay• Performance evaluation with realistic I/O
workload– Trace I/O requests during actual execution– Replay traced I/O patters on hypothetical systems
2015/06/19 41TPC-C benchmark (SF=100)
I/O Replay (TPC-C) on Various Devices
2015/06/19 42
HDD
SATA SSD
PCIe SSD
Tips on Simulation• Technique of what-if analysis based on
some assumptions– Underlying models, workload generation
(random number distribution), omitted details, ...
– Only proper assumptions can lead to meaningful results
• Validating assumptions and results with modeling/measurement is important
2015/06/19 43
Basic Techniques
2015/06/19 44
Modeling
MeasurementSimulation Validate
Wrap Up• Why performance evaluation matters– The key process to understand systems
• Stay goal-oriented• Use basic techniques effectively– Modeling, measurement and simulation– Validate each other
• Experiences develop your philosophy
2015/06/19 45