Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by Umesh Bellur

Profiling Application Performance

and Resource Utilization

in Virtualized Environments

Umesh Bellur

Puru Kulkarni

http://www.cse.iitb.ac.in/synerg

Systems and Networks Research Group

Department of Computer Science and Engineering

Indian Institute of Technology Bombay

SYNERG, Dept. of CSE, IIT BombayFeb 16th, 2011 2

Abstract physical resources as virtual objects

o Virtual machines execute on virtual systems

o Virtual machine monitor multiplexes physical resources across virtual resources

Virtualization based Provisioning

Benefits

Direct: Rapid provisioning, Interoperability, Performance

Isolation, Increased resource utilization

Derived: Server Consolidation, Flexible resource

provisioning, Efficient replication, backup and migration


Virtualisation-based Provisioning Issues

Physical to Virtual

Resource requirements mapping

and modelling

Capacity planning

VM placement policies

Flexible Resource Provisioning

SLA-based migration

Power-aware consolidation

Cloud bursting

VM Optimisations

Shared memory based I/O for

co-located VMs

Network I/O guarantees

VMs for resource-constrained

platforms

Deployment & Manageability

Resource accounting in Clouds

Prototype Testbed setups


Provisioning & Capacity Planning in Virtual Environments

Some questions of interest

Application behaviour and resource usage under different workloads?

Throughput, response time, network, CPU, memory ...

With finite resources which SLAs can be satisfied?

How will the application behave with varying resources?

What is the impact of VM movement on application performance and resource usage levels?

SYNERG, Dept. of CSE, IIT Bombay 501/12/10

Why separate VM-specific tool?

Existing tools not virtualisation aware

All I/O through privileged VM

Impact of VM migration

Dom0 DomU

Xen Hypervisor

Hardware Workload generation and

app. performance separate

from resource monitoring


State of the art for virtual environments

Resource monitoring tools

o xenmon, xentop, sysstat, Hyperic etc.

Load generation benchmarks

o SPEC, TPCW ...

Isolated from profiling framework

o Load generation and control

o Resource configuration control

o User-level performance metrics

Require an integrated tool (for VM worlds)

oLoad generation + Resource configuration +

Measurement and monitoring


Problem Definition

Profile performance and resource utilization levels of applications in virtual environments

Multi-tier, multi-VM based (web-based) application

Implications on

o Capacity planning

o Migration and Consolidation Strategies

virtPerf*

o Application profiling tool for virtualized environments

o Built on top of AutoPerf** – a similar tool for physical environments

o Current scope: Xen and kvm

* The tool forms the thesis of a current MTech student - Prajakta Patil.

** S. Shirodkar, “Autoperf: An automated load generator and performance measurement tool for

multi-tier software systems,”, Master’s thesis, IIT Bombay, Mumbai, India, June 2007


virtPerf Architecture

Workload generator + Measurement tool

Load

generator

Controller

requests DomU

DomU

responses

DomU

Dom0

profiling

agent

virtPerf master

Physical machine 1

Physical machine 2

virtPerf slave


Load Generator and VM Initialisation

Load

generator

Controller

load

level

requests

- Multi-client load

- Warm-up

detection

DomU

Dom0

responses

ready

Initialize

- Set resource

configurations

- IP to VM id mapping

Load characteristics

Transaction details

Warmup => Stable response time

Thread startup, caching etc.

Agents initialize VMs according to resource profile


Run Termination and Measurements

Load

generator

Controller

requests DomU

Dom0

responses

DONE

STOP

profiling

<network, disk, cpu,

memory, service

demand>

...

Load characteristics

Transaction details

Resource

usage summary

- Collate performance

metrics

- Select next load level or

VM resource configuration

xentop at Host (Dom0)

sysstat and ps at Guest (DomU)

SYNERG, Dept. of CSE, IIT BombayFeb 16th, 2011

MSD of a Profiling Run


Metrics Collected

Resource usage metrics (averaged over duration of run)

CPU utilization

Network & disk I/O

Memory usage

Performance metrics

Average response time

Throughput

Service demand


virtPerf Features

Load control

o #clients, think time distributions, total number of requests issued

o AutoPerf provides automatic load level section, warmup detection

VM resource configuration

o #cores, CPU allocation (min, max), CPU pinning

o To both Dom0 and each of the DomUs individually

Impact of Migration

o Resource requirements, Performance hit (drop in throughput, increase in response time)

VM Collocation

o Resource contention impact & VM Collocation benefits


virtPerf Validation

• Metrics reported by virtPerf for resource consumptions and service performance known apriori.

• CPU Utilization:

– A service that can consume a specific amount of CPU with a knob to vary this from 10-90%.

– virtPerf reports utilization within 3% of this at any level.

• Network I/O:

– A service will read/write a known amount of I/O at a specified rate.

– virtPerf reports back this number with a 400 byte overhead per request caused by HTTP and the virtPerf control overhead

• Disk I/O:

– Similar to Network I/O and results are within 2% of expected values


Experimental Setup

PMs: Quad Core Intel i5 Intel(R) Core(TM) i5 CPU, 760 @ 2.80GHz3 GB RAM

Virtual platforms: Xen

WebCalendar & RUBiS - 2 tier applications with a MySQL backend and

a HTTP-based front-end


Throughput Saturation

webcal saturates at 64 req/s

virtPerf can automatically detect saturation

CPU utilization of VM (DomU) ~400% (4 core m/c)


Sensitivity to Think time distribution

Think time distribution: Poisson

Mean: <80ms, 150 ms, 800 ms>

Close to service

demand =>high

arrival rate

Throughput

saturates with ~12

usersThink time >> service

demand =>Low arrival

rate

Throughput saturates

with

~72 users


VM Configuration (#cores)

1 core vs 4 cores

Maximum Throughput 16 req/s : 65 req/s

Response time 900 ms : 3000 ms

Performance metrics ratio 1 : 4

VM CPU Utilization~80 % : 375 %


VM Configuration (placement)

3 configurations

o Single VM app (c1)

o 2 VMs on 1 PM (c2)

o 2 VMs on 2 PMs (c3)

Response time

o c1 << c2 << c3

Throughput

o c1 >> c2 >> c3


VM Resource Configuration (CPU availability)

CPU for DomU (guest VM): 150% and 300%

Achieved Throughput doubles

(25 req/s to 50 req/s)

Response time approximately

half with 300% CPU

WebCalendar


VM Resource Configuration for RUBiS

• Impact of increasing CPU for the Web Tier felt ONLY AFTER DB

Tier has sufficient CPU (260%). Else DB Tier is the bottleneck

• Throughput goes from 90 to 156 requests/sec at the DB tier

receiving upto 340% and the web tier going from 10 to 30%

Web Tier

CPU

Database Tier

CPU

Response Time

(miliseconds)

Throughput

(request/sec)

10% 20% 5902 8.43

10% 100% 1088 45.49

10% 180% 585 79.89

10% 260% 540 85.43

10% 340% 507 90.16

30% 20% 5525 9.07

30% 100% 1058 45.72

30% 180% 555 83.55

30% 260% 366 121.90

30% 340% 278 155.21


VM Migration while under Load


Effects of Migration


Core Pinning

• VirtPerf can pin VMs on specificcores of a machine.

• Makes a difference to applicationperformance.

Peak throughput almost doubles when the DBTier is given it’s own core compared to whenall the cores are shared amongst the VMs – Dom0and 2 app VMs sharing 2 cores.


virtPerf Summary

Workload generator + Application profiler

Feature set (profiling with ...)

o Load distribution control

o VM resource configuration and control

o Performance and resource metrics with migration

o Profiling with VM interference (to do)

Demonstration of virtperf capabilities and correctness via lab set up with a couple of multi tier applications


Future Work

virtperf features addition

o Session, probabilistic request workloads

o Profiling with interfering VMs

o Evaluation and testing with kvm

Experimentation

o Testing at scale

o Black-box vs. Gray-box profiling - Remove profiling within VM

o Bottleneck identification with multi-tier applications

Open-source release

http://www.cse.iitb.ac.in/synerg


Other Virtualisation Related Projects@synerg

Flexible resource provisioning

o Affinity-aware resource profiling and prediction

o Benchmarking, estimating and tuning cost of migration

p2v placement tool

o Design and evaluation of virtualisation-aware placement techniques

vSim

o A simulator for virtualisation-based systems

Network I/O optimisations for collocated VMs


Questions?umesh, [email protected]

Acknowledgements:

1. Prof. Varsha Apte who seeded the idea with AutoPerf

2. Past MTech students Shrirang Shirodkar, Vaibhav Selot

Documents

Apache Hadoop India Summit 2011 talk "Profiling Application Performance" by Umesh Bellur