Optimize Oracle RDBMS on VMware Guy Harrison Director, R&D Melbourne [email protected] @guyharrison

Optimize Oracle RDBMS on VMware

Guy HarrisonDirector, R&D Melbournewww.guyharrison.net [email protected]@guyharrison

http://www.guyharrison.net/

mailto:[email protected]

http://www.twitter.com/guyharrison

http://www.twitter.com/guyharrison

Introductions

Blue

Yellow

Red

0 10 20 30 40 50 60 70 80

Star trek shirt fatality analysis

Pct

11

©2011 Quest Software, Inc. All rights reserved..

Agenda

• Motivations for Virtualization

• VMware ESX resource management:

• Memory

• CPU

• IO

• Paravirtualization (OVM) vs Hardware Assisted

Virtualization (ESX)

• RAC on VMware

12


Motivations for Virtualization

• Better utilization of server resources• Reduced power consumption

Server Consolidation

• Fewer physical machines• Backup, cloning, rapid provisioningManageability

• Adjust resources on demand• A complement to the physical “grid”

visionElastic computing

13


Resistance to Database virtualization

• Virtual CPU & IO• Sharing of virtual resources Performance

• Large databases too big for a single VM

• RAC-style clustering problematic Scale

• Oracle’s stance often misunderstood

• See MyOracleSupport 249212.1Support

14


DB virtualization is happening

15


Oracle virtualization is lagging....

SQL Server

File and Print Servers

IIS

Active Directory

Apache

Exchange

Sharepoint

Java

Oracle

None of the Above

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Which of the following do you run in VMs?

Quest survey of vFoglight users , 2010

16


ESX Memory management

17


Managing ESX memory

• ESX can “overcommit” memory• Sum of all VM physical memory allocations > actual ESX physical memory

• Memory is critical to Oracle server performance• SGA memory to reduce datafile IO

• PGA memory to reduce sort & hash IO

• ESX uses four methods to share memory:• Memory Page Sharing

• Memory compression

• “Ballooning”

• ESX swapping

• DBA needs to carefully configure to avoid disaster

Configuring VM memory

Relative Memory Priority for this VM

Minimum Memory for this VM

VMs Compete for memory in this range

Maximum memory for the VM (dynamic)

19


Monitoring VM memory

ESX Swap

ESX virtual memory

ESX physical memory

VM VM virtual memory

ESX swap

Effective VM physical memory

ESX and VM memory

ESX Swap

ESX virtual memory

ESX physical memory

VM

Apparent VM physical memory

ESX swap


Vmmemctl

“balloon”

VM SwapVM Swap

ESX Ballooning

ESX Ballooning

As memory grows, ESX balloon driver (vmmemctl) forces VM to page out

memory to VM swapfile

ESX Ballooning

• Inside the VM, paging to the

swapfile is observed.

• The guest OS will determine

which pages are paged out

• If LOCK_SGA=TRUE, then the

SGA should not be paged.

ESX Swap

ESX virtual memory

ESX physical memory

ESX swap

ESX Swapping

VM virtual memory

VM


ESX Swap

ESX virtual memory

ESX physical memory

ESX swap

Apparent VM physical memory

VMEffective VM physical memory

ESX Swapping

ESX Swapping

ESX swaps out VM memory to ESX swapfile

ESX Swapping

• Within the VM, swapping cannot

be detected.

• ESX will determine which

memory pages go to disk

• Usually occurs when VMware

tools are not installed

• Even if LOCK_SGA=TRUE,

SGA memory might be on disk

Avoiding Ballooning and swapping

memory reservations help avoid ballooning or ESX

swapping

29


Ballooning vs. Swapping

Swingbench workload running on Oracle database – from VMWare whitepaper: http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf

http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf

30


VMware memory recommendations

• Paging or swapping of PGA or SGA is almost always a Very Bad

ThingTM.

• Use memory reservations to avoid swapping or ballooning

• Install VMware tools to allow ballooning instead of swapping

• Set Memory reservation = PGA+SGA+process Overhead

• Be realistic about memory requirements:

• In physical machines, we are used to using all available memory

• In VM, use only the memory you need, freeing up memory for other VMs

• Oracle advisories (or Spotlight) can show you how much memory is

needed

• Reduce VM reservation and Oracle memory targets in tandem to

release memory

32


ESX CPU management

33


ESX CPU management

• If more virtual CPUs than ESX CPUs, then vCPUs will

sometimes wait for physical CPU

• Time “stops” inside the VM when this occurs

• For multi-CPU VMs, it’s (nearly) all or nothing.

• A vCPU can be in one of three states:• Associated with an ESX CPU but idle

• Associated with an ESX CPU and executing instructions

• Waiting for ESX CPU to become available

• Shares and reservations determine which VM wins access to

the ESX CPUs

Configuring VM CPU

VMs compete for CPU in this range

Shares determine relative CPU allocated when competing

CPU utilization VM

“CPU Ready” is the

amount of time VM

spends waiting on

ESX for CPU

Inside the VM, CPU

stats can be

misleading

SMP for vCPUs

• ESX usually has to

schedule all vCPUs for a

VM simultaneously

• The more CPUs the

harder this is

• Some CPU is also

needed for ESX

• More is therefore not

always better

(Thanks to Carl Bradshaw for letting me reprint this diagram from his Oracle on VMWare whitepaper)

http://www.dspmanagedservices.co.uk/doc-oracle-on-vmware



37


ESX CPU performance comparisons

Vs 2 core 1.8 GHz physical machine

VT enabled

38


Programmatic performance

PLSQL

PLSQL compiled

Java Stored Proc

Simple C program

Standalone Java

0 5 10 15 20 25 30 35 40 45 50

ESX 2 VCPU 2.26 GHz VT enabled

ESX 2 CPU 3.5 GHz no-VT

Physical 2 CPU 1.8 GHz

Elapsed Time (s)

NB: Not a benchmark! Just some informal measurements!!

39


Programmatic performance (2)

ESX 2 CPU 3.5 GHz no-VT

ESX 2 VCPU 2.26 GHz VT enabled

0 50 100 150 200 250 300 350

333

72

Elapsed time Pct relative to Physical CPU adjusted for GHz

40


ESX CPU recommendations

• Use up to date chipsets and ESX software

• Allocate as few VCPUs as possible to each VM

• Use reservations and shares to prioritise access to ESX CPU

• Monitor ESX Ready time to determine the “penalty” of competing

with other virtual machines

41


ESX IO management

Typical VMWare disk configuration

IO Resource Allocation

• Disk shares can be used to prioritize IO bandwidth.

• This is poorly implemented prior to vSphere 4.1

44


Storage IO Control

• Prior to vSphere 4.1:• disk shares could be used only at the VM level, and only within a single

ESX host

• vSphere 4.1 Storage IO Control (SIOC):• Manages disk share priorities for all VMs attaching to the same

datastore

• Is triggered by high (“congested”) latency

• Can be enabled globally at the datastore level

• Enables equitable distribution even when set to defaults

Storage IO Control

46


vSphere 4.1 SIOC

47


SOIC won’t make up for a poorly configured IO layout

Performant VMware disk configuration

49


Optimal configuration

• See “Oracle Database Scalability in VMware® ESX” at

www.vmware.com/oracle

• Each virtual disk directly mapped via RDM to dedicated

RAID 0 (+1) group

41 Spindles!

http://www.vmware.com/oracle

50


ESX IO recommendations

• Follow normal best practice for physical disks

• Avoid sharing disk workloads • Dedicated datastores using VMFS

• Align virtual disks to physical disks?

• Consider Raw Device Mapping (RDM)

• Consider SIOC in vSphere 4.1

• If you can’t optimize IO, avoid IO:• Tune, tune, tune SQL

• Prefer indexed paths

• Memory configuration

• Don’t forget about temp IO (sorts, hash joins)

51


Shameless plugs

53


54


55


Paravirtualization vs Hardware Virtualization

56


Paravirtualization vs “Hardware Virtualization”

• Virtualization is not emulation....

• Where-ever possible, Hypervisor runs native code from

OS against underlying hardware

• Because a virtualized operating system is running

outside privileged x86 “ring 0”, direct calls to hardware

need special handling.

• The three main approaches are:• Full Virtualization (VMWare on older hardware)

• ParaVirtualization (Xen, Oracle VM)

• Hardware Assisted Virtualization (Intel VT, AMD-V)

Full virtualization

• Hardware calls from the VM

are handled by the

hypervisor by:• Catching the calls as they occur at

run time

• Re-writing the VM image at load

time (binary translation)

• Requires no special

hardware

• Supports any guest OS

• Relatively Poor performance

• Used by ESX on older chip-

sets

VM

Hypervisor

Hardware

Ring 0

Hardware Assisted virtualization

• Intel VT and AMD-V chips

add a non-root mode Ring

0.

• VM can issue instructions

from non-root Ring 0.

• CPU can divert these to

hypervisor

• No changes to OS required

• Good performance

• Requires modern chipsets

Hypervisor

Hardware

VM

Ring 0

Root Mode Non-Root Mode

Paravirtualization

• VM operating system is rewritten to translate device calls to “hypercalls”

• Hypercalls are handled by a special VM (dom0 in Xen/OVM)

• Good performance but requires modified VM OS

• Xen can use either paravirtualization or hardware assist

Hypervisor

Hardware

VM(domU)

Ring 0VM

(dom0)

60


RAC and ESX

61


Paravirtualization, ESX and RAC

• Prior to 11.2.0.2, Oracle relied on paravirtualized kernels to

maintain time synchronization for RAC clusters.

• From 11.2.0.2 Oracle uses Cluster Time Synchronization

Service (CTSS) to maintain clock sync, and this works on

ESX

• Therefore, Oracle supports RAC on Vmware ESX only from

11.2.0.2 onwards

• See Oracle MySupport Note 249212.1

https://support.oracle.com/CSP/ui/flash.html#tab=KBHome(page=KBHome&id=()),(page=KBNavigator&id=(bmDocID=249212.1&viewingMode=1143&from=BOOKMARK&bmDocDsrc=KB&bmDocTitle=Support%20Position%20for%20Oracle%20Products%20Running%20on%20VMWare%20Virtualized%20En

62


References

• Latest version of this presentation:• http://www.slideshare.net/gharriso/optimize-oracle-on-vmware-5271530

• My blog (www.guyharrison.net ):• http://guyharrison.squarespace.com/blog/2010/2/22/memory-

management-for-oracle-databases-on-vmware-esx.html

• http://guyharrison.squarespace.com/blog/2010/4/9/esx-cpu-optimization-

for-oracle-databases.html

• http://guyharrison.squarespace.com/blog/2010/7/12/stolen-cpu-on-xen-

based-virtual-machines.html

• http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf

• http://www.vmware.com/files/pdf/Oracle_Databases_on_vSphere_Deployment_Tips.pdf

• http://www.vmware.com/files/pdf/techpaper/VMW-vSphere41-SIOC.pdf

http://www.slideshare.net/gharriso/optimize-oracle-on-vmware-5271530

http://www.guyharrison.net/

http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf

http://www.vmware.com/files/pdf/Oracle_Databases_on_vSphere_Deployment_Tips.pdf

http://www.vmware.com/files/pdf/techpaper/VMW-vSphere41-SIOC.pdf

COLLABORATE 12April 22-26, 2012

Mandalay Bay Convention CenterLas Vegas, Nevada, USA

www.collaborate12.orgwww.collaborate12.ioug.org

http://www.collaborate12.org/

http://www.collaborate12.ioug.org/

Documents

Optimize Oracle RDBMS on VMware Guy Harrison Director, R&D Melbourne [email protected] @guyharrison