<Insert Picture Here>

Oracle Real Application Clusters (RAC) on Oracle VM

Nitin Vengurlekar,

Markus Michalewicz,

Daniel Dibbets

• RAC is the cornerstone of Oracle’s Grid infrastructure

• RAC is the enabler of Grid

• Higher Availability

• RAC systems can be configured to have no single point of failure

• Scalability

• RAC allows database systems to scale-out rather than having to scale-up

• Affordability

• Can be built using low-cost servers and low-cost storage

• Manageability

• RAC with EM, allows users to manage a cluster of database servers as if they were single systems.

Oracle RAC Overview

• Improved server consolidation

• RAC databases with underutilized CPU resources or peaky CPU utilization can be consolidated with other similar workloads

• Rapid provisioning of RAC servers and resources

• RAC nodes can be quickly deployed including fully patched and configured system images

• Sub-capacity licensing

• Oracle VM can be configured using hard partitioning

• Hard partitioning allows you to license only CPUs used by the partition instead of all CPUs on a physical server

• Virtual Cluster

• Create RAC environment on a single physical server

• Perfect environment for test, Q/A, or demo clusters

Benefits of RAC on Oracle VM (OVM)

Service

public network

Node1

Operating System

Oracle Clusterware

instance 1

ASM

VIP1

Listener

Node 2

Operating System

Oracle Clusterware

instance 2

ASM

VIP2

Listener

Service Node n

Operating System

Oracle Clusterware

instance n

ASM

VIPn

Listener

Service

/…/

Redo / Archive logs all instances

shared storage

Database / Control files

OCR and Voting Disks

Managed by ASM

RAW / Block Devices

Oracle RAC Architecture

• Configuration• Oracle Database 10.2.0.4 – 32bit

• Guest OS : Oracle Enterprise Linux 5.1 or higher

• Requires Oracle VM 2.1.2

• Requires para-virtualized mode guests

• Static mode support only

• Live Migration, dynamic resource changes (CPU / Memory) currently not supported

• Timeline for Certification

• RAC-OVM Template coming soon

• 64bit – CYQ1

• 11g – CYQ2

• Dynamic resources changes – CYQ2

• Live Migration – possibly CYQ2

Oracle VM – RAC Database Support

• Why RAC on OVM Certification is needed

• Oracle Clusterware / RAC is very time and scheduling sensitive

• Follow best practices and guidelines to avoid surprises

• Validation/Certification - Areas of focus:

• Clock drift

• Scheduling

• Timeout sensitivity

• Dynamic changes

• OS profile and statistics

• CPU, Memory affinity

• …

Server Virtualization Certification

• Size as you would a bare-metal server

• Same traditional questions:

• How much memory do I need?

• How many CPUs do I need?

• How much I/O and network bandwidth should I dedicate?

Sizing and Configuration – Guest and Dom0

• Minimum of 2 Oracle VM servers

• Ensure enough memory to accommodate DomUcomponents (OS, Oracle Clusterware, and RAC databases)

• Do not overload Dom0 with unnecessary packages,

products and workload

• Configure Time synchronization

• Configure NTPD to run in Dom0 and DomU

• Set guests to manage their own system clocks by setting xen.independent_wallclock=1 in /etc/sysctl.conf file.

RAC on OVM – Server Configuration

• CPU configuration

• Assign a minimum of 1 core to Dom0.

• Certain circumstances may require to assign 1> core to Dom0. Follow guidelines in Oracle VM Server User’s Guide

• Over-committing CPUs or cores is not supported

• Don‘t create more vcpu‘s then real CPU‘s in one DomU

• Dynamic VCPU and memory changes are currently not

supported

RAC on OVM – Server Configuration cont’d

D o m a in - 0

G u e s t D o m a in 1

D a t a b a s e 1

D a t a b a s e 1I n s t a n c e 1

A S M I n s t a n c e+ A S M 2

A S M I n s t a n c e+ A S M 1

D a t a b a s e 1I n s t a n c e 2

O r a c l e C l u s t e r W a r e

O r a c le E n t e r p r i s e L i n u x

O r a c l e C lu s t e r W a r e

O r a c l e E n t e r p r i s e L in u x

G u e s t D o m a i n 2

RAC on OVM – Non-Production Deployment

Domain-0

Guest Domain 1

Database 1

Instance 1

Database 2

Instance 1

Database 1

Instance 2

Database 2

Instance 2

ASM Instance

+ASM1

ASM Instance

+ASM1

ASM Instance

+ASM2

ASM Instance

+ASM2

Oracle ClusterWare

Oracle Enterprise Linux

Oracle ClusterWare

Oracle Enterprise Linux

Oracle ClusterWare

Oracle Enterprise Linux

Oracle ClusterWare

Oracle Enterprise Linux

Guest Domain 2

Domain-0

Guest Domain 1

Guest Domain 2

Database 1

Database 2

RAC on OVM – Production Deployment

• Two or more HBAs need for redundancy

• HBAs should be configured as multipath-ed devices in Dom0

• Only physical disks based VBD are supported

• File or block based storage configuration okay for single instance

• Configure device persistency in Dom0

• Generally provided by multipathing driver

• If not available, use udev in DomU

• Optionally configure physical block devices (in DomU)

as ASMLIB disks

RAC on OVM – Disk Configuration

# xen config file example for RAC Guest Domain

name = “vmrac1"

memory = “8192"

disk = [

'phy:/dev/mapper/mpath3p1,xvda,w!',

'phy:/dev/mapper/mpath4p1,xvdb,w!',

'phy:/dev/mapper/mpath5p1,xvdc,w!',

]

vif = [

'mac=00:16:3E:00:00:08, bridge=xenbr0',

'mac=00:16:3E:10:A5:96, bridge=xenbr1',

]

vfb = ["type=vnc,vncunused=1"]

uuid = "3d6f1de4-626c-e02a-42a1-458c9c17e728"

bootloader="/usr/bin/pygrub"

vcpus=8

on_reboot = 'restart'

on_crash = 'restart'

RAC on OVM – Disk Configuration

D o m a i n - 0

O r a c l e V M

G u e s t 1 G u e s t 2

p o r t - 0 p o r t - 1

S A N S t o r a g eP r o c e s s o r 0 P r o c e s s o r 1

S A N S w i t c h 1 S A N S w i t c h 2

M u l t i p a t h i n g D r i v e r

V B D D r i v e r V B D D r i v e r

L U N - 1

/ d e v / s d a

/ d e v / x v d a / d e v / x v d a

S A N H o s t B u s A d a p t e r

A S M I n s t a n c e+ A S M 1

A S M I n s t a n c e+ A S M 2

RAC on OVM – ASM Disk Configuration

Domain-0

Oracle VM

Guest 1 Guest 2

port-0 port-1

SAN StorageProcessor 0 Processor 1

SAN Switch 1 SAN Switch 2

Multipathing Driver

VBD Driver VBD Driver

LUN-1

/dev/sda

/dev/xvda /dev/xvda

SAN Host Bus Adapter

OCFS filesystem

RAC on OVM – OCFS Disk Configuration

• Use dedicated network devices • For each host a minimum of two 1Gb Ethernet NICs is recommended

• For production RAC environments provide

network traffic separation for public and private

• Extra Ethernet NICs required if network based storage connectivity

(iSCSI or NFS) is used

• These networks can be shared with multiple RAC VM guests, but network bandwidth and capacity needs to reviewed

• For complete network redundancy, 4-6 NICs recommended

• Bond ethN interfaces in Dom0 for HA and/or bandwidth

RAC on OVM – Network Configuration

# xen config file example for RAC Guest Domain

name = “vmrac1"

memory = “8192"

disk = [

'phy:/dev/mapper/mpath3p1,xvda,w!',

'phy:/dev/mapper/mpath4p1,xvdb,w!',

'phy:/dev/mapper/mpath5p1,xvdc,w!',

]

vif = [

'mac=00:16:3E:00:00:08, bridge=xenbr0',

'mac=00:16:3E:10:A5:96, bridge=xenbr1',

]

vfb = ["type=vnc,vncunused=1"]

uuid = "3d6f1de4-626c-e02a-42a1-458c9c17e728"

bootloader="/usr/bin/pygrub"

vcpus=8

on_reboot = 'restart'

on_crash = 'restart'

RAC on OVM – Network Configuration

e th 0 e th 1 e th 2 e th 3

x e n b r0 x e n b r1

e th 0 e th 1 e th 0 e th 1

b o n d 0 b o n d 1

D o m a in -0

P u b lic

n e tw o rk

P u b lic n e tw o rk

P r iv a te n e tw o rk

P r iv a te n e tw o rk

O ra c le V M

G u e s t 1 G u e s t 2

P u b lic N e tw o rk

S w itc h 1

P u b lic N e tw o rk

S w itc h 2

P riv a te N e tw o rk

S w itc h 1

P riv a te N e tw o rk

S w itc h 2

RAC on OVM – Network Configuration

eth0 eth1 eth2 eth3

xenbr0 xenbr1

eth0 eth1 eth0 eth1

bond0 bond1

Domain-0

Public network

Public network

Private network

Private network

Oracle VM

Guest 1 Guest 2

Public NetworkSwitch 1

Public NetworkSwitch 2

Private NetworkSwitch 1

Private NetworkSwitch 2

eth4 eth5

xenbr2

bond2

Storage NetworkSwitch 1

Storage NetworkSwitch 2

eth2eth2

Storage network

Storage network

NetApp Filer:

filer01.us.oracle.com

Exported volume:volume01

filer01.us.oracle.com:/volume01/u01/oradata /u01/oradata

RAC on OVM – NFS based Virtual Cluster

• Disk Configuration• Make sure device persistancy is correctly setup with udev in Dom0

• Make sure permissions are set persistently and correct using udev in DomU

• Network Configuration• Make sure bridges are configured correctly in Dom0

• Use brctl show to verify

• Ensure firewall in DomU does not affect interconnect traffic

• The default firewall in the Oracle supplied Linux templates blocks RAC traffic.

• Either make it a trusted device:iptables –A input –i ethN –j ACCEPT, or disable firewall

Typical Installation Issues

[root@strab03 ~]# xm list

Name ID Mem VCPUs State Time(s)

Domain-0 0 751 4 r----- 7225.2

OVM_EL5U1_X86_64_PVM_10GB_RAC11 1 10240 2 -b---- 15.8

OVM_EL5U1_X86_64_PVM_10GB_RAC12 2 10240 2 -b---- 13.9

[root@strab03 ~]# brctl show

bridge name bridge id STP enabled interfaces

xenbr0 8000.feffffffffff no vif2.0

peth0

vif1.0

vif0.0

xenbr1 8000.feffffffffff no vif2.1

peth1

vif1.1

vif0.1

• Example for having both RAC guests on one server.

• Make sure the same bridges, using the same physical networks are used in both RAC domU.

Verifying Network Setup

• Each RAC node must be deployed on a separate physical server for production environments

• Do not over-commit CPUs

• Configure disk multi-pathing and NIC bonding in Dom0

• To sustain required I/O throughput, ensure sufficient

CPU resources are allocated to hypervisor

RAC on OVM – Best Practices Summary

• Gold Image includes

• EL 5 update 2

• RAC stack - CRS/ASM/Db 11.1.0.7 plus current bundle

• Configured ASM diskgroup with seed database

• RAC Cloning to enable/provision additional nodes

• Boot script will prompt for:

• Hostname

• IP address

• Network devices

• Requires 6 block devices – to be used for ASM and OCR/Voting Disk

RAC on OVM – Templates

RAC-OVM Template

• Allows you to build a fully patched RAC node in one shot

• No need to patch & apply one off patches

• Creates a “production ready” clustered domain

• Consists of two Phases

• Prep phase

• Deploy Phase

• Entire underlying process hidden from user

• User only prompted for required input values on guest creation

• Template package will include a complete tar’ed Clusterware Stack (CRS/ASM/RDBMS)

• Two ORACLE_HOMEs - CRS and RDBMS

Process Flow

• TAR extract• Clusterware home

• RAC home for ASM

• RAC home for RAC-RDBMS

• Build 1 node cluster on first Virtual machine• Execute clone script for Clusterware

• Run clone.pl with proper variables from the template

• Runs root.sh

• Runs orainst.sh – oraInventory directory

• Execute clone script for ASM and database

• Runs clone.pl

• Runs root.sh

• Runs netca and dbca in silent mode

• Oracle RAC on OVM Technical White Paper:

http://www.oracle.com/technology/products/database/clusterware/pdf/oracle_

rac_in_oracle_vm_environments.pdf

• RAC internal website – including best practices:http://rac.us.oracle.com

• Oracle RAC on oracle.com:

http://otn.oracle.com/rac

• Oracle Clusterware on oracle.com:

http://otn.oracle.com/clusterware

• Oracle ASM on oracle.com:

http://otn.oracle.com/asm

More Information