Techniques and Best Practices - Oracle · An example of light consolidation would be something like: "I want to consolidate 30 databases, but I also need a few cores to run some ETL,

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Database Consolidation with Oracle Exadata Techniques and Best Practices

Manish Shah Sr. Principal Product Manager, Exadata Oracle Corporation Ajit Singh Director, Data Technology Platforms New York Stock Exchange (NYSE: ICE)

Copyright © 2015, Oracle and/or its affiliates. All rights reserved.

Program Agenda

Why Consolidate On Exadata

How To Be Successful With Consolidation On Exadata

Customer Success Story - NYSE

1

2

2

3


Program Agenda




1

2

3

3

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. 4

Odds Of dying in a lifetime*

1 in 98 1 in 7178

*2008 National Safety Council Report

4 People 250 People


Redundant Database Servers

Redundant Storage

Redundant Connectivity

Redundant and hot swappable power supply

Redundant Components Continuous Application Availability through hardware

Faults

5

• FANs • PDUs • Batteries • Switch • Disk • Flash • Motherboards • DIMMs


Reduced HA Brownouts • Fast node detection and failover • Automatic detection of Exadata storage failures with low application impact • Automatic detection of Exadata network failures with low application impact

Data Protection • Automatic Hard Disk scrub and repair • Exadata H.A.R.D

Quality Of Service • I/O Latency capping for read and write operations • Exadata Cell I/O Timeout Threshold • Health Factor for predictive failed disk • Identification of underperforming disks and automatic removal • I/O Resource Management • Network Resource Management

Overview of Exadata Specific HA Features

6


Performance

• Exadata Smart Flash Logging • Active Bonding Network • Exadata Smart Write Back Flash Cache Persistent After Cell Restarts

Management

• Flash and Disk Life Cycle Management Alerts • Cell Alert Summary • LED Notification for Storage Server Disk Removal • Drop Hard Disk for replacement • Drop BBU for replacement • Minimize or eliminate False disk failures

7


0

50

100

150

200

40 DB 160 DB

x86

X86

Exadata

How many more databases can be consolidated on

Exadata?

177 ms 132 ms

4X Density With Faster Response

Time

8

Avg. Response Time

0

50

100

150

200

250

300

168 non-CDB

252 PDB

PDB

50% Higher Density

How many more PDB can be consolidated than non-

CDB?

212 ms 14 ms

Avg. Response Time

Exadata Non-CDB


The Exadata “Community Effect”

• Standard Technologies • Standard Configurations • Standard Integration • Standard Tuning • Standard Support

9

Exadata Takes Standardization to the Next Level

Server Vendor A

Storage Vendor B

Network Vendor C

Database Vendor D

OS Vendor E

VM Vendor F

The New Global Standard A Company Standard vs

Oracle Public Cloud Oracle Development & Support

1,000’s of Customers and Partners

The Scalability of Telecom, The Availability of Banking, The Security of Government


Program Agenda




1

2

10

3


8 Key Success Factors



Availability Service Levels for Unplanned and Planned Outages Oracle MAA Availability Tiers

BRONZE

SILVER

GOLD • Comprehensive HA and Disaster Protection • Recovery in seconds with zero or near-zero data loss

• High Availability (HA) for Recoverable Local Outages • Backups plus redo for Oracle data protection

• Basic Service Restart • Backups plus redo for Oracle data protection

PLATINUM • Zero Outage for Platinum Ready Applications • Zero data loss

13


Zero Application Outage for Platinum Ready Applications Platinum Tier

Site A Site B

Backups

Backups

Outages masked from applications, in-flight transactions preserved

– Application Continuity Zero data loss failover, LAN or WAN

– Active Data Guard / Far Sync Bi-directional replication and zero

downtime maintenance – Oracle GoldenGate

Online patching for applications – Edition-based Redefinition

Automated workload management – Oracle Global Data Services

Tape

Data Guard sync

Tape

Application Continuity

GoldenGate

RAC RAC

14



• What criteria should you use to determine # of CDBs? – Prod v/s Non-prod? – Business Functions? – Business SLAs / Common maintenance window? – All of the above?

• It “Depends” ! –

Determine Strategy to move to Multitenant

How many CDBs should you have?

No. I am not going to say that.


Goal

25 Prod 75 Non Prod

ABC Company 100 Databases

5 DW 20 OLTP 10 DW 65 OLTP


25 Prod 75 Non Prod



Metal Tier Prod Non Prod

DW OLTP DW OLTP

Platinum 1 5 Gold 2 10 Silver 2 5 2 20

Bronze 8 45

Total 5 20 10 65

Apply Reference Architecture

Start With Database Categorization


• Separate CDBs for OLTP and DW workloads

• Separate CDBs for different 12c versions. Limit the number of Oracle database homes to a maximum of 5.

• Separate CDBs for Prod and Non-Prod HA Tier. • Consider separate CDB for PDBs that don’t need flash performance so you can turn

IORM Flash cache and Flash log off for that CDB when needed.

• Keep no more than two variants per patch set (e.g. 12.1.0.2.7 – Apr PSU and 12.1.0.2.10 – July PSU)

Guidelines for Separating PDBs into CDBs


25 Prod 75 Non Prod



Metal Tier Prod Non Prod

DW OLTP DW OLTP

Platinum 1 5 Gold 2 10 Silver 2 5 2 20

Bronze 8 45

Total 5 20 10 65

Apply CDB Guidelines

Start With Database Categorization Apply Reference Architecture



Engineered Systems – Capacity Utilization and Planning (ES-CUP)

Program Details

• Free program offered by Oracle for Exadata Customers

• New Customers

• Analyze existing workload

• recommend Exadata machine configuration

• Existing Customers • Recommend solutions to address current challenges • Provide roadmap for ongoing capacity planning exercise

• Engagement Process

• Contact your regional Oracle Sales Team

22


Engineered Systems – Capacity Utilization and Planning

Conf-call to review objectives

Document engagement scope

Planning and timeline

Customer to describe near-term growth estimates

ACTI

VITI

ES

Kick-Off Data Collection Analyze Readout

Oracle send data collection scripts

Customer execute data collection scripts and send logs to Oracle

Customer execute Exachk and send logs to Oracle

Oracle to analyze information

Crosscheck with Best Practices

Generate Exadata Capacity Utilization and Planning document

Generate roadmap

Remote or onsite delivery of Exadata Capacity Utilization and Planning

Review objectives Knowledge transfer Documentation

shared

ES-CUP

Step 1 Step 2 Step 3 Step 4



Consolidation Workbench

Enterprise Manager

25

25 Prod 75 Non Prod





Resources To Be Managed

CPU

Memory

Disk

Flash

Exadata Resource Management

CDB1

CDB2

Instance 2

Instance 2

CDB1

CDB2

Instance 1

Instance 1

P1 P2 P3

P4 P5 P6

Exadata Quarter Rack X5-2 27

P1 P2 P3

P4 P5 P6

Compute Node 1 Compute Node 2

Storage Node 1

Storage Node 2

Storage Node 3


CDB1 CDB2

P1 P2 P3 P4 P5 P6

OLT

P

BATC

H

OLT

P

BATC

H

OLT

P

BATC

H

OLT

P

BATC

H

OLT

P

BATC

H

OLT

P

BATC

H

HOW DO I ALLOCATE BETWEEN

HOW DO I ALLOCATE SPACE AND BANDWIDTH BETWEEN

CPU

FLASH

DISK HOW DO I ALLOCATE IO REQUESTS BETWEEN

Exadata Resource Management

28

Storage Node 1

Storage Node 2

Storage Node 3

CDB1

CDB2

Instance 1

Instance 1

P1 P2 P3

P4 P5 P6

Compute Node


Most Important For Consolidation Purpose

Configured on database Configured on database

Configured on database





Between CDBs

PDBs Within CDBs

Workloads Within PDBs

CPU

FLASH

DISK

Instance Caging CDB Resource

Plan

CDB Resource Plan

CDB Resource Plan

Database Resource Plan



Inter-Database IORM

Inter-Database IORM

Configured on Storage Cell

Configured on Storage Cell

Storage Node 1

Storage Node 2

Storage Node 3

CDB1

CDB2

Instance 1

Instance 1

P1 P2 P3

P4 P5 P6

Compute Node



Best practices for Exadata planned maintenance

• Leverage Exachk for simple software planning • Leverage zero downtime software maintenance • Usage Out of Place and Standby First Updating • Leverage Lights Out Patching with notification function • Take advantage of Exadata Engineered defaults

31



OVM On Exadata Use Cases and Benefits

Caveats:

• Exadata is optimized to be a Database Machine. The primary goal of OVM on Exadata is to provide a higher level of Isolation between databases.

• Exadata is not a middleware consolidation platform. Consider PCA* or SuperCluster for that purpose.

Recommendation by Oracle:

• Focus on consolidating database with perhaps some light consolidation of other applications. An example of light consolidation would be something like: "I want to consolidate 30 databases, but I also need a few cores to run some ETL, management, and other tools and don't want to deploy another server just for these" .

OVM on Exadata was created for databases but it is a full fledge feature that can be used for any valid reason a customer wants. It is available at no additional cost.

33

*Private Cloud Appliance


VM VM

Isol

atio

n

Efficiency

Database 12c Multitenant

Many DBs in one Server

Virtual Machines

Dedicated DB Servers

• VMs have good Isolation but poor efficiency and high management – Separate OS, memory, CPUs, and patching – Isolation without need to trust DBA,

System Admin

• Database consolidation in a single OS is highly efficient but less isolated – DB Resource manager isolation adds no

overhead; must trust admins to configure

• Best strategy is to combine VMs with database native consolidation – Multiple trusted DBs/PDBs in a VM – Few VMs per server to limit overhead

DATAB

ASE IN

-MEM

OR

Y

FINANCE VM

SUPPLY CHAIN

VM

SALES VM

OVM On Exadata Use Cases


Use Cases:

• Hosting, Cloud, Cross department consolidation

• OVMs are trusted partitions – use for license cost reduction/compliance (e.g. create a VM cluster with few cores to match what you are licensed for – best for Oracle options e.g. partitioning)

• Government compliance for workload isolation – PCI/DSS requirements can be met by using OVM, VLAN Tagging and IB Partitioning.

• Host a small app or a tool that needs isolation

No Additional Cost

SALES

SUPPLY CHAIN

FINANCE

DB 11.2 and 12c X2-2, X3-2, X4-2, X5-2

35

OVM On Exadata Use Cases


• Exadata OVM allows licensing of Virtual CPUs (Trusted Partitions)* – At least 40% of usable cores on the machine must be licensed for Oracle software – Cannot use both OVM and CoD to license less than 40% of cores – A server must be virtualized or non-virtualized – not mixed – OVM has no License cost; support is included with premier hardware support

36

* http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf

Virtualized Virtualized

VM1A 12 cores

(24 vCPU)

VM2A 24 cores

(48 vCPU)

DB EE

Non-Virtualized

DB EE RAC

Adv Security

14 cores

DB EE DB IM

VM1AB 12 cores

(24 vCPU)

VM2B 24 cores

(48 vCPU)

DB EE DB EE RAC

Adv Security

60% CoD

Cluster

RAC RAC

Cluster

OVM On Exadata Configuration Guidelines

http://www.oracle.com/us/corporate/pricing/partitioning-070609.pdf�




Hardware X2-2 X3-2 X4-2 X5-2 Physical Memory /Node (default/max)

72 GB 144 GB (max)

256 GB 512 GB (max)

256 GB 512 GB (max)

256 GB 768 GB (max)

Min Mem per VM 8 GB min + DB or app memory Max Mem per VM 512 GB Mem Defaults (GB) Small - 16; Medium - 32; Large – 64 (Templates adjustable at install time)

Physical CPU threads/node* 24 32 48 72 Min vCPUs per VM 2 Max vCPUs per VM Physical Threads - 2 Default Template vCPUs Small - 2; Medium - 4; Large – 8 (Templates adjustable at install time)

Total Available Disk Space for all DomU

700 GB 700 GB 1.6 TB 1.6 TB

Minimum Used Disk Space per VM

100 GB Sparse allocated per VM for initial deployment. 90 GB required to run one Oracle database version (8 GB required per DB Version). 10 GB available for any other usage.

OVM On Exadata Configuration Guidelines

*1 thread = 1 vCPU

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. DB

FS

RECO

DATA DBFS

RECO

DATA

OVM Cluster 1

OVM Cluster 2

Node 3 (Virtual) Node 4 (Virtual)

DBFS

RECO

DATA

Node 1 (Physical) Node 2 (Physical)

Physical Cluster

Exadata Half Rack Split into Physical and Virtual Clusters

38

OVM On Exadata Example


DBFS

RECO

DATA

DBFS

RECO

DATA

Node 3 (Virtual) Node 4 (Virtual)

Dom 0 Dom 0

OVM Cluster 1

OVM Cluster 2

Dom U1

Dom U2

Dom U1

Dom U2

39

OVM On Exadata Example


Database Server: Bare Metal / Physical versus OVM

OVM Database Server

dom0

Exadata (Linux, Xen, fw)

domU-3

Exadata (Linux)

Oracle GI/DB homes domU-2

Exadata (Linux)

Oracle GI/DB homes domU-1

Exadata (Linux)

Oracle GI/DB homes

Bare Metal / Physical Database Server

Exadata (Linux, fw)

Oracle GI/DB homes

No change to Storage Grid, Networking between OVM and Bare Metal

OVM On Exadata Software Architecture Comparison


OVM Patching on Exadata • Must maintain patching on both Dom0 and DomU • Grid Infrastructure and RDBMS patching is handled in the same manner as non-virtualized system

• Exadata Software updates on DomU may include updates to both, Exadata Platform Software and OS or it may include update to any one component

• Dom0 only needs Exadata Software Updates

41

OVM On Exadata Maintenance and Patching

New York Stock Exchange Oracle Exadata – Our Journey

Ajit Singh, Director, Data Technology Platforms

New York Stock Exchange (NYSE) and Intercontinental Exchange (ICE)

NYSE - world’s largest stock exchange by market capitalization ($28 trillion)

NYSE is owned by Intercontinental Exchange (ICE) - Architects of the world’s markets

11 exchanges, 7 clearing houses 12,000 + listed contracts and securities

Diverse markets span futures & options on interest rates, commodities, indexes and FX, as well as equities and equity options.

We transact on average in <200 mics.

DATA ENVIRONMENT

November 17, 2015 44

Data volume: ~50-60 billion transactions per day (at peak)

Data size: ~ 15 TB of data per day (at peak)

All data for equities, ETFs, options markets and the feed to the street

Data for trading system latency, other markets

Data services responsible for managing all the data and data platforms including Exadata

Oracle Database environment on Exadata


Oracle - primary transactional database platform supporting all NYSE businesses

OLTP and Data Mart environments - 250+ databases, over 500 instances

Total database size - 350+ TB

Daily transaction volume - ~1000 million transactions

Large transactional databases: ~10-20 TB databases generating about ~ 300 GB of log per day

Large Data Mart: >100 TB in size (after HCC compression for historical data)

CONSOLIDATION TO EXADATA

2004: 2006: 2014:

17 Exadata – 12 for NYSE

NYSE Exadata footprint

EXADATA FOR CONSOLIDATION


Some examples: Dev: ~60 databases consolidated on a full rack of X3-2 with an average size of 1.6 TB QA: ~70 QA consolidated on to a full rack of X3-2

with an average size of 2 TB

Prod: ~ 50 databases consolidated on a full rack of X3-2 with an average size of 9.5 TB Mix of OLTP and data marts

Leveraging exadata for performance


Performance gains of 2x – 25x

Trade Reports database – 180,000 messages/sec - ~ 50% increase in processing speed

For the same application, completion time decreased by 66%

Data Mart - daily transactions of 2 TB applied to standby in real-time

Data Loads – processing times decreased by 50%

Data Mart – hybrid columnar compression - 4x compression gain vs ACO. 15-16x compression achieved on raw data using HCC

Full Backup time of a ~150 TB data mart reduced by 85% (over Infiniband switches)

Utilize SmartScan capabilities

Migration


Consolidation planning

Database load compatibility analysis setup appropriate resources – parameters, limits

Migration to Exadata using Data Guard, RMAN Closing and RMAN Duplicate

Plan hardware - Dedicated network

Test it out – connectivity, test with application and RAT testing for performance validation

CONSIDERATIONS


Coordinate downtime

Risk Mitigation: Manage capacity Oracle Homes – 2 Load Isolation Instance caging, IORM deployed to manage resources consolidated

Security isolation

Rolling upgrades of databases

Rolling Exadata bundle patches – cell servers, components, compute nodes and clusterware

Observe utilization Re-evaluate database parameters

Reduced time to market

Platform as a service

Database as a service

Reduced cost Hardware, licensing operations

BENEFITS OF CONSOLIDATION


Documents

Techniques and Best Practices - Oracle · An example of light consolidation would be something like: "I want to consolidate 30 databases, but I also need a few cores to run some ETL,