EXADATA CLOUD MACHINE - s3.amazonaws.com · what’s it really like? andy colvin accenture enkitecgroup rmoug 2018 exadata cloud machine

WHAT’S IT REALLY LIKE?

ANDY COLVINACCENTURE ENKITEC GROUPRMOUG 2018

EXADATA CLOUD MACHINE

ANDYCOLVIN

Professionally• 10+ years at Enkitec/Accenture• Oracle ACE• Co-author of Expert Oracle Exadata• Background in systems, network, DBA

Unprofessionally• Husband / father / travel agent• Arsenal supporter• Beer snob• BBQ aficionado

Twitter - @acolvinEmail - [email protected] - blog.oracle-ninja.com

EXADATA CLOUD OFFERINGS

Copyright © 2017 Andy Colvin. All rights reserved. | 4

EXADATA CLOUD OFFERINGSWHAT OPTIONS DO I HAVE?Exadata now comes in 3 major flavors

TraditionalHosting

Exadata Cloud Service

Exadata Cloud @ Customer


EXADATA CLOUD OFFERINGSTRADITIONAL EXADATA Exadata exactly the way you’ve known it• Traditional “on premises” Exadata

• Standard Exadata, the way it’s been done for years

• You own the asset, and can manage it to your needs• Hire partners for managed services• Install additional software


EXADATA CLOUD OFFERINGSEXADATA CLOUD SERVICEExadata for the Cloudy McCloudface folks• Exadata running in Oracle's public cloud

• Great if your applications are already in Oracle's cloud

• Administrative tasks are performed via Oracle's cloud console or API• System deployment• Database creation• Configuration of backups• Patching

• Equipment is owned by Oracle• Exadata resides in an Oracle datacenter

• Supports either OCI or classic interfaces/APIs


EXADATA CLOUD OFFERINGSEXADATA CLOUD MACHINEJust like Oracle cloud – in your datacenter• Exadata running in your datacenter

• If your applications can't move to public cloud, this is a solution• Assets are owned and maintained by Oracle

• Includes all benefits of Exadata Cloud service – in your datacenter• Subscription pricing

• Bursting

• Scalable

• Inexpensive cost of database management packs

• You *do* give up some control by choosing Cloud @ Customer• Only Oracle has access log in to physical equipment (cells, compute nodes, switches)


EXADATA CLOUD OFFERINGSCOMPARING TRADITIONAL VS CLOUD

Hosted Exadata• Owned – capital expense• Customer supports entire stack• Elastic configurations• Bare metal or virtualized

configuration• Extreme Flash or High Capacity

storage• Monitoring / alerting performed by

Oracle Platinum Services

Exadata Cloud• Subscription model• Oracle manages infra components• Cookie-cutter sizes• Only supports virtualization – single

cluster• Only high capacity storage is offered

at this time• Monitoring / alerting performed by

Oracle Cloud Operations

EXADATA CLOUD MACHINE – ACCESS AND MANAGEMENT


EXADATA CLOUD MACHINEORACLE CLOUD CONTROL PLANEOne small thing….• Exadata Cloud Machine requires the Oracle "Cloud Control Plane"

• The control plane is what makes an Exadata Cloud Machine "cloudy"

• This is provided via an Oracle Cloud Machine ("OCM")• The OCM is the mechanism for providing billing/utilization metrics, identity management, API

access, and virtual networking management

• One OCM can manage multiple Exadata Cloud Machines in a single metro area

• If you are running geographically distant sites, each site needs an OCM

• OCM gives access to all of the remaining Oracle Cloud offerings• OCM can be used to provide IaaS, database cloud service (non-Exadata), object storage, and

more


EXADATA CLOUD MACHINEORACLE CLOUD CONTROL PLANEConnecting the control plane• OCM connectivity can be a challenge

• The control plane is what makes an Exadata Cloud Machine "cloudy"

• This is provided via an Oracle Cloud Machine ("OCM")• The OCM is the mechanism for providing billing/utilization metrics, identity management, API

access, and virtual networking management

• One OCM can manage multiple Exadata Cloud Machines in a single metro area

• If you are running geographically distant sites, each site needs an OCM

• OCM gives access to all of the remaining Oracle Cloud offerings• OCM can be used to provide IaaS, database cloud service (non-Exadata), object storage, and

more


EXADATA CLOUD MACHINEADVANCED SUPPORT GATEWAYOracle connects via a support gateway• The Advanced Support Gateway ("ASG") allows Oracle access to your

network• The ASG serves the same purpose as a platinum gateway in a traditional hosted environment

• All traffic in and out of the ASG is encrypted with SSL• The ASG logically resides in a DMZ network range, with specific firewall

ports open externally and internally• This 1U server should be placed in a rack outside of the OCM and ECM

frames• You will need 1 ASG per location


EXADATA CLOUD MACHINEMULTIPLE SITES IN A METRO AREA

DallasDC1 DC2

New YorkDC1


EXADATA CLOUD MACHINEMANAGEMENT MODEL

domUCompute Node

1(VM)

dom0Compute Node

2

Exadata Storage Server n

Exadata Storage Server 2


IB Switch 1 IB Switch 2

dom0Compute Node

1

domUCompute Node

2(VM)

fs

hr

dw

Exadata Cloud Machine

Managed by Oracle Cloud

Operations

Managed by Customer


EXADATA CLOUD OFFERINGSADMINISTRATIVE ACCESS

ComponentType

CustomerAccess?

OracleAccess?

Database Yes No

Compute Node Virtual Machine Yes No

Compute Node Hypervisor No Yes

Exadata Storage Servers No Yes

InfiniBand Switches No Yes

Management Switch No Yes

Power Distribution Units No Yes

EXADATA CLOUD MACHINE –CONFIGURATION


EXADATA CLOUD OFFERINGSRACK CONFIGURATIONS

Eighth Quarter Half FullCompute Nodes 2 2 4 8

OCPU 16-68 22-84 44-168 88-336

Max RAM 240GB 720GB 720GB 720GB

Storage Servers 3 3 6 12

Raw Disk 42.7TB 85.4TB 170.9TB 341.7TB

Flash Capacity 19.2TB 38.4TB 76.8TB 153.6TB


EXADATA CLOUD MACHINENETWORK CONFIGURATIONExadata Cloud Machine separates networks by ownership• domU compute nodes use networks routable to customer

• Customer management (administration)

• Database client access

• Database backup

• Storage InfiniBand partition

• Cluster InfiniBand partition

• All other components reside on a private management network• Oracle manages these components, so they aren't accessible to you

• Compute node management / ILOM

• Storage server management / ILOM

• InfiniBand Switches


EXADATA CLOUD MACHINENETWORK CONFIGURATION

domUCompute Node

1(VM)

dom0Compute Node

2


IB Switch 1 IB Switch 2

dom0Compute Node

1

domUCompute Node

2(VM)

Exadata Cloud Machine



PDU2

PDU1

TOR Switches

OCM

Client Switches

Customer AdminDatabase ClientBackupInfiniBandCloud Admin

Networks


EXADATA CLOUD MACHINENETWORK CONNECTIVITY REQUIREMENTSNewer Exadata Cloud Machine models include top of rack switches• High speed connectivity

• Connect with 10GbE or via MPO for 40gbps• 10GbE required at a minimum

• Systems include top-of-rack ("TOR") switches• Reduced cabling to each rack

• Run cables to the TOR switches• TOR switches provide cabling to compute nodes

• Reduced cabling to each rack• Run cables to the TOR switches


EXADATA CLOUD MACHINENETWORK CONNECTIVITY

Sample connectivity diagram• Connections use 4-way

MPO to 10GbE connection• Cables provided by Oracle


EXADATA CLOUD MACHINEDNS CONFIGURATIONDNS is controlled/maintained by Oracle• You are assigned a <cloudid.>oraclecloudatcustomer.com domain

• Your DNS servers must be configured to forward to DNS servers inside of the OCM• You can create your own internal aliases, but the Oracle DNS is the source of truth


EXADATA CLOUD MACHINETAILORING TO FIT YOUR NEEDSAs cloud factor increases, customizability decreases• All database homes must be owned by "Oracle" user

• Problematic when sharing DR and nonproduction• Groundwork is laid by installing GI as "grid" user, so may be supported in future

• Diskgroups must follow a predetermined pattern:• 80% DATA, 20% RECO, 0% SPARSE• 40% DATA, 60% RECO, 0% SPARSE• 60% DATA, 20% RECO, 20% SPARSE – only available in Oracle public cloud• 35% DATA, 50% RECO, 15% SPARSE – only available in Oracle public cloud

• There are also diskgroups for ACFS filesystems and the cluster files• No custom directory structures for Oracle homes – must follow Oracle's

guidelines


EXADATA CLOUD MACHINEIMPLEMENTATION TIMELINEThe cloud is agile, but it rolls in slowly• Plan including Oracle's responsible tasks included 33 steps• By comparison, traditional Exadata plan includes 6 steps

• Most time is spent planning, installing, and configuring OCM• Additional time is needed to add system to Oracle's management infrastructure

• Think of this as doing an entire Platinum Services implementation before using your new Exadata


EXADATA CLOUD MACHINEIMPLEMENTATION STEPSHigh level implementation steps (*planned*)1. Customer orders Exadata and Oracle Cloud Machines2. ECM/OCM/ASG are shipped from the factory3. Customer and Oracle complete PCMA form and OEDA

a. PCMA includes information about cloud and management networks, DNS domain, as well as ASG configuration information

b. Without the PCMA, nothing can be done

4. Oracle takes PCMA information, sends to engineers for deployment5. Field engineers perform installation of ASG, OCM, and ECM

a. Connect racks to power and networkb. Install ASG in separate rack and prepare for connectivity

6. ACS remotely configures OCM and ECM for deployment, databases begin to get created7. Oracle enables service subscription, billing begins

Typical timeframe of 30 days

between steps 5 and 7


EXADATA CLOUD MACHINEPUTTING IT ALL TOGETHER

EXADATA CLOUD MACHINE – A REAL WORLD STORY


CASE STUDYCASE STUDY BACKGROUNDExisting Exadata customer, making the cloud jump• Large direct retail company• Currently owns 2 Exadata racks

• X2-2 quarter rack – development / DR• X3-2 quarter rack – production

• 3 key applications on Exadata• E-Business Suite handling all ERP• Web backend database for public and private sites • Warehouse distribution management system

• Small IT shop• 4 DBAs• 5 Linux administrators


CASE STUDYWHY NOW?Exadata systems were due for a refresh• Deploying a new datacenter in 2017• End of support lifecycle nearing

• Extended support was nearing for the X2-2• Desire to move to newer equipment that requires less maintenance

• No more battery replacements• Worry about flash card EOL

• More powerful hardware in the X6-2• 34X flash capacity and 3X flash IOPS of X2-2• 4X flash capacity and 1.9X flash IOPS of X3-2


CASE STUDYDECISION TIME…To go cloud, or not to go cloud…• Decision was between buying new Exadata racks, or cloud at customer

• Public cloud was not an option due to security requirements and desire to host applications in-house

• Decision was between buying new Exadata racks, or cloud at customer• Public cloud was not an option due to security requirements and desire to host applications in-

house


CASE STUDYDECISION FACTORS

• No difference from current setup

• More flexibility for configuration

• Speedy deployment

• Nothing new to learn

• Great from an accounting perspective

• Access to all licensed options

• Elevated support from Oracle

• They can tell the other groups about how "cloud" they are

• Large capital expenditure

• Pay per licensed option

• Customer owns a depreciating asset

• Very few customers running Oracle cloud @ customer

• Early adopter tax

• More restrictive configuration requirements

• Lack of total control

• Longer deployment model

Hosted Exadata Cloud @ Customerpro

con

WHAT DID THEY DO?!?!?


CASE STUDYTHEY WENT CLOUD


CASE STUDYHERE COMES THE CLOUDPurchase is made in late May 2017• Customer chooses two Exadata X6-2 quarter racks

• One in existing datacenter• One in new datacenter to be built out in mid/late 2017• They see themselves having plenty of lead time for implementation

• They also need a control plane• There is no desire to use it for cloud services, though• All applications are either on VMware or bare metal• Oracle sweetens the deal to include one OCM at each location


CASE STUDYIMPLEMENTATIONHigh level implementation steps (*actual*)1. Customer orders Exadata and Oracle Cloud Machines (May)2. Customer is sent multiple copies of PCMA (June, July)

a. Network and security requirements for the OCM and ASG are discussed for the first time, customer's security team does not initially agree to the security model.

b. Customer doesn't have necessary 10GbE ports in legacy datacenter for OCM and ECM

c. During this time, power for the OCM is changed, no more small single-phase PDUs – customer balks at power cost for OCM, may remove one OCM from the order.

3. Customer and Oracle work PCMA form and assuage security's fears (August)a. Nothing can be done until new datacenter is online – mid-October

4. Exception for smaller PDUs granted, ECM/OCM/ASG are shipped from the factory (September/October)

5. Network team gets new datacenter online (November)6. Installation begins the week of December 4, 20177. Service activated February 2018


CASE STUDYLESSONS LEARNEDWhat have they learned?• Documentation wasn't easy to find

• Look at https://docs.oracle.com/en/cloud/cloud-at-customer for documentation - *not* Oracle Cloud Machine

• Too much new tech at once can be overwhelming• Network and security teams were in the middle of deploying new datacenter when cloud services came in to

play – this created significant delays to the project

• Customer had trouble implementing platinum services before – going with cloud at customer was more difficult

• Would they do it the same way now?

https://docs.oracle.com/en/cloud/cloud-at-customer

WRAPPING UP


I’M GETTING EXADATA CLOUD @ CUSTOMERWHAT WILL I DO?!?!It’s going to take a little while to get started…• Implementation timelines are noticeably longer

• The first implementation at a site requires building out an OCM – this takes time• Think of it as a full platinum deployment *before* you set up the Exadata

• Subsequent machines are deployed very quickly• Similar to a traditional Exadata deployment – provide network and power, do your homework,

and it’s simple!


I’M GETTING EXADATA CLOUDWHAT WILL I DO?!?!Don’t worry about performance differences between traditional and cloud• It’s still the same Exadata rack

• Fewer cells, but still same performance features

• OVM on Exadata doesn’t show a performance impact• Testing shows that a single OVM cluster performs on par with

• You don’t have to worry about noisy neighbors• Each Exadata (public cloud or cloud @ customer) is dedicated to a single cluster

• Storage servers are not shared

• Compute nodes aren’t shared


I’M GETTING EXADATA CLOUDWHAT WILL I DO?!?!You’ll still have plenty of work to do• Oracle doesn’t manage the database layer

• Oracle has ownership of physical compute nodes, storage servers, and InfiniBand• There’s plenty of room for you to manage what’s inside the VM

• Patching is still a challenge• New tools!• Patching framework is (mostly) automated, but requires frequent updates to the toolkit

(exadbcpatchmulti)• Oracle is backlogged on patching cloud infrastructure – patches aren’t available instantaneously

Documents

EXADATA CLOUD MACHINE - s3.amazonaws.com · what’s it really like? andy colvin accenture enkitecgroup rmoug 2018 exadata cloud machine