Oracle Open World September 28 – October 2, 2014 Jim McKinstry Oracle Consulting Practice Director Bill Callahan Director, Product and Technology

Oracle Open WorldSeptember 28 – October 2, 2014

Jim McKinstryOracle Consulting Practice Director

Bill CallahanDirector, Product and Technology

2

©2014 CCC Information Services Inc. All Rights Reserved. Confidential and Proprietary.

MAA Overview Who is CCC?Initial State ArchitecturePhase 1: Introduction of Engineered Systems Phase 2: Current State ArchitecturePhase 3: The Future

Agenda

3



Agenda

4


Oracle Maximum Availability Architecture (MAA) TiersAvailability Service Levels for Unplanned and Planned Outages

BRONZE

SILVER

GOLD Comprehensive HA and Disaster ProtectionZero or near-zero data loss

High Availability (HA) for Recoverable Local Outages Data protected as of last backup

Single Instance Database, Basic Service Restart Data protected as of last backup

PLATINUM Zero Outage for Platinum Ready ApplicationsZero data loss

5


Oracle Engineered Systems significantly simplify the architecture with pre-built and pre-tested configurations.Oracle Engineered Systems provide no single point of failure architectures.Oracle Maximum Availability Architecture ensures best practices to minimize or eliminate unplanned and planned downtime (including rolling upgrades).Oracle provides bundle patches that are pre-tested to minimize the risk of downtime.

MAA provides the performance and scalability to support CCC’s growth.

Oracle Engineered Systems: Increase availability Decrease TCO Increase scalability Increase performance

Benefits of MAA with Engineered Systems for CCC

6



Agenda


AUTO PHYSICAL DAMAGE—INSURANCE

CASUALTY

EVALUATE AND REVIEW

first partybill review

third partybill review

RepairConsumer New Opportunities

REPAIR

INTAKE

dispatch

APD—Insurance IS & AIS Capabilities

intelligence

OPTIMIZE

auditworkflow

strategicparts

sourcingestimatingtablet

estimating

EVALUATE AND REVIEW

predictivecausation

injurycausation

INTAKE RESTORE AND RESETTLE

medicaladvisoryservices

injuryevaluation predictive

OPTIMIZE

analytics

partsprocurement

repairworkflow

REPAIR STATUS

status

predictive open shop

INTAKE

salvage

RESTOREAND

RESETTLE

consultingintelligence

OPTIMIZE

dispatchtotal loss

guided touraudit

workflownext gen

scheduling

strategicparts

sourcingestimatingtotal loss

next

EVALUATE AND REVIEW

Who is CCC?

8



Agenda

9


CCC 2011 Initial Environment

Oracle/SUN M5000 and M4000 ServersCold Failover (4 hour RTS/? RPO)Oracle 10g Database Single Instance

Block SAN Storage30TB Combined storage across 20 databases45TB for Claim Folder/EDS

Performance IssuesRegular Performance disruptions caused by SAN Contention Reprocessing of customer transactions could

take as much as 24 hours

Customer Portal Experience – Poor Some queries related to search took 15-20

minutes – resulted in browser timeouts

10


CCC 2011 Initial Environment (cont’d)

Data WarehouseRedbrick (10TB On Disk Footprint)ETL/Reports 6+ hoursMicroStrategy – 30+ minutes

Web/Application LayerOracle/SUN SPARC servers (V490, E2900, M4000) Oracle 10g SOA/OC4J Applications

DR Using SAN ReplicationRepurposed Load Test Systems at DR Site for DR

11



Agenda

12


Exadata Implementation 2012Migrated – X2-2 Full Rack in Prod and

DRClaim Folder7 SOA Dehydration StoresData Warehouse8 other application databases

RAC for HAImplemented Services for quick failover

Storage reduction – 30TBImplemented Secure File LOBS

Reduced Application incidents by 60% where IO had been a bottleneck

Storage Offload (Automatic!)Portal Search < 30 seconds.

Data Warehouse1.8TB On Disk – With HCC!ETL/Reports < 1 hrMicrostrategy - < 5 min

DR Oracle Dataguard

Exadata X2-2 Full Rack

Exadata X2-2 Full Rack

13


DR – Exadata databasesDatabase recovery time for DR reduced from 4 hours to 30 minutes Recovery resources reduced from 3 DBA team members to 1 DBA

Patching Goal was to get to Platinum Support CertificationFull Frame (Storage Cells, CRS, RDBMS, IB) requiredPainful – requiring a 16 hour down time schedule Bug in CRS prevented rolling CRS Patching – Caused random nodes to

restart Reluctance to perform Cell Patching rolling – due to single frame in

production

Paid Off – It wasn’t for nothing! Control File Caching to Flash Cache – Huge relief in log file sync waits Significant buffer busy “global transaction manager” improvements

We need a local standby

Year 1 with Exadata

14


Half Rack Exalogic980+ Application EARs Migrated off of OC4J

60% overcapacity for Redundancy

Eliminated 30 SPARC Servers in Production

Implemented SDP over Infiniband for communication to Exadata Databases

40% Improvement in application Performance

DR ZFS Storage ReplicationEnables rapid provisioning of Prod Images over Load Test nodes

Exalogic Implementation

15



Agenda

16


Local DataGuard Physical StandbyX3-2 Full Rack – Used for PrimaryImplemented in 30 days

RTS and RPO Improvements30 minute recovery (full frame switchover)<15 minutes RPO

Patching ImpactStandby first Patching optionAchieved Platinum Support Certification

Performance20% improvement in CPU utilizationEliminated remaining IO contentionBackups run from Standby (Must have Active DG)

DR Oracle Dataguard

Exadata Implementation 2013

Exadata X2-2 Full Rack Exadata X2-2 Full Rack

17


Year 2-3 with Exadata

Platinum Services - PatchingDirectly coordinated patching with Platinum SupportPlatinum will make you plan, this is a good thing.

Patching First half was painful again, but in a different way– We had a schedule, but did not complete platinum fast enough to

schedule them!– They worked hard to support us, but the last minute details stung.– Local Standby masked all of it from our customers.

Second half (Happening right now!)– Planning since May– Plans have been set since 3 weeks after July Patch set– I am here, not there, which has to tell you something.

18



Agenda

19


Years ? with Exadata – What is next?

Storage! - We have grownClaim folder is 104TB now

Standbys Improving RTS– Considered Logical Standby – but too many unsupported datatypes.– Tuning Data Guard to bring switchover down to 10 min.– Automating steps – Our Offshore will execute our next DR

Weblogic Active Grid Link– Implemented in phases since January 2014– Should finish the across the board implementation as final apps

move to Weblogic

21


Thank You

Documents

Oracle Open World September 28 – October 2, 2014 Jim McKinstry Oracle Consulting Practice Director Bill Callahan Director, Product and Technology