Exalytics @ WellPointThe Fastest Essbase, Hyperion Planning, and

OBIEE Performance Imaginable.

Jonathon Eastman

WellPoint, Inc.

Introduction. Why Oracle Exalytics. Business Case. Extreme Performance Infrastructure. Lessons Learned… including performance. Questions???

AGENDA

Health Benefits Company

37 Million Members Q1 2014

$71 Billion Revenue 10K 2013

≈ 50K Associates and Contractors.

About WellPoint

Jonathon Eastman, Director EPM/BI. Center of Excellence – EPM.

● Cross functional team of CPAs to technical experts.● We manage system tools and processes critical for

the forecasting and budgeting, internal management reporting, and external GAAP and Statutory Reporting functions.

We report to CAO even though we perform many functions traditionally considered “IT”.

About Me And Our Department

Oracle EPM @ WellPoint

Uber reliant on Essbase. 2500 named Essbase users. 40 Reporting Cubes, 500+ Production Cubes. Reporting Tools.

● Financial Reporting/Workspace● Excel Add-In/Smart View.● Interactive Reporting (Dashboards).● OBIEE with mobility.

Hyperion Planning. 300 users. EPM 11.1.2.3.500 OBIEE 11.1.1.6

EPM @ WellPoint

interRel Consulting. WellPoint Consultants

last 7 years. We won the Oracle

Innovation award together @ OpenWorld 2013 (EPM/BI).

WellPoint’s Partner.

Why Oracle Exalytics – Part 1.

Over 1000 pages. Expected acquisitions and new business

requirements. More data, metadata, and processes.

Aging and complex infrastructure.

13 AIX servers and 5 windows servers.

OS Patching and upgrades costly and time consuming.

Why Oracle Exalytics – Part 2.

Why Oracle Exalytics – Part 3.

Tested, Certified, Packaged, Deployed, Upgraded, Managed, and Supported Together.

AIX Last Tested, Last Released.

Performance!!!!

Business Case

Align with an initiative.● Amerigroup acquisition for initial 3 Exalytics boxes.● ACA Exchanges for 4th Exalytics box.● Easier to find budget.

Replace aging infrastructure.● Easy way to get concrete numbers.● Priced internally against WellPoint Standard

Offerings.● We did not shop numbers against outside vendors.

WellPoint’s Business Case – Part 1.

Roughly 25% cost savings. Can’t release hard numbers. Includes new software

● Converted to BI Foundation Suite ● Received Mobility and others.● Majority of our cost.

Better Servers.● More CPU (80 cores to 120 cores).● More RAM (640 GB to 3 TB).

WellPoint’s Business Case – Part 2.

Savings were enough to justify. Normalize server CPU/RAM specs. Take out the software portion. Apply performance ratio to the cost. Administrative savings.

● Server consolidation (patching, upgrades, monitoring, etc.).

Subjective savings● Better lifecycles due to performance improvements.

WellPoint’s Business Case – Part 3.

Extreme Performance Infrastructure.

WellPoint’s New Infrastructure – Part 1

F5 – to load balance for to EPM web applications (Round Robin).

3 Exalytics X2-3 boxes (No Flash Storage).

ZFS storage device.● EPM web application

share (RM1, LCM).● Essbase App Directory

WellPoint’s New Infrastructure – Part 2

Added 4th Exalytics Server.

Oracle VM. DEV, TEST, and

PROD Fully utilize box.

Lessons Learned

Budget correctly. Transceivers do not come with the Exalytics

boxes.● 10G networking (We did not use Infiniband).● Required on the switch side and on the server side.

Fiber Channel cards and transceivers do not come with the ZFS appliance.

Virtualization and ASR (automated service request) require commodity OEL box (linux).

Load balancing/Fail over potential costs.

Lessons Learned – Procurement Part 1.

Lessons Learned – Procurement Part 2.

Who does What?

Lessons Learned – Procurement (RACI)

Activity WLP IT WLP FDSM WLP IBM OracleGovernance R I C CUser Provisioning R I    Backup & Restore A A R  Job Scheduling A A R  EPM & BI Suite Application Admin & Support I R    

EPM & BI Suite Application Installation

I R    

Exalytics Installation and configuration.

A A A R

Monitoring   A C  RGeneral Administration I A   RProactive OS & Firmware Patching

A A   R

24hr Hardware Failure ResolutionA A   R

         

Power, Networking, Disaster Recovery.

Final Roles and Responsibilities.

Hardware failure.

Information Technology – Governance, User Security.

EPM CEE – Everything EPM, Associate Support

Exalytics OS Patching, admin support.

Not Like Apple computer all ready to go. Now ships with Oracle Linux (5.5) pre-installed. Oracle Virtual Linux

now option as fixed price install. Done on-site. 1 week to install OBIEE/Time Ten on three Exalytics boxes.

Need to fill out Pre-Installation Questionnaire. Host Names, static ip addresses, DNS Servers, gateways. Logistics and Facility access. Database configuration.

Create SS Schemas, Temporary dba access to run OBIEE RCU. Port Configuration.

Oracle ACS performs installation. OBIEE, TimesTen, Essbase, but not all of EPM. (Like Planning)

Lessons Learned – Installation.

Exalytics comes with HugePages memory set really high to support TimesTen Oracle In-Memory Database.

Essbase/EPM does not use and it can potentially degrade performance.

Lessons Learned – Configuration Part 1

Serious Error~(1013000)~Unable to Create Request Server Thread

JVM hs_err_pid.log Check pid_max for linux OS.

● cat /proc/sys/kernel/pid_max● Output: kernel.pid_max = 32768

Change two ways.● sysctl -w kernel.pid_max=4194303● echo 4194303 > /proc/sys/kernel/pid_max

Lessons Learned – Configuration Part 2

500 Gigabyte linux tmpfs for TimesTen.● This in combination with Hugepages greatly reduces

available memory.

Load balancing requires a shared drives for LCM and RM1 directories.● Oracle Clustered file systems supported.● SAMBA/NFS not supported.

If using a ZFS, need technical expertise configuring Multipath.

Oracle VM – Guest Size Limitation (Patch Soon).

Lessons Learned – Configuration Part 3

Lessons Learned – Performance!!!

Benchmark Hardware IBM Power 595 (9119-FHA)

32 Core 64-Bit Power6 Processers running @ 5.0 GHZ.

4 cores dedicated to Virtual IO. Split into 4 separate LPARS 96 Gigabytes RAM 2 Terabytes EMC Storage AIX 6.1

ASO Reporting.● 9 Dimensions.● 4 Attributes.● 35K members

BSO Calcs.● 11K Block Size.● 8 Dimensions.● 5 Sparse.

Our Benchmark Process

Saw over 2-3X improvement across the board.

Serial calculations represented roughly 200,000 lines of calculation script code.

Export = 2 Gigabytes @ level 0.

Lessons Learned – BSO Performance.

Lessons Learned – BSO Performance

Optimizing data cache and index caches improved performance.

As today, must empirically optimize cache settings on a per application basis.

Performance could get worse if cache is set too high.

Buffer loads saw a 2X increase. Already loading in memory.

Commits and aggs performed 3-4X better

Setting the Pending Cache Size Limit improved performance. Set too high and performance went down.

Lessons Learned – ASO Performance

MDX Query times had the biggest variability improvement – from 2-8x better

95% of our queries are fast, so we looked for large dynamic Hierarchies with lots of formulas for test.

2-4x performance improvement report batch bursting in workspace.

Lessons Learned – Query Performance.

Local storage cannot handle multiple simultaneous batch processes (non-flash).

WellPoint originally looked at volatile RAMDISK before san was supported.

ZFS performance equaled RAMDISK. Running 4 calc dims at once with 200,000 lines of

calc script code. (Screenshot to follow).

Lessons Learned – Storage Performance.

Lessons Learned – ZFS Performance.

3000 IOPS 3000 IOPS

Midday Direct Admin and Headcount Batch Job to create two ASO databases. 5X Faster.

Midday P&L, Balance Sheet, Cash Flow, statistical metrics. Many bso calculations exported into an aso reporting cube. 2X Faster.

Nightly Batch process for all our “GAAP” databases. Cost Allocations, Direct Admin, Headcount, Enrollment, Quality Improvement, MLR Rebates, MLR Segments, P&L. 2.5X Faster.

Lessons Learned – Batch Performance.

Performance hit over our x2-3 servers connected to ZFS.● XEN overhead.● Exalytics X3-4 come with 22 93G flash drives.● LVM (Logical Volume Manager)

Oracle ACS coming by end of year to tune linux and zfs.

Contact me if you curious about the Oracle’s hypervisor’s impact on performance.

Lessons Learned – Virtualization Performance.

Can you realize even better gains? Hardware

CPU – Number of cores, Processor Speed, Cache, etc…. Hard Disk IO – Stand alone disk, performance RAID, Storage Area Networks

(SAN), RAMDISK.

Cube Design – Index, Structure, and Cache optimization. Script optimization – Calculating the minimum data

necessary, utilizing CPU efficiently, etc… Load Optimization – Buffers, parallel loads, etc…. Batch process optimization – Take advantage of parallelism.

Lessons Learned – Performance Variability.

Great Performance compared to high-end hardware 2-3x with cost savings.

You can see even better gains if using lessor hardware.

Exalytics has some configurations slanted towards OBIEE that may need to be altered if you focus more on EPM.

You still need to optimize you application caches, outlines, and data loads.

Takeaways.

Questions?

Feel free to contact me.● Jonathon.eastman@wellpoint.com