Latha Krishnaswamy | Director, Product Management | Exalogic

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Restricted

PeopleSoft High Performance and MAA Guidelines on Oracle Exalogic and Oracle Exadata' [CON7985]

Latha Krishnaswamy| Director, Product Management | ExalogicDennis McCormick, Systems Manager at City and County of San FranciscoAaron DeLosreyes, Director, Oracle Technology, Cognizant

Oct 2nd, 2014

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Restricted 2

Safe Harbor StatementThe following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.


Program AgendaPeopleSoft on Exalogic - Anti-pattern traps to avoid….

What is Exalogic Elastic Cloud Software?

Exalogic Elastic Cloud Software – PeopleSoft Optimizations

PeopleSoft Reference Architecture on Exalogic

MAA Best practices PeopleSoft on Exalogic

Case Studies – City of San Francisco, Cognizant

Demo

Summary

1

2

3

4

5

6

7

0


• Multiple PeopleSoft Environments– Port conflict, session state– Hardware (F5) & Software Load

Balancers (Apache) vs OTD on Exalogic

• Non-separation of JVMs – PIAs for FS, TE, HR – traffic crisscrossing– Integration Broker on same JVM as PIA

• Scheduler logs – For recovery & DR, log file share

locations and continuous replications

• Choice of protocol – Not TCP over JDBC, but rather SDPoIB– JOLT over RDMA - PT 8.53+

• PeopleTools mid-tier– Shared storage use by different VMs

• Patching/upgrading multiple VMs

– COBOL is chatty. Scheduler on Exalogic.

• PeopleSoft Application database– PUM, SYS DB, SES DB, migrate to

Exadata

PeopleSoft on Exalogic: Anti-Patterns Observed…

Let’s begin our journey to embark on best ways to onboard PeopleSoft workload on Exalogic & Exadata…….


Program Agenda

What is Exalogic Elastic Cloud Software?1

2

3

4





Demo

Summary

5

6

7

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | 6

Exalogic Elastic Cloud System Software

Exalogic Control

= Exalogic Elastic Cloud Software

En

terp

rise

Man

ager

CoherenceWebLogic Tuxedo

Exabus Integration

Traffic Director

Exabus Integration

Exabus Integration

Exabus Integration

Middleware and Business Applications

Exalogic Elastic Cloud Hardware

Exabus

Physical Oracle Linux/Solaris

Oracle VM 3 for Exalogic

Oracle Linux Guest OS


PeopleSoft on Exalogic – Building Blocks

Exalogic Control

= Exalogic Elastic Cloud Software

En

terp

rise

Man

ager

Application Server Tuxedo

PIA WebLogicScheduler/ Integration

BrokerExabus

Integration

Traffic Director

Exabus Integration

Exabus Integration

Exabus Integration

PeopleSoft Mid-Tier Deployment

Exalogic Elastic Cloud Hardware

Exabus

Physical Oracle Linux/Solaris

Oracle VM 3 for Exalogic

Oracle Linux Guest OS


Exalogic Elastic Cloud Software Optimizations for PeopleSoft Workloads

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

Tuxedo on Exalogic Increased Performance from Exabus

• Eliminates BRIDGE process as bottleneck

• Enables greater scalability of Tuxedo domains

• 7x throughput• No application changes

needed

TuxedoTransactions/sec.

Standard Hardware Exalogic

7X

13,680

95,595

EECS 1.0

2.0X

StandardHardware Exalogic

Tuxedo Applications

BRIDGE

BRIDGE

dge

Single TCP Connection

Over Ethernet Exabus


Sockets Direct Protocol (SDP) SupportBypasses TCP/IP Stack

Tuxedo Domain 2Tuxedo Domain 2

Jolt Clients

Node 1

Server 1Server 1

Server 2Server 2

Node 2

Server 1Server 1

Server 2Server 2

DB

WebLogic ServerWebLogic Server

Tuxedo Domain 1Tuxedo Domain 1

Node 1

Server 3Server 3

Server 4Server 4

/WS Clients


• JDBC & Data Source Optimizations– Active GridLink for RAC, Socket Direct

Protocol (SDP) for JDBC

• I/O & Work Manager Optimizations– Scattered reads / gathered writes,

optimized self-tuning thread pool

• Cluster State Replication Optimizations– Lazy deserialization, multiple channels

for replication

• SDP Optimizations– Network channels, JDBC connectivity

• Web Application Scalability and Performance Optimizations– Oracle HTTP client connection

management

• Faster Cluster Recovery– Rapid death detection and failover

• Documentation:– http://docs.oracle.com/middleware/1212/core/

FMWLC/products2.htm#CHDGCFDC

Exalogic Optimizations in WebLogic Server

http://docs.oracle.com/middleware/1212/core/FMWLC/products2.htm

http://docs.oracle.com/middleware/1212/core/FMWLC/products2.htm


• Coherence Optimizations– Coherence*Web caching performance

optimizations– Use of optimized point-to-point

communications in Coherence clusters– IMB transport option on Linux

• Documentation:– http://docs.oracle.com/middleware/1212/cohe

rence/COHAG/deploy_checklist.htm#COHAG5300

• Tuxedo Optimizations– Efficient algorithm for Self-Tuning Lock

Mechanism (SPINCOUNT)– Optimized performance of intra-node

communication (shared memory instead of IPC queue, reduced buffer copies)

• Documentation:– http://docs.oracle.com/cd/E53645_01/tuxedo/

docs12cr2/exalogic/exalogic.html#wp1092546

Exalogic Optimizations in Coherence and Tuxedo

http://docs.oracle.com/middleware/1212/coherence/COHAG/deploy_checklist.htm



http://docs.oracle.com/cd/E53645_01/tuxedo/docs12cr2/exalogic/exalogic.html

http://docs.oracle.com/cd/E53645_01/tuxedo/docs12cr2/exalogic/exalogic.html

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal 13

Oracle Traffic Director – OverviewBuilt-in Application Delivery Controller for Load Balancing over InfiniBand

Request, content-based routing Request rate acceleration Request rate & connection limiting Quality of service tuning

Built-in HA on engineered systems Dynamic reconfiguration

HTTP Reverse proxy Support for SSL 3.0 and TLS 1.0 Web-application firewall

Built-in HTTP, TCP load balancing Native Exabus integration SSL & TLS offloading Content caching & HTTP

Compression Throughput

SecurityHigh Performance

Quality of ServiceHigh Availability


Performance Test Results + AnecdotesHigh Performance – PeopleSoft on Exalogic & Exadata

Blog Entry URL: https://blogs.oracle.com/wernerwatschke/

http://moafshar.blogspot.com/2013/10/middleware-optimizations-for-exalogic.html

Commodity


Program Agenda


2

3

4





Demo

Summary

5

6

7

1


PeopleSoft on Exalogic (Virtual) & ExadataReference Architecture

• vServers on Exalogic• Infiniband connection• Shared Storage (ZFSSA)• Common PeopleTools• VM Templates • Multiple PIA - WLS• Multiple AppServers• Multiple Schedulers• Multiple DB (App, SES)• Multiple Workloads• Application Isolation• Better consolidation• Patch/Upgrade faster• Database RAC nodes• ……


PeopleSoft F5, PIA, OTD configuration on Exalogic virtual

• SSL Configured on F5 and OTD• PeopleSoft PIA’s use F5 virtual address for SSL config• PIA, Tuxedo, OTD Admin Node servers deployed on Exalogic Infiniband channel• OTD Failover groups configured on EoIB Channel• OTD used in Active – Passive configuration


PeopleSoft OVM Template for ExalogicSimplified Provisioning, Easy Patching and High Performance

Supports PeopleTools 8.53, 8.54

Rapidly test drive new PeopleTools release features: Fluid UI, Activity Guides

Includes PIA and App Server – Weblogic 11g and Tuxedo 10gR3

Application agnostic – can support any PeopleSoft App Tier

New template for every patch-set simplifying patching going forward

High Performance – parity with bare metal

Complete Instance Management

Reconfigure and save Gold images


Apps Life Cycle on Exalogic - Simplified

• Patch apps using OVM Template Apps Patch sets - save 100s of hours and ensure consistency across App, Middleware and Hardware

• Manage Apps, Middleware, DB, Servers and Storage from single EM console

• Scale Up and Scale down app instances on Exalogic using EM and Apps OVM Templates

• Platinum Services resolve issues faster and reduce risk

Application Operations

EM for Apps

Data Center Operations

- EM for Exalogic

Drill Through Views


Program Agenda


2

3

4


Reference Architecture



Demo

Summary

5

6

7

1


VMs with shared storage on Exalogic virtual Save Time, Eliminate Mistakes, Follow Best Practices

• Shared PS_HOME, PS_APP_HOME on ZFSSA • Facilitates patching & upgrading• Can scale up VMs depending on load

• Typical Issues during Load Test• Port locking• Crisscrossing of OTD failover groups for PIAs

• PeopleSoft MAA Best practices• Clear demarcation of VMs for each PIA• PIA, AppServer per env i.e. FS, TE, HR• Integration Broker on their own VMs• OTD Active Passive setup• Storage Replication

• ZFS storage shares scheduled replication• Scheduler logs continuous replication

• PeopleSoft MAA OTN White Paper

http://www.oracle.com/technetwork/database/availability/peoplesoft-maa-2044588.pdf


Push Button Apps Deployment with ExceleratorsSave Time, Eliminate Mistakes, Follow Best Practices

Test-Dev App Cloud

Production Ready Oracle App per MAA

InfrastructureInstall & Config Directives

App Topology Design Provision VMs Configure App

VMs

Application

Storage Networks

ExceleratorExcelerator

Exalogic

Exadata

ApplicationInstall & Config Directives

Push-ButtonApplication Infrastructure

Service

Delivered as Fixed Price ACS Service

Available Now PeopleSoft Commerce

EBS (H2FY15)

Automated App Install From Days to hours Create Test Dev Cloud Apply MAA Best Practices Reduce manual tasks and errors


Program Agenda


2

3

4


Reference Architecture



Demo

Summary

5

6

7

1

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.

San Francisco’s Exa StorySan Francisco Overview

Exa Upgrade Objectives and Criteria

Exa Upgrade Implementation Timeline

Then and NowPerformanceMaximum Availability Architecture

Looking Back: Challenges and Lessons Learned

Final Thoughts


City of San FranciscoPeopleSoft 9.0 (Ben, HR, TL, Absence, Payroll,eSuites)Original 9.0 Implementation: Aug 201225 Environments on HCM 9.0 and HCM 9.2 (Upgrade)Production= 211 GB (compressed) 6 GB growth

biweekly30k employees, 100,000k covered lives, $100m payrollController’s eMerge (PeopleSoft) Division = 40 FTE

Responsible for application (4 FTE), OS (2 FTE) and hardware Department of Technology: =200 FTE

Responsible for database, storage, network and datacenters


Upgrade Project Objectives 1. Refresh Hardware2. Move to new Production Data Center3. Move to new Disaster Recovery/Test/Dev

Data Center4. Simplify Architecture


Review and Analyze Procure Install & Configure Test & Develop Deploy OptimizeMay-13 Jun-13 Jul-13 Aug-13 Sep-13 Oct-13 Nov-13 Dec-13 Jan-14 Feb-14 Mar-14 April-14 May-14 June-14 July-14 Aug-14 Sept-14

San Francisco’s Exa Implementation TimelineInception

Excite Process KickoffMay 9th

Install at New Test/DEV/DR Data Center

Feb 10-21st

UpgradeExadatas to 1/4 rack

Exalogic new 1/8 rack

Vendor Selected

November 5th

Install at New Production Data Center

Dec 12th -20th

RFQ IssuedOctober 25th

Go-Live June 6th

Apply April PatchMay 19-22nd

Apply July PatchAug. 15th-28th

Infrastructure

• Moved to 2 new data centers• Built Test/DEV/DR from floor-up• Established connectivity

Datacenter Lease Signed

July 10th

1G Connection Established

Jan 17th

Operating/Database System

• Upgraded from PT 8.5.0 to 8.5.3• Implemented RAC• Implemented Active Dataguard• Expanded OEM usage• Implemented GoldenGate

Hardware

• 2 1/8 rack Exadatas• 2 1/8 rack Exalogics• UCS for Windows


Reference ArchitectureSimplify the visio!


Hardware Refresh: Key Criteria1. Create a stable, scalable base for future work2. Increase performance 3. Implement MAA/DR4. Reduce Total Cost of Ownership: How?

Simplify architecture across data centers: reduce training timeIncreased automation via OEM: more time for value added workOne vendor: less time troubleshooting Increased performance: less overtime, more time for Q/A

Other options considered: new blades, Vblock, hybrid model


Challenge Recommendation Oracle Stated Benefit CCSF Observation

Complex Architecture Database Tier

Install Exadata Engineered Solution lower cost and risk

Has been met or exceeded

Complex ArchitectureApplication Tier

Install ExalogicEngineered Solution with predefined PeopleSoft templates

Has been met or exceeded, some issues with upgrade templates

Live Reporting & Robust DR

Install Golden GateActive DataGuard

Fast and Flexible Data Replication

Has been met or exceeded

Single Points of Failure

Install Real Application Clusters (RAC)

High availability reduce risk

Has been fully met

Staff overloadLeverage Enterprise Manager

Automated proactive infrastructure monitoring, Mgmt, and diagnostics

In progress, so far so good

Staff overload Leverage PeopleSoft Mgmt Pack

Comprehensive, application performance & config Mgmt

In progress, so far so good

Updating/Patching system Oracle Support

Removes complexity and transfer responsibility to Oracle from project team

Has been met, some issues with front line team


MAA: Highlighted ExamplesRolling patching: no overtime and no outages for end users

Failover: DR test was planned and implemented in a few hours, SunGard test took months of planning time

Replaced parts in production chassis during workweek

Exachecks: proactive monitoring, great Oracle Support

Use of ZFS Exalogic Shares via Vservers to manage images of PeopleSoft HOME & Oracle HOME: makes everything easy


Maximum Availability

Component Then Now Comment

RAC No Yes Unable to implement on blades

ExaCheck MAA Report

No Yes Regularly Run and track faults/warnings lets us be proactive

ActiveDataguard

Yes Yes With PeopleTools 8.53.09 have secondary to keep online sessions active

Templates No Yes Night and day difference in server creation turn around times hours not days or months with full procurement

Patching No Yes Ability to patch during workweek and transfer responsibility (hardware/OS)


PerformanceMore than just run time improvements….

Trouble shooting is much faster and smoother

End User Experience: “the pages jump at you”

Templates: spun up PeopleSoft 9.2 Environments in ½ a day, previous lifecycle would have been months or weeks

Increased from 1,000 end users in June to 3,00o today, with plans to grow to 30,000 in early 2015


Highlighted Tasks Benchmark time

Exadata time

Time Gained % change

Create Pay Sheet 14 4 10 250%Load Time Data 45 25 20 80%Prelim Calc # 1 48 20 28 140%Pre Confirm Audit Report 27 3 24 800%Pay Confirmation 62 21 41 195%Absence Process (Entitlement-Calc) 22 10 12 120%Create Pay Checks (XML Testing) 10 5 5 100%Create Pay Advice (XML Testing) 48 25 23 92%Distribute cost of Time & Labor 38 30 8 27%

CCSF Payroll Register report 73 18 54 304% Retirement services files 39 24 15 62% Talx E-Pay 109 36 72 196% Talx Unemployment file 20 11 9 75%

Performance Improvements


Time Savings7 hours

5.5 hoursPayroll Processing

Performance Improvements

1.5 hoursDataGuard Backup

Payroll Processing Timeline* Tues Wed Thurs

Before Hours 6 10 8

end time 7:00 PM 9:00 PM or later 5:00 or later Tues Wed Thurs

After Hours 5 8 5

end time 6:00 PM 6:00 PM or earlier 3:00 PM*excludes application and database processes that run after payroll.

Payroll Processing Improvements: how we used the 7 hoursReturn to more normal work life balanceMore time for quality assurance and data entry tasksWill return slack time to end user departments: goal is 1:00 PM

on Thursday


Looking Back….Recommendations…

Plan for growth from day oneTraining should be “just in time” hard to relate to scale of

changeDon't deviate from best practice; listen don’t do it

the old wayDevelop documentation and procedures that work for your

organization there is no one size fits all Oracle to do…

Improve documentation for first timers 800 page user manuals are too dense: need quick

reference guidesImprove Platinum Gateway experience Issues with 8.53.10 template during initial move and build


Final ThoughtsAre we confident we made the right decision? YESWhy?

Total Cost of OwnershipScalability: ability to scale up and scale down is a game

changerOne vendor is really, really nice

What’s next?Purchased two new Exadata 1/8 racks and new

Exalogic 1/8 rack for Oracle Identity Manager and other applications

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.38

Run Better. Run Different.

Maximum Availability Architecture at Cognizant PeopleSoft SystemsAaron De Los Reyes Sr. Director, Oracle Technology Cognizant, Inc

Oct 2, 2014

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.39 | ©2012, Cognizant

39

Global

Large

Critical

Cognizant’s

PeopleSoft ESA, Fin, HR on Exalogic MAA

And, got problematic

• Peak time slowness

• More and more servers

• Lack of ability to scale

• Longer maintenance window

http://www.google.co.in/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&docid=TJ_FdgOwzBNHSM&tbnid=pHu2B_wG74AA-M:&ved=0CAUQjRw&url=http://www.globalyoungacademy.net/projects/glosys-1&ei=6VYIVMrDCdGSuATlw4GABA&bvm=bv.74649129,d.c2E&psig=AFQjCNGMt9wwrQhwRHBw22t517IlBlZrYQ&ust=1409918974693830

http://www.google.co.in/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&docid=TJ_FdgOwzBNHSM&tbnid=pHu2B_wG74AA-M:&ved=0CAUQjRw&url=http://www.globalyoungacademy.net/projects/glosys-1&ei=6VYIVMrDCdGSuATlw4GABA&bvm=bv.74649129,d.c2E&psig=AFQjCNGMt9wwrQhwRHBw22t517IlBlZrYQ&ust=1409918974693830

Copyright © 2014, Oracle and/or its affiliates. All rights reserved.40 | ©2012, Cognizant

40

Benefits

50% improved downstream integration processing

Server consolidation – 5:1

Maintenance window reduced to few hours

Support effort down 50%

Better CPU utilization – 3:1

Solution?

Exalogic upgrade on web layer

Oracle Exalogic

Lintel / Weblogic

| ©2012, Cognizant 41

Didn’t quite stop there…Performance woes

- 10-15 slow incidents/month

- Batch failures / over runs

- Slow online pages due to DB2 table locking

- Long running SQLs

Inability to scale for 2015+ volume

3 Options, 1 Pick

DB Layer upgrade to Exadata

Oracle Exadata

DB2/AIX

| ©2012, Cognizant 42

Implementation, Outcomes & BenefitsStrict timelines

Conversion of ~6,000 SQLs

Constantly changing production system

Only <3 days available for cut over

Full DR

60 hours cut over

DR upgraded same time as production

No major post production issues

– Financials close batch jobs complete 56% faster

– Payment batch jobs run 93% faster

– Customer billing batch jobs run 55% faster

– RTO for DR down rom hours to near-zero

– Transactions with <1s response 75% from 30%

– Batch window reduced to ~4 hours from 10+ hours

– Huge floor, and power saving

– 24x7 availability

43


Cognizant’s Application Landscape

Cognizant’s PeopleSoft Implementation for Corporate functions include -

PeopleSoft Financials/Enterprise Services Automation - GL, AP, AR, AM, CM, PO , T&E, CA, BI, RM, PC

PeopleSoft HCM – Core HR, Absence Management, Time and Labor, Payroll, ePerformance, Self-Service, Workforce Administration, Recruitment, eCompensation, Profile Management

PeopleSoft ELM for Cognizant Academy

User base – 180K , World wide

Database size : ESA – 2 .5 TB, HCM – 1 TB, ELM – 800 GB

Peak user load: ESA – 15K bi-weekly, HCM – 40K semi-annually, ELM – 10K (irregular peaks)

PeopleSoft Suite

44


Architecture

Primary:

App Tier: ½ Rack Exalogic

DB Tier: ¼ Rack Exadata (High Performance)

Stand by:

App Tier: 1/8th Exalogic

DB Tier: ½ Rack Exadata (High capacity)

Applications:

PS HCM 9.1, PS Financials 9.1, PS ELM 9.1

Environment Management

Oracle EM for PeopleSoft

Applications Monitoring for Health & Perf:

Oracle EM for PeopleSoft and Oracle Real User Experience (RUEI)

Technology ComponentsDeployment for 3 Applications (Financials, HCM, ELM)


Maximum Availability Architecture: PeopleSoft Deployment at Cognizant

45

SLA: 99.9% uptimePrimary Site: MAA

Tier HA option

Weblogic Cluster/Distributed membersPIA Distributed across nodesTuxedo (App) Distributed across nodes

Tuxedo (IB)Master/Slave with distributed domains

Tuxedo (Batch) Distributed across nodesDatabase Cluster/Distributed members

Stand-by Synchronization

Tier HA optionApp Tier Exalogic to Exalogic using ZFSDB Tier Data Guard RPO Near ZeroRTO Less than 2 hours


Maximum Availability at Cognizant

46

Tier Server High Availability Type

PeopleSoft Web Tier Exalogic

Weblogic cluster Load Balanced across multiple nodes Coherence for session replications

Active/ActivePeopleSoft App Tier Exalogic Load Balanced across multiple nodes Active/ActivePeopleSoft DB Exadata Oracle RAC Active/Active

Stand by Exalogic Exalogic ZFS File System replication Active/Active

Stand by Exadata Exadata Oracle Data Guard Active/ActiveLoad Balancer F5 Cluster Active/Active

Maintenance SLA Type SLA adherence High Availability

EXA Maintenance EXA PatchesRolling upgrade, fully online with no service outage Active mode

Database Maintenance Backup, Stats, PatchesDB Backup is performed at local DG copyand Production is kept online always Active mode

PS Weblogic maintenance Recycing of PIA domains Online maintenance only Active modePS App Server maintenance Recycling of App domains Online maintenance only Active modeApplication upgrades Tools upgrades, Bundles, App upgrades **PeopleSoft Lite for critical functions only **Limited service


Summary• Exalogic is engineered systems platform ideal for PeopleSoft workloads• Software optimizations and shared storage help scale & serve multiple envs• Virtual servers house various components of PeopleSoft – PIA, Tuxedo,

integration broker, process scheduler…• Multiple VMs ensure High Availability & Workload Isolation• PeopleSoft MAA best practices ensure lower Recovery times (RTO)• PeopleSoft Exalogic Virtual Templates available for PeopleTools release• PeopleSoft Excelerator for rapid provisioning using automation


Documents

Latha Krishnaswamy | Director, Product Management | Exalogic