How Cisco IT Migrated from HP Computing Systemd2zmdbbm9feqrf.cloudfront.net/2012/usa/pdf/COCDCT-2765.pdfCisco Public How Cisco IT Migrated from HP Superdome to the Cisco Unified Computing

© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public

How Cisco IT Migrated from HP

Superdome to the Cisco Unified

Computing System

Jag Kahlon, IT Architect, Cisco IT

Sean McKeown, Technology Solutions Architect, Cisco DC Sales

2

© 2012 Cisco and/or its affiliates. All rights reserved.Presentation_ID Cisco Public 3

42


42?a)The answer to the Ultimate Question of Life, the

Universe, and Everything

b)The ASCII value of the asterisk (*) character

c) The size in TB of Cisco IT’s E-Business Suite mission-

critical database

d)The value in billions $USD of business supported by the

above database


Jag Kahlon Technology Solutions Architect for Cisco Data

Center sales

Former IT Director, Data Center Operations @Oracle

15 years of IT experience, 8 in management for Oracle DC operations

Have worn many operational hats:

Server Management

Storage Management

Backup and Recovery

Change Management

Major Incident and Problem Management

Living & Education: Reside in Colorado with wife and two children, originally from Washington, DC; MS in astrophysics from Northwestern University, BS in physics & theology from Georgetown University

• Architect for Cisco IT

• 18 years of IT experience

• Experience in architecture and design on large cross functional projects

• Currently providing technical leadership on the HP Superdome to UCS transition within Cisco IT

• Living & Education: Reside in Fremont, CA with wife and two children. Have worked on both coasts for various large organizations within IT. Originally from India, BS in Electrical Engineering from Punjab University

Sean McKeown


Topics

Cisco IT overview

HPUX PA-RISC and Itanium lights out

Cisco Unified Computing System

Methodology: planning, strategy and execution

Case studies and lessons learned


Cisco IT Overview

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8

locations incountries

offices

data centers and server rooms

employees

of data center space

of UPS power to raised floors

servers virtualized in new DCs, overall

Virtualization goal =


* Count covers all lifecycles and all stages of blade commissioning. Data as of October 27, 2011

Allen, TX (200)

Green Park,

UK (40) Amsterdam

(275)

Bangalore

(447)

Carrolton, TX (296)

Irvine, CA (72) Lawrenceville, GA (35)

Mountain View, CA (64)

Richardson, TX (2230)

RTP, NC (424)San Jose, CA (1486)

< 100 UCS Blades

200-500 UCS Blades

> 1400 UCS Blades

6400+ Total Cisco UCS Blades Deployed Globally*


HPUX Lights Out


Cisco HPUX Lights Out

Before

• Approximately 200 HP Superdome and mid-range HP-UX servers in the U.S. and Europe

• Hosting Oracle Real Application Clusters (RAC) and Oracle eBusiness Suite.

• Of 29 production Superdomes, 17 were PA-RISC and 12 Itanium

After

• Goal: None

• Current status: only a handful of legacy environments awaiting decommission

Objective: Remove proprietary platforms running HP-UX from the enterprise and migrate applications to UCS


Cost Avoidance

Cisco IT saved 85% in CAPEX with the Quote to Cash migration

HPUX Roadmap

PA-RISC EOL’d

Itantiumdevelopment uncertain

Software support dwindling

Standardization

Single platform: x86 on UCS

Two OS’s: Linux and Windows

Consolidation

Fewer, modern data centers

HP to UCS Migration – Business drivers


HP to UCS Migration – Technology drivers

Manageability

Management and support increasingly difficult

Cisco IT can rapidly provision a replacement by applying a Cisco UCS Manager service profile to a different blade

Scalability

Cisco IT was hitting their Superdomes’ limits for vertical scalability for key applications

Horizontal scale-out deemed too costly and painful

Environmental

Cisco UCS occupied one-sixth the data center space of the HP systems that it replaced, consumed 65 percent less power, and used 40 percent fewer cables

UCS

Enterprise level performance and scalability

Industry standard x86 + Linux architecture





A Single System That Unifies Compute: Industry standard x86

Network: Wire once for SAN, NAS, iSCSI

Virtualization: Control, scale, performance

Low latency 10GbE network ideal for RAC

Embedded Management Service profiles enable swappable compute

Dynamic resource provisioning

Ability to integrate with broad partner ecosystem

Energy and Cost Efficient Fewer servers, switches, adapters, cables

Lower power and cooling requirements


Cisco Unified Computing SystemInnovation unites computing, networking, virtualization, and storage access

Cisco UCS Traditional Blade Server

vs.

Traditional

Blade Server

Offerings


Installing Typical Servers

LAN

SAN

•RAID settings

•Disk scrub actions

•Number of vHBAs

•HBA WWN assignments

•FC Boot Parameters

•HBA firmware

•FC Fabric assignments for HBAs

•QoS settings

•Border port assignment per vNIC

•NIC Transmit/Receive Rate

Limiting

•VLAN assignments for NICs

•VLAN tagging config for NICs

•Number of vNICs

•PXE settings

•NIC firmware

•Advanced feature settings

•Remote KVM IP settings

•Call Home behavior

•Remote KVM firmware

•Server UUID

•Serial over LAN settings

•Boot order

•IPMI settings

•BIOS scrub actions

•BIOS firmware

•BIOS Settings


Simplify the Data Center: UCS Service Profiles

LAN

SAN

Serv

ice

Pro

file


Service Profiles Hardware State Abstraction – Oracle Examples

RAC node 2

HTTP Server ERP

RAC node 1 RAC node 4RAC node 3

ERP

Benefits

• Enables consistent setup, reduces service calls and downtime

• Add new instance in minutes, not hours or days

• Immediately test if workload is CPU or memory constrained

• Automatically load profile when new server added to system

• No LAN, SAN Zoning or any reconfiguration


Blades and Rack-Mount Servers Managed by

UCS Manager


UCS Solution Focus

Compute

Network

Virtualization

Storage

Operating Systems

Applications

Information

Cisco UCS B-Series

Cisco UCS Manager

Cisco Nexus® Family

Switches

NetApp FAS

10 GE and FCoE

Complete Bundle

VBLOCK FLEXPOD Standard Configurations

Infrastructure

OS/ Hypervisor

VDI Databases Enterprise Apps Business Analytics & DW RISC Migration

Unified Computing

HANA &

BWA

http://en.wikipedia.org/wiki/File:Suse_logo.svg

http://www.mysql.com/

http://www.ibm.com/us/en/


42 TB $42 B

© 2012 Cisco and/or its affiliates. All rights reserved.CODCT-2765 Cisco Public

Migrating Oracle from HPUX to UCS:

Methodology and Successes


Dual Core 90nm

Quad Core 65nm

2005 2005 2006 2007 2008 2010 2011

Eight Core 45nm

Single Core 90nm

Ten Core 32nm

Xeon 7040 Six Core

45nm

Xeon 7500

Xeon 7400

Xeon 3.66GHz

Dual Core 65nm

Xeon E7-4800

Xeon 7300Xeon 7100

Source: Intel internal OLTP database workload performance estimates as of 15 April 2011. Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.

2000% Performance Increase in CPU power since 2005

Cost

UCS servers are significantly cheaper

Less maintenance and support cost

Reliability

Clustering and virtualization provide better reliability

Vertically scaled nodes increase chances of failure

Independence

X86 platform and Linux are not tied to a single vendor

Performance

x86 chipsets have surpassed proprietary performance

levels, especially for database work


Large ERP Environments

Legacy systems reaching technology end of life and scalability limits

Increasing maintenance and support costs for older platforms

Proprietary (e.g. HP-UX, VFS) vs. “open” (Linux)

Data Center migration: Technology/architecture “rethink” opportunity

Oracle de-support for Itanium

Cisco UCS provides better performance at considerably lower cost

Large ERP DB Size Migration Date

Cisco Customer Care Center Service Sales 40 TB Live March 4th (Q3FY12)

The Configuration & Ordering Experience 3 TB Done - Q1FY12

E-Customer – Customer Master 8 TB Done - Q3FY11

Quote and order entry (QTC) 6 TB/4 TB Done – Q3FY11

Goal to Commissions 4 TB Done – Q1FY11

Plan to Build ,Manufacturing & Logistics. 1 TB Done = Q1FY11

Business Intelligence systems 28 TB/ 3 TB Done – Q2FY12

Partner Ordering Tool 2.5 TB Q4FY12

Note: Cisco follows FY – (Aug – Jul)


Cisco IT – Large ERP Systems analysis

Based on Cisco IT internal benchmarking

Value Add for IT

•Reduction in cost

•Better scalability

•Accelerate move off HP RISC platform30%

growth

UCS 4 node RAC 8 cores

SJPROD PEAK Req

QTCPRD PEAK Req

ECUST PEAK Req

C3 PEAK Requirements – Three SD 128 cores (40% - 30% - 60%)

CORE Peak Req

ODS Peak Req

G2C Peak Req

BVPROD Peak Req

SDMPRD Peak Req

ICW PEAK Req

1 node RAC HP IA Superdome 64 core

2 node RAC HP IA Superdome 32 Core

2 node RAC HP IA Superdome 16 Core

UCS 1 node 32 cores

UCS 8 node RAC 32 cores

3 node RAC HP IA Superdome 64 core/128 Cores/128 cores

CFN PEAK Req


Source System (HP Superdome) Target System (Cisco UCS B200)

1. 2 Node HP Itanium Superdome SD64B, Oracle RAC cluster with 32 CPU Core * 64 GB Memory each

2. E-Business version 12.0.4 using Oracle Incentive Compensation (OIC) module

3. Oracle database version 10.2.0.4

4. Tier-II DC

1. 4 Node Cisco UCS B200 M1, Oracle RAC cluster with 8 CPU Core* 48 GB Memory each

2. E-Business version 12.0.4 using Oracle Incentive Compensation (OIC) module

3. Oracle database version 11.2.0.1

4. Tier-III DC

G2C Migration Details• Goals to Commission (G2C) application supports

Cisco Sales compensation platform• No additional certification was required. UCS

hardware is already certified for target Oracle E-Business Suite applications

• No application re-architecture was required• One 10th the tile space, less than one third the

power utilization and less than half the data cables

HP Superdome to Cisco UCS Migration

Used Oracle DG for DC Migration

0.00

5.00

10.00

15.00

20.00

25.00

# of DC Tile Space Power Utilization

(KW) Number of Data Cables

4.00

13

24

0.40

4

10

Source HP Superdomes (2 qty)

Target Cisco UCS Servers (4 qty)


Quote to Cash Migration Benefits

100% of Cisco revenue flows through the Quote to Cash systems

UCS occupies one-sixth the data center space that replaced legacy

Consumes less than 65% power and uses 40% fewer cables

Legacy Current (Cisco UCS)

Production Server

Configuration

2 HP-UX Superdomes

(San Jose)

2 Cisco UCS B440 M1 Blade Servers

(Richardson))

Operating System HP-UX Version 11.11 Red Hat Enterprise Linux Version 5.5

CPU Cores 80 64

Memory 160 GB 512 GB

Performance Capacity for current workload Capacity for double the current workload

Database Oracle RAC 9i Oracle RAC 10g

Oracle E-Biz 11.5.9 11.5.9

Used Oracle DG for DC Migration

0

5

10

15

20

25

DC Rack spaces

Power Utilization (KW)

Number of Data Cables*

2 HPUX Superdomes plus peripheral 2 Cisco UCS Server B440


Cisco IT Superdome to UCS Instances and Cost

Cisco UCS – The Right Solution For Oracle

Capital Cost

reduction of

more than

75%

Saving

$167,200/yr

Of SW

maintenance

cost


User Experience Before and After

0

50

100

150

200

250

300

350

400

Cisco Invoice Interface -

US

Autoinvoice Import

Program

General Ledger

Transfer Program

Revenue Recognition

Order Import Aging - 7 Buckets - By

Account Report

Customer Interface -

US

Customer Interface -

JPN

Cisco Holds -Gross Dollars Report

Cisco Open RMA Report

Cisco Back Log Report

Cisco Backlog-AR

Report

63

326

168 172

103

19

268

50

108 107

60 62

HP Superdome (Feb 2011): Maximum Runtime (minutes) Cisco Unified Computing System (Mar 2011): Maximum Runtime (minutes) Gain

0

20

40

60

80

100

120

Aged Account Summary (Search by customer)

Aged Account Summary (Search by Bill To)

SOM Scrubber (Hold Criteria for AMS)

Change Order Cross Scrubber (By SO Order)

Change Order Scrubber (SO Search)

Order Inquiry (Search by SO number)

Unallocated Report –Download Report

26

6

19

67

20

52

13

Time on HP Superdome (Seconds) Time on Cisco Unified Computing System (Seconds) Percent Improvement

Batch

OLTP


Migrating a 40 TB Oracle RAC E-Business

Database from HP to UCS


1/3 data center space usedSupports Cisco Service Sales, Advance

Service Mgmt, Technical Service Delivery47 kW

power saved

23% of Cisco revenue ($10B annually) 60% reduction in CAPEX/OPEX

per annum

C3 Migration Benefits

Source (Legacy) Target (Cisco UCS)3 Legacy RISC-based IA Superdomes 12 Cisco UCS B440 blade servers

Production Server Configuration

HP-UX v11.23 Red Hat Enterprise Linux v5.5Operating System

384 384CPU Cores

1536 GB 3072 GBMemory

Capacity: current workload Capacity: 1.5 times the current workloadPerformance

3 node RAC Oracle 10g (CRS 10g) 12 node RAC Oracle 10g (CRS 11gR2)Database

N+1 ArchitectureLegacy: Cisco UCS B440:

15 4vs.

74 KW 27 KWvs.

241 100vs.

27% less Tile space

36% less Power

41% less cables


Migrate C3 Platform from San Jose to Richardson

Richardson, Texas

C3 Platform

Technical ServicesService Delivery and

Parts Fulfillment

Advanced ServicesManagement

of Services Projects Engagement

Service Sales20% of Cisco Revenue

BusinessValue

Improved User

ExperienceIncreased performance

Cisco on CiscoShowcase UCS &

Cisco Data Center

Capabilities

AgilityReduce time-to-market for

new business capabilities

ResiliencyEliminate Cisco Services

single source of failure

Application PortfolioRationalized

• Reduce complexity• Improve supportability• EOL unneeded items

Cost Optimization & Risk Avoidance

• Servers at CPU capacity• Support and license

fees are costly

San Jose, California


Legacy C3 Environment

• Highly active, revenue impacting OLTP database• Based on EBS 11i/10g ~ 200+ bolt-on applications• 3 RAC Node HP IA64 Superdomes • 128 Cores each @ 1.5 GHz, 512 GB RAM, EMC SAN• ~ 36 TB, expected 40 TB soon• Redo of 3 TB/day; peaks 4 to TB• ~ 8000+ sessions from Conc. managers and JVM connections• Reached hard limit on vertical scalability!

Considerable Cost to increase capacity

or move to next-gen Superdomes

with

Multiple life cycle env


8 Node UCS RAC

Required:

8 node certification Services Setup, RAT Testing

Capacity Analysis

Split Option ALL UCS

Required:

Business Buy in for Split Option

Services Setup, Capacity Analysis

Split Option UCS and HPUX in RCDN

Required:

Business Buy in for Split Option

Services Setup, Capacity Analysis

HPUX in RCDN as is with UCS provided for App failover

Required:

Migration and planning for Application failover

If Not Possible

If Not Possible

If Not Possible

1

2

3

4

Option DC OPS DBA APP

1

2

3

4

Option of last resort

2,206

4,626

378

0

500

1,000

1,500

2,000

2,500

3,000

3,500

4,000

4,500

5,000

AWR TPS Interconnect (MBPS)

1 Node

2 Nodes

Based on 0.8 RAC factor for 2nd node and 0.7 factor for subsequent nodesTwo Node TPS actual tests

Based on Cisco IT internal testing for heavy Database loads


C3 Migration Analysis

Oracle 10g with 11iLinux RHEL 5.5Oracle ASMServices based configurationApplications segregation with data model

Oracle 10g with 11iHPUX 11.21Raw VolumesServices based configurationApplications segregation with data model

Itanium 128 core node 1

Itanium 128 core Node 2

UCS 32 core core 3 node


• High Operational impact of Scaling horizontally with Superdomes – limit 128 cores•Considerable capital cost to migrate to next generation•Very high support cost•High downtime to replace parts or upgrade


Itanium 128 core Node 3

36 TB Data

Over 10000 TPS

Peak IO 5 GB/s

Less that 10 ms

average response

time

Application Profiling

App segregation and Load Balancing

Physical Design Verification

Load testing and validation

RATto test


(Known): Capture metrics at system (host, database) level.

(Known): Capture metrics at database session level by process.

Need a bridge between system and processes running within the database. Different types of metrics; TPS, CPU, IO vs. Logical IO, Physical Reads, Physical Writes

(Derived): Group processes by application (logical grouping); typically based on business process

alignment and data affinity.

Application grouping is bridge. Logical IO most closely relates to TPS. Generate ratio of a processes Logical IO out of total Logical IO for all processes. Trend metrics over a period of time.

How to connect our processes and systems within a large system so that we can distribute and scale? We leverage a blend of known facts and derived ratios.






C3 groups

Application Group TPS Node

EBS Concurrent Managers

Contract Management 1050 3

E-Biz and Misc CM Jobs 781 4

Quoting & Ordering 1565 5

Service Request, Service Supply Chain, Projects 1194 6

E-Biz Online (Forms, OA Web, Utility, Admin) 835 1, 2

Infra Processes, Adhoc Requests, Misc 758 2, 1

Online Technical Services (Service Request, Service Supply

Chain, Projects)785 7, 8, 9

Online Service Sales (Contract Management, Quoting

&Ordering, Config)1329

10, 11,

12

Total 8297 12





Application Group TPS Node

E-Biz Online (Forms, OA Web, Utility, Admin) 835 1

Online Technical Services (Service Request, Service Supply

Chain, Projects)785 1

EBS Concurrent Managers (Critical Technical Services Jobs) 430 1

EBS Concurrent Manager

Online (non-critical) + Batch + Failover

(Upstream/Downstream)1390 2

E-Biz and Misc CM Jobs 155 2

Infra Processes, Adhoc Requests, Misc 758 (1, 2, 3)

Online Service Sales (Contract Management, Quoting &

Ordering, Config)1329 3

EBS Concurrent Managers (Critical Service Sales Jobs) 2615 3

Total 8297 3

Before After

Superdome to UCS Application Mapping


Node2 Node4 Node1 Node3 Node7 Node5

CM1 CM2OTH

Order MgmtService Request

Projects

CCS-MSC

Service Agreements

Quoting

Node6 Node9 Node11 Node8 Node10 Node12

E-business Apps Forms/Java ( All)TPS: 835.12IO: 802.35 MBIOPS: 51KData: All

EBIZDB links, Infra/ Desktop clients, 3rd party, batch jobs etc.TPS: 757.45IO: 822.6 MBIOPS: 51KData: Other/NA

Conc. MgrGroup 1TPS: 899IO: 517.95 MBIOPS: 32KData: SA/IB

CM3 CM4

Conc. MgrGroup 2TPS: 681IO: 146.7 MBIOPS: 9KData: SA/IB + Quote

Conc. MgrGroup 3TPS: 1257IO: 422.55 MBIOPS: 27KData: Quote

Conc. MgrGroup 4TPS: 973IO: 761.85 MBIOPS: 48KData: All Remaining

Service RequestTPS: 552.1IO: 404.55 MBIOPS: 13K

Order ManagementTPS: 201.46IO: 121.95 MBIOPS: 17K

ProjectsTPS: 19.82IO: 12.15 MBIOPS: 4K

Service Agreement /Install BaseTPS: 459.42IO: 164.25 MBIOPS: 5K

QuotingTPS: 869.24IO: 317.7 MBIOPS: 10K

Miscellaneousonline

cci/x, perlonline

Misc DataTPS: 10.77 IO: 5.4 MB

IOPS: 15K

!ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1

!ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1

!ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1


Current C3 Environment – (N+1) with 50% more capacity

• Highly active, revenue impacting OLTP database• Based on EBS 11i/10g ~ 200+ bolt-on applications• 12 RAC Node Cisco UCS• 32 Cores each Xeon 7500, 256 GB RAM, EMC SAN• ~ 40 TB DB size• Redo of 5 TB/day• ~ 8000+ sessions from Conc. managers and JVM connections• Significantly increased horizontal and vertical scalability

Minimal Cost to increase capacity or

move to next gen UCS

with

Multiple life cycle env


SAN DW1

SAN DW2

SAN GW1

SAN GW2

SAN GW3

SAN GW4

SAN SW1

SAN SW2

SAN SW3

SAN SW4

San PC: 64 Gbps

San PC: 40 Gbps

Data Frame

Frame

Other

Apps

Frame

Set B

Frame

Redo Mirror

12 x DB

nodes

S L O T

1

S L O T

5

S L O T

3

S L O T

7

S L O T

2

SLOT6

S L O T

4

S L O T

8

!

U C S 5 1 0 8

O K F A I L O K F A I L O K F A I LO K F A I L

CO NS OLE

Cisco UCS 6140X P

PS

2P

S1

STAT

L1 MGM T 0

L2 MGM T 1

4 8 12 16

3 7 11 15

2 6 10 14

1 5 9 13

2 0 24 28 32

19 23 27 31

1 8 22 26 30

17 21 25 29

36 40

35 39

34 38

33 37

1 00-24 0V~9 .2 A

50 -6 0 H z

SLO

T2S

LOT3

1 /2/4 G F I BRE CHANNE L

N 1 0 - E 0 0 8 01 2 3 4 5 6 7 8


N 1 0 - E 0 0 8 01 2 3 4 5 6 7 8

CO NS OLE

Cisco UCS 6140X P

PS

2PS

1

STAT

L1 MGM T 0

L2 MGM T 1

4 8 12 16

3 7 11 15

2 6 10 14

1 5 9 13

2 0 24 28 32

19 23 27 31

1 8 22 26 30

17 21 25 29

36 40

35 39

34 38

33 37

1 00-24 0V~9 .2 A

50 -6 0 H z

SLO

T2S

LOT3


N 1 0 - E 0 0 8 01 2 3 4 5 6 7 8


N 1 0 - E 0 0 8 01 2 3 4 5 6 7 8

FI-A

FI-B

8 x vHBA

1&3&5&7

8 x vHBA

2&4&6&8

EMC

SET A

EMC

EMC

EMC

EMC

EMC

Redo Frame

Vsan 2

Vsan 1

FA to MDS 192 GbpsMinimal Cost to increase capacity

or move to next gen UCS

UCS throughput:FC (Storage): 128 GbpsEth (Network): 160 GbpsPer BladeFC: 128/12=10.66 GbpsEth: 160/12= 13.33 Gbps

If there is a Fabric Interconnect outage solution will still meet requirements

MDS 9513 – 13 modules (24, 48,..) Edge Core topologyData Frame VmaxCache – 512 GB Mirrored

RA

C

Priv

ate

netw

orkP

ublic

netw

ork

12 UCS B440Cores – 384Xeon CPU – 2.2 GHzFE Node – VMCM FE Node – B250(4)

0

2

4

6

8

10

12

C3 Data Frame Requirements today

C3 Data Frame requiremnts with 30% growth

Frame Capability tested (R/O)

Frame capability R/W (EMC input)


UCS Projections (based on RAT Testing and Production Stats)

Growth % 12.00% 46.1% 6% 5% 5% 3% -5% 4% 6% 6% 6% 5% 5% 5%

RAT CPU utilization Dec

9th data

Other system cpu load added

Projected cpuutilization Q4FY11

Q1FY12 Q2FY12 Q3FY12 Q4FY12 Q1FY13 Q2FY13 Q3FY13 Q4FY13 Q1FY14 Q2FY14 Q3FY14 Q4FY14

12 node Production Configuration

25.0% 37.0% 54.1% 57.4% 60.4% 63.7% 65.5% 62.4% 64.9% 68.6% 72.5% 76.6% 80.0% 83.6% 87.4%

Projected Peak utilization(15-20% net safety margin is reflected in the numbers)

0.00

10.00

20.00

30.00

40.00

50.00

60.00

2:5

1:0

2

2:5

3:0

2

2:5

5:0

2

2:5

7:0

2

2:5

9:0

2

3:0

1:0

2

3:0

3:0

2

3:0

5:0

2

3:0

7:0

2

3:0

9:0

2

3:1

1:0

2

3:1

3:0

2

3:1

5:0

2

3:1

7:0

2

3:1

9:0

2

3:2

1:0

2

3:2

3:0

2

3:2

5:0

2

3:2

7:0

2

3:2

9:0

2

3:3

1:0

2

3:3

3:0

2

3:3

5:0

2

3:3

7:0

2

3:3

9:0

2

3:4

1:0

2

3:4

3:0

2

3:4

5:0

2

3:4

7:0

2

3:4

9:0

2

3:5

1:0

2

3:5

3:0

2

3:5

5:0

2

3:5

7:0

2

3:5

9:0

2

4:0

1:0

2

4:0

3:0

2

4:0

5:0

2

4:0

7:0

2

4:0

9:0

2

4:1

1:0

2

4:1

3:0

2

4:1

5:0

2

4:1

7:0

2

4:1

9:0

2

4:2

1:0

2

4:2

3:0

2

4:2

5:0

2

4:2

7:0

2

4:2

9:0

2

4:3

1:0

2

4:3

3:0

2

4:3

5:0

2

4:3

7:0

2

4:3

9:0

2

4:4

1:0

2

4:4

3:0

2

4:4

5:0

2

4:4

7:0

2

4:4

9:0

2

4:5

1:0

2

4:5

3:0

2

4:5

5:0

2

4:5

7:0

2

4:5

9:0

2

5:0

1:0

2

5:0

3:0

2

5:0

5:0

2

5:0

7:0

2

5:0

9:0

2

5:1

1:0

2

5:1

3:0

2

5:1

5:0

2

5:1

7:0

2

5:1

9:0

2

5:2

1:0

2

5:2

3:0

2

5:2

5:0

2

5:2

7:0

2

5:2

9:0

2

EBS - N1 - Average 3.63

EBS,Misc-N2 - Average 7.11

Others,Misc-N3 - Average 4.4

CM1 - N4 - Average 9.29

CM2 - N5 - Average 6.08

CM3-N6- Average3.07

CSSD - N7 - Average 8.11



BASE - N10 - Average 17.13



RAT Testing from Dec 9th data

Actual utilization :- 40% for CM nodes, 20% for others


Application Performance

0

5

10

15

20

25

Search Service

Request by User Profile

Search Service

Request by XML

Search Service

Request by Manager

Search Service

Request by Customer

Search Service Request

Search Service

Request by User Profile

Search Service

Request by Contracts

HP Production SJ to SJ UCS Production RCDN to RCDN

0

2

4

6

8

10

12

14

Adding Install Site

on to a Quote

Create 2tier SMS3 Quote

Generate Quote

Moving Items

Install Sites (Move

Products)

Renewal Quote

Creation

Contract Number Search

Serial Number Search

Bill To Id Search -X-

Track

Quote Summary

Page

Serial Number Search -CSCC Page

Create New Quote

Contract Number

Search in Advance Search Page

HP Production SJ to SJ UCS Production RCDN to RCDN

1.62

1.37

1.2

1.25

1.3

1.35

1.4

1.45

1.5

1.55

1.6

1.65

HP UCS

Average Run Time (minutes) per exeution of Conc

programs running >=1 mt but < 30 mts 39.59

30.88

0

5

10

15

20

25

30

35

40

45

HP UCS

Average Run Time (minutes) per exeution of Conc

programs running > 30 mts

Co

ncurre

nt P

rogra

ms E

xecutio

n

Co

mpa

rison

Onlin

e tra

nsactio

ns E

xecu

tion

Com

paris

on

Sec

Sec


Instance Crash issue before and after

Node2

Node4

Node1 Node3 Node7 Node5

CM1 CM2OTH

CCS-MSC

Service Agreements Quoting

Node8Node10

Node12

EBIZ CM3 CM4

Miscellaneousonline

!ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!A

03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1

!ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1

!ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1!

ResetConsole

UCS B440 M1

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

A03-0300

GA

2

10k S

AS

300G

B

!

!

SLOT

2SLOT

1

Order Mgmt

Service Request Projects

Node6 Node9Node11

EBIZ,OTS, CM partial, MISC

CM most, MISC

Service Mgmt, CM partial, MISC

Instance Crash on any node• Significant impact to business• Manual failover with degradation• Downtime for business• Considerable effort from Infra

Instance Crash on any node• No impact to business• Automated failover with no degradation• No Downtime for business• Minimal effort from Infra

N+1 Architecture


Copy Export RMAN read RMAN write

�

4 TB / Hr�

�

4.5 TB / Hr�

�

40 TB

San Jose Data Center

Node1 Node2 Node3

Richardson Data Center

Storage

Frames

Richardson Data Center

Standby

Instance

Rman Write

Rman Read

Copy

ActivityTime Taken in

Hours

EMC recoverpoint sync up from San Jose

to Texas DC and validation of database0.5

DB backup before and verification on HP

server1

Pre-migration activities like re-sync domain

indexes, purge OKC tables, generate and

prepare XXTS scripts

1.5

Copy data files to Cisco UCS 8

RMAN conversion 1

Import metadata 5

DB Auditing 2

Validation and auditing

Auto configuration on DB nodes 3

Auto configuration on 11 FE nodes 10

Total time taken 32


Earlier we had multiple issues almost every day with C3 and there were two dedicated DBA

Now we are down to one DBA and have given him a other databases to manage as things are so quiet

Do we need to be worried about our job security

Coffee room discussion


Horizontal Vertical

Proposed C3 UCS Configuration allows for adding additional 12 nodes dynamically

FY09 - UCS 8 Core

FY10 - UCS 16 Core

FY11 - UCS 32 Core

FY12 UCS 40 Core plus

Additional cores for performance improvement

Increased processing power

Scalable architecture to meet C3 current and future needs

C3 configUCS B440

Solution design throughput:At 100% availability FC (Storage): 128 GbpsEth (Network): 160 Gbps

Two FI IO (6140): Per BladeFC: 128/12=10.66 GbpsEth: 160/12= 13.33 Gbps

At 50% FI availabilityThroughputFC (Storage): 64 GbpsEth (Network): 80 Gbps

Per BladeFC: 64/12= 5.33 GbpsEth: 80/12= 6.66 Gbps

Cisco IT has put a $42 B plus business on UCS


http://www.cisco.com/go/migratetoucs

Documents

How Cisco IT Migrated from HP Computing Systemd2zmdbbm9feqrf.cloudfront.net/2012/usa/pdf/COCDCT-2765.pdfCisco Public How Cisco IT Migrated from HP Superdome to the Cisco Unified Computing