Upgrading Server Hardware in UCS without

re-installing the OSRavi Mishra

Technical Marketing Engineer, CSPG

@ravmishr

BRKCOM-2016

• Software Definition of Servers: UCS Service Profiles

• Server Refresh Cycles and Traditional Re-Imaging

• UCS Service Profile Migration from UCS M1/M2 to UCS M3 blades

• Service Profile Migration from M3 to M4 Server

• Special Consideration

Agenda

UCS Building Blocks

8765

UC

S 2

208X

P

4321

IOM 22xxFEX 2232PP

UCS 6248UP

UCS 6296UP

UCS ManagerEmbedded– manages entire system

UCS Fabric Interconnect48 Port 10Gb FCoE96 Port 10Gb FCoE

UCS Fabric Extender/IO ModuleRemote line card

UCS Blade Server ChassisFlexible bay configurations

UCS Blade/Rack ServerIndustry-standard architecture

UCS Virtual AdaptersChoice of multipleadapters

UCS Server Architecture

• Typical Infrastructure: Assign a business use to a chassis

• Some chassis filled and requires new purchases of chassis, LAN I/O modules, SAN I/O modules, Management modules, Intra-DC cabling all required

• Open capacity in other chassis are stranded

FI Running UCSM FI Running UCSM

No Single Point of Failure in Infrastructure

Compute Lifecycle Management on UCS Overview

Server Refresh Lifecycle

• Server hardware EoS

• Modernize IT infrastructure

• Standard Compliance/data retention requirement

• Operational excellence

• New business opportunities

• Typical problems with older Infrastructure

• High Costs

• Increased inefficiencies

• Complexity

• Security gaps

Older Server Inefficient and Costly

• Lower Performance, Higher Energy Consumption

(Source: IT Metrics: Align IT Investment Levels with Strategy Using Run, Grow, Transform and Beyond. Gartner [March 23, 2012])

Intel Tick-Tock Model

New

Micro-

architecture

Xeon®

5500

45nm

Xeon®

5600

32nm

New

Process

Technology

TOCK TICK

Intel® MicroarchitectureCodename Nehalem

Xeon®

E5- 2600

New

Process

Technology

Xeon® E5- 2600 v2

32nm

New

Micro-

architecture

22nm

TOCK TICK

Intel® MicroarchitectureCodename Sandy

Bridge

22nm

New

Micro-

architecture

Broadwell

14nm

New

Process

Technology

TOCK TICK

Intel® MicroarchitectureCodename Haswell

M1 M2 M3 M3* M4*M3 with v2 CPU

Xeon® E5- 2600 v3

0

200

400

600

800

1000

1200

1400

1600

2006Xeon 5160

(2C,3GHz, 80W)

2007X5272

(2C, 3.4GHz,80W)

2007X5365(4C,

3GHz, 150W)

2008X5492

(4C, 3.4GHz,150W)

2009X5570

(4C, 2.93GHz,95W)

2010X5680

(6C, 3.33GHz,130W)

2012E5-2690

(8C, 2.9GHz,135W)

2013E5-2697 v2

(12C, 2.7GHz,130W)

2014E5-2699 v3

(18C, 2.3GHz,145W)

SP

EC

int*

_ra

te_

bas

e20

06

pe

rfo

rma

nc

eIntel® Xeon® Processor E5-2600 Product FamilyHistorical 2S Integer Throughput Performance

Up to 22x increase in integer throughput on Intel® Xeon® since 2006

Higher is betterUp to

22X

Source as of Sept 8 2014: results published or submitted to www.spec.org. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Per formance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more information go to http://www.intel.com/performance Intel does not control or audit the design or implementation of third party benchmark data or Web sites referenced in this document. Intel encourages all of its customers to visit the referenced Web sites or others where similar performance benchmark data are reported and confirm whether the referenced benchmark data are accurate and reflect performance of systems available for purchase. *Other names and brands may be claimed as the property of others

10

Benefits of refreshing old servers

Better Performance

Fewer Servers

Less Space

Better Security

Software

Licencing &

Compatibility

Less Power

Lifecycle Management for Compute in UCS

• Key server instances – moved to modern HW as needed

• “Moved Up” to latest generations

• Servers coming off support

• Servers ending lease

• Latest Servers on Mission Critical Nodes

• Evacuated hardware can move down the tree

• VM infrastructure adds value here

• No need to re-image the VM’s as they go on new server types

• UCS brings this model to all server types

• Not just virtualized

• We have the depth to keep consistency in key areas

Traditional ServersHardware Centric

• Servers are specifically purchased for applications

• Hardware dictates the identity of the servers

• Management of servers handled individually or through add-on software LAN

SAN

•RAID settings

•Disk scrub actions

•Number of vHBAs

•HBA WWN assignments

•FC Boot Parameters

•HBA firmware

•FC Fabric assignments for HBAs

•QoS settings

•Border port assignment per vNIC

•NIC Transmit/Receive Rate

Limiting

•VLAN assignments for NICs

•VLAN tagging config for NICs

•Number of vNICs

•PXE settings\

•NIC firmware

•Advanced feature settings

•Remote KVM IP settings

•Call Home behavior

•Remote KVM firmware

•Server UUID

•Serial over LAN settings

•Boot order

•IPMI settings

•BIOS scrub actions

•BIOS firmware

•BIOS Settings

UCS Service ProfilesWorkload Centric

• Servers are specifically purchased for additional capacity and provide service elasticity

• Server identities are defined by the service profile

• Management of servers provided through the very system that defines them (UCSM).

LAN

SAN

UCS Service Profiles: Configuration Portability

Service ProfileIdentity for a serverSIM Card

Identity for a phone

Fundamental Enabler of Service Profile

Traditional Server Refresh Steps

Traditional Server Refresh Steps - Preparation

Prepare new Physical Server

• Physical mounting and cabling

• Establishing Network Access configuration facing new hardware

• Establishing any Storage Access configuration facing new hardware

• BIOS settings

• Build RAID volumes / Configure SAN Booting

• Update any new onboard firmware

Traditional Server Refresh Steps - Install and Config

• Install New OS on Physical Server

• PXE Installer / Virtual Media / Physical Media / LUN Cloning / others

• Treat the Installed OS

• IP’s, Masks, Gateways, NIS, Domains, etc.

• Installation and Configuration of Application

• Rebuild the databases, key application identity etc.

• Move services away from hardware to be retired

Time: Few weeks to month(s)

UCS Service Profile Migration

UCS Server Refresh Process

• Prepare OS for migration

• Disassociate Service Profile from Old Server

• Associate Service Profile to New Server

• Boot the OS and Clean up

From Weeks

To Hour(s)

No Server/Application Reinstall

Server Hardware Upgrade in UCS

• Prepare OS for Migration

• Backup of the network configuration

• Upgrade the UCS Infrastructure & UCSM

• Determine enic/fnic version based on HCL

• Driver Injection :- download the UCS driver DVD from cisco.com and update the driver of old server for target firmware version

Backup of Network Configuration

• Manually capture the adapter MAC address with corresponding network configuration.

• Use the following command to back up your network configuration from a DOS command prompt on the Windows server:

• netsh interface dump > netcfg.dat

• Optional for ESXi and RHEL but recommended

How to check UCS firmware version

• Multiple ways to check the firmware version of the blade.

• One way is to click the server and click on Installed firmware.

In the above screen, the server BIOS is 1.4

Upgrade the UCS Infrastructure & UCSM

• For support of new hardware, upgrade of UCS infrastructure and UCS manager may be required.

• Obtain Software bundle from cisco.com. There are 3 types of bundle• Cisco UCS Infrastructure Software Bundle—Required for all Cisco UCS domains (for FI/IOM/UCS

Manager),

• Cisco UCS B-Series Blade Server Software Bundle—Required for all Cisco UCS domains that include blade servers

• Cisco UCS C-Series Rack-Mount UCS-Managed Server Software Bundle—Only required for Cisco UCS domains that include integrated rack-mount servers

• Firmware can be upgrade via manual or automated (recommended) process

• Please refer latest UCS upgrade guide• http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/upgrading/from2-1/to2-

2/b_UpgradingCiscoUCSFrom2-1To2-2.html

Manual vs Firmware Auto Install

Upgrade with Auto Install

• Helps the customer create infrastructure and host firmware packages with the aid of a wizard

• to group the components in a downloaded bundle from Cisco.com.

• available on or after UCS 2.1 software release.

Upgrade manually

• Manual process where specific sequence of upgrade needs to be followed.

• You can choose to upgrade some server endpoints, such as adapters, manually

Cisco Hardware Compatibility List (HCL)

• Cisco maintains a UCS Hardware Compatibility List (HCL) to the level of maintenance releases. Cisco does not modify this for the patch releases as only defects are addressed. This can be found at:

• UCS HCL Toolhttp://www.cisco.com/web/techdoc/ucs/interoperability/matrix/matrix.html

• UCS v2.2(4):http://www.cisco.com/c/dam/en/us/td/docs/unified_computing/ucs/interoperability/matrix/b_hcl_B_rel2-24.pdf

• UCS v2.2(3d):http://www.cisco.com/c/dam/en/us/td/docs/unified_computing/ucs/interoperability/matrix/r_hcl_B_rel2-23.pdf

• General Page with these and others: http://www.cisco.com/en/US/products/ps10477/prod_technical_reference_list.html

UCS Driver Injection

• Driver Injection is a process in which you prepare an old server for service profile migration to a new server

• This process involved identifying a upgradable version of Cisco UCS Virtual Interface (VIC) Card -Ethernet NIC (enic) and Fiber channel NIC (fnic) driver for a particular OS.

• The firmware version can be determined from the OS CLI or GUI interface like windows device manager.

• After identifying the deriver version, please follow the OS driver update process to upgrade driver of enic/fnic interface for that OS.

How to check enic/fnic version

Windows

• Device Manager Network Adapter

• Device Manager Storage HBA

How to check enic/fnic version

Linux• For the eNIC driver, cat /sys/module/enic/version

• For the fNIC driver, cat /sys/module/enic/version

Vmware

Driver Version of enic/fnic before upgrade

• Let take an example, As per UCS hardware and software interoperability matrix, UCS B200M2 blade (source blade) with Windows Server 2008R2 SP1 on UCS Adapter firmware version 2.1(3) should below driver version

• Now lets check the interoperability matrix for firmware version 2.2.3. This is the firmware version we are going to use for new B200M3 blade (target blade) in this example.

Driver comparison between M2 and M3

Driver Version require to update

M2 Server

M3/M4 Server

M3/M4 Server

Server Hardware Upgrade in UCS

Disassociate Service Profile from old blade

• After the dis-association, take out the old blade/server and reinsert new blade on that slot.

• Can use any slot in the same or different chassis within same UCS domain

Associate Service Profile to new blade

• Check the UCS firmware version of the target blade, this should match the firmware version driver you upgraded to.

• Associate the old service profile from old to new blade. Wait for association to complete. Please check for any service profile association related faults in UCS Manager.

Server Hardware Upgrade in UCS

• Boot Up the OS and Clean up

• Boot the OS into the new blade.

• Update the drivers of other new devices on new blade with Intel driver update tool. More info

• UCS driver Installation for common Operating Systems

• Check the network settings and reconfigure those settings if changed (OS dependent)

Its demo time!!!!

http://bit.ly/brkcom2016

Special Consideration

Cisco VIC driver for ESXi

• In some cases, drivers on the Cisco ISO aren't digitally signed by VMware so upgrade with VUM might fail.

• Two options

• Create custom ISO with –NoSignatureCheck enabled

• Download signed driver from Vmware website

• Enic driver for ESXi 5.5https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI55-CISCO-ENIC-21262&productId=353

• Fnic driver for ESXi 5.5https://my.vmware.com/web/vmware/details?productId=353&downloadGroup=DT-ESXI55-CISCO-FNIC-16012B

PCI Re-enumeration

• With Cisco VIC card every vNIC presented as PCI device to the OS.

• PCI re-enumeration possible if

• Most OS (Windows/Linux)

• Number of VIC cards changes (for example from Palo to VIC1240/VIC1280)

• Add/remove number of vNICS

• Always take backup of network config to capture mac address to IP address mapping before service profile migration

vCon Assignments changes

• The Service Profile was designed to have mobility between servers with different resources.

• vCons were designed to provide an abstraction of physical adapters to provide a mechanism for this mobility.

• When building a service profile there are Implicit (system) and Explicit (user) assignments of vNICs and vHBAs to vCONs

vCon1 vCon2 vCon3 vCon4

eth

0

eth

1

hba

0

eth

2

eth

3

hba

1

eth

4

eth

5

Eth

6

Eth

7

Adapter 1 Adapter 2Adapter 1!

ResetConsole

UCS B440 M1/M2

A03

-030

0GA

2

10k

SA

S

300G

B

!

A03

-030

0GA

2

10k

SA

S

300G

B

!

A03

-030

0GA

2

10k

SA

S

300G

B!

A03

-030

0GA

2

10k

SA

S

300G

B

!

!

SLOT

2SLOT

1

• By default the system will place vNICs and vHBAs across 4 vCons and map vCons to adapters.

• Initially this was round robin, but changed to linear in version 2.1.x and above.

• The change allowed for consistent PCIe numbering when the number of adapters in the system changed.

vCon1 vCon3 vCon2 vCon4

eth

0

eth

1

hba

0

eth

2

eth

3

hba

1

eth

4

eth

5

Eth

6

Eth

7

Adapter 1 Adapter 2Adapter 1!

ResetConsole

UCS B440 M1/M2

A03

-030

0GA

2

10k

SA

S

300G

B

!

A03

-030

0GA

2

10k

SA

S

300G

B

!

A03

-030

0GA

2

10k

SA

S

300G

B!

A03

-030

0GA

2

10k

SA

S

300G

B

!

!

SLOT

2SLOT

1

vCon Assignments (rond-robin)

Firmware version older than 2.0 in Windows

• In some scenario, with Windows OS and firmware/driver version older than 2.0, you might see STOP 0x7B trap screen.

• Windows not able to load the drivers from CDDB.

• Windows Critical Device Database (CDDB) missing some entries in PCI id tree in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CriticalDeviceDatabase

Workaround for CDDB issue

• CDDB with M81KR (Palo) and firmware version 2.0+

• CDDB with Palo and firmware version 1.4, no entries for VIC12x0

• Workaround to export the registry entry, take proper backup and swap the digits in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CriticalDeviceDatabase\PCI#VEN_1137&DEV_0045&SUBSYS_00481137

Swap “4” and “8” here

Assumptions for this test

• Hardware Configuration

• UCS B200 M2 with M81KR VIC, UCS B200M3 with VIC1240 and B200M4 with VIC 1340.

• Similar steps can also be applicable for VIC 1280 and C series rack server.

• Moving from 3rd party NICs to VIC won’t work.

• Moving from 3rd party CNA to VIC might work

• UCS Blade OS is booting from SAN and not using the local drives on the blade

Software configuration for test

• Software Configuration

• Test covered migration for Windows Server 2008R2, Red Hat Enterprise Linux 6.1 (RHEL) & VMware ESXi 5.5update2.

• Similar process can be used for other version of these OS and other supported Operating Systems.

• Test did not covered any or some aspects of application testing

• Success criteria was booting the OS and network/storage connectivity

SP Migration based on boot image option

UCS Servers with Local JBOD Boot

• PCI tree for I/O construction possible with VIC adapters

• No Virtual Disks Configured from Physical

• UCS Service Profile migration works

• We just move the disks physically

• Disk support on the matrix for the Server type must be verified

UCS M2 server with Local RAID Boot

• Not fully supported for older server (M2)

• Differing LSI software stacks are the key inhibitor

• Utilities to overcome are out there

• LSI and Cisco development agree that these will not be supported

• UCS Service Profile can be utilized still without change

• Cisco UCS RAID Guide clearly explains this caveat

B200M2

B200M3

UCS M3 server with Local RAID Boot

• Not all combination of drives tested

• Proceed with caution in this case

• UCSB-MRAID12G is mostly backward compatible with metadata of M3 servers but customer are advised to test and take backup of data before migration.

B200M3

B200M4

UCS Servers with SAN Boot

• Supported within UCS

• LUN Masking and Zoning all are unchanged

• Service Profile writes the adapter, and its identities onto the new hardware

• Must be within UCS HCL throughout

• Already Imaged OS’ and Service Profile Operations

• Prepare OS for migrations

• Disassociation

• Association

• OS boot and clean up

Summary

• Cisco UCS provides stateless computing with the concept of Service profile.

• Seamless compute life cycle management with Cisco UCS.

• Cisco ASIC technology allows this over a wider span of hardware variations and bare-metal instances

Participate in the “My Favorite Speaker” Contest

• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)

• Send a tweet and include

• Your favorite speaker’s Twitter handle @ravmishr

• Two hashtags: #CLUS #MyFavoriteSpeaker

• You can submit an entry for more than one of your “favorite” speakers

• Don’t forget to follow @CiscoLive and @CiscoPress

• View the official rules at http://bit.ly/CLUSwin

Promote Your Favorite Speaker and You Could be a Winner

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect.

Q&A

Thank you

Data Center / Virtualization Cisco Education Offerings

Course Description Cisco Certification

Cisco Data Center CCIE Unified Fabric

Workshop (DCXUF);

Cisco Data Center CCIE Unified Computing

Workshop (DCXUC)

Prepare for your CCIE Data Center practical exam with hands on lab

exercises running on a dedicated comprehensive topology

CCIE® Data Center

Implementing Cisco Data Center Unified Fabric

(DCUFI);

Implementing Cisco Data Center Unified

Computing (DCUCI)

Obtain the skills to deploy complex virtualized Data Center Fabric and

Computing environments with Nexus and Cisco UCS.

CCNP® Data Center

Introducing Cisco Data Center Networking

(DCICN); Introducing Cisco Data Center

Technologies (DCICT)

Learn basic data center technologies and how to build a data center

infrastructure.

CCNA® Data Center

Product Training Portfolio: DCAC9k, DCINX9k,

DCMDS, DCUCS, DCNX1K, DCNX5K, DCNX7K

Get a deep understanding of the Cisco data center product line including

the Cisco Nexus9K in ACI and NexusOS modes

For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com

Cloud Cisco Education OfferingsCourse Description Cisco Certification

Designing the FlexPod Solution (FPDESIGN);

Implementing and Administering the FlexPod

Solution (FPIMPADM)

Learn how to design, implement and administer FlexPod solutions FlexPod Design Specialist;

FlexPod Implementation &

Administration Specialist

UCS Director (UCSDF) Learn how to manage physical and virtual infrastructure using

orchestration and automation functions of UCS Director.

Cisco Prime Service Catalog Learn how to deliver data center, workplace, and application services in an

on-demand, automated, and repeatable method.

Cisco Intercloud Fabric Learn how to implement end-to-end hybrid clouds with Intercloud Fabric

for Business and Intercloud Fabric for Providers.

Cisco Intelligent Automation for Cloud Learn how to implement and manage cloud deployments with Cisco

Intelligent Automation for Cloud

For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com