PowerHA SystemMirror for AIX: New Features and Best Practice

© 2013 IBM Corporation

IBM Systems and Technology Group Technical Symposium

Auckland New Zealand | August 14 – 17, 2013

AdvancedTechnical

Skills

AHY24

PowerHA SystemMirror for AIX: New Features and Best Practice

Antony “Red” Steel - ATS

Auckland New Zealand | August 14 - 17 2013

#include <std_disclaimer.h>•These notes have been prepared by an Australian, sobeware of unusual spelling and pronunciation.

AdvancedTechnical

Skills

© 2013 IBM CorporationPowerHA SystemMirror



AdvancedTechnical

Skills

Contents● Introduction to PowerHA – Standard and Enterprise

● PowerHA maintenance and features

● PowerHA directions




AdvancedTechnical

Skills

Agenda● PowerHA Standard and

Enterprise Editions

● Cluster Aware AIX

● General changes

● What's new in PowerHA 7.1.1 and 7.1.2

● Walk through PowerHA configuration and demo of application

Standard Edition

Enterprise Edition

Centralised Management C-SPOC

Cluster resource management

Shared Storage management

Cluster verification framework

Integrated disk heartbeat

SMIT management interfaces

AIX event/error management

Integrated heartbeat

PowerHA DLPAR HA management

Smart Assists

Multi Site HA Management

PowerHA GLVM async mode

IBM Metro Mirror support

IBM Global Mirror support DS8700

EMC SRDF sync/async pending

Hitachi Truecopy

Stretched or linked clusters

DS8000 Hyper Swap




AdvancedTechnical

Skills

Introduction to PowerHA SystemMirror Standard● Introduction to PowerHA

● What is high availability

● Planning – designing high availability

● Features of PowerHA to keep your applications available

HACMP 5.5 went EOS 30/4/2012

GeoRM went EOS 30/9/2009

PowerHA SystemMirror 6.1AIX 7.1 with RSCT 3.1.0.0 Fixpack 6 Aug 2011AIX 6.1 TL2 with RSCT 2.5.4.0 GA Oct 2009AIX 5.3 TL9 with RSCT 2.4.12.0 EOS Sept 2014

PowerHA SystemMirror 7.1AIX 7.1 with RSCT 3.1.0.1 Fixpack 4 Sept 2011AIX 6.1 TL6 with RSCT 3.1.0.1 GA Sept 2010 EOS Sept 2014

PowerHA SystemMirror 7.1.1AIX 7.1 TL1 SP2 Fixpack 1 Feb 2012AIX 6.1 TL7 SP2 GA Dec 2011 EOS N/A

PowerHA SystemMirror 7.1.2AIX 7.1 TL2 SP1 Fixpack 1 Feb 2013AIX 6.1 TL8 SP1 GA Nov 2012 EOS N/A

StandardEdition

5765 H23

StandardEdition

5765 H23

StandardEdition

5765 H23

StandardEdition

5765 H39

EnterpriseEdition

5765 H40

EnterpriseEdition

N/A

EnterpriseEdition

5765 H24

EnterpriseEdition

N/A




AdvancedTechnical

Skills

●Introduction to High Availability

Causes of downtimeApplica-tion errorsOperating system er-rorsHardware failureOperator error

● PowerHA SystemMirror for AIX Standard Edition

● Cluster management for the data centre– Monitors, detects and reacts to events– Establishes a heartbeat between the systems– Enables automatic switch-over

● IBM shared storage clustering– Can enable near-continuous application service– Minimize impact of planned & unplanned outages– Ease of use for HA operations

● Smart Assists – application agents– Out of the box deployment for SAP and other popular applications

● Mature Product– 22 Major releases (averaging one a year)– Over 12,000 customers worldwide

● PowerHA SystemMirror for AIX Enterprise Edition– Cluster management for the Enterprise– Multi-site cluster management– Includes the Standard Edition function

Standish Group Research 2008-2010




AdvancedTechnical

Skills

Introduction to High Availability● High availability is:

– The reduction to close to zero for downtime (not fault tolerance)– Solution may address planned or unplanned down time– Solution need not be fault tolerant but should be fault resistant– Solution should eliminate single points of failure (SPOF)

● PowerHA is not the answer if– Cannot afford any downtime – life critical systems - Need a fault tolerant solution– Environment is not secure

● Many users with root access– Then environment is not stable

● Change management is not respected● You do not have trained administrators● Procedures are not well documented● Environment is prone to user fiddle factor

– Applications cannot be controlled● Scripts cannot be used to start/stop and recover applications




AdvancedTechnical

Skills

Eliminate single points of failure by:

● Node– Using multiple nodes

● Power source– Using multiple circuits or un-interruptible power supplies

● Network adapter– Using redundant network adapters and bonding (etherchannel etc)– Network Using multiple networks to connect nodes / clients

● TCP/IP subsystem– Using non-IP networks to connect nodes

● Disk adapter– Using redundant disk adapter or multipath hardware– Disk Using multiple disks with mirroring or raid

● Application– Adding node for takeover; configuring application monitor– VIO server Implementing dual VIO servers

● Site– Adding an additional site




AdvancedTechnical

Skills

Setting realistic expectations

What is considered an outage in your environment?

Unexpected downtime

Maintenance Tasks

What are the desired:– RTO – Recovery Time Objective– RPO – Recovery Point Objective

Availability Downtime90% (1-nine) 36.5 days/year99% (2-nines) 3.65 days/year99.9% (3-nines) 8.76 hours/year99.99% (4-nines) 52 minutes/year99.999% (5-nines) 5 minutes/year99.9999% (6-nines) 31 seconds/year




AdvancedTechnical

Skills

Building for availability

Infrastructure planning– Power Redundancy; I/O Drawers; SCSI Backplane; SAN HBAs / Multipathing; Virtualized

or Dedicated Deployments; Backup Strategies; Application Fallover Protection

LPM– Live move of OS/Application between frames; Workload management; Energy

management; Hardware management

Partition Suspend/Resume– Resume where stopped; suspend low priority workloads; Firmware updates without

stopping / restarting the application

Charm– Available on high end models (>= 770)– Perform CHARM during low-use periods– LPM critical partitions to other servers if possible– Depending on the repair, IBM may recommend quiescing critical applications on running

partitions– Have current backups before beginning, and make sure all configuration redundancy

requirements have been met– Use PowerVM Suspend / Resume to reduce CPU and active memory




AdvancedTechnical

Skills

Introduction to High Availability

● Planned– Maintenance– Upgrades– Testing– Development

● Unplanned– User Error– Application Failure– Component Failure– Operating System Failure– Environmental Disasters

PowerHA will help to mask or eliminate

Becoming a moreimportant area -PowerHA as anadministration tool

LPM is an alternative for But not for(or software upgrades etc)




AdvancedTechnical

Skills

You cannot let sleeping clusters lie

● Why touch the system ?? – has been working now for 2 years...– Hardware may need to be upgraded (6 monthly f/w update – 1/year may not be

concurrent).– Replacement hardware may be at unrecognisable firmware levels..– Application may need to be upgraded, which may require new software levels or fixes– OS and/or application out of support– Business expands– PowerHA designed to manage/support upgrade process

● Rolling upgrades● Snapshot conversions




AdvancedTechnical

Skills

High Availability options● One site – HA

– PowerHA SystemMirror● Dual servers, shared storage● Site only single point of failure

● Disaster Recovery– Replication

● GLVM● Storage / Database

– PowerHA SystemMirror Enterprise Ed.

● PowerHA managing application and storage replication

– GLVM– SVC; Storewise;

MetroMirror; GlobalMirror– EMC SRDF / Hitachi

TrueCopy/HUR

>> Planning and preparation




AdvancedTechnical

Skills

Cluster Aware AIX

● IBM Cluster products (RSCT, PowerHA, VIOS...) use CAA– CAA is a toolset – doesn’t form a cluster (now concept of quorum or fencing nodes –

but provides tools to manage these)

● All interfaces are monitored– lscluster -i

● All nodes monitored– lscluster -m

● Changes from 2010– No consistent view of devices– SolidDB no longer used– No zones / sub-clusters– Secure communication between nodes– Deadman switch (DMS)– A node is detected if isolated – can generate an AHAFS event or crash the node– clctrl -tune -o deadman_mode (clctrl -tune -L to list)




AdvancedTechnical

Skills

Cluster topology

NodeB NodeCNodeA

network2

network1nA_n1_boot1

nA_n2_boot1nA_n2_boot2

nA_n1_boot2

hdisko hdiskphdiskn

Repository disk

nB_n1_boot1

NodeC




AdvancedTechnical

Skills

Cluster topology

NodeB NodeCNodeA

network2

network1nA_n1_boot1


nA_n1_boot2

hdisko hdiskphdiskn

Repository disk

nB_n1_boot1

NodeCNodeANodeANodeA

RG1

app_mon1

vg2

RG2

app_mon2

vg1

nA_alnC_al

rg1_n1_svc1rg1_n2_svc1

rg2_n1_svc1

rmt0

rg2_n2_svc1




AdvancedTechnical

Skills

Cluster topology

NodeB NodeCNodeA

network2

network1nA_n1_boot1


nA_n1_boot2

hdisko hdiskphdiskn

Repository disk

nB_n1_boot1

NodeCNodeANodeANodeA

RG1

app_mon1

vg2

RG2

app_mon2

vg1

nA_alnC_al

rg1_n1_svc1rg1_n2_svc1

rg2_n1_svc1

rmt0

rg2_n2_svc1Policies

Policies




AdvancedTechnical

Skills

Cluster behaviour

● Resource Group Policies– Startup

● Online on home node only● Online on first available● Online on all available● Start up distribution

– Failover● Failover to next node in the list● Failover using Dynamic node

priority (CPU, Paging space, Disk IO, Adaptive (user defined))

● Bring offline– Fallback

● Fallback to higher priority node● Never fallback

● Resource group dependencies

● IP distribution preferences

● Inter site management policies– Online on Both Sites– Online on Either Site– Prefer Primary Site– Ignore




AdvancedTechnical

Skills

Resource Group dependencies● Online on same node dependency

– Resource groups come online on the same node

● Parent child dependency– Child will come online after the parent is stable, will go offline if the parent goes

offline. Can have up to 3 levels

● Online on different node dependency– High, intermediate and low– High will force intermediate and low to move, intermediate will force low to move– Same priority cannot come online on same node– Same priority will not cause a movement

DB

n1

App

n2

Test

n3

App

n2 n3

On same node dependencyDB: n1,n3,n2 HighApp: n2,n3,n1 IntermediateTest n3,n2,n1 Low

Parent / ChildDB – parent; App - Child Test App

n2 n3

DB




AdvancedTechnical

Skills

IP distribution preferences

● Collocation– All Service labels will be on the same “adapter”

● Collocation with persistent– all service labels will be on the same “adapter” as the persistent IP.

● Collocation with Source– all service labels will be on the same “adapter” and the customer can choose the source

IP of the outgoing packets

● Anti-collocation– all resources of this type will be allocated on the first “adapter” which is not already

serving (or serving the least number of) addresses

● Anti-collocation with 1st Source– Same as above with the service IP being the source address of all outgoing packets.

● Anti-collocation with Persistent Labels– service labels will almost never be on the same “adapter” as the persistent IP, that is,

service will occupy a different interface as long as one is available, but if no other is available then they will occupy the same interface.

● Anti-collocation with Persistent Labels and Source– Same as above with all outgoing packets having the service IP as the source address.




AdvancedTechnical

Skills

Are you using PowerHA features

● Are you aware of / using– Fast failure detection– File collections– Application monitoring

● Startup, long running or both● Process or custom

– CSPOC– Cluster Test tool

● Remember that in the new versions of PowerHA, the developers used feedback from the field/PMRs to fix common problems

missing heartbeat check




AdvancedTechnical

Skills

Cluster Aware AIX (cont)

● Debugging– snap caa

● Logging via syslog– lscluster -s for stats

● lsattr -El cluster0– Obtains node and repository disk UUID

● /usr/lib/cluster/clras lsrepos– Lists valid cluster repository disks

● /usr/lib/cluster/clras sfwinfo -d hdisk2– Displays storage framework UUID for disks

● /usr/lib/cluster/clras dumprepos– Displays contents of cluster repository disk




AdvancedTechnical

Skills

Cluster Aware AIX● Kernel based

● A set of services/tools embedded in AIX to help manage a cluster of AIX nodes and/or help run cluster software on AIX– IBM cluster products (including RSCT, PowerHA, and the VIOS) will use and/or

call CAA services/tools.– CAA services can assist in the management and monitoring of an arbitrary set of

nodes and/or running a third-party cluster.

● CAA does not form a cluster by itself. It is a tool set.– There is no notion of quorum. (If 20 nodes of a 21 node cluster are down, CAA

still runs on the remaining node).– CAA does not eject nodes from a cluster. CAA provides tools to fence a node but

never fences a node and will continue to run on a fenced node

● Requires a repository disk (protected at the storage level)

● By default all interfaces monitored

● snap caa to collect PD data




AdvancedTechnical

Skills

Cluster Aware AIX (cont)● All nodes are monitored.

– Cluster Aware AIX tells you what nodes are in the cluster plus information on those nodes ,including state. A special “gossip” protocol is used over the multicast address to determine node information and implement scalable reliable multicast. No traditional heartbeat mechanism is employed. Gossip packets travel over all interfaces, including storage.

● CAA monitors both communication interface states and points-of-contact between nodes on a node-by-node basis– A point-of-contact indicates that a node has received a packet from the other node

over the interface.– A point-of-contact “up” state indicates that the packet flow continues between the

nodes.– A point-of-contact “down” state indicates that the packet flow does not continue

between the nodes, even though the interface may be in an “up” state.

● Note: The ability to monitor this particular condition is very important. An interface in the “up” state and a point-of-contact in a “down” state can occur because of hardware or other network issues between these particular nodes.




AdvancedTechnical

Skills

Cluster Aware AIX (cont)● Cluster disks.

– CAA has information on all disks in the cluster – including their state. (3rd party disks do not participate in the monitoring).

● SolidDB and cluster disk naming dropped in 2010

● In 2011 added:– Deadman switch for isolated nodes – tuneable and response options.– 3rd party disk support added– Synchronous changes allowed across the cluster– Improved logging and RAS tools

● In 2012 added:– 2 sites – Linked or stretched clusters

● Stretched Cluster (Single CAA cluster; Single Repository Disk; Require multi-cast across 2 sites; Cluster communication:- Networks, SAN, or Disk)

● Linked Cluster (Linked CAA cluster; 2 Separate Repository Disks; One local repository on each site; Synchronized between sites; Cluster communication:- Networks)




AdvancedTechnical

Skills

Res

ourc

e M

onito

ring

and

Con

trol

Group ServicesResource Manager

PowerHA 6.1

RSCT

AIX

Topology Services




AdvancedTechnical

Skills

Res

ourc

e M

onito

ring

and

Con

trol

Group ServicesResource Manager

PowerHA 7.1

RSCT

AIX CAA




AdvancedTechnical

Skills

● PowerHA 6.1● Heartbeat Rings: detailed protocol

– Leader, Successor, Mayor etc– Difficult to add/delete nodes

● Requires IP aliases management in the subnet

● PowerHA 7.1● Multicast based protocol

– Discover and use as many adapters as possible– Use network and SAN as needed– Adapt to the environment: delay, subnet etc

● Kernel based cluster message handling

Cluster aware AIX – Topology managementHost 2

Host 3

Host 4

Host 1

MULTICAST

Host 2

Host 3

Host 4

Host 1




AdvancedTechnical

Skills

Default Multi Channel Health Management

Network

SAN

HeartbeatsReliable Messaging

Heartbeats Reliable Messaging

Host 1 Host 2

Minimal Setup

Multiple channels of communication– Network– SAN– Central Repository

Heartbeats

First line of Defence

Second line of Defence

Third line of Defence Cluster Repository

3 lines of (redundant) independant communications




AdvancedTechnical

Skills

Configure SAN heart beating in virtual environment




AdvancedTechnical

Skills

General Changes

● Disk Handling Changes – ECMVG required– Existing volume groups automatically converted

● No user action required, no override allowed● Done by call to cl_makecm out of node_up● C-SPOC creates all volume groups as ECM● Either “Fast Disk Takeover” or “Concurrent Access”● Active/Passive mode used for non-concurrent resource groups

– No SCSI-2 disk reserves set or broken● Most disk differences now irrelevant● Disk reserve handling code – cl_disk_available – retained for migration● Fast path through code if ECM and no reserves




AdvancedTechnical

Skills

PowerHA 7.1.1

● Key dates:– Announce: October 12– General Availability: December 16

● Lifecycle information: – http://www-01.ibm.com/software/support/lifecycle/index_h.html

● Offerings:– Standard Edition has “base” function plus Smart Assists– New features added to Enterprise Edition 6.1 (only) – no 7.1 EE

● RSCT and AIX requisites– AIX 6.1 TL 7 with bos.cluster.rte 6.1.7.2 (SP2) APAR IV09929

ORAIX 7.1 TL 1 with bos.cluster.rte 7.1.1.2 (SP2) APAR IV09868

– RSCT 3.1.1.0 – works with either versions of AIX




AdvancedTechnical

Skills

PowerHA 7.1.1 (cont)

● New features – Standard edition– Security features– Encrypted Filesystem, Role Based Access Control, LDAP

● Smart Assists– Expanded middleware support including SAP MaxDB HotStandby and Websphere

MQ Series

● IBM Systems Director plug-in– Extends features available through Director

● Cluster Aware AIX– New features

● Miscellaneous updates– CSPOC enhancements, migration, synchronous application startup




AdvancedTechnical

Skills

PowerHA 7.1.1 (cont)

● New features – Enterprise edition (6.1)– XIV replication support– Global Mirror support enhancements– Enterprise Edition 6.1 requires SystemMirror 6.1 with Service Pack 7

● SP7 and new install images available from FixCentral– http://www.ibm.com/support/fixcentral/aix/selectFixes follow the links to select

IV11782 (packaging APAR) – New support included in existing genxd fileset (updates only)




AdvancedTechnical

Skills

PowerHA 7.1.1 Smart Assist support

DB2 Enterprise Edition 9.5 9.7WAS 6.1 6.1WAS N/D 6.1 6.1HTTP Server 6.1 6.1TSM 6.1 6.2TDS 5.2 6.3

4.5.1 4.5.1Lotus Domino Server 8.5.1

11g r1Oracle Application Server 10g r2SAP

V7.6- Oracle 10g r2 10g r2- DB2 9.7MQ Series 7.0.1.5AIX print server AIX 6.1AIX DHCP AIX 6.1AIX DNS AIX 6.1

SystemMirror 7.1.0 SystemMirror 7.1.1

Filenet

Oracle Databse

SAP ERP netweaver 2004s SAP SCM 7.0 with Netweaver 7.0 EHP1 for FVT SAP SCM 7.0 with Netweaver 7.0 EHP1 for SVT

- MaxDB




AdvancedTechnical

Skills

PowerHA 7.1.1 Federated Security

● All user, RBAC, encrypted FS credentials in a central store– Can use existing LDAP or Windows server

● Role based access (RBAC)– Roles:

ha_admin: Administratorha_op: Operatorha_mon: Monitorha_view: Viewer

● Support for Encrypted filesystems– Shared filesystem or LDAP for keystore




AdvancedTechnical

Skills

PowerHA 7.1.2● Version 7.1.2 offers both a Standard and an Enterprise Edition.

– The Enterprise Edition provides for Disaster Recovery solutions with both host based mirroring and storage based mirroring

● IPv6 support is enabled with this version for v7 product

● Simpler to deploy and easier to manage multi-site configurations with IBM Systems Director, intuitive interfaces, multi-site install wizard– Stretched Cluster; Cluster wide AIX commands, kernel based event

management single repository multicast communications – Linked Clustering; cluster wide AIX commands, kernel based event

management, linked clusters with unicast communications & dual repositories

● HyperSwap capability is introduced. – HyperSwap with DS8800 storage subsystems provides for continuous

availability against storage failures.

● Cluster Split/Merge technology for managing split-site policy scenarios




AdvancedTechnical

Skills

PowerHA 7.1.2● Cross Site Mirroring using LVM mirror pools

● Enhancements to the Director plugin to facilitate the use of these new features

● Software Levels Required:– OS – AIX 6.1 TL8 SP1– OS – AIX 7.1 TL2 SP1– PowerHA SystemMirror 7.1.2 SP1– Additonal software requirements for Enterprise Editionand HyperSwap – PowerHA

SystemMirror 7.1.2 APAR IV27586




AdvancedTechnical

Skills

PowerHA 7.1.2

● High Availability and disaster recovery across multiple sites– PowerHA SystemMirror for AIX Enterprise Edition – Adds long distance failover for Disaster Recovery– Low cost host based mirroring support– Extensive support for storage array replication– Short distance (Campus to 80-100km) deployment: Synchronous– Long distance ( >100km) deployment: Asynchronous

New York London

Site 1

Network

Site 2

Fiber

Host Mirroring

Storage Mirroring

Enterprise Edition

Replication Technology Sync Async

Host Replication

Geo LVM

Storage Array Replication

IBM DS8K Series Storage - PPRC

SVC, Storevize,

XIV

EMC – SRDF *

Hitachi – Universal Replicator,Truecopy *

HP – Continuous Access *




AdvancedTechnical

Skills

PowerHA 7.1.2 Multi Sites Stretched Cluster

Linked Cluster

Inter site communication Multicast Unicast

Repository disk Shared Separate

Cluster Communication NetworksSANDisk

NetworksSAN in future

Cross site LVM mirroring

HyperSwap

Multi site Conncurrent RG with HyperSwap

Fig 1: Multi Sites with Stretched Cluster

Site 1 Site 2

RepositoryDisk

Site 1

RepositoryDisk 1

Site 2

RepositoryDisk 2

Links

Fig 2: Multi Sites with Linked Clusters

Standard Enterprise

Multi Site Definition Site Service IP Site Policies

Stretched Cluster

Linked Clusters

HADR with Storage Replication Management

HyperSwap




AdvancedTechnical

Skills

Tie breaker support

● PowerHA 7.1.2 Tie BreakerSupport– Separate Site Split and Merge

policies– Split/Merge: Tie Breaker policy– FC/iSCSI Tie Breaker– SCSI 3 reservation disk– Losing side is quiesced

Site 1Cluster

Site 2

Shared DiskTie Breaker

SCSI or iSCSI

Site 3

Policy Setting Split Merge Comments

Tie Breaker Tie break Holder side wins

Majority Rule >N/2 side wins

In case of N/2, side that includes node with the smallest node id

Manual Manual steps needed for recovery to continue

More suited for Linked Clusters




AdvancedTechnical

Skills

HyperSwap Technology

SyncMirror

Application

Hyperswap

Cluster

Primary DS8KSite 1

Secondary DS8KSite 2

Legend:

Active Path

Passive Path

Continuous Availability against Storage failures

Substitutes storage secondary to take the place of failed primary device– Non-disruptive - applications keep running– Key value add to HA/DR deployments

– Unplanned HyperSwap:

• Continuous Availability against storage failures

– Planned HyperSwap:

• Storage Maintenance without downtime

• Storage migration without downtime

Customer Benefits

HA/DR




AdvancedTechnical

Skills

HyperSwap Support by AIX-PowerHA

• HyperSwap device configuration transparent to application– Application can continue to use the device as before

Primary DS8KSecondary DS8K

/dev/hdiskX /dev/hdiskY

Application/LVM/Middleware

ConfigureHyperSwap

Primary DS8KSecondary DS8K

/dev/hdiskX /dev/hdiskY

Application/LVM/Middleware

/dev/hdiskX

SYNCSYNC

HyperSwap Pair

SYNC




AdvancedTechnical

Skills

HyperSwap Multi Site Deployments: Oracle RAC Example

Active-Active Sites (Future)– Active-Active workload across sites– Continuous availability of site compute

infrastructure and storage outages– Oracle RAC long distance deployment

Compute Node outages: – Active-Active workload provides continuous availability

Storage outages:– HyperSwap provides continuous availability

Fig 2: Active-Active HyperSwap

Oracle RACSite 1

N1-1 N1-2

S1

(Active) (Active)

N2-1 N2-2

S2

(Active) (Active)

< 100 KM

Site 2

Fig 1: Active-Passive HyperSwap

Oracle RACSite 1

N1-1 N1-2

S1

(Active) (Active)

N2-1 N2-2

S2

(Passive) (Passive)

< 100 KM

Site 2

PowerHA Cluster

SYNC

SYNC

Active-Passive Sites– Active-Active workload within a site– Active-Passive across sites– Continuous availability for site storage outages




AdvancedTechnical

Skills

Pre-requisites

● Additional AIX Fileset Requirements:– bos.cluster.rte ← CAA Fileset– bos.cluster.solid ← Solid DB (not required in 7.1.1)– bos.ahafs ← Autonomic Health Advisor Filesystem

● ECM VGs are Required in 7.1– not pre-reqd but required ← bos.clvm.enh

● Configuration Files– /etc/cluster/rhosts on the node where cluster will be created– /etc/hosts the nostname is the first alias for that IP address

● Topology services daemon is no longer used– CAA uses Scalable Reliable Multicast (SRM) for monitoring all network and storage

interfaces using a single cluster-wide multicast IP address

● Can automatically define Multicast Address for you– Range 224.0.0.0 –239.255.255.255




AdvancedTechnical

Skills

Implementation differences

● New LAN Switch Settings– IP Multicasting Enabled

● Address Automatically selected during cluster configuration● Set on Network Switches

– IGMP_snooping Enabled● Will reduce the amount of Multicast Traffic on LAN switches

● TME must be enabled on HBAs to leverage SAN heartbeating– List of supported Adapters in the slide notes– Additional steps for virtual HBAs (later slide)

● Repository Disk requirement– CAA Requirement (documented size has changed)

● This value can now be altered to 512MB or higher (max is 460GB)● Larger disks will only result in wasted space

– VSCSI volumes are supported




AdvancedTechnical

Skills

Implementation differences (cont).

● All network adapters will be discovered and used– To exclude adaters, use:

/etc/cluster/ifrestrict:en4en5

● IPAT via Aliasing Only– No IPAT via Replacement– No Heartbeating over Aliases

● Network types supported– Ether– Infiniband (soon)– Notice that FDDI, TMSSA, TMSCSI and others are gone

● Removed Serial Network Types– RS232 Serial network– Disk heartbeat networks– No Multi-node disk heartbeat

mping to testbroadcast




AdvancedTechnical

Skills

Zoning requirements for HBA heartbeating

WWPN

WWPN

Storage subsystem

WWPN

WWPN

WWPN

WWPN

WWPN

WWPN

optionalheartbeatzone

sharedstoragezone




AdvancedTechnical

Skills

Zoning requirements for HBA heartbeating

WWPN

WWPN

Storage subsystem

WWPN

WWPN

WWPN

WWPN

WWPN

WWPN

optionalheartbeatzone

individualsharedstoragezone




AdvancedTechnical

Skills

Tools

● Cluster test tool

● Application availability analysis tool

● File collections

● Automatic cluster verification

● Automatic Error Notification (can also be customized)

● Auto-corrective/Self healing clusters

● Custom Pager notification methods (including SMS)

● OEM Volume and Filesystem Support (Veritas) and Custom disk methods

● Non-disruptive startup (create cluster around existing environment)

● Cluster snapshots to save/restore clusters (XML format allows easy editing)




AdvancedTechnical

Skills

Cluster test tool

● Automated test plan– Important part of install process– Still important as regular procedure once in production– Many cluster administrators believe testing too time consuming and costly– Lack of testing leads to failures– Conducts a series of tests and then analyzes them– Will start all nodes, then preform node down with and without takeover on random

nodes; network and application down. There are some limitations.

● Custom test procedure - user defined plan

● Designed to test the configuration, not the operation of the cluster manager




AdvancedTechnical

Skills

Tests

● NODE_UP: start one or more nodes

● NODE_DOWN_FORCED: stop a node forced

● NODE_DOWN_GRACEFUL: stop one or more nodes

● NODE_DOWN_TAKEOVER: stop a node with takeover

● CLSTRMGR_KILL: catastrophic software failure

● NETWORK_DOWN_LOCAL: stop a network on a node

● NETWORK_UP_LOCAL: restart a network on a node

● SERVER_DOWN: stop an application server

● WAIT: pause testing

● RG_ONLINE, RG_OFFLINE, RG_MOVE, RG_MOVE_SITE:Resource Group online,offline, move and site move

● JOIN_LABEL, FAIL_LABEL: Interface fail and join

● VG_DOWN: loss of VG




AdvancedTechnical

Skills

Tests (cont)

● NETWORK_UP/DOWN_LOCAL: Local network up and down

● SITE_UP, SITE_DOWN_GRACEFUL, SITE_DOWN_TAKEOVER: site up and down graceful or takeover

● SITE_ISOLATION, SITE_MERGE: Site isolation and re-integration

● Non-IP networks now tested

1/10/2005_07:20:24: 11/10/2005_07:20:24: | Validate NODE_UP11/10/2005_07:20:24: 11/10/2005_07:20:24: Event node: ALL11/10/2005_07:20:24: Configured nodes: ha1 ha211/10/2005_07:20:24: Event 2: NODE_DOWN_GRACEFUL: NODE_DOWN_GRACEFUL,node1,Stop cluster services gracefully on a node11/10/2005_07:20:24: 11/10/2005_07:20:24: | Validate NODE_DOWN_GRACEFUL11/10/2005_07:20:24: 11/10/2005_07:20:24: Event node: ha111/10/2005_07:20:24: Configured nodes: ha1 ha211/10/2005_07:20:24: Event 3: NODE_UP: NODE_UP,node1, Restart cluster services on the node that was stopped11/10/2005_07:20:24: 11/10/2005_07:20:24: | Validate NODE_UP11/10/2005_07:20:24: 11/10/2005_07:20:24: Event node: ha111/10/2005_07:20:24: Configured nodes: ha1 ha211/10/2005_07:20:24: Event 4: NODE_DOWN_TAKEOVER: NODE_DOWN_TAKEOVER, node2,Stop cluster services with takeover on a node




AdvancedTechnical

Skills

Application availability analysis tool

Application Availability Analysis Type or select values in entry fields. Press Enter AFTER making all desired changes. [Entry Fields]* Select an Application [test_appl01] +* Begin analysis on YEAR (19702038) [2005] #* MONTH (0112) [01] #* DAY (131) [01] #* Begin analysis at HOUR (0023) [00] #* MINUTES (0059) [00] #* SECONDS (0059) [00] #* End analysis on YEAR (19702038) [2005] #* MONTH (0112) [06] #* DAY (131) [30] #* End analysis at HOUR (0023) [23] #* MINUTES (0059) [59] #* SECONDS (0059) [59] #

Analysis begins: Saturday, 01Jan2005, 00:00Analysis ends: Thursday, 30June2005, 23:59Application analyzed: test_appl01Total time: 180 days, 23 hours, 59 minutes, 59 secondsUptime: Amount: 180 days, 22 hours, 58 minutes, 29 seconds Percentage: 99.97% Longest period: 98 days, 16 hours, 48 minutes, 3 secondsDowntime: Amount: 0 days, 1 hours, 1 minutes, 30 seconds

Good log for initial PD




AdvancedTechnical

Skills

Upgrade considerations

● Non-Disruptive Upgrade functionality is NOT available to get to 7.X– Can use non-disruptive upgrade to load patches:

● ie. Source 7.1.0.1 to Target 7.1.0.4

● Migration to 7.X releases is different than prior releases– Migration is disruptive– Requires the use of clmigcheck utility– Requires some reconfiguration of cluster topology

● If running older versions of HA you have a decision to make:– Migrate or Start at PowerHA version 7.1.0 or 7.1.1– Migrating from 7.1.0 to 7.1.1 is disruptive– (7.1.1 requires newer AIX levels which provide CAA enhancements)




AdvancedTechnical

Skills

Designing High Availability

● Designing High Availability– A spare should be available for every single hardware and software component that is

required to keep application running● No ‘Single Point of Failure”

– Whilst a generally accepted principle, not always adhered to– Cut to reduce cost– effects of the failure of a single component not always thought through

eg single adapter networks, no serial/failed serial network– Nodes– Power feed– Storage– Networks– Adapters– Administrators (good documentation 'clear' design)– Applications




AdvancedTechnical

Skills

PowerHA usability changes

● Mount Guard– A new JFS2 facility to help prevent accidental double mounts– LVM and CAA can help, but not ensure– A second mount, without an intervening unmount will be rejected. Mount state is

maintained on the disks– Set by chfs option, can be changed by chfs and logredo– Available in bos.filesystems 7.1.1 or 6.1.7– Available in HA 6.1 and 5.5

● Private Networks– Reserve a network for Oracle – Oracle needs a network with no heartbeat etc.– PowerHA < 6.1 supported, 7.1 didn’t– PowerHA 7.1.1 restores ability to make network as private.




AdvancedTechnical

Skills

PowerHA usability changes (cont)

● Application start in debug mode

● DARE Progress Indicators– What’s going on, and when is it done– Terminal is locked– Back ported to HA 6.1 and 5.5

● Mirror Pools– NB: PowerHA 7.1 didn’t support Xsite mirroring, but PowerHA 7.1.1 has concept of

sites and uses Mirror Pools to handle cross site mirroring

● Renaming Physical Volumes optional since 7.1

● Shared physical volumes can be given consistent names across the cluster

● Cannot be part of a VG when renamed

● Foreground Application Start

● Application server can now be started in foreground

● Simplifies design of scripts, but poor scripts can lead to config_too_long

● Startup in Debug mode – warning exit code currently not checked.

● User can respond immediately to start failure




AdvancedTechnical

Skills


● Network changes– New ways to specify the source IP address for outgoing network traffic– The following are the new policies for Service IP Distribution Preference:

● Anti-Collocation with 1st Service– Each Service label will be placed on a different adapter and the service

address is the source address of all outgoing traffic● Collocation with 1st Service

– All the Service labels are placed on one adapter and the customer can choose an address as a source for all outgoing traffic

● Anti-collocation with Persistent with 1st Service– Each service label will be the source address

● The swap adapter will use the new “transfer” option of ifconfig– This should help with problems associated with default and user specified routes

● CLCOMD now uses all unrestricted interfaces




AdvancedTechnical

Skills


● Two-Node disk heartbeat– Easy set up, change and test (only 5.5 and 6.1)

● New Heartbeat Tuning Parameters– Grace Period: The amount of time (seconds) the node will wait before marking a

node as DOWN. Accepted values are between 5 and 30 Seconds.– Failure Cycle: The frequency of the heartbeat. Accepted values are between 1 and

20 seconds– Settings apply to all networks across the cluster.

● Notes on Migration– Check carefully as not many configurations can be migrated




AdvancedTechnical

Skills

PowerHA usability changes (cont)● Repository Resiliency

– In PowerHA 7.1.0, the node shuts down on when the repository disk fails● Disk failure or lost connection

– CAA will provide Repository Resiliency● Requires AIX 6.1.7 SP4 or AIX 7.1.0 SP3, PowerHA 7.1.1 SP1● Node continues running even on repository disk failure, using locally cached

information● Kept in the kernel● User can provide a new disk on which to rebuild the repository● No changes allowed while repository is out of service

– On repository failure● Message posted to hacmp.out● Repeated on config_too_long pattern● DARE and sync continue to function, but any CAA topology changes are

rejected– User must recognise repository failure, and allocate a new disk

● SMIT path under Manage the Cluster -> Select a new Repository Disk




AdvancedTechnical

Skills

7.1 - clmgr – cluster command line

● Director plug-in neededs a consistent interface for SystemMirror.– Simplify management of clusters from Director– Reduce maintenance overhead

● Replacement to CLVT– Current Smart Assists utilize CLVT

● Overcomes previous CLVT limitations– Limited trace output and logging; Difficult to use

● clmgr is a hard link to clvt, clvt is a binary– the *only* binary in the clmgr code base, – all other code is ksh93

/usr/es/sbin/cluster/utilities/clmgr

● Added 100% tracing coverage, with multiple levels– all STDERR output is written to “/var/hacmp/log/clutils.log”

● Fully globalized; uses the “command.cat” message catalog. Added dozens of consistent error messages, and large amounts of automatic help– Consolidated the set of supported actions and attributes




AdvancedTechnical

Skills

7.1 - clmgr – cluster command line (cont)

● Supported actions– add– delete– manage– modify– move– offline– online– query– recover– sync– view

– file_collection– fallback_timer– volume_group *– logical_volume *– file_system *– physical_volume *– method*– report– snapshot– tape

● Supported object classes– cluster– site– node– interface– network– resource_group– service_ip– persistent_ip– application_controller – application_monitor– dependency

* incomplete coverage of features




AdvancedTechnical

Skills

clmgr examples

7801p20:/usr/local/scripts# clmgr online cluster WHEN=now MANAGE=auto \ BROADCAST=false CLINFO=true FORCE=false FIX=interactively

<snip>

s7801p22: Aug 12 2012 21:04:35 Checking for srcmstr active...s7801p22: Aug 12 2012 21:04:35 complete.s7801p22: Aug 12 2012 21:04:35 s7801p22: /usr/es/sbin/cluster/utilities/clstart: called with flags -m -G -i -B -As7801p22: s7801p22: Aug 12 2012 21:05:10 s7801p22: Completed execution of /usr/es/sbin/cluster/etc/rc.clusters7801p22: with parameters: -boot -N -A -i interactively -P cl_rc_cluster.s7801p22: Exit status = 0




AdvancedTechnical

Skills

s7801p20:/usr/local/scripts# lscluster -cCluster query for cluster pleiades returns:Cluster uuid: 527e26c4-99b8-11e1-a0e3-1293071a2808Number of nodes in cluster = 3 Cluster id for node s7801p20 is 1 Primary IP address for node s7801p20 is 10.2.55.120 Cluster id for node s7801p21 is 2 Primary IP address for node s7801p21 is 10.2.55.121 Cluster id for node s7801p22 is 3 Primary IP address for node s7801p22 is 10.2.55.122Number of disks in cluster = 0Multicast address for cluster is 228.2.55.120

lscluster command

● lscluster flags-i Lists the cluster configuration interfaces on the local node.-n Allows the cluster name to be queried for all interfaces-s Lists the cluster network statistics on the local node.-m Lists the cluster node configuration information.-d Lists the cluster storage interfaces.-c Lists the cluster configuration.




AdvancedTechnical

Skills

clRGinfo command

s7801p20:/usr/local/scripts# clRGinfo -p

Cluster Name: pleiades

Resource Group Name: test_rgNode Group State ---------------------------- --------------- s7801p20 ONLINE s7801p21 OFFLINE s7801p22 OFFLINE




AdvancedTechnical

Skills

lssrc command output changed

s7801p20:/usr/local/scripts# lssrc -ls clstrmgrESCurrent state: ST_STABLEsccsid = "$Header: @(#) 61haes_r710_integration/14 43haes/usr/sbin/cluster/hacmprd/ main.C, hacmp, 61haes_r710, 1038A_61haes_r710 2010-08-27T05:11:44-05:00$"i_local_nodeid 0, i_local_siteid -1, my_handle 1ml_idx[1]=0 ml_idx[2]=1 ml_idx[3]=2 There are 0 events on the Ibcast queueThere are 0 events on the RM Ibcast queueCLversion: 12local node vrmf is 7103cluster fix level is "3"The following timer(s) are currently active:Current DNP valuesDNP Values for NodeId - 1 NodeName - s7801p20 PgSpFree = 126661 PvPctBusy = 0 PctTotalTimeIdle = 98.523127DNP Values for NodeId - 2 NodeName - s7801p21 PgSpFree = 127610 PvPctBusy = 0 PctTotalTimeIdle = 98.945318DNP Values for NodeId - 3 NodeName - s7801p22 PgSpFree = 126483 PvPctBusy = 0 PctTotalTimeIdle = 98.801866




AdvancedTechnical

Skills

cldump command

s7801p20:/usr/local/scripts# cldump

cldump: Waiting for the Cluster SMUX peer (clstrmgrES)to stabilize.............Failed retrieving cluster information.

There are a number of possible causes:clinfoES or snmpd subsystems are not active.snmp is unresponsive.snmp is not configured correctly.Cluster services are not active on any nodes.

Refer to the HACMP Administration Guide for more information.




AdvancedTechnical

Skills

cltopinfo command

s7801p20:/usr/local/scripts# cltopinfoCluster Name: pleiadesCluster Connection Authentication Mode: StandardCluster Message Authentication Mode: NoneCluster Message Encryption: NoneUse Persistent Labels for Communication: NoRepository Disk: caa_private0Cluster IP Address: 228.2.55.120There are 3 node(s) and 1 network(s) definedNODE s7801p20: Network net_ether_02 srvc1 10.2.50.120 s7801p20b 172.3.1.20NODE s7801p21: Network net_ether_02 srvc1 10.2.50.120 s7801p21 10.2.55.121NODE s7801p22: Network net_ether_02 srvc1 10.2.50.120 S7801p22 10.2.55.122<snip>Resource Group test_rg Startup Policy Online On Home Node Only Fallover Policy Fallover To Next Priority Node In The List Fallback Policy Never Fallback Participating Nodes s7801p20 s7801p21 s7801p22 Service IP Label srvc1




AdvancedTechnical

Skills

Cluster wide execution

clcmd lssrc -g caa

-------------------------------NODE s7801p22-------------------------------Subsystem Group PID Status cld caa 6750432 active clcomd caa 7012576 active clconfd caa 7798794 active solidhac caa 6815926 active solid caa 8847410 active

-------------------------------NODE s7801p20-------------------------------Subsystem Group PID Status cld caa 5832952 active clcomd caa 6553816 active solid caa 7929910 active solidhac caa 8454150 active clconfd caa 8388622 active

● Command is– /usr/sbin/clcmd

● Provided by CAA, distributes command to all nodes (or a subset of the nodes) in cluster (or clusters)– Similar to dsh




AdvancedTechnical

Skills

IBM Systems Director: PowerHA management interface

No charge plug-inMasks complexityCentral management Real-time statusSmart Assist integrationDeployment wizards

73




AdvancedTechnical

Skills

PowerHA 7.1.2 Director Plugin Enhancements

● Wizards– Cluster Create Wizard

• Single Site and Multi Site deployment– Resource Group Creation Wizard

• Custom and Smart Assist based RG deployment

– SAP liveCache HotStandby solution Wizard– Federated Security Setup Wizard– Volume Group Create Wizard

• Support for LVM Mirror Pools– Replication (Mirror) Group Wizard

• HyperSwap Setup

● Management Enhancements– Repository Disk/s Management– Resource Groups management

• Snapshots, networks, log files etc– Reports Management– Notifications management– Event driven callouts– Capacity upgrade based fallovers– HyperSwap Management– File collections




AdvancedTechnical

Skills

PowerHA 7.1.2 Director Plugin: Multi Site Management




AdvancedTechnical

Skills

76

System Director Plug-in: Basic Architecture

Three-tier architecture provides scalability:

User Interface Management Server Director Agent

Director Server

Central point of control

Supported on AIX, Linux, and Windows

Agent manager

User Interface

Web-based interface

Command-line interface

PowerHA

AIX

DirectorAgentP D

P D

P D

P D

P D

P D

P D

Discovery of clusters and resources

Director Agent

Automatically installed on AIX 7.1 & AIX V6.1 TL06

Secure communication




AdvancedTechnical

Skills

System Director Plug-in – Getting Started




AdvancedTechnical

Skills

All communication interfaces are monitored• Cluster Aware AIX tells you what interfaces have been discovered on a node and

information on those interfaces including state

All cluster disks are monitored• Cluster Aware AIX tells you what disks are in the cluster and information on

those disks including state• All monitors implemented at a low-level of the AIX kernel, therefore they are

largely insensitive to system load

All nodes are monitored• Cluster Aware AIX tells you what nodes are in the cluster and information on

those nodes including state. A special “gossip” protocol is used over the multicast address to determine node information and implement scalable reliable multicast. No traditional heartbeat mechanism is employed. Gossip packets travel over all interfaces including storage.

Monitoring Services




AdvancedTechnical

Skills

LVM Split Site (Cross Site) Equivalent

● Assumes SAN connected disks and nodes at two locations

● Define shared volume group with super strict mirror pools– Mirror pool for each location– Disks must be manually assigned to each mirror pool

● Knowing which disks are where is a user responsibility– LVM mirrors logical volume between two locations– Resource group definition should allow forced varyon

● In the event of node and disk loss at one location– Volume group forced on line at other location by PowerHA

● Mirror pool set up guarantees a local copy of the data– Manual recovery of repository using Repository Resiliency




AdvancedTechnical

Skills

Problem determination● # clctrl -tune -L

NAME DEF MIN MAX UNIT SCOPE ENTITY_NAME(UUID) CUR pleiades(361d4ace5eb011e291f01293071a2807) 240config_timeout 240 0 2G1 seconds c ndeadman_mode a c nhb_src_disk 1 1 3 chb_src_lan 1 1 3 chb_src_san 2 1 3 clink_timeout 30000 0 1171K milliseconds c nnode_down_delay 10000 5000 600000 milliseconds c nnode_timeout 20000 10000 600000 milliseconds c npacket_ttl 32 1 64 c nremote_hb_factor 10 1 100 crepos_mode e c nsite_merge_policy p c




AdvancedTechnical

Skills

Problem determination● # snap caa

– Creates /tmp/ibmsupt/caaContains data from each node in Data/data_timetar tvf s7801p20.tar

– drwxrxrx 0 0 0 Jan 30 09:55:30 2013 s7801p20/– rwrr 0 0 1123 Jan 30 09:55:31 2013 s7801p20/LOG– rwrr 0 0 2554 Jan 30 09:55:30 2013 s7801p20/bootstrap_repository– rwrr 0 0 978 Jan 30 09:55:30 2013 s7801p20/caa_tunables– rwrr 0 0 194671 Jan 30 09:55:29 2013 s7801p20/clcomd_log.Z– rwrr 0 0 5618196 Jan 30 09:55:30 2013 s7801p20/clcomddiag_log.Z– rwrr 0 0 1362 Jan 30 09:55:30 2013 s7801p20/detail_repository– rwrr 0 0 548 Jan 30 09:55:30 2013 s7801p20/lscluster_clusters– rwrr 0 0 6144 Jan 30 09:55:30 2013 s7801p20/lscluster_network_interfaces– rwrr 0 0 1968 Jan 30 09:55:30 2013 s7801p20/lscluster_network_statistics– rwrr 0 0 2484 Jan 30 09:55:30 2013 s7801p20/lscluster_nodes– rwrr 0 0 1067 Jan 30 09:55:30 2013 s7801p20/lscluster_storage_interfaces– rwrr 0 0 76 Jan 30 09:55:30 2013 s7801p20/lsrepos_all– rwrr 0 0 396 Jan 30 09:55:30 2013 s7801p20/swfinfo_uuids– rwrr 0 0 10017023 Jan 30 09:55:28 2013 s7801p20/syslog_caa– rwrr 0 0 93 Jan 30 09:55:30 2013 s7801p20/system_proc_version– rwrr 0 0 30 Jan 30 09:55:30 2013 s7801p20/system_uname




AdvancedTechnical

Skills

Moving to Disaster Recovery

● Requirements for HADR Solution– Recovery Time Objective

● Time application is unavailable– Recovery Point Objective

● Last data point at which production is recovered in event of a failure– Planned downtime

● Maintenance / Testing– Geographic dispersion

● To meet compliance regulations– Ease of management

● Degree of skill required compared with practicality of swaps– Ease of deployment

● Desire from customers for a simple solution– Integration and support

● Degree of integration with the OS and application will affect the success of failover




AdvancedTechnical

Skills

Summary of changes

PowerHA 6.1– DSCLI Metro Mirror VIOS– Packaging & Pricing Changes– p6/p7 CoD DLPAR Support– EMC SRDF Integration– GLVM Config Wizard– Full IPV6 Support

PowerHA 7.1– Cluster Aware AIX– IBM Director Integration– Hitachi TrueCopy & HUR async Integration– DS8700 Global Mirror Integration– Drop topology services for MultiCast protocol– Storage Monitoring– HADR Storage Framework

PowerHA 7.1.1– CAA Repository Resilience– JFS2 Mount Guard support– SAP Hot Standby Solution– Federated Security– SAP & MQ Smart Assists– XIV Replication Integration– Director Plug-in Updates

PowerHA 7.1.2– Cluster Aware AIX

● IPv6, Rolling upgrade, Linked clusters

– IBM Systems Director plug-in● New wizards, 2 site clusters,

– Enterprise Edition,● Linked and stretched clusters● Split / merge site options with

tie-breaker● Hyperswap

– Support for DS8k for 2 sites




AdvancedTechnical

Skills

PowerHA roadmap

● PowerHA release Life cycle strategy– Current model: Major release every year

● Requires ISV certification for every major release– New model: Implement technology level release strategy

● Major releases as necessary● Minor release updates (Technology Leve 0 to Major release)● At lease two technology levels per major release

– Proposed● Additional 2 year service offering for last TL (under review)

● New command– halevel -s




AdvancedTechnical

Skills

Support planning




AdvancedTechnical

Skills

Roadmap

PowerHA SystemMirror 7.1.1 TL01

● SAP liveCache Hot Standby Solution

● PowerHA federated security

● MQSeries smart assist

● Technology level release

PowerHA Enterprise Edition 6.1

● HA/DR support for XIV

PowerHA 7.1 director plugin

● Federated security management

● Replicated storage management

● Wizards update– SAP liveCache HotSwap– GLVM express wizard– Multi-site cluster wizard

PowerHA SystemMirror 7.1.3

● VM HA management– VM restart– VM DR restart

PowerHA Enterprise Edition 7.1+

● PowerHA failover reversal

● 3 or more sites support

● Operator override support


● Three site support

● Cluster modeling

● Failover reversal

● LPAR HA management

2011 2012 2013X X X

PowerHA SystemMirror 7.1.2

● Smart Assists– Weblogic; Sybase; Peoplesoft

● PowerHA Enterprise Edition 7.1+

● Hyperswap HA/DR


● Hyperswap HA/DR support

● Cluster modeling

● LPAR HA management




AdvancedTechnical

Skills

References● IBM PowerHA SystemMirror for AIX v7.1

– http://www.redbooks.ibm.com/abstracts/sg247845.html

● PowerHA Web site:– www.ibm.com/systems/power/software/availability/

● PowerHA portal– http://www-03.ibm.com/systems/power/software/availability/aix/index.html

● Online Documentation– http://www-03.ibm.com/systems/p/library/hacmp_docs.html

● PowerHA SystemMirror Marketing Page– http://www-03.ibm.com/systems/power/software/availability/aix/index.html

● PowerHA landing page on IBM.com

● http://www-03.ibm.com/systems/power/software/availability/aix/index.html

● PowerHA technical forum– https://www.ibm.com/developerworks/community/forums/html/forum?id=1111111

1-0000-0000-0000-000000001611● PowerHA Comments & Questions:

– [email protected]

Thanks to Shawn, Mike and the US team for notes and detailed information.




AdvancedTechnical

Skills

Other useful info

● IBM Technology Service Offering for PowerHA SystemMirror XD deployment – http://www-935.ibm.com/services/us/index.wss/offering/its/a1000032

● Redbooks– SG24-7739 : PowerHA for AIX Cookbook– SG24-7841 : Exploiting IBM PowerHA SystemMirror Enterprise Edition– SG24-7845 : IBM PowerHA SystemMirror 7.1 for AIX

● RedGuide– High Availability and Disaster Recovery Planning: Next-Generation Solutions for Multi server IBM

Power Systems Environments● http://www.redbooks.ibm.com/abstracts/redp4669.html?Open

● Education: PowerHA for AIX Implementation, Configuration and Administration AN610 – Go to www.ibm.com/services/learning, search for AN610 or PowerHA – coming soon

● GLVM white paper– www.ibm.com/systems/resources/systems_p_os_aix_whitepapers_pdf_aix_glvm.pdf

● clmgr white paper– www.ibm.com/systems/resources/systems_power_software_availability_clmgr_tech_guide.pdf

● IBM storage virtualization offerings– www.ibm.com/systems/storage/virtualization

● Wiki– http://www.ibm.com/developerworks/wikis/display/WikiPtype/High%20Availability




AdvancedTechnical

Skills

PowerHA SystemMirror for AIX v7.1 Two-Node Quick Configuration Guide– http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102216

Current Redbook– http://

www.redbooks.ibm.com/redbooks.nsf/searchsite?SearchView&query=powerha

Redbook if using PowerHA Enterprise Edition with Hitachi TrueCopy– http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS5098

Disaster recovery using IBM Storwize family storage with IBM PowerHA SystemMirror Enterprise Edition 7.1

– http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102245– Implementing PowerHA with Storwize V7000

Tips for Configuring PowerHA on Flex System POWER7 Compute Nodes– http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102181

89

Other useful info




AdvancedTechnical

Skills

Resources – matrices and cross references

PowerHA Hardware Support Matrix– http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105638

PowerHA for AIX Version Compatibility Matrix– http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD101347

PowerHA Enterprise Edition Support Cross Reference– http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105440

90

http://www-03.ibm.com/systems/power/software/availability/aix/index.html

http://www-03.ibm.com/systems/power/software/availability/aix/index.html

https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000001611

https://www.ibm.com/developerworks/community/forums/html/forum?id=11111111-0000-0000-0000-000000001611




AdvancedTechnical

Skills

Resources – demo videos

Configuring PowerHA v7.1.2 using IBM Systems Director Demo http://www.youtube.com/watch?v=zxHURigatQc Apply Updates (Service Packs) to an active PowerHA 7.1.2 cluster http://www.youtube.com/watch?v=fZpYiu8zAZo PowerHA cluster test tool demonstration http://www.youtube.com/watch?v=zZHhCXhg1L8 Dynamically add a node into an active PowerHA cluster http://www.youtube.com/watch?v=bV9JdzPWTVQ PowerHA Enterprise Edition with XIV replication failover http://www.youtube.com/watch?v=RJ5O0030agM

91




AdvancedTechnical

Skills

Resources - DeveloperWorks

PowerHA cluster migration to POWER7 (Chris Gibson – IBM)– http://www.ibm.com/developerworks/aix/library/au-cluster-migration/index.html

PowerHA 7.1 heartbeat over SAN (Talor Holloway – Advent One)– http://www.ibm.com/developerworks/aix/library/au-aix-powerha-heartbeat/index.htm

l

92

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102216


http://www.redbooks.ibm.com/redbooks.nsf/searchsite?SearchView&query=powerha

http://www.redbooks.ibm.com/redbooks.nsf/searchsite?SearchView&query=powerha

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS5098

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS5098



http://www-03.ibm.com/support/techdocs/atsmastr.nsf/5cb5ed706d254a8186256c71006d2e0a/541e93f4faff21438625785e005eab09/$FILE/IBM%20Storwize%20V7000%20with%20PowerHA%20Standard%20final1.pdf





http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102181

http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102181




AdvancedTechnical

Skills

Edison Group whitepaper – The Value of Deep Integration

http://www-03.ibm.com/systems/power/advantages/whypower/powerha.html

93

“…Such deep integration enables innovative features unavailable in other products… In addition, because the clustering solution and operating system evolve together, any flaws in the synthesis between the two discovered in the field are addressed, and the fixes are baked into the next release of the product. This ensures a product that continually improves over time into an extremely robust HA clustering solution.”

http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD105638


http://w3-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD101347








AdvancedTechnical

Skills

Case study - Robert Wood Johnson University Hospital

Download case study

94

OverviewConsistently ranked as one of “America’s Best Hospitals” by U.S. News & World Report, Robert Wood Johnson University Hospital provides state-of-the-art care through a wide range of health care services. The 610-bed hospital based in New Brunswick, New Jersey functions as one of the nation’s leading academic medical centers and is the only Level 1 Trauma Center for Central New Jersey.

Business needTo remain competitive and ensure business continuity, Robert Wood Johnson University Hospital needed to improve IT performance and implement a failover system to ensure reliable data access.

SolutionThe hospital deployed IBM Power 740 Express servers running IBM AIX, IBM PowerHA SystemMirror for AIX, IBM System Storage DS4300 and NTT DATA Optimum Revenue Cycle Management software.

BenefitsHospital staff and patients noticed vast performance improvements in accounts and records systems, and IT staff ensured data access by reducing failover time from several hours to five minutes.

http://www.youtube.com/watch?v=zxHURigatQc

http://www.youtube.com/watch?v=zxHURigatQc

http://www.youtube.com/watch?v=fZpYiu8zAZo

http://www.youtube.com/watch?v=fZpYiu8zAZo

http://www.youtube.com/watch?v=zZHhCXhg1L8

http://www.youtube.com/watch?v=zZHhCXhg1L8

http://www.youtube.com/watch?v=bV9JdzPWTVQ

http://www.youtube.com/watch?v=bV9JdzPWTVQ

http://www.youtube.com/watch?v=RJ5O0030agM

http://www.youtube.com/watch?v=RJ5O0030agM




AdvancedTechnical

Skills

IBMTECHU.COM IBMTECHU.COM IBM STG Technical Universities & Symposia web portal

ibmtechu.com/nz

download password: nz2013

KEY FEATURES...– Create a personal agenda using the agenda planner– View the agenda and agenda changes– Use the agenda search to find the sessions and/or – Download presentations– Submit Session and Conference Evaluations

Questions ?

? Antony (Red) Steel - ATSSenior IT Specialist IBM Aust/[email protected]+61 41980 3049

AHY24 PowerHA SystemMirror for AIX: New Features and Best Practice

http://www.ibm.com/developerworks/aix/library/au-cluster-migration/index.html

http://www.ibm.com/developerworks/aix/library/au-cluster-migration/index.html

http://www.ibm.com/developerworks/aix/library/au-aix-powerha-heartbeat/index.html

http://www.ibm.com/developerworks/aix/library/au-aix-powerha-heartbeat/index.html

Documents

PowerHA SystemMirror for AIX: New Features and Best Practice