IBM · IBM Resiliency Orchestration Administrator's Guide © IBM Corportation 2017, 2018 0 Contents Introduction

Administrator’s Guide

Version 7.2 SP4

IBM Resiliency Orchestration Administrator's Guide

DISCLAIMER

IBM believes that the information in this publication is accurate as of its publication date. The information is subject to change without notice.

COPYRIGHT

© IBM Corporation 2017, 2018. All Rights Reserved.

Printed April 2018

Use, copy, and distribution of any IBM software described in this publication need an applicable software license.

No part of this product or document may be reproduced, stored in a retrieval system, or transmitted, in any form by any means, electronic, mechanical, photocopy, recording, or otherwise, without prior written authorization of IBM and its licensers, if any.

TRADEMARK INFORMATION

IBM is a trademark of IBM Corporation.

All other trademarks used in this publication are the property of their respective holders.


© IBM Corportation 2017, 2018 0

Contents

Introduction .................................................................................... 13

IBM Resiliency Orchestration 7.2 SP4 ............................................................. 13

Purpose ............................................................................................................ 13

Getting Started ............................................................................... 14

Using Online Help .......................................................................... 15

About Online Help ............................................................................................ 15

Launching Online Help ..................................................................................... 15

The Browser Environment ................................................................................ 15

Hiding/ Showing the Navigation Pane .............................................................. 15

Navigation Bar ........................................................................................................ 15

Breadcrumbs .......................................................................................................... 15

Working Area .......................................................................................................... 16

Right Pane.............................................................................................................. 16

Administration Overview ......................................................................................... 16

About IBM Resiliency Orchestration ....................................................................... 16

Log Out .................................................................................................................. 17

Help ........................................................................................................................ 17

Quick Start .............................................................................................................. 17

Changing Password ............................................................................................... 18

Reset Password ..................................................................................................... 18

Architecture Overview ................................................................... 20

Overview .......................................................................................................... 20

Key Features .................................................................................................... 20

Continuity Monitoring .............................................................................................. 20

Continuity Management .......................................................................................... 21

Test Exercise Management .................................................................................... 21

Continuity Reports .................................................................................................. 21

Discovery................................................................................................................ 21

Support for Heterogeneous technologies: ........................................................ 22

Support for Clusters ................................................................................................ 22

Support for Software and Hardware Data Protection Technologies ........................ 22

Storage-based replication ....................................................................................... 22

Switch-based replication ......................................................................................... 22

Host-based replication ............................................................................................ 22

Key Benefits ........................................................................................................... 23

How IBM Resiliency Orchestration Works? ............................................................. 24

Discovery ....................................................................................... 25

Discovery Overview.......................................................................................... 25

Sites Overview ................................................................................................. 25

Home Page ...................................................................................................... 26



Subsystems Overview ...................................................................................... 27

Groups Overview .................................................................................................... 27

Recovery Group ..................................................................................................... 28

Application Group ................................................................................................... 28

Status of a Group ................................................................................................... 29

Execution Mode ...................................................................................................... 29

Execution Status..................................................................................................... 29

Recovery Status ..................................................................................................... 31

Continuity Operations on a Group .................................................................... 32

Business Continuity Modes .......................................................... 33

Business Continuity States ............................................................................... 34

About RPO, RTO ............................................................................................. 36

Recovery Point Objective ....................................................................................... 36

Recovery Time Objective ........................................................................................ 36

Compute Current App RPO .................................................................................... 36

Compute current Data RPO .................................................................................... 36

Compute current RTO and display the desired RTO and estimated RTO values .... 37

Pre- requisites ........................................................................................................ 40

Additional Support .................................................................................................. 40

Starting and Stopping IBM Resiliency Orchestration Server Services .......................................................................................... 41

On Server Restart ............................................................................................ 41

Starting IBM Resiliency Orchestration Agents ............................ 42

Starting Agents on the UNIX Server ................................................................. 42

Stopping IBM Resiliency Orchestration Agents on Window Server .................. 43

Stopping Agents on the UNIX Server ............................................................... 43

Verifying IBM Resiliency Orchestration Agents .......................... 45

Verifying Agents on Windows Server ............................................................... 45

Verifying Agents on UNIX Server ..................................................................... 45

Verifying the processes of Agents on UNIX Server .......................................... 46

Refreshing Details ............................................................................................ 46

Configuration ................................................................................. 49

Configuring IBM Resiliency Orchestration ........................................................ 49

Tomcat Configuration ....................................................................................... 49

Steps to enable HTTPS in tomcat: .......................................................................... 49

Steps to disable HTTP in tomcat ............................................................................ 50

Steps to change the port numbers 8080 or 8443 to different ones .......................... 50

Steps to Enable Compression in Tomcat Server..................................................... 51

Tomcats logs rotation ............................................................................................. 52

Agent Configuration.......................................................................................... 52

Site Controller .................................................................................................. 52

Adding Site Controller to Resiliency Orchestration Server ............................... 54



Agentless ............................................................................................................... 54

Pre- requisites ........................................................................................................ 55

Enabling SSH on Unix/Linux subsystem .......................................................... 55

Enabling WMI on Windows subsystem (High level) ......................................... 55

Enabling WMI on Windows subsystem (Detailed information): ........................ 55

Change/ Replace Ownership: ................................................................................. 58

Open Custom Range of ports WMI/ RPC: .............................................................. 58

Range configuration for WMI/RPC .......................................................................... 59

Configuring WAN Agent .......................................................................................... 66

User Management .......................................................................... 68

Setting up Users ............................................................................................... 68

Basic User Role management .......................................................................... 68

Adding Users .......................................................................................................... 68

Modifying Users ...................................................................................................... 71

Deleting Users ........................................................................................................ 71

Setting System Options for Users ........................................................................... 72

Editing User Preferences ........................................................................................ 73

Configuring LDAP ................................................................................................... 73

Creating pre-packaged roles for IBM Resiliency Orchestration ............................... 75

Creating custom roles for IBM Resiliency Orchestration ......................................... 75

Configuring AD ....................................................................................................... 76

Division of the User Management System .............................................................. 80

Known Limitations .................................................................................................. 83

Listing Configured Users ........................................................................................ 83

User Reports .......................................................................................................... 84

Appendix ................................................................................................................ 85

Configuring Notifications ......................................................................................... 88

Email Notification .................................................................................................... 88

SMS Notification ..................................................................................................... 89

SNMP Notification .................................................................................................. 89

Notification List ....................................................................................................... 89

Deleting Notification List ......................................................................................... 91

Configuring SNMP Trap Forwarder ........................................................................ 91

Configuring Email Server ........................................................................................ 92

Configuring Email Templates .................................................................................. 93

Business Process Integration ................................................................................. 94

Listing Business Process Schedules ...................................................................... 94

Adding Business Process ....................................................................................... 95

Configuring Business Process ................................................................................ 95

Custom Event ......................................................................................................... 96

Importing custom event .......................................................................................... 96

Exporting custom event script ................................................................................. 96

Importing / Exporting custom event policy workflow ................................................ 97



Scheduling.............................................................................................................. 98

Managing Business Process .................................................................................. 98

Deleting Business Process ..................................................................................... 99

Continuity Workflows .............................................................................................. 99

Working with Actions ............................................................................................ 101

Creating new workflow.......................................................................................... 114

Deleting Workflow ................................................................................................. 114

Importing or Exporting a Workflow ........................................................................ 115

Scheduling Business Process Workflow / Drills .................................................... 116

Automatic Retry on Failure ................................................................................... 119

Single Stepping Actions in Workflows ................................................................... 120

Synchronizing Workflow with RPO/RTO ............................................................... 120

Publish/ Save as a Draft ....................................................................................... 121

Previewing Workflow ............................................................................................ 122

Workflow Version History/ Rollback ...................................................................... 122

Workflow Key Value List ....................................................................................... 122

Workflow Configuration Limitations ....................................................................... 125

Configuring Approver List ..................................................................................... 125

Recovery Automation Library (RAL) .......................................... 127

AWS ............................................................................................................... 128

Create Route Table .............................................................................................. 128

Attach Amazon Volume ........................................................................................ 129

AmazonS3BucketCleanup .................................................................................... 129

Attach Security Group .......................................................................................... 130

Attach Elastic IP ................................................................................................... 131

AWS Instance Status Checks RAL ....................................................................... 132

AWS Instance Terminate RAL .............................................................................. 132

Create Amazon Instance ...................................................................................... 133

Create Amazon Volume ........................................................................................ 134

Create Amazon Snapshot ..................................................................................... 136

Create Internet Gateway ....................................................................................... 137

Create Image RAL ................................................................................................ 137

Create Route Table .............................................................................................. 138

Create Subnet RAL .............................................................................................. 138

Create Security Group .......................................................................................... 140

Create Amazon S3 Bucket ................................................................................... 141

Delete Amazon Volume ........................................................................................ 142

Detach Amazon Volume ....................................................................................... 142

Delete Amazon Snapshot ..................................................................................... 143

Import EC2 RAL ................................................................................................... 143

Import EBS Volume RAL ...................................................................................... 145

Start AWS Instance .............................................................................................. 147

Stop Amazon Instance.......................................................................................... 147

S3 Bucket CleanUp .............................................................................................. 148



VMware VCenter ............................................................................................ 148

Add VMDK to Virtual Machine .............................................................................. 148

Create Snapshot of Virtual Machine ..................................................................... 149

Create Clone of Virtual Machine ........................................................................... 149

Delete Snapshot of Virtual Machine ...................................................................... 150

Mount Datastore on Host ...................................................................................... 150

Power on Virtual Machine ..................................................................................... 151

Power off Virtual Machine ..................................................................................... 151

Reboot Virtual Machine ........................................................................................ 151

Reset Virtual Machine .......................................................................................... 152

Re-configure CPU Allocation of Virtual Machine ................................................... 152

Re-configure Memory Allocation of Virtual Machine .............................................. 153

Remove VMDK from Virtual Machine ................................................................... 153

Restore Snapshot of Virtual Machine .................................................................... 154

Rescan HBA ......................................................................................................... 154

Shutdown Virtual Machine .................................................................................... 155

Unmount Datastore from Host .............................................................................. 155

Add Virtual Machine to Inventory .......................................................................... 156

Vmware Host Mount Datastore ............................................................................. 157

Vmware Host Unmount Datastore ........................................................................ 158

Add Disk to VM ..................................................................................................... 159

Remove Disk from VM .......................................................................................... 160

Change Port Group Mapping for Virtual Machine.................................................. 161

Download VMX File .............................................................................................. 162

Upload VMX File................................................................................................... 163

Add VM to Cluster ................................................................................................ 163

Rescan HBAs for Cluster ...................................................................................... 164

Datastore Status ................................................................................................... 165

Vmware Execute Guest OS Command ................................................................. 166

Guest OS Customization ...................................................................................... 167

VMDK Details ....................................................................................................... 167

Execute Cluster Ops ............................................................................................. 167

DB2 ................................................................................................................ 167

Activate Database ................................................................................................ 167

Deactivate Database ............................................................................................ 168

Execute SQL with Connection .............................................................................. 168

Execute SQL Without Connection ........................................................................ 169

Takeover .............................................................................................................. 169

File ....................................................................................................................... 229

Process ................................................................................................................ 244

Repeatable ........................................................................................................... 249

Replication Level .................................................................................................. 250

Replication............................................................................................................ 436

IBM Internal (This is not for field use.) .................................................................. 438



Workflow .............................................................................................................. 455

Trigger Site Level Drill .................................................................................... 480

Trigger Site Level FO ..................................................................................... 481

Zerto ............................................................................................................... 482

Zerto Logon .......................................................................................................... 482

Zerto Logoff .......................................................................................................... 482

Zerto Change Recovery Host ............................................................................... 482

Zerto Delete VRA ................................................................................................. 483

Zerto Failover ....................................................................................................... 483

Zerto Failover Test ............................................................................................... 483

Zerto Force Sync VPG.......................................................................................... 484

Zerto Launch New VRA ........................................................................................ 484

Zerto Resume VPG .............................................................................................. 485

Zerto Unprotect VPG ............................................................................................ 485

Zerto Wait for Task Complete ............................................................................... 486

Zerto Is VPG Exist ................................................................................................ 486

Zerto Create VPG Settings ................................................................................... 486

Zerto Delete VPG ................................................................................................. 486

Low Touch Oracle DG .................................................................................... 487

Supported Features .............................................................................................. 487

Overview .............................................................................................................. 487

Prerequisites ........................................................................................................ 488

Low Touch Wizard - Primary Server ..................................................................... 489

Primary Database ................................................................................................. 491

AWS Information .................................................................................................. 493

Low Touch Execution workflow ............................................................................. 496

Configuration .................................................................................................. 499

Configuring RPO/ RTO ......................................................................................... 499

Configuring Notification Regarding Events............................................................ 502

Configuring Event Policy ....................................................................................... 503

Adding Polar Event Script ..................................................................................... 505

Site Ticker ...................................................................................................... 506

Show Expired Ticker ............................................................................................. 507

Add Site Ticker ..................................................................................................... 507

Vault Framework ............................................................................................ 508

Introduction ........................................................................................................... 508

Configuring Vault .................................................................................................. 510

Vault Agent ........................................................................................................... 520

Vault Features ...................................................................................................... 521

Subsystems IBM Supports ................................................................................... 525

Dashboards .................................................................................................... 526

DR Operational Dashboard ............................................................................ 526

Application Summary ............................................................................................ 527



Application Details ................................................................................................ 529

Dashboard Snapshot ............................................................................................ 530

DR Manager Dashboard ................................................................................ 530

Application Summary ............................................................................................ 530

Application Details ................................................................................................ 534

Manage ......................................................................................... 535

Filters ............................................................................................................. 535

Group Health .................................................................................................. 536

Sites ............................................................................................................... 536

Application Group ........................................................................................... 539

AG Listing ............................................................................................................. 541

AG Details ............................................................................................................ 541

Relationship .......................................................................................................... 542

Recovery Group ............................................................................................. 543

Recovery Group ................................................................................................... 543

RG Listing............................................................................................................. 544

RG Details ............................................................................................................ 545

Relationship in RG ................................................................................................ 546

Continuity Workflows ...................................................................................... 547

Executing Workflows ...................................................................................... 548

Executing Workflow .............................................................................................. 548

Health Monitoring ........................................................................................... 551

Viewing Group Relationship ................................................................................. 551

RPO and RTO Monitoring .............................................................................. 551

Monitoring Replication .................................................................................... 552

Viewing Recovery Group Replication Details ................................................. 553

Monitor ......................................................................................... 556

Filters ............................................................................................................. 556

Group Health .................................................................................................. 557

Sites ............................................................................................................... 558

Application Group ........................................................................................... 559

Application Group ................................................................................................. 559

AG Listing ............................................................................................................. 563

AG Details ............................................................................................................ 563

Relationship .......................................................................................................... 563

Recovery Group ............................................................................................. 565

Recovery Group ................................................................................................... 565

RG Listing............................................................................................................. 566

RG Details ............................................................................................................ 566

Relationship in RG ................................................................................................ 567

Continuity Workflows ...................................................................................... 568

Executing Workflows ...................................................................................... 569



Executing Workflow .............................................................................................. 569

Health Monitoring ........................................................................................... 572

Viewing Group Relationship ................................................................................. 572

RPO and RTO Monitoring .............................................................................. 573

Continuity Monitoring Overview ...................................................................... 574

Monitoring Replication .................................................................................... 574

Viewing Recovery Group Replication Details ................................................. 575

Events ........................................................................................... 576

Event Alarm .................................................................................................... 576

View Events ................................................................................................... 576

Viewing Group Events .................................................................................... 579

Filtering Events .............................................................................................. 580

Complete Listing of Events ............................................................................. 581

Viewing Sequence of Events .......................................................................... 585

System Events ............................................................................................... 586

Monitoring Events........................................................................................... 587

Monitoring Users and Passwords for raised Events ....................................... 587

Recovery Automation .................................................................. 590

Managing Groups ........................................................................................... 590

Group Maintenance - Working with Different Modes of Group .............................. 590

Business Continuity Operations ............................................................................ 594

Start and Stop Replication .................................................................................... 609

Managing Events .................................................................................................. 613

Drills ............................................................................................. 620

Configuring Drills ............................................................................................ 621

Switchover ............................................................................................................ 621

Switchback ........................................................................................................... 622

Custom Drills ........................................................................................................ 623

IntegrityCheck Test Place holder .......................................................................... 625

Listing Drill Schedules .......................................................................................... 625

Executing Drills .............................................................................................. 625

Drills State ............................................................................................................ 628

Drill Listing ............................................................................................................ 630

Tracking Drills ....................................................................................................... 630

Changing Drills Mode ........................................................................................... 631

Drills Interfering with BCO .................................................................................... 631

Identifying Group under Unrestricted Test Mode .................................................. 632

Effect of Drills on Group Status ............................................................................. 632

Viewing Drills ........................................................................................................ 632

Viewing Drills for a Group ..................................................................................... 632

Viewing automation status .................................................................................... 633

Viewing Drills History Report ................................................................................ 634



Viewing Workflow Calendar .................................................................................. 634

Viewing Recent Execution Drills Status ................................................................ 635

Drills Management ................................................................................................ 636

Scheduled ...................................................................................................... 637

Viewing Scheduled ............................................................................................... 637

Scheduling Workflows .......................................................................................... 637

Executing ....................................................................................................... 638

Executing ............................................................................................................. 638

Summary ........................................................................................................ 640

Viewing Summary ................................................................................................. 640

Select Workflow .................................................................................................... 641

Viewing workflow details ....................................................................................... 641

Editing Workflows ................................................................................................. 642

Viewing Published workflow by logged in user ............................................... 643

Viewing Published workflow by logged in user ...................................................... 643

Working with Workflow Manager ................................................ 645

Working with Workflow Manager .................................................................... 645

Workflow Execution Page .............................................................................. 645

Resuming workflow after IBM Server failure .................................................. 646

Dealing with Failures of Workflow Execution .................................................. 647

Stopping Workflows........................................................................................ 648

Approving/Rejecting a workflow before execution .......................................... 650

Approving a workflow ........................................................................................... 650

Rejecting a workflow ............................................................................................. 650

DryRun ........................................................................................................... 650

Mode .................................................................................................................... 651

Process ................................................................................................................ 651

Workflow Enhancements ................................................................................ 651

Confirmation of Workflow Execution ..................................................................... 651

Dryrun Execution .................................................................................................. 652

Nested Workflow (Navigation to Workflow within a Workflow) ............................... 652

View Workflows .................................................................................................... 652

Workflows Visual Distinction ................................................................................. 652

Stopping/Aborting Parent Workflows .................................................................... 653

Propagation of Nested Workflow Status ............................................................... 653

Parent Workflow Summary Counts ....................................................................... 653

Glimpse of Nested Workflows ............................................................................... 653

Validating and Fortifying Nested Workflows .......................................................... 653

Validation ..................................................................................... 654

View Rules ..................................................................................................... 654

Listing Validation Rules ........................................................................................ 654

Tasks .............................................................................................................. 654



Listing Tasks ........................................................................................................ 654

Creating Tasks ..................................................................................................... 656

Modifying Tasks.................................................................................................... 657

Deleting Tasks ...................................................................................................... 657

Executing Tasks ................................................................................................... 658

Viewing Execution History .................................................................................... 658

Reports ......................................................................................... 661

Reports Overview ........................................................................................... 661

You can generate the following reports: ................................................................ 661

Viewing Reports ............................................................................................. 662

Printing Reports ............................................................................................. 673

Internet Explorer: .................................................................................................. 673

Firefox: ................................................................................................................. 674

Exporting a Report ......................................................................................... 674

Customizing Reports ...................................................................................... 675

Group Summary Report ................................................................................. 675

Test Summary Report .................................................................................... 678

To generate Test Summary Report: ...................................................................... 678

Custom Report Framework ............................................................................ 681

Data Capturing Module to Generate Custom Reports ........................................... 681

Advanced Reports Module ................................................................................... 689

Audit Logging ................................................................................................. 696

Viewing Audit Log Report ..................................................................................... 698

Scheduling Audit Log Report ................................................................................ 699

Listing Scheduled Audit Log Reports .................................................................... 701

Modifying Audit Log Report Schedule ................................................................... 701

Deleting Audit Log Report Schedule ..................................................................... 703

Discovery ..................................................................................... 705

Sites ............................................................................................................... 705

Adding Organization ............................................................................................. 705

Adding Sites ......................................................................................................... 706

Modifying Sites ..................................................................................................... 708

Deleting Site ......................................................................................................... 709

Site Listing ............................................................................................................ 709

Subsystem ..................................................................................................... 710

Auto Discovery ..................................................................................................... 711

Credentials ........................................................................................................... 711

Create New Credential Using SSH ....................................................................... 712

Agent Upgrade ..................................................................................................... 714

Prerequisites .................................................................................................. 715

Component & WAN ........................................................................................ 718

Datasets ............................................................................................................... 724

Protection Schemes ............................................................................................. 745



Modifying Protection Schemes ....................................................................... 781

Deleting Protection Schemes ......................................................................... 782

Agent Node Configuration .............................................................................. 782

Converged ...................................................................................................... 782

UCS Directors ...................................................................................................... 783

UCS Director vDC Map for DR ............................................................................. 786

Management Service ..................................................................................... 789

Adding Management Service ................................................................................ 789

Modifying Management Service ............................................................................ 792

Deleting Management Service .............................................................................. 794

Management Service List ..................................................................................... 794

SRA ................................................................................................................ 795

SRA Discovery ..................................................................................................... 795

SRA Listing ........................................................................................................... 797

ADC Profile .................................................................................................... 797

Adding ADC Profile ............................................................................................... 798

Modifying ADC Profile .......................................................................................... 800

Deleting ADC Profile ............................................................................................. 800

Listing ADC Profile ............................................................................................... 801

Group Creation ............................................................................................... 801

Groups ................................................................................................................. 801

Creating / Editing Groups ..................................................................................... 803

Creating Recovery Groups .................................................................................. 804

Viewing Recovery Group Details .......................................................................... 809

Editing Recovery Group ........................................................................................ 810

Application Group Workflow Auto Generation ....................................................... 812

Network Component ............................................................................................. 814

Support for Cluster ............................................................................................... 814

Creating an Application Group .............................................................................. 815

Viewing Application Group details ........................................................................ 818

Editing Application Group ..................................................................................... 818

Deleting a Group .................................................................................................. 819

IBM Resiliency Orchestration Logs ............................................ 822

Working with Logger....................................................................................... 822

Logger Overview ............................................................................................ 823

Setting Debug Level ....................................................................................... 825

Retaining Log ................................................................................................. 826

Operational History......................................................................................... 827

Purge Log Now ..................................................................................................... 828

Setting Log File Size ...................................................................................... 830

Fetching Log Files .......................................................................................... 831

System Capture ............................................................................................. 831



Listing Log Files ............................................................................................. 834

Filtering Log Files and Advanced Log Fetching ............................................. 835

Fetching Log files using CLI tools .................................................................. 836

Resiliency Orchestration Log Fetching / Extracting ............................................... 836

Resiliency Orchestration Log Admin ..................................................................... 840

Admin Tasks ................................................................................ 841

Administration Summary ................................................................................ 841

Viewing the Administration Tasks Summary .................................................. 841

Adding an IBM Resiliency Orchestration Instance or Server .......................... 842

Managing Resiliency Orchestration Instances ............................................... 842

Viewing Resiliency Orchestration Instances ................................................... 842

Deleting an Resiliency Orchestration Instance ............................................... 843

Licensing ...................................................................................... 844

Introduction .................................................................................................... 844

Licensing Introduction .................................................................................... 844

Procuring and Uploading License .................................................................. 845

Viewing License information procured .................................................................. 845

Uploading License ................................................................................................ 846

Upgrading license ................................................................................................. 847

View the Uploaded License ............................................................................ 848

Enabling / Disabling for a Group .................................................................... 849

License Usage ............................................................................................... 849

Home Page Options ....................................................................................... 854

High Availability of IBM Resiliency Orchestration Server ........ 856

Continuity Operation Recovery ...................................................................... 856

Continuity Recovery ....................................................................................... 856

Event Management Recovery ........................................................................ 858

Drills Recovery ............................................................................................... 858

IBM Resiliency Orchestration Meta Data Recovery ....................................... 858

1. Group Status Recovery .................................................................................... 859

2. Continuity Mode Recovery ................................................................................ 859

3. Event Policy Recovery ...................................................................................... 859

IBM Resiliency Orchestration Server Recovery ........................ 861

Resiliency Orchestration Server Failover ....................................................... 861

Resiliency Orchestration Server Fallback ....................................................... 862

IBM Resiliency Orchestration Server Failover configured for UCS Director .......... 863

Monitoring Server Recovery ................................................................................. 864

Metadata Replication using Automated Script ...................................................... 867

Metadata Replication with Manual Steps .............................................................. 872

Script Design ........................................................................................................ 879

Backup ................................................................................................................. 883



Server Memory Management ............................................................................... 886

Interoperability Matrix ................................................................. 888

IBM Resiliency Orchestration Monitoring ....................................................... 888

Normal Monitoring ................................................................................................ 888

Reverse Monitoring .............................................................................................. 890

IBM Resiliency Orchestration Recovery Automation ...................................... 892

Normal Management ............................................................................................ 892

Reverse Management .......................................................................................... 896

IBM Resiliency Orchestration Test Exercises ................................................. 899

Switchover/Switchback ......................................................................................... 899

3rd Party Integration .................................................................... 902 Email Notification .................................................................................................. 902

SMS Notification ................................................................................................... 902

SNMP Notification ................................................................................................ 902

Glossary ....................................................................................... 903



Introduction

IBM Resiliency Orchestration 7.2 SP4

IBM Resiliency Orchestration 7.2 SP4 is an industry leading software product for Business

Continuity that addresses Disaster Recovery (DR) challenges. The IBM Resiliency Orchestration

automates DR workflows by inter-operating with several industry leading Database, Replication,

and Cluster Products and provides comprehensive Disaster Recovery Solution Management.

Purpose

This manual is your guide for the operation and maintenance of the product. The document gives

a detailed description of:

All menu commands, icons and links available in the product.

The terminologies used.

Procedures to create, modify and delete various entities.

Procedure to maintain an interface with a variety of features in order to accomplish a

particular task.

Thus the manual helps you to use the product with ease and makes you familiar with IBM

Resiliency Orchestration.



Getting Started

If you have not started the IBM Resiliency Orchestration services during the installation, start

them manually before invoking the GUI. Refer to Start or Stop IBM Resiliency Orchestration

Server Services.

Perform the following steps to start working with IBM Resiliency Orchestration.

1. Enter the following URL on the Internet Explorer browser:

http://:8080/PanacesGUI

Note:

Enter the following URL on the Internet Explorer browser to work in secure mode:

https://:8443/PanacesGUI

The following page appears:

2. Enter User name and Password in the respective fields.

Note:

The super administrator configures the user names.

3. Click Submit to open the IBM Resiliency Orchestration console.



Using Online Help

About Online Help

The console includes an HTML-based, cross-platform online help system. It provides information

about using IBM Resiliency Orchestration features and enables you to find information about a

specific task.

For best results, view the help in Internet Explorer version 5.0 or later and Mozilla.

Launching Online Help

You can launch the online help system by clicking Help > Help Contents.

The online help opens in your default Web browser. In the Web browser, the left pane contains

the Contents, Index, Search and Glossary tabs. Navigate to different topics by clicking on

them.

The Browser Environment

You can use the browser's navigation aids to navigate through the online help. You can resize the

browser window to the desired size.

Use the Back button of the browser to return to the previously viewed topic. Use the Forward

button to go to the topic that was displayed prior to going back.

Hiding/ Showing the Navigation Pane

If you want to hide the navigation pane in the online help system, which includes the Contents,

Index, Search, and Glossary tabs, click at the top left of the navigation pane. If you have

hidden the navigation pane and want to see it again, click the Contents, Index, Search, or

Glossary button.

Navigation Bar

The Navigation Bar on the IBM Resiliency Orchestration window provides shortcut options to

navigate through the different features of the product.

The Navigation Bar provides the following tabs. Click any of these tabs to use the relevant

features of IBM Resiliency Orchestration.

Monitor

Manage

Drills

Reports

Discover

Admin

Logout

NOTE:

At any point, if you want to go to Home page, click the IBM Resiliency Orchestration icon.

Breadcrumbs

The IBM Resiliency Orchestration windows display breadcrumbs, a type of secondary navigation

that enables users to identify their location in the application. Breadcrumbs provide a trail of the



pages visited by the users. Links to these pages help the user to access the required page from the

current location with a single click.

Breadcrumbs appear horizontally below the navigation and title bars.

Working Area

This is the key functional area of the IBM Resiliency Orchestration window. This area displays

windows for different features depending on the tabs chosen on the navigation bar. You can also

view the current execution windows and perform different BCO operations.

Right Pane

This pane displays the list of configured Groups along with a drop-down menu from which you

can select the criteria for displaying the Group information.

This information is displayed on the Right pane for any tab chosen on the navigation bar.

Additionally, you can perform certain configuration operations by clicking the link or by selecting

different options displayed in the pane. For example, on the Notifications window, click Add

Notification List in the right pane to set up a new notification list to receive Event notifications.

Administration Overview

IBM Resiliency Orchestration Administration console allows you to perform various

administration activities like creating, modifying or deleting the users, configuring notification

lists, agents, and backup manager, managing logs and handling IBM Cloud Continuity™ server

failover.

Navigate to the IBM Resiliency Orchestration Administration console by clicking the Admin

link on the navigation bar.

It displays the following links.

Go to Users

Go to Notifications

GO to Agents

Go to Logs

Go to Backup

Go to Server Failover

Go to Directory Server Details (Applicable only for Advanced User Role Management)

Go to System Events

Go to License

Go to Operational History

About IBM Resiliency Orchestration

You can view the product license information by clicking Help > About on the navigation bar.

This window displays the following information about the IBM Resiliency Orchestration:

Current Version details

IBM Resiliency Orchestration Environments

Server Details

Acknowledgements

Version History



Alternatively, you can view the version of IBM Resiliency Orchestration by entering the

following command at $EAMSROOT/bin location in the IBM Resiliency Orchestration server:

./panaces version

Log Out

Logs out the current user from IBM Resiliency Orchestration. You are then taken to the IBM

Resiliency Orchestration Login page to log on to the product again.

Help

Select Help to navigate through the following options in the secondary navigation bar.

Contents & Index - Provides quick accessibility to the online information about the

product.

About IBM Resiliency Orchestration - Displays the following information on IBM

Resiliency Orchestration:

Product information

Version details

Copyright information

NOTE:

You can also view the version of IBM Resiliency Orchestration by entering the following

command at $EAMSROOT/bin location in IBM Resiliency Orchestration server:

./panaces version

Quick Start - Guides to short, simple introductory steps to IBM Resiliency Orchestration

application.

Quick Start

You can quickly create site, subsystem, group and user by clicking Help > Quick Start on the

navigation bar. The Quick Start page consists of Quick Start and Group Overview tabs.

Click the Quick Start tab to perform the following tasks:

Button Click

Create Site To create a new site. Sites are the geographical locations where Continuity Group subsystems physically exist. A valid group must have at least one Primary and one DR site locations defined. For more information about Create Site, refer to Adding Sites.

Create Subsystem To create a new subsystem. Group subsystems consist of Components, Datasets and Protection Schemes. A continuity group must have at least one each of these subsystems on both Primary and DR sites. For more information about Subsystems, refer to Discovering Subsystem.

Create Group To create a new group. IBM Resiliency Orchestration has 3 types of groups with a



Button Click

hierarchical model. An AG comprises of many RGs. 3 Site Group essentially comprises of 3 AGs [Primary, Secondary and Remote] and these AGs contain RGs. For more information about Groups, refer to Creating / Editing Groups.

Changing Password

The Resiliency Orchestration administrator creates the login credentials of the user, the user

needs to login to the Resiliency Orchestration system to change the password. If the user has

logged in for the first time, changing password is mandatory. The following figure displays the

Resiliency Orchestration Change Password screen, which prompts the user to change the

password.

Note

The Resiliency Orchestration administrator creates the first login credentials for a user.

Reset Password

The Super Administrator only can reset the password of a user. The Super Administrator can

initiate the change password option through the admin screen. Then the user receives an email to

the email id configured in the profile, as shown in the following figure:



The user needs to click the password reset link to reset the password. The Super Administrator

needs to use this procedure to unblock the user.



Architecture Overview

Overview

IBM Resiliency Orchestration is the industry leading application, continuity management

software for open system applications and databases. It is a framework based continuity

management software that enables consistent operational interfaces, and reduces the expertise

required to configure and operate the application continuity and disaster recovery solutions.

IBM Resiliency Orchestration encapsulates all the required intelligence – processes and

procedures required to interface with application, replication mechanism, servers, storage, and

networks to automate deployment and operation of the continuity solution.

IBM Resiliency Orchestration offers a single console view of enterprise IT continuity health. It

interfaces with the various layers of the continuity solutions and reports deviation on Recovery

Point Objective (RPO) and Recovery Time Objective (RTO) of the application, in real time.

With its application aware framework, IBM Resiliency Orchestration significantly reduces the

operational costs associated with the management of continuity assets, increases the productivity

of the operations staff and provides continuity assurance for mission critical applications.

IBM Resiliency Orchestration provides coherent and consistent binding of the state of

applications and status of data at the time of an outage, so that recovery is quick, reliable and

needs little or no expertise.

Key Features

Key features of IBM Resiliency Orchestration include:

Disaster recovery management for multi-site DR environments with the following capabilities:

Continuity Monitoring

Provides a dashboard view of the real time status of disaster recovery environment including the health of the sites, applications, replication mechanisms and readiness of the

disaster recovery systems for recovery.

Provides visual representation of the entire disaster recovery environment.

Application aware Event Manager provides support for multiple severity events.

Supports multiple notification mechanisms, including EMS, emails and text messages.

Computes and tracks current recovery point for each application and reports deviations against set RPO.

Detects application environment and configuration changes and provides alerts on impact.

Provides real time reports on data lag between primary and DR systems.

Provides events dashboard across all applications.

Correlates various infrastructure failures and events and provides support for root cause analysis.

Monitors and provides real time information on WAN link utilization and health.

Estimates the recovery time for the various components of the solution – server, application, data recovery, data consistency. Provides breakup of recovery steps, and

estimated recovery time for each of these steps.



Performs real time monitoring of replication infrastructure and provides replication details including replicated data sizes, failure and success events and vendor specific replication

object details.

Provides database specific transaction information on primary and remote systems.

Provides uniform replication monitoring interface to all replicators.

Continuity Management

Provides best practice template based application provisioning into the Continuity infrastructure.

Provides extensive customization capability of the templates using a visual workflow editor.

Provides automated failover and fallback management, integrating into industry standard databases and applications.

Provides automated switchover and switchback management.

Provides interfaces to start and stop continuity and replication operations for maintenance purpose.

Provides RPO management to maintain the current data loss within the continuity objectives. Replication can be prioritized for applications managed by IBM Resiliency

Orchestration.

Provides unified replication management, integrating into third party replication products.

Provides template based policies and automated execution of policies for Events.

Provides control to perform operations at an individual application level as well as group of applications.

Test Exercise Management

Provides intrusive and non-intrusive testing capability and test DR readiness without impacting production services.

Allows test suite customization to meet specific environments.

Provides pre-packaged template based test exercises.

Restores the replication state after performing testing.

Displays real time status of the test execution.

Facilitates testing on snapshot copy apart from the target copy on the DR site.

Displays “pre-test” checks to improve test success.

Provides option to start, stop, abort, roll back and continue test.

Provides per application test dashboard.

Continuity Reports

Graphical and tabular reports on various continuity management and monitoring areas including RPO, RTO, Continuity operations, data replication and WAN utilization.

Provide daily, weekly, monthly, quarterly and yearly reports per the configurable options.

Provides comprehensive filters based on RPO, RTO (list of filters to provide).

Discovery

Provides a complete inventory of DR infrastructure including servers, replication software, database, network and applications.



Provides auto discovery for replicators.

Provides relationship and detailed configuration of discovered Subsystems and binds them as groups.

Provides pre-packaged templates to assist in modeling the DR infrastructure.

Support for Heterogeneous technologies:

Support for Clusters

Clustering is the common term for distributing a service over a number of servers in order to

increase fault tolerance and to support loads larger than a single server is able to handle. It is often

used for large scale and mission critical applications where there can be no downtime. A cluster

contains servers which share state at some level.

IBM Resiliency Orchestration supports Hardware Clustering and Software Clustering:

High availability cluster (also called Failover clusters, Hardware Clustering, etc) are used to make sure no loss of service occurs when the primary server fails. Usually, in this case,

secondary hardware is present to take over.

Software Clustering (also called Application Clusters) is a method of turning multiple computer servers into a cluster (a group of servers that acts like a single system). Clustering

software is installed in each of the servers in the group. Each of the servers maintains the

same information and collectively they perform administrative tasks such as load

balancing, determining node failures, and assigning Failover task.

Hence, IBM Resiliency Orchestration supports application clusters by monitoring the application

cluster nodes, and ensuring when the entire cluster is not available to enable remote disaster

recovery site take over.

Support for Software and Hardware Data Protection Technologies

In real-time data protection, the data is protected on a real time basis wherein the data from the

host server is copied onto the remote location as soon as the data is changed, over the network.

IBM Resiliency Orchestration supports the following types of protection technologies:

Storage-based replication

Storage-based replication does not operate on the host computer. This kind of replication occurs

at the storage device level. The disk arrays manage the replication of data between sites, relieving

the hosts of the replication process.

Switch-based replication

The replication is at the storage network level. The switch replicates the data from one site to a

switch that receives the data at another site.

Host-based replication

Host-based replication typically resides at the file system or logical volume level within the

operating system. Like storage-based offerings, it's usually transparent to the application, but

certainly not transparent to the host operating system or hardware. Advantages of this method of

replication include, ability to replicate to heterogeneous storage and similar (not necessarily

identical) systems and ability to reduce required bandwidth.

File based Replication Technologies

Host based file replicator products



The modified data at the primary site is identified based on the file modification time stamps and

shipped over the network using the file replicator. This method combines the best of the benefits

of remote mirroring technologies and software protection technologies.

Database Replication Technology

Data replication is at the logical level by re-constructing the database transactions at the

remote server. This method has the advantage of minimizing the bandwidth required. It

also supports protection against rolling disasters and user errors.

Support for Databases and Cluster Applications

IBM Resiliency Orchestration monitors all the servers and applications assets across

the sites and ensures workflow co-ordination among them.

IBM Resiliency Orchestration performs database setup on the near site with an

initial copy of the database from the production to deploy new applications into DR.

Support for Infrastructure

IBM Resiliency Orchestration is configured into various devices like servers, storage,

network, and WAN to monitor the continuity health of the devices.

Key Benefits

IBM Resiliency Orchestration enables organization to effectively manage the continuity solutions

and provide continuity assurance to the business with the following advantages:

Accurate state information on the health of continuity systems, help IT managers resolve

problems quickly.

Automated monitoring significantly reduces the load on the operations staff and increases

their productivity.

Automated continuity management assures application continuity and makes recovery

reliable and uniform across the enterprise.

Deep application integration reduces the load on various IT administrator/ operations staff

and increases their productivity.

Provisioning of new applications into continuity assets is very fast and non-intrusive to

production.

Disaster readiness verification becomes practical and achievable with test automation.

Cost avoidance in DR drills, as more tests can be performed within the same period with

less vendor support.

Increased uptime for the production applications, as production systems are not impacted

during testing.

CIOs get consolidated business view of all the systems covered under Continuity.

Detailed historical reports enable department and business unit heads identify deviations

with internal processes and enable corrective measures.

SLA management becomes easy with real-time information availability on various

elements of the continuity infrastructure.

By automating several monitoring functions, IBM Resiliency Orchestration increases the

productivity of the skilled administrative staff.



Assists in identifying problems through event correlation and helps fix the DR systems

timely.

Automated template based solutions makes provisioning of new applications into DR,

quick and non-intrusive to IT operations.

Automated continuity operations and RPO management assures continuity and makes the

recovery uniform across the enterprise.

Pre-defined and automated execution of policies ensures uniform event response across

the enterprise, irrespective of the time of day and non-availability of people and expertise.

Automated testing ensures zero errors.

No production downtime - Test DR site readiness without impacting production services.

Comprehensive reports provide a high level test status, execution details and differences

with past testing result.

How IBM Resiliency Orchestration Works?

IBM Resiliency Orchestration architecture is a distributed approach that ties together business

goals, DR workflow and solution infrastructure management to deliver an enterprise class

continuity manager product.

The Business Layer provides a way to specify business goals and monitor and manage how well

they are met. Specifically, the Business Layer:

Translates application continuity goals to RPO metrics.

Translates time of day based responses to coordinated action for each part of the solution.

The Process Layer provides proven, industry tailored templates for automating DR workflow

involved in deploying and supporting IT continuity solution. Specifically, the Process Layer:

Standardize interactions and automates sequences of steps required for data consistency

and application continuity.

Provides Policy engine to program and automate time of day based responses to events.

Automates tasks such as scheduled testing, report generation a

Documents

IBM · IBM Resiliency Orchestration Administrator's Guide © IBM Corportation 2017, 2018 0 Contents Introduction