Oracle® Real Application ClustersAdministration and Deployment Guide

11g Release 2 (11.2)

E16795-07

August 2010

Oracle Real Application Clusters Administration and Deployment Guide, 11g Release 2 (11.2)

E16795-07

Copyright © 1999, 2010, Oracle and/or its affiliates. All rights reserved.

Primary Author: Richard Strohm

Contributing Authors: Troy Anthony, Lance Ashdown, Ram Avudaiappan, Prasad Bagal, Mark Bauer, Anand Beldalker, Eric Belden, Gajanan Bhat, David Brower, George Claborn, Carol Colrain, Jonathan Creighton, Rajesh Dasari, Steve Fogel, Richard Frank, GP Prabhaker Gongloor, Wei Hu, Yong Hu, Dominique Jeunot, Sameer Joshi, Raj K. Kammend, Sana Karam, Roland Knapp, Ken Lee, Karen Li, Barb Lundhild, Venkat Maddali, Bill Manry, Gaurav Manglik, John McHugh, Saar Maoz, Matthew Mckerley, Markus Michalewicz, Anil Nair, Philip Newlan, Michael Nowak, Muthu Olagappan, Bharat Paliwal, Hanlin Qian, Mark Ramacher, Kevin Reardon, Dipak Saggi, Sudheendra Sampath, Viv Schupmann, Daniel Semler, Ara Shakian, Cathy Shea, Khethavath P. Singh, Kesavan Srinivasan, Janet Stern, Leo Tominna, Peter Wahl, Tak Wang, Richard Wessman, Douglas Williams, Mike Zampiceni, Michael Zoll

This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited.

The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing.

If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable:

U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms set forth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065.

This software is developed for general use in a variety of information management applications. It is not developed or intended for use in any inherently dangerous applications, including applications which may create a risk of personal injury. If you use this software in dangerous applications, then you shall be responsible to take all appropriate fail-safe, backup, redundancy, and other measures to ensure the safe use of this software. Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software in dangerous applications.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

This software and documentation may provide access to or information on content, products, and services from third parties. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third-party content, products, and services. Oracle Corporation and its affiliates will not be responsible for any loss, costs, or damages incurred due to your access to or use of third-party content, products, or services.

iii

Contents

Preface ............................................................................................................................................................... xix

Audience..................................................................................................................................................... xixDocumentation Accessibility ................................................................................................................... xixRelated Documents ................................................................................................................................... xxConventions ............................................................................................................................................... xxi

What's New in Oracle RAC Administration and Deployment? ..................................... xxiii

Oracle Database 11g Release 2 (11.2.0.2) New Features in Oracle RAC.......................................... xxiiiOracle Database 11g Release 2 (11.2.0.1) New Features in Oracle RAC.......................................... xxiv

1 Introduction to Oracle RAC

Overview of Oracle RAC ........................................................................................................................ 1-1Overview of Oracle Clusterware for Oracle RAC.............................................................................. 1-3Overview of Oracle RAC Architecture and Processing .................................................................... 1-4

Understanding Cluster-Aware Storage Solutions......................................................................... 1-4Overview of Connecting to Oracle Database Using Services and VIP Addresses ................... 1-4About Oracle RAC Software Components..................................................................................... 1-5About Oracle RAC Background Processes..................................................................................... 1-6

Overview of Automatic Workload Management ............................................................................... 1-7Overview of Installing Oracle RAC ..................................................................................................... 1-9

Understanding Compatibility in Oracle RAC Environments ..................................................... 1-9Overview of Oracle RAC Installation and Database Creation ................................................. 1-10Overview of Extending the Grid Foundation and Oracle RAC Software .............................. 1-10

Overview of Managing Oracle RAC Environments....................................................................... 1-11About Designing and Deploying Oracle RAC Environments.................................................. 1-12About Administrative Tools for Oracle RAC Environments ................................................... 1-12About Monitoring Oracle RAC Environments ........................................................................... 1-13About Evaluating Performance in Oracle RAC Environments................................................ 1-14

2 Administering Storage

Overview of Storage in Oracle RAC..................................................................................................... 2-1Optimal Flexible Architecture ............................................................................................................... 2-2Data File Access in Oracle RAC ............................................................................................................ 2-2Redo Log File Storage in Oracle RAC .................................................................................................. 2-3Automatic Undo Management in Oracle RAC ................................................................................... 2-3

iv

Oracle Automatic Storage Management in Oracle RAC .................................................................. 2-4Storage Management in Oracle RAC .............................................................................................. 2-4Modifying Disk Group Configurations for Oracle ASM in Oracle RAC................................... 2-5Oracle ASM Disk Group Management ........................................................................................... 2-5Configuring Preferred Mirror Read Disks in Extended Distance Clusters ............................... 2-5Converting Single-Instance Oracle ASM to Clustered Oracle ASM........................................... 2-6Administering Oracle ASM Instances with SRVCTL in Oracle RAC......................................... 2-6

3 Administering Database Instances and Cluster Databases

Tools for Administering Oracle RAC ................................................................................................... 3-1Overview of Oracle RAC Database Administration..................................................................... 3-2

Converting an Administrator-Managed Database to a Policy-Managed Database.......... 3-3Administering Oracle RAC with Oracle Enterprise Manager..................................................... 3-4Administering Oracle RAC with SQL*Plus ................................................................................... 3-4

Changing the SQL*Plus Prompt ............................................................................................... 3-5How SQL*Plus Commands Affect Instances .......................................................................... 3-5

Administering Oracle RAC with SRVCTL ..................................................................................... 3-6Starting and Stopping Instances and Oracle RAC Databases ......................................................... 3-7

Starting and Stopping with Oracle Enterprise Manager.............................................................. 3-8Starting Up and Shutting Down with SQL*Plus .......................................................................... 3-8Starting Up and Shutting Down with SRVCTL............................................................................. 3-9

Verifying That Instances are Running .............................................................................................. 3-10Terminating Sessions On a Specific Cluster Instance.................................................................... 3-10Overview of Initialization Parameter Files in Oracle RAC .......................................................... 3-12

Setting SPFILE Parameter Values for Oracle RAC..................................................................... 3-12Parameter File Search Order in Oracle RAC............................................................................... 3-13Backing Up the Server Parameter File ......................................................................................... 3-14

Initialization Parameter Use in Oracle RAC .................................................................................... 3-14Parameters That Must Have Identical Settings on All Instances ............................................. 3-17Parameters That Have Unique Settings on All Instances.......................................................... 3-17Parameters That Should Have Identical Settings on All Instances.......................................... 3-18

Quiescing Oracle RAC Databases...................................................................................................... 3-19Administering Multiple Cluster Interconnects on Linux and UNIX Platforms ....................... 3-20

Recommendations for Setting the CLUSTER_INTERCONNECTS Parameter...................... 3-20Usage Examples for the CLUSTER_INTERCONNECTS Parameter....................................... 3-21

Customizing How Oracle Clusterware Manages Oracle RAC Databases ................................. 3-22Advanced Oracle Enterprise Manager Administration ................................................................. 3-23

Using Oracle Enterprise Manager Grid Control to Discover Nodes and Instances.............. 3-24Other Oracle Enterprise Manager Capabilities........................................................................... 3-24Administering Jobs and Alerts in Oracle RAC ........................................................................... 3-25

Administering Jobs in Oracle RAC ....................................................................................... 3-25Administering Alerts in Oracle RAC with Oracle Enterprise Manager .......................... 3-26Performing Scheduled Maintenance Using Defined Blackouts in Oracle Enterprise Manager.................................................................................................... 3-26

4 Administering Oracle RAC One Node

Adding an Oracle RAC One Node Database ...................................................................................... 4-2

v

Converting Databases.............................................................................................................................. 4-3Converting a Database from Oracle RAC to Oracle RAC One Node......................................... 4-3Converting a Database from Oracle RAC One Node to Oracle RAC......................................... 4-3

Online Database Relocation................................................................................................................... 4-4

5 Introduction to Automatic Workload Management

Overview of Automatic Workload Management ............................................................................... 5-2Automatic Workload Repository ........................................................................................................... 5-3Service Deployment Options................................................................................................................. 5-3

Using Oracle Services ........................................................................................................................ 5-4Service Characteristics....................................................................................................................... 5-4

Service Name............................................................................................................................... 5-4Net Service Name ....................................................................................................................... 5-5Service Edition............................................................................................................................. 5-5Service Management Policy ...................................................................................................... 5-5Database Role for a Service ....................................................................................................... 5-5Instance Preference ..................................................................................................................... 5-5Server Pool Assignment............................................................................................................. 5-6Load Balancing Advisory Goal for Runtime Connection Load Balancing......................... 5-6Connection Load Balancing Goal ............................................................................................. 5-7Distributed Transaction Processing ......................................................................................... 5-7Subscription to High Availability Events................................................................................ 5-7Transparent Application Failover Policy ................................................................................ 5-7

Service Usage in an Oracle RAC Database..................................................................................... 5-8Resource Profile for a Service.................................................................................................... 5-8Database Resource Manager Consumer Group Mappings for Services............................. 5-8Performance Monitoring by Service with AWR..................................................................... 5-8Parallel Operations and Services .............................................................................................. 5-8Oracle Streams and Oracle RAC............................................................................................... 5-9

Default Service Connections............................................................................................................. 5-9Connection Load Balancing ................................................................................................................... 5-9

Client-Side Load Balancing............................................................................................................ 5-10Generic Database Clients ........................................................................................................ 5-10

Other Client-Side Connection Features ....................................................................................... 5-10JDBC Clients ............................................................................................................................. 5-11Oracle Call Interface Clients................................................................................................... 5-11

Server-Side Load Balancing........................................................................................................... 5-11Fast Application Notification.............................................................................................................. 5-12

Overview of Fast Application Notification ................................................................................. 5-12Application High Availability with Services and FAN............................................................. 5-13Managing Unplanned Outages..................................................................................................... 5-13Managing Planned Outages .......................................................................................................... 5-14Fast Application Notification High Availability Events ........................................................... 5-14Using Fast Application Notification Callouts ............................................................................. 5-16

Load Balancing Advisory..................................................................................................................... 5-16Overview of the Load Balancing Advisory................................................................................. 5-17Configuring Your Environment to Use the Load Balancing Advisory................................... 5-17

vi

Load Balancing Advisory FAN Events ....................................................................................... 5-18Monitoring Load Balancing Advisory FAN Events................................................................... 5-18

Oracle Clients That Are Integrated with Fast Application Notification .................................... 5-19Overview of Oracle Integrated Clients and FAN....................................................................... 5-19Enabling JDBC Clients for Fast Connection Failover................................................................. 5-20

ONS for JDBC Clients.............................................................................................................. 5-20Configuring FCF for JDBC/OCI and JDBC Thin Driver Clients ...................................... 5-21

Enabling JDBC Clients for Runtime Connection Load Balancing ........................................... 5-22Enabling Oracle Call Interface Clients for Fast Connection Failover ...................................... 5-22Enabling Oracle Call Interface Clients for Runtime Connection Load Balancing................. 5-23Enabling ODP.NET Clients to Receive FAN High Availability Events.................................. 5-24Enabling ODP.NET Clients to Receive FAN Load Balancing Advisory Events.................... 5-25

Services and Distributed Transaction Processing in Oracle RAC ............................................... 5-25Overview of XA Transactions and DTP Services ....................................................................... 5-26Benefits of DTP Services for XA Transactions ............................................................................ 5-26Using DTP Services with Oracle RAC ......................................................................................... 5-27Configuring DTP Services.............................................................................................................. 5-27Relocating DTP Services in Administrator-Managed Databases............................................. 5-28

Administering Services ........................................................................................................................ 5-28Overview of Service Administration............................................................................................ 5-28Administering Services with Oracle Enterprise Manager ........................................................ 5-30Administering Services with SRVCTL......................................................................................... 5-30

Creating Services with SRVCTL ............................................................................................ 5-31Starting and Stopping Services with SRVCTL..................................................................... 5-31Enabling and Disabling Services with SRVCTL.................................................................. 5-31Relocating Services with SRVCTL......................................................................................... 5-32Obtaining the Statuses of Services with SRVCTL ............................................................... 5-32Obtaining the Configuration of Services with SRVCTL..................................................... 5-32

Measuring Performance by Service Using the Automatic Workload Repository .................... 5-33Automatic Workload Repository Service Thresholds and Alerts ................................................ 5-34

Example of Services and Thresholds Alerts ............................................................................... 5-34Enable Service, Module, and Action Monitoring ....................................................................... 5-35

6 Configuring Recovery Manager and Archiving

Overview of Configuring RMAN for Oracle RAC ............................................................................ 6-1Configuring the RMAN Snapshot Control File Location ................................................................ 6-1Configuring RMAN to Automatically Backup the Control File and SPFILE .............................. 6-2Crosschecking on Multiple Oracle RAC Nodes................................................................................. 6-3Configuring Channels for RMAN in Oracle RAC............................................................................. 6-3

Configuring Channels to Use Automatic Load Balancing........................................................... 6-3Configuring Channels to Use a Specific Channel.......................................................................... 6-3

Managing Archived Redo Logs Using RMAN in Oracle RAC ....................................................... 6-4Archived Redo Log File Conventions in Oracle RAC....................................................................... 6-5RMAN Archiving Configuration Scenarios........................................................................................ 6-5

Oracle Automatic Storage Management and Cluster File System Archiving Scheme ............ 6-6Advantages of the Cluster File System Archiving Scheme ................................................. 6-6Initialization Parameter Settings for the Cluster File System Archiving Scheme ............. 6-6

vii

Location of Archived Logs for the Cluster File System Archiving Scheme ....................... 6-7Noncluster File System Local Archiving Scheme.......................................................................... 6-7

Considerations for Using Noncluster File System Local Archiving.................................... 6-7Initialization Parameter Settings for Noncluster File System Local Archiving ................. 6-7Location of Archived Logs for Noncluster File System Local Archiving........................... 6-8File System Configuration for Noncluster File System Local Archiving............................ 6-8

Changing the Archiving Mode in Oracle RAC .................................................................................. 6-8Monitoring the Archiver Processes....................................................................................................... 6-9

7 Managing Backup and Recovery

RMAN Backup Scenario for Noncluster File System Backups....................................................... 7-1RMAN Restore Scenarios for Oracle RAC .......................................................................................... 7-1

Cluster File System Restore Scheme................................................................................................ 7-2Noncluster File System Restore Scheme......................................................................................... 7-2Using RMAN or Oracle Enterprise Manager to Restore the Server Parameter File (SPFILE) 7-2

Instance Recovery in Oracle RAC ......................................................................................................... 7-3Single Node Failure in Oracle RAC................................................................................................. 7-3Multiple-Node Failures in Oracle RAC .......................................................................................... 7-3Using RMAN to Create Backups in Oracle RAC........................................................................... 7-3Channel Connections to Cluster Instances ..................................................................................... 7-4Node Affinity Awareness of Fast Connections ............................................................................ 7-4Deleting Archived Redo Logs after a Successful Backup............................................................. 7-5Autolocation for Backup and Restore Commands........................................................................ 7-5

Media Recovery in Oracle RAC............................................................................................................. 7-5Parallel Recovery in Oracle RAC .......................................................................................................... 7-6

Parallel Recovery with RMAN......................................................................................................... 7-6Disabling Parallel Recovery.............................................................................................................. 7-6

Disabling Instance and Crash Recovery Parallelism ............................................................. 7-6Disabling Media Recovery Parallelism.................................................................................... 7-6

Using a Fast Recovery Area in Oracle RAC ........................................................................................ 7-7

8 Cloning Oracle RAC to Nodes in a Cluster

Introduction to Cloning Oracle RAC ................................................................................................... 8-1Preparing to Clone Oracle RAC............................................................................................................. 8-2Cloning Oracle RAC to Nodes in a Cluster......................................................................................... 8-3

Deploying Oracle RAC Database Homes....................................................................................... 8-3Locating and Viewing Log Files Generated During Cloning.......................................................... 8-6

9 Using Cloning to Extend Oracle RAC to Nodes in the Same Cluster

About Adding Nodes Using Cloning in Oracle RAC Environments ............................................ 9-1Cloning Non-Shared Oracle Homes on Linux and UNIX Systems ................................................ 9-1Cloning Shared Oracle Homes on Linux and UNIX Systems ......................................................... 9-2Cloning Oracle Homes on Windows Systems .................................................................................... 9-3

viii

10 Adding and Deleting Oracle RAC from Nodes on Linux and UNIX Systems

Adding Oracle RAC to Nodes with Oracle Clusterware Installed .............................................. 10-1Adding Oracle RAC Database Instances to Target Nodes........................................................ 10-2

Using DBCA in Interactive Mode to Add Database Instances to Target Nodes ............ 10-3Using DBCA in Silent Mode to Add Database Instances to Target Nodes ..................... 10-4

Deleting Oracle RAC from a Cluster Node...................................................................................... 10-5Deleting Instances from Oracle RAC Databases ........................................................................ 10-5

Using DBCA in Interactive Mode to Delete Instances from Nodes ................................. 10-6Using DBCA in Silent Mode to Delete Instances from Nodes .......................................... 10-7

Removing Oracle RAC ................................................................................................................... 10-8Deleting Nodes from the Cluster .................................................................................................. 10-8

11 Adding and Deleting Oracle RAC from Nodes on Windows Systems

Adding Oracle RAC to Nodes with Oracle Clusterware Installed .............................................. 11-1Adding Oracle RAC Database Instances to Target Nodes........................................................ 11-2

Using DBCA in Interactive Mode to Add Database Instances to Target Nodes ............ 11-3Using DBCA in Silent Mode to Add Database Instances to Target Nodes ..................... 11-4

Deleting Oracle RAC from a Cluster Node...................................................................................... 11-4Deleting Instances from Oracle RAC Databases ........................................................................ 11-5

Using DBCA in Interactive Mode to Delete Instances from Nodes ................................. 11-6Using DBCA in Silent Mode to Delete Instances from Nodes .......................................... 11-7

Removing Oracle RAC ................................................................................................................... 11-7Deleting Nodes from the Cluster .................................................................................................. 11-8

12 Design and Deployment Techniques

Deploying Oracle RAC for High Availability ................................................................................. 12-1Best Practices for Deploying Oracle RAC in a High Availability Environment.................... 12-2Consolidating Multiple Applications in a Database or Multiple Databases in a Cluster..... 12-3Scalability of Oracle RAC............................................................................................................... 12-4

General Design Considerations for Oracle RAC ............................................................................ 12-5General Database Deployment Topics for Oracle RAC ................................................................ 12-5

Tablespace Use in Oracle RAC...................................................................................................... 12-6Object Creation and Performance in Oracle RAC ..................................................................... 12-6Node Addition and Deletion and the SYSAUX Tablespace in Oracle RAC........................... 12-6Distributed Transactions and Oracle RAC.................................................................................. 12-6Deploying OLTP Applications in Oracle RAC ........................................................................... 12-7Flexible Implementation with Cache Fusion .............................................................................. 12-7Deploying Data Warehouse Applications with Oracle RAC.................................................... 12-7

Speed-Up for Data Warehouse Applications on Oracle RAC........................................... 12-7Parallel Execution in Data Warehouse Systems and Oracle RAC .................................... 12-8

Data Security Considerations in Oracle RAC ............................................................................. 12-8Transparent Data Encryption and Wallets........................................................................... 12-8Windows Firewall Considerations ........................................................................................ 12-9

13 Monitoring Performance

Overview of Monitoring and Tuning Oracle RAC Databases...................................................... 13-1

ix

Monitoring Oracle RAC and Oracle Clusterware ...................................................................... 13-2The Cluster Database Home Page ......................................................................................... 13-2The Interconnects Page ........................................................................................................... 13-3The Cluster Performance Page............................................................................................... 13-3

Tuning Oracle RAC Databases...................................................................................................... 13-4Verifying the Interconnect Settings for Oracle RAC...................................................................... 13-4Influencing Interconnect Processing ................................................................................................. 13-5Performance Views in Oracle RAC.................................................................................................... 13-6Creating Oracle RAC Data Dictionary Views with CATCLUST.SQL......................................... 13-6Oracle RAC Performance Statistics ................................................................................................... 13-6Automatic Workload Repository in Oracle RAC Environments ................................................. 13-6Active Session History Reports for Oracle RAC ............................................................................. 13-7

Overview of ASH Reports for Oracle RAC................................................................................. 13-7ASH Report for Oracle RAC: Top Cluster Events...................................................................... 13-8ASH Report for Oracle RAC: Top Remote Instance................................................................... 13-8

Monitoring Oracle RAC Statistics and Wait Events ....................................................................... 13-8Oracle RAC Statistics and Events in AWR and Statspack Reports.......................................... 13-8Oracle RAC Wait Events ................................................................................................................ 13-9Monitoring Performance by Analyzing GCS and GES Statistics ............................................. 13-9

Analyzing the Effect of Cache Fusion in Oracle RAC ........................................................ 13-9Analyzing Performance Using GCS and GES Statistics ................................................... 13-10

Analyzing Cache Fusion Transfer Impact Using GCS Statistics ............................................ 13-10Analyzing Response Times Based on Wait Events .................................................................. 13-11

Block-Related Wait Events.................................................................................................... 13-11Message-Related Wait Events .............................................................................................. 13-12Contention-Related Wait Events ......................................................................................... 13-12Load-Related Wait Events .................................................................................................... 13-13

A Server Control Utility Reference

Using SRVCTL ......................................................................................................................................... A-2Overview of SRVCTL ....................................................................................................................... A-3Operational Notes for SRVCTL....................................................................................................... A-4

Usage Information ..................................................................................................................... A-4Character Set and Case Sensitivity of Object Values ............................................................ A-4Summary of Tasks for Which SRVCTL Is Used .................................................................... A-5Using SRVCTL Help.................................................................................................................. A-6Privileges and Security.............................................................................................................. A-6

Additional Topics on SRVCTL........................................................................................................ A-8Deprecated Subprograms or Commands ...................................................................................... A-9

SRVCTL Command Reference ........................................................................................................... A-11add..................................................................................................................................................... A-14

srvctl add asm........................................................................................................................... A-14srvctl add cvu ........................................................................................................................... A-15srvctl add database .................................................................................................................. A-15srvctl add filesystem................................................................................................................ A-17srvctl add gns............................................................................................................................ A-18srvctl add instance ................................................................................................................... A-18

x

srvctl add listener..................................................................................................................... A-19srvctl add network................................................................................................................... A-20srvctl add nodeapps ................................................................................................................ A-20srvctl add oc4j ........................................................................................................................... A-21srvctl add ons............................................................................................................................ A-22srvctl add scan .......................................................................................................................... A-22srvctl add scan_listener ........................................................................................................... A-23srvctl add service ..................................................................................................................... A-23srvctl add srvpool .................................................................................................................... A-26srvctl add vip ............................................................................................................................ A-27

config................................................................................................................................................. A-29srvctl config asm ...................................................................................................................... A-29srvctl config database .............................................................................................................. A-30srvctl config filesystem............................................................................................................ A-31srvctl config gns ....................................................................................................................... A-31srvctl config listener................................................................................................................. A-32srvctl config nodeapps ............................................................................................................ A-32srvctl config oc4j....................................................................................................................... A-33srvctl config ons ....................................................................................................................... A-33srvctl config scan...................................................................................................................... A-33srvctl config scan_listener....................................................................................................... A-33srvctl config service ................................................................................................................. A-34srvctl config srvpool ................................................................................................................ A-35srvctl config vip........................................................................................................................ A-35

convert .............................................................................................................................................. A-37disable ............................................................................................................................................... A-38

srvctl disable asm..................................................................................................................... A-38srvctl disable cvu...................................................................................................................... A-39srvctl disable database............................................................................................................. A-40srvctl disable diskgroup.......................................................................................................... A-40srvctl disable filesystem .......................................................................................................... A-40srvctl disable gns...................................................................................................................... A-41srvctl disable instance.............................................................................................................. A-41srvctl disable listener ............................................................................................................... A-42srvctl disable nodeapps........................................................................................................... A-42srvctl disable oc4j ..................................................................................................................... A-43srvctl disable ons...................................................................................................................... A-43srvctl disable scan .................................................................................................................... A-43srvctl disable scan_listener ..................................................................................................... A-43srvctl disable service................................................................................................................ A-44srvctl disable vip ...................................................................................................................... A-45

downgrade ....................................................................................................................................... A-46srvctl downgrade database..................................................................................................... A-46

enable ................................................................................................................................................ A-47srvctl enable asm...................................................................................................................... A-47srvctl enable cvu....................................................................................................................... A-48srvctl enable database.............................................................................................................. A-48

xi

srvctl enable diskgroup........................................................................................................... A-49srvctl enable filesystem ........................................................................................................... A-49srvctl enable gns....................................................................................................................... A-49srvctl enable instance............................................................................................................... A-50srvctl enable listener ................................................................................................................ A-50srvctl enable nodeapps............................................................................................................ A-51srvctl enable oc4j ...................................................................................................................... A-51srvctl enable ons ....................................................................................................................... A-52srvctl enable scan ..................................................................................................................... A-52srvctl enable scan_listener ...................................................................................................... A-52srvctl enable service................................................................................................................. A-53srvctl enable vip ....................................................................................................................... A-53

getenv................................................................................................................................................ A-55srvctl getenv asm...................................................................................................................... A-55srvctl getenv database ............................................................................................................. A-55srvctl getenv listener................................................................................................................ A-56srvctl getenv nodeapps ........................................................................................................... A-56srvctl getenv vip....................................................................................................................... A-57

modify............................................................................................................................................... A-58srvctl modify asm..................................................................................................................... A-58srvctl modify cvu ..................................................................................................................... A-59srvctl modify database ............................................................................................................ A-59srvctl modify filesystem.......................................................................................................... A-61srvctl modify gns...................................................................................................................... A-62srvctl modify instance ............................................................................................................. A-62srvctl modify listener............................................................................................................... A-63srvctl modify network............................................................................................................. A-64srvctl modify nodeapps .......................................................................................................... A-64srvctl modify oc4j ..................................................................................................................... A-65srvctl modify ons...................................................................................................................... A-66srvctl modify scan .................................................................................................................... A-66srvctl modify scan_listener ..................................................................................................... A-67srvctl modify service ............................................................................................................... A-67srvctl modify srvpool .............................................................................................................. A-70

relocate.............................................................................................................................................. A-72srvctl relocate cvu .................................................................................................................... A-72srvctl relocate database ........................................................................................................... A-72srvctl relocate gns..................................................................................................................... A-73srvctl relocate oc4j .................................................................................................................... A-74srvctl relocate scan ................................................................................................................... A-74srvctl relocate scan_listener .................................................................................................... A-75srvctl relocate server................................................................................................................ A-75srvctl relocate service............................................................................................................... A-76

remove .............................................................................................................................................. A-77srvctl remove asm .................................................................................................................... A-78srvctl remove cvu..................................................................................................................... A-78srvctl remove database............................................................................................................ A-78

xii

srvctl remove diskgroup......................................................................................................... A-79srvctl remove filesystem ......................................................................................................... A-79srvctl remove gns ..................................................................................................................... A-80srvctl remove instance............................................................................................................. A-80srvctl remove listener .............................................................................................................. A-81srvctl remove nodeapps.......................................................................................................... A-81srvctl remove oc4j .................................................................................................................... A-82srvctl remove ons ..................................................................................................................... A-82srvctl remove scan.................................................................................................................... A-82srvctl remove scan_listener .................................................................................................... A-83srvctl remove service ............................................................................................................... A-83srvctl remove srvpool.............................................................................................................. A-83srvctl remove vip ..................................................................................................................... A-84

setenv ................................................................................................................................................ A-85srvctl setenv asm ...................................................................................................................... A-85srvctl setenv database.............................................................................................................. A-85srvctl setenv listener ................................................................................................................ A-86srvctl setenv nodeapps............................................................................................................ A-86srvctl setenv vip ....................................................................................................................... A-87

start.................................................................................................................................................... A-88srvctl start asm.......................................................................................................................... A-88srvctl start cvu .......................................................................................................................... A-89srvctl start database ................................................................................................................. A-89srvctl start diskgroup .............................................................................................................. A-90srvctl start filesystem............................................................................................................... A-91srvctl start gns........................................................................................................................... A-91srvctl start home....................................................................................................................... A-92srvctl start instance .................................................................................................................. A-92srvctl start listener.................................................................................................................... A-93srvctl start nodeapps ............................................................................................................... A-93srvctl start oc4j .......................................................................................................................... A-94srvctl start ons........................................................................................................................... A-94srvctl start scan ......................................................................................................................... A-94srvctl start scan_listener .......................................................................................................... A-95srvctl start service..................................................................................................................... A-95srvctl start vip ........................................................................................................................... A-96

status ................................................................................................................................................. A-98srvctl status asm ....................................................................................................................... A-98srvctl status cvu........................................................................................................................ A-99srvctl status database............................................................................................................... A-99srvctl status diskgroup.......................................................................................................... A-100srvctl status filesystem .......................................................................................................... A-100srvctl status gns ...................................................................................................................... A-101srvctl status home .................................................................................................................. A-101srvctl status instance.............................................................................................................. A-101srvctl status listener ............................................................................................................... A-102srvctl status nodeapps........................................................................................................... A-102

xiii

srvctl status oc4j ..................................................................................................................... A-103srvctl status ons ...................................................................................................................... A-103srvctl status scan .................................................................................................................... A-103srvctl status scan_listener ..................................................................................................... A-104srvctl status server ................................................................................................................. A-104srvctl status service ................................................................................................................ A-105srvctl status srvpool............................................................................................................... A-105srvctl status vip ...................................................................................................................... A-106

stop .................................................................................................................................................. A-107srvctl stop asm........................................................................................................................ A-107srvctl stop cvu......................................................................................................................... A-108srvctl stop database................................................................................................................ A-108srvctl stop diskgroup............................................................................................................. A-109srvctl stop filesystem ............................................................................................................. A-109srvctl stop gns......................................................................................................................... A-110srvctl stop home ..................................................................................................................... A-110srvctl stop instance................................................................................................................. A-111srvctl stop listener .................................................................................................................. A-112srvctl stop nodeapps.............................................................................................................. A-112srvctl stop oc4j ........................................................................................................................ A-113srvctl stop ons......................................................................................................................... A-113srvctl stop scan ....................................................................................................................... A-114srvctl stop scan_listener ........................................................................................................ A-114srvctl stop service................................................................................................................... A-115srvctl stop vip ......................................................................................................................... A-116

unsetenv.......................................................................................................................................... A-117srvctl unsetenv asm ............................................................................................................... A-117srvctl unsetenv database ....................................................................................................... A-117srvctl unsetenv listener ......................................................................................................... A-118srvctl unsetenv nodeapps ..................................................................................................... A-118srvctl unsetenv vip................................................................................................................. A-119

upgrade........................................................................................................................................... A-120srvctl upgrade database ........................................................................................................ A-120

B Troubleshooting Oracle RAC

Where to Find Files for Analyzing Errors........................................................................................... B-1Managing Diagnostic Data in Oracle RAC ........................................................................................ B-2Using Instance-Specific Alert Files in Oracle RAC .......................................................................... B-2Enabling Tracing for Java-Based Tools and Utilities in Oracle RAC ............................................ B-3Resolving Pending Shutdown Issues.................................................................................................. B-3How to Determine If Oracle RAC Instances Are Using the Private Network ............................ B-3

Glossary

Index

xiv

xv

List of Tables

3–1 How SQL*Plus Commands Affect Instances ......................................................................... 3-63–2 Descriptions of V$ACTIVE_INSTANCES Columns ......................................................... 3-103–3 Initialization Parameters Specific to Oracle RAC............................................................... 3-153–4 Parameters That Should Have Identical Settings on All Instances ................................. 3-185–1 FAN Event Types.................................................................................................................... 5-145–2 Event Parameter Name-Value Pairs and Descriptions...................................................... 5-145–3 FAN Parameters and Matching Session Information........................................................ 5-155–4 Load Balancing Advisory FAN Events................................................................................ 5-186–1 Archived Redo Log Filename Format Parameters................................................................ 6-56–2 UNIX/NFS Location Log Examples, Noncluster File System Local Archiving ............... 6-86–3 UNIX/NFS Configuration for Shared Read Local Archiving Examples........................... 6-88–1 clone.pl Script Parameters ........................................................................................................ 8-48–2 Environment Variables Passed to the clone.pl Script ........................................................... 8-58–3 Cloning Parameters Passed to the clone.pl Script................................................................. 8-68–4 Finding the Location of the Oracle Inventory Directory...................................................... 8-710–1 Variables in the DBCA Silent Mode Syntax ........................................................................ 10-512–1 Oracle Executables Requiring Windows Firewall Exceptions ........................................ 12-9A–1 String Restrictions for SRVCTL Object Names..................................................................... A-5A–2 Deprecated Commands and Options for SRVCTL .............................................................. A-9A–3 Summary of SRVCTL Commands........................................................................................ A-11A–4 Object Keywords and Abbreviations ................................................................................... A-12A–5 srvctl add Summary ............................................................................................................... A-14A–6 srvctl add cvu Options ........................................................................................................... A-15A–7 srvctl add database Options .................................................................................................. A-16A–8 srvctl add filesystem Options................................................................................................ A-18A–9 srvctl add gns Options ........................................................................................................... A-18A–10 srvctl add instance Options ................................................................................................... A-19A–11 srvctl add listener Options..................................................................................................... A-19A–12 srvctl add network Options................................................................................................... A-20A–13 srvctl add nodeapps Options ................................................................................................ A-21A–14 srvctl add ons Options ........................................................................................................... A-22A–15 srvctl add scan Options.......................................................................................................... A-23A–16 srvctl add scan_listener Options........................................................................................... A-23A–17 srvctl add service Options ..................................................................................................... A-24A–18 srvctl add srvpool Options .................................................................................................... A-27A–19 srvctl add vip Options............................................................................................................ A-27A–20 srvctl config Summary ........................................................................................................... A-29A–21 srvctl config asm Options ...................................................................................................... A-30A–22 srvctl config database Options.............................................................................................. A-30A–23 srvctl config filesystem Options............................................................................................ A-31A–24 srvctl config gns Options ....................................................................................................... A-31A–25 srvctl config listener Options ................................................................................................ A-32A–26 srvctl config nodeapps Option.............................................................................................. A-32A–27 srvctl config service Options ................................................................................................. A-34A–28 srvctl config vip Options........................................................................................................ A-35A–29 srvctl convert database Options............................................................................................ A-37A–30 srvctl disable Summary.......................................................................................................... A-38A–31 srvctl disable asm Options..................................................................................................... A-39A–32 srvctl disable cvu Options ..................................................................................................... A-39A–33 srvctl disable database Options ............................................................................................ A-40A–34 srvctl disable diskgroup Options.......................................................................................... A-40A–35 srvctl disable filesystem Options.......................................................................................... A-41A–36 srvctl disable gns Options...................................................................................................... A-41

xvi

A–37 srvctl disable instance Options ............................................................................................. A-42A–38 srvctl disable listener Options............................................................................................... A-42A–39 srvctl disable nodeapps Options .......................................................................................... A-42A–40 srvctl disable oc4j Options ..................................................................................................... A-43A–41 srvctl disable service Options................................................................................................ A-44A–42 srvctl disable vip Options ...................................................................................................... A-45A–43 srvctl downgrade database Options .................................................................................... A-46A–44 srvctl enable Summary........................................................................................................... A-47A–45 srvctl enable asm Option ....................................................................................................... A-48A–46 srvctl enable cvu Options ...................................................................................................... A-48A–47 srvctl enable database Options ............................................................................................. A-48A–48 srvctl enable diskgroup Options........................................................................................... A-49A–49 srvctl enable filesystem Options ........................................................................................... A-49A–50 srvctl enable gns Options....................................................................................................... A-50A–51 srvctl enable instance Option ................................................................................................ A-50A–52 srvctl enable listener Options................................................................................................ A-50A–53 srvctl enable nodeapps Options............................................................................................ A-51A–54 srvctl enable oc4j Options ...................................................................................................... A-51A–55 srvctl enable service Options................................................................................................. A-53A–56 srvctl enable vip Options ....................................................................................................... A-54A–57 srvctl getenv Summary .......................................................................................................... A-55A–58 srvctl getenv asm Options ..................................................................................................... A-55A–59 srvctl getenv database Options............................................................................................. A-55A–60 srvctl getenv listener Options ............................................................................................... A-56A–61 srvctl getenv nodeapps Options ........................................................................................... A-56A–62 srvctl getenv vip Options....................................................................................................... A-57A–63 srvctl modify Summary ......................................................................................................... A-58A–64 srvctl modify asm Options .................................................................................................... A-59A–65 srvctl modify cvu Options ..................................................................................................... A-59A–66 srvctl modify database Options ............................................................................................ A-60A–67 srvctl modify filesystem Options.......................................................................................... A-62A–68 srvctl modify gns Options ..................................................................................................... A-62A–69 srvctl modify instance Options ............................................................................................. A-63A–70 srvctl modify listener Options............................................................................................... A-63A–71 srvctl modify network Options............................................................................................. A-64A–72 srvctl modify nodeapps Options .......................................................................................... A-65A–73 srvctl modify oc4j Options..................................................................................................... A-66A–74 srvctl modify ons Options ..................................................................................................... A-66A–75 srvctl modify scan_listener Options..................................................................................... A-67A–76 srvctl modify service Options for Moving a Service.......................................................... A-68A–77 srvctl modify service Options for Changing an Available Instance to

a Preferred Instance ................................................................................................................ A-69A–78 srvctl modify service Options for Changing Available and Preferred Status of

Multiple Instances................................................................................................................... A-69A–79 srvctl modify service Options ............................................................................................... A-70A–80 srvctl modify srvpool Options .............................................................................................. A-71A–81 srvctl relocate Summary ........................................................................................................ A-72A–82 srvctl relocate cvu Options .................................................................................................... A-72A–83 srvctl relocate database Options ........................................................................................... A-73A–84 srvctl relocate gns Options .................................................................................................... A-74A–85 srvctl relocate oc4j Options.................................................................................................... A-74A–86 srvctl relocate scan Options................................................................................................... A-74A–87 srvctl relocate scan_listener Options.................................................................................... A-75A–88 srvctl relocate server Options................................................................................................ A-75A–89 srvctl relocate service Options .............................................................................................. A-76

xvii

A–90 srvctl remove Summary......................................................................................................... A-77A–91 srvctl remove database Options............................................................................................ A-78A–92 srvctl remove diskgroup Options......................................................................................... A-79A–93 srvctl remove filesystem Options ......................................................................................... A-79A–94 srvctl remove instance Options............................................................................................. A-80A–95 srvctl remove listener Options .............................................................................................. A-81A–96 srvctl remove nodeapps Options.......................................................................................... A-81A–97 srvctl remove oc4j Options .................................................................................................... A-82A–98 srvctl remove ons Options..................................................................................................... A-82A–99 srvctl remove service Options............................................................................................... A-83A–100 srvctl remove vip Options ..................................................................................................... A-84A–101 srvctl setenv Summary........................................................................................................... A-85A–102 srvctl setenv asm Options...................................................................................................... A-85A–103 srvctl setenv database Options ............................................................................................. A-86A–104 srvctl setenv listener Options ................................................................................................ A-86A–105 srvctl setenv nodeapps Options............................................................................................ A-87A–106 srvctl setenv vip Options ....................................................................................................... A-87A–107 srvctl start Summary .............................................................................................................. A-88A–108 srvctl start asm Options ......................................................................................................... A-89A–109 srvctl start cvu Options .......................................................................................................... A-89A–110 srvctl start database Options ................................................................................................. A-90A–111 srvctl start diskgroup Options .............................................................................................. A-90A–112 srvctl start filesystem Options............................................................................................... A-91A–113 srvctl start gns Options .......................................................................................................... A-91A–114 srvctl start home Options....................................................................................................... A-92A–115 srvctl start instance Options .................................................................................................. A-92A–116 srvctl start listener Options.................................................................................................... A-93A–117 srvctl start nodeapps Options ............................................................................................... A-94A–118 srvctl start scan Options......................................................................................................... A-95A–119 srvctl start scan_listener Options.......................................................................................... A-95A–120 srvctl start service Options .................................................................................................... A-96A–121 srvctl start vip Options........................................................................................................... A-96A–122 srvctl status Summary............................................................................................................ A-98A–123 srvctl status asm Options....................................................................................................... A-99A–124 srvctl status cvu Options........................................................................................................ A-99A–125 srvctl status database Options .............................................................................................. A-99A–126 srvctl status diskgroup Options.......................................................................................... A-100A–127 srvctl status filesystem Options .......................................................................................... A-100A–128 srvctl status gns Options...................................................................................................... A-101A–129 srvctl status home Options .................................................................................................. A-101A–130 srvctl status instance Options.............................................................................................. A-102A–131 srvctl status listener Options ............................................................................................... A-102A–132 srvctl status oc4j Options ..................................................................................................... A-103A–133 srvctl status scan Options .................................................................................................... A-104A–134 srvctl status scan_listener Options ..................................................................................... A-104A–135 srvctl status server Options ................................................................................................. A-104A–136 srvctl status service Options................................................................................................ A-105A–137 srvctl status srvpool Options............................................................................................... A-106A–138 srvctl status vip Options ...................................................................................................... A-106A–139 srvctl stop Summary............................................................................................................. A-107A–140 srvctl stop asm Option ......................................................................................................... A-108A–141 srvctl stop database Options ............................................................................................... A-109A–142 srvctl stop diskgroup Options............................................................................................. A-109A–143 srvctl stop filesystem Options............................................................................................. A-110A–144 srvctl stop gns Options......................................................................................................... A-110

xviii

A–145 srvctl stop home Options................................................................................................