11gr2 Cluster Upgrade

Oracle 11gR2 Clusterware Upgrade on HPUX

A Success Story

Presented by : Syed Jaffer Hussain Slide # 1

Disclaimer


This views/content in this slides are those of the author and do not necessarily reflect that of Oracle Corporation and/or its affiliates/subsidiaries. The material in this document is for informational purposes only and is published with no guarantee or warranty, express or implied..

This material should not be reproduced or used without the authors' written permission.

Foreword


The goal of this presentation is to explain you the best practices that we had adopted/followed/practiced to upgrade successfully our FOUR Oracle 10.2.0.4 clusterware environments to Oracle 11g R2 on HPUX Servers.

What will be covered

o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Introduction to our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.


Know your presenter

Syed Jaffer HussainDatabase Support Manager20 years IT experience11+ years as an Oracle DBAOracle ACE DirectorOracle 10g Certified Master(OCM)Oracle 10g RAC Certified ExpertOCP v8i,9i,10g & 11gITIL v3 Foundation CertifiedAuthored Oracle 11g R1/R2 Real Application Clusters EssentialsTwitter: @sjaffarhussainhttp://jaffardba.blogspot.com



o WHY upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.


Why upgrade? – A burning question

Is it mandatory to upgrade an existing environment with every Oracle new release?


Why upgrade?


Why upgrade?


Situations that perhaps drives you to oppose an upgrade:

Legacy application. Doesn’t want to take the risk. Lack of skills and support.

Why upgrade?

Situations that are likely to drives you to embrace for an upgrade:

Organization standards. Oracle release support policy. Tantalizing new features. Bug fixes.



o Why upgrade?o Upgrade compatibility MATRIX.o Upgrade pattern [best practices].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.


Upgrade compatibility matrix


Courtesy from Oracle CorporationThanks to Roy Swonger & Mike Dietrich






Minimum Version of the database that can be directly upgraded to Oracle 11g Release 2

Source Database Target Database9.2.0.8 or higher 11.2.x10.1.0.5 or higher 11.2.x10.2.0.2 or higher 11.2.x11.1.0.6 or higher 11.2.x

The following database version will require an indirect upgrade path.

Source Database ---> Upgrade Path for Target Database--->Target Database7.3.3 (or lower)-----> 7.3.4 ---> 9.2.0.8 ---->11.2.x8.0.5 (or lower)----> 8.0.6 ---> 9.2.0.8 ---->11.2.x8.1.7 (or lower)----> 8.1.7.4---> 10.2.0.4---->11.2.x9.0.1.3 (or lower)----> 9.0.1.4-- ->10.2.0.4---->11.2.x9.2.0.7(or lower)---->9.2.0.8---->11.2.x


o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [BEST PRACTICE].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.


Upgrade pattern [BEST PRACTICE]


UPGRADEClusterware

+ ASM

Upgrade pattern [best practice]


UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software



UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases



UPGRADEClusterware

+ ASM

INSTALLRDBMS

Software

UPGRADEdatabases

Post upgrade


o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Overview of our SETUP.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.


Overview of our setup


HP Superdome 1 Intel(R) Itanium 2 9000 series EMC Symmetrix DMX-4

CISCO Switch

NetBackup 6.5.6



8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes



8 nodes

Production 1 Production 2

6 nodes

7 nodes

Non-Production 1 Non-Production 2

3 nodes

•31 databases•51 instances

•4 logical processors•22GB Memory









V11.23/var/opt

V11.31/var/opt

HP UX upgrade from V11.23 TO v11.31

The framework that we have followed for the upgrade



V11.23/var/opt

V11.31/var/opt


EMC Storage Configurationupdate




V11.23/var/opt

V11.31/var/opt


EMC Storage Configurationupgrade

ClusterwareDatabasesupgrade


10.2.0.4 11.2.0.2


o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post upgrade steps.o Upgrade aftermath stories.o References.




Preparation is the headstone to every successful installation orupgrade projects.

Prepare for upgrade – prerequisites checks


Verify:o OS certification for Oracle 11gR2.o OS kernel, packages.o Additional OS groups.o Oracle base location.o GIH location.

Set GI home different from OB. Configure Single Client Access Name(SCAN)

o Maximum of 3 IP’s (same across nodes).o DNS round robin resolution.o Must be on the subnet as Public and VIP IP’s.o Not recommended to configure in HOSTS file.

root user access UNSET the following

o ORACLE_BASEo ORACLE_HOMEo ORACLE_CRS_HOMEo TNS_ADMINo ORA_NLS10

Prepare for upgrade – prerequisites checks


Space:o Additional space for [/u00] mount point.o Enough /tmp space

Back up:o Back up Oracle homes (CRS, ASM)o Back up Voting disk.o Back up OCR.

CRS version:o Remember, the crs active and software version

must be the same.crsctl query crs activeversioncrsctl query crs softwareversion

Nodes readiness:runcluvfy.sh stage –pre crsinst –n nodes_list –fixup –fixupdir <dirname> -verbose




Clusterware, ASM upgrade – What’s new?


Grid Infrastructure

Out-of-Place andIn-place upgrades

Clusterware

Automatic Storage Management (ASM)

Clusterware, ASM upgrade

Is it necessary to upgrade Clusterware AND ASM together?


Clusterware, ASM upgrade


Clusteware + ASM =

no rolling upgrade

Clusterware and ASM upgrade
























ORACLE BASE must be different from the SW

location……







Cause : False warning generated by unpublished CVU bug 10024549Solution: The warning can be ignored















Can I run the rootupgrade.sh script in PARALLEL across nodes?



Always run the script on the LOCAL node first.




Run the script in parallel on all nodes, except the LAST node.




Run the script in parallel on all nodes, except the LAST node.

Run the script on the LAST node.



Shuts down the existing clusterware stack.

Brings up the new version clusterware stack.

Up on successfully running the script on the LAST node, the CRS active/release version will set to the new version.

rootupgrade.sh



Average rootupgrade.sh execution time on each node was just about 15 min.

Took more time on couple nodes, mostly while stopping the existing cluster stack.

Look closely for the logs under /u00/app/11.2.0/grid/cfgtoollogs.









Cried for HELP!



Internet

Cried for HELP!



Internet MOS

Cried for HELP!



ASM alert.log, trace files & log files

Internet MOS

Cried for HELP!

Doesn’t help much!



Life Savior

MOS [ID 398374.1]



SQL> ORACLE instance shut down.SQL> Disconnected

[PRSR][1]prsr_get_enabled_key: OCR api procr_open_key failed for key DATABASE.ASM.node01.+asm1.ENABLED with 4[ PRSR][1]OCR error code : 4[ PRSR][1]OCR error msg : PROC-4: The cluster registry key to be operated on does not exist.2009-06-23 16:12:26.332: [ RACG][1] [28830][1][ora.node01.ASM1.asm]: prsr_fetch_asm_config() error, rc = 8

2009-06-23 16:12:27.443: [ RACG][1] [28830][1][ora.node01.ASM1.asm]:SQL*Plus: Release 10.2.0.4.0 - Production on Tue Jun 23 16:12:26 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.

Enter user-name: Connected to an idle instance.

SQL> ORA-01506: missing or illegal database nameSQL> Disconnected

cd $ORA_ASM_HOME/log/hostname/racg/ora.<hostname>.ASM.asm.log



What might have gone wrong???

ASM instance name registered as ASMn instead of +ASMn in the OCR.

Could be due to a manual entry.

GI is very particular about + in contrast to previous cluster version.

Bug 5644904: WRONG ASM KEY PROC-4: THE CLUSTER REGISTRY KEY TO BE OPERATED ON DOES NOT EXIST



Remedy

Stop ASM and dependent instances.

Remove dependency on ASM instance.

Remove ASM from OCR.

Add ASM back in with ‘+’ correct name

Add dependency to ASM.

Startup ASM.

Resume the failed step.



In action srvctl stop asm –n <hostname>

srvctl modify instance –d DBNAME –i <instance_name> -f [do it for all active instances on the node]

srvctl remove asm –n <hostname> -i <ASM_INSTANCE1>

srvctl add asm –n <hostname> -i <+ASM_INSTANCE1>-o <ORA_ASM_HOME>

srvctl modify instance –d <DBNAME> -i <instance_name>-s +ASM_INSTANCE1

[do it for all active instances on the node] srvctl start asm –n <hostname>



In action srvctl stop asm –n <hostname>

srvctl modify instance –d DBNAME –i <instance_name> -f [do it for all active instances on the node]

srvctl remove asm –n <hostname> -i <ASM_INSTANCE1>

srvctl add asm –n <hostname> -i <+ASM_INSTANCE1>-o <ORA_ASM_HOME>

srvctl modify instance –d <DBNAME> -i <instance_name>-s +ASM_INSTANCE1

[do it for all active instances on the node] srvctl start asm –n <hostname>

Repeat on all nodes!!!





Post upgrade checks – active/software version

#crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.2.0.2.0] # crsctl query crs releaseversionOracle High Availability Services release version on the local node is [11.2.0.2.0]



Post upgrade checks – cluster stats across nodes

# crsctl check cluster -all**************************************************************node01:CRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online**************************************************************node02:CRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online**************************************************************node03:CRS-4537: Cluster Ready Services is onlineCRS-4529: Cluster Synchronization Services is onlineCRS-4533: Event Manager is online**************************************************************



Post upgrade checks – query voting disk

# ocrcheck

Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 1051708 Used space (kbytes) : 27048 Available space (kbytes) : 1024660 ID : 1837247566 Device/File Name : /dev/rdsk/oracle/ocr/ln1/ora_ocr_001 Device/File integrity check succeeded Device/File Name : /dev/rdsk/oracle/ocr/ln1/ora_ocr_002 Device/File integrity check succeeded

Device/File not configured



Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

Up to FIVEOCR files



Post upgrade checks – query voting disk

# crsctl query css votedisk

## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. ONLINE 3ee7c55614f24fdcfff130e1abf94384 (/dev/rdsk/oracle/vote/ln1/ora_vote_001) [] 2. ONLINE a2c4e40a31f97f4bff6d65d558d03be8 (/dev/rdsk/oracle/vote/ln1/ora_vote_002) [] 3. ONLINE 1c9ed71914a54f9bbf55dd1c86048628 (/dev/rdsk/oracle/vote/ln1/ora_vote_003) []



Post upgrade checks – b/g and daemon processes#ps -ef |grep d.bin oracle 14458 1 0 15:14:12 ? 0:41 /u00/app/11.2.0/grid/bin/oraagent.bin oracle 14462 1 0 15:14:13 ? 0:00 /u00/app/11.2.0/grid/bin/mdnsd.bin root 14539 1 0 15:14:28 ? 0:24 /u00/app/11.2.0/grid/bin/cssdagent oracle 14549 1 0 15:14:29 ? 0:01 /u00/app/11.2.0/grid/bin/diskmon.bin -d -f root 14541 1 0 15:14:28 ? 4:17 /u00/app/11.2.0/grid/bin/orarootagent.bin root 14732 1 0 15:15:48 ? 7:35 /u00/app/11.2.0/grid/bin/crsd.bin reboot oracle 14469 1 0 15:14:15 ? 0:11 /u00/app/11.2.0/grid/bin/gpnpd.bin oracle 14739 1 0 15:15:49 ? 3:21 /u00/app/11.2.0/grid/bin/evmd.bin oracle 14535 1 0 15:14:25 ? 2:51 /u00/app/11.2.0/grid/bin/gipcd.bin oracle 14545 1 0 15:14:29 ? 4:42 /u00/app/11.2.0/grid/bin/ocssd.bin root 20675 1 0 15:36:03 ? 4:28 /u00/app/11.2.0/grid/bin/orarootagent.bin root 14595 1 0 15:14:50 ? 3:06 /u00/app/11.2.0/grid/bin/octssd.bin root 13999 1 0 15:13:20 ? 2:47 /u00/app/11.2.0/grid/bin/ohasd.bin reboot root 14533 1 0 15:14:24 ? 0:26 /u00/app/11.2.0/grid/bin/cssdmonitor oracle 20671 1 0 15:36:03 ? 0:08 /u00/app/11.2.0/grid/bin/appagent.bin oracle 20685 1 0 15:36:03 ? 0:26 /u00/app/11.2.0/grid/bin/oraagent.bin

#ps -ef |grep init.d

root 66 0 0 Feb 9 ? 0:00 pagetable_init_daemon oracle 8645 15234 0 19:31:02 pts/0 0:00 grep init.d root 13977 1 0 15:13:19 ? 0:00 /bin/sh /sbin/init.d/init.ohasd run



Remember

You can easily resume/downgrade from a failed or successful upgrade.

Can fall back to the pre 11gR2 version.

You hardly need a 3-step procedure to do that.

Remove OLD homes


Removing OLD cluster home

Although it’s a best practice to keep theOLD_HOME for a while after a successfulupgrade, you can use the following to remove the OLD_HOMES.

export ORACLE_HOME=$OLD_HOME

## detach OLD_HOME$OLD_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=$OLD_HOME

#confirm $OLD_HOME is removed from central inventory:$OLD_HOME/OPatch/opatch lsinventory -all

## remove files in OLD_HOME manually:/bin/rm -rf $OLD_HOME

unset ORACLE_HOME




11gR2 RDBMS software installation


cluvfy stage -pre dbinst -n node01,node02,node03 -d $ORACLE_HOME –verbose

Verify the READINESS of the nodes for DB software installation



































How to manage the pre 11gR2 databases under 11gR2 GIH?



root@node01: /u00/app/11.2.0/grid/bin # ./crsctl pin css -n node01 node02 node03

CRS-4664: Node usiogd01 successfully pinned.CRS-4664: Node usiogd02 successfully pinned.CRS-4664: Node usiogd03 successfully pinned.

PINNING NODES

You need to PIN the nodes in order to run your pre 11gR2 databases UNDERThe Grid Infrastructure umbrella.



o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post DB upgrade steps.o Upgrade aftermath stories.o References.

Database upgrade – Caution

Ensure the database backed up thoroughly BEFORE you kick off the db upgrade procedure.


Database upgrade – upgrade paths


UPGRADE

SQL> @catupgrade.sql

• RAC aware and most recommended for RAC dbs.• Also can be run in –silent mode.

Database upgradeFor a smooth rise, follow the best practice, run @OH_11G/rdbms/admin/utlu112i.sql********************************************************************** Tablespaces: [make adjustments in the current environment] ********************************************************************** --> SYSTEM tablespace is adequate for the upgrade. .... minimum required size: 788 MB --> UNDOTBS1 tablespace is adequate for the upgrade. .... minimum required size: 43 MB --> SYSAUX tablespace is adequate for the upgrade. .... minimum required size: 566 MB --> TEMP tablespace is adequate for the upgrade. .... minimum required size: 61 MB

PURGE DBA_RECYCLEBIN .... must be executed immediately prior to executing your upgrade. . ********************************************************************** Recommendations ********************************************************************** Oracle recommends gathering dictionary statistics prior to upgrading the database. To gather dictionary statistics execute the following command while connected as SYSDBA: EXECUTE dbms_stats.gather_dictionary_stats;


Database upgrade

As a best practice, run @OH_11G/rdbms/admin/utlu112i.sql********************************************************************** Components: [The following database components will be upgraded or installed] ********************************************************************** --> Oracle Catalog Views [upgrade] VALID --> Oracle Packages and Types [upgrade] VALID --> JServer JAVA Virtual Machine [upgrade] VALID --> Oracle XDK for Java [upgrade] VALID --> Real Application Clusters [upgrade] VALID --> Oracle Workspace Manager [upgrade] VALID --> OLAP Analytic Workspace [upgrade] VALID --> OLAP Catalog [upgrade] VALID --> EM Repository [upgrade] VALID --> Oracle Text [upgrade] VALID --> Oracle XML Database [upgrade] VALID --> Oracle Java Packages [upgrade] VALID --> Oracle interMedia [upgrade] VALID --> Spatial [upgrade] VALID --> Data Mining [upgrade] VALID --> Expression Filter [upgrade] VALID --> Rule Manager [upgrade] VALID --> Oracle OLAP API [upgrade] VALID


Database upgrade


Database upgrade


Database upgrade


Database upgrade


Database upgrade


Listener will be created

on all nodes

Database upgrade


Database upgrade


Database upgrade


Note 453903.1 - Enabling and Disabling Oracle Database Vault in UNIX Note 453902.1 - Enabling and Disabling Oracle Database Vault in WINDOWS $ cd $ORACLE_HOME/rdbms/lib$ make -f ins_rdbms.mk dv_off$ cd $ORACLE_HOME/bin$ relink all (For RAC installations, run these commands on all nodes.) #cd /u00/app/oracle/product/11.2.0/db_1/rdbms/lib # make -f ins_rdbms.mk dv_off /usr/ccs/bin/ar d /u00/app/oracle/product/11.2.0/db_1/rdbms/lib/libknlopt.a kzvidv.o /usr/ccs/bin/ar cr /u00/app/oracle/product/11.2.0/db_1/rdbms/lib/libknlopt.a /u00/app/oracle/product/11.2.0/db_1/rdbms/lib/kzvndv.o

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=453903.1

https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=453902.1

Database upgrade


Database upgrade


Database upgrade


Database upgrade


Database upgrade


Database upgrade


Database upgrade





Database post upgrade – list


Run @utlrp.sql to compile ANY invalid objects.

Run @utlu112s.sql to check post upgrade components status.

Adjust time zone data in the database.

Gather statistics for system and fixed objects.

Amend scripts [if any] to affect the new OH settings.

Verify the database upgrade srvctl config database –d <DBNAME>

Back up the database.

Modify the COMPATIBLE init. Parameter to 11gR2.

Database upgrade – measuring upgrade


Database upgrade – trivia


Upgraded over 100 databases successfully using MANUAL and DBUA methods.

Avg. database upgrade time was just about 2 hrs.

Upgrade using DBUA for RAC DB proved a wise decision.

Nearly identical duration between CLI and DBUA upgrade methods.

Duration is typically proportional to the number of installed database components.

Database upgrade – CLI upgrade steps






Upgrade aftermath stories


Snapshot Controlfile must be placed on a shared storage for a RAC database.

ORA-24247: network access denied by access control list (ACL)Solution: UTL_TCP, UTL_HTTP, UTL_SMTP, UTL_MAIL is used

grant the privileges explicitlyDBMS_NETWORK_ACL_ADMIN.CREATE_ACL

ADD_PRIVILEGE ASSIGN_ACL

ASM instance resource status MIGHT turn into INTERMEDIATE withCHECK TIMED OUT state_details [BUG 11807012]Solution: crsctl modify resource “ora.asm” –attr “CHECK_TIMEOUT=132”MOS [ID 1323679.1]

IMPDP fails with ORA-29913, 31011, 19202, LPX-00217[BUG 11877267]Solution: no solution… workaround exists

ORA-07445: exception core dump [evaopn3()]Solution: _simple_view_merge, _complex_view_merging=false


o Why upgrade?o Upgrade compatibility matrix.o Upgrade pattern [best practices].o Overview of our setup.o Prepare for upgrade.o Clusterware and ASM upgrade.o 11gR2 RDBMS software installation.o Database upgrade.o Post DB upgrade steps.o How to proceed from failed upgrade. o Upgrade aftermath stories.o References.


References


Oracle Database Upgrade Path Reference List [ID 730365.1] Complete Checklist for Manual Upgrades to 11gR2 [ID 837570.1] Complete checklist to upgrade the database to 11g R2 using DBUA [ID 870814.1] Master Note for Real Application Clusters (RAC) Oracle Clusterware and Oracle Grid Infrastructure [ID 1096952.1] How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure on Linux/Unix [ID 969254.1] Oracle Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1] Do You Need to Relink Oracle Clusterware When Upgrading the Operating System? [ID 743649.1]

Oracle 11g R1/R2 Real Application Clusters Essentials

Move to Oracle Database 11g – The whole Story blogs.oracle.com/UPGRADE http://download.oracle.com/docs/cd/E11882_01/server.112/e17222/toc.htm http://www.oracle.com/technetwork/database/upgrade/index.html http://oukc.oracle.com/static09/opn/core09/oracle9i_database/95960/060311_95960/index.htm

http://download.oracle.com/docs/cd/E11882_01/server.112/e17222/toc.htm

http://www.oracle.com/technetwork/database/upgrade/index.html

http://oukc.oracle.com/static09/opn/core09/oracle9i_database/95960/060311_95960/index.htm

A big thank you allfor

listening ...


You can write me at [email protected]

Documents

11gr2 Cluster Upgrade