Oracle 11.2.0.3 Upgrade - Draft Studios - A Splash of Color · Web viewRun the Oracle 11g Release 2 Pre-Upgrade Tool Fix Warnings Found from Running the Pre-Upgrade Tool Installation

WeSaySo! Corporation

Oracle 11.2.0.2 to 11.2.0.3 Upgrade

Group 4Minh Vo, Susan Champigny, Ganapathi S. Santhana

9/22/2012

1 University AvenueLowell, MA 01854

Phone: 978 934-4000

WeSaySo! Corporation 1

IntroductionTo ensure a successful upgrade of our Production Database Server running on Oracle 11.2.0.2 to Oracle 11.2.0.3, we have followed best practices recommended by Oracle Corporation. The upgrade will be installed out-of-place using the DBUA (Database Upgrade Assistant) method with a fallback to Manual method only if required. DBUA was chosen to help keep user typos and errors to a minimum.

To know what to expect of the project, we replicated the server and performed three separate test upgrades to solidify our upgrade plan. Each of our group members took part in one of these test environments until success was achieved. Individually we ran into various issues, which have been documented in the Appendix section. Google and Oracle Support documents helped the most at this stage.

When we felt comfortable with the entire upgrade process, we proceeded to upgrade the Production environment. This report attempts to document our final upgrade process on the Production machine.


Plan of Action PREPARATION

o Check for Oracle Environment Certification – Fedora 12 (Constantine) i686

o Download the Latest (11.2.0.3) Software and Critical Patch Updates (July 2012) from Oracle Support

o Test and Verify the Environment

PRE-UPGRADE

o Check for Hidden Parameters

o Run the Oracle 11g Release 2 Pre-Upgrade Tool

o Fix Warnings Found from Running the Pre-Upgrade Tool

INSTALLATION

o Install the Oracle 11.2.0.3 Software

UPGRADE

o Perform the Upgrade Using Database Upgrade Assistant (DBUA)

PATCH

o Patch to the Latest Critical Patch Update (July 2012) - 14038787

POST-UPGRADE

o Re-Verify and Spot-Check the Upgrade

o Conclusion

APPENDIX

o Adding a 20GB Hard Drive to the Production Server (VM Environment)

o Resolving Display Issues When Attempting to Start OUI from Non-X Terminal

o Oracle Enterprise Manager Error: Complaining About Missing Password File

o OPatch Failed With Error Code 73

o


PreparationCHECKING FOR ORACLE ENVIRONMENT CERTIFICATION

We begin our preparation stage by checking certification of Oracle 11.2.0.3 on our Production environment Fedora Linux Release 12 (Constantine). Visiting https://support.oracle.com and signing in takes us to the My Oracle Support portal. Although Fedora Linux Release 12 is not on the list of Oracle Certified Operating Systems, many end users have recorded successful installations online:

http://www.oracle-base.com/articles/11g/oracle-db-11gr2-installation-on-fedora-12.php http://forums.fedoraforum.org/showthread.php?t=210324

http://forums.fedoraforum.org/showthread.php?t=210324

http://www.oracle-base.com/articles/11g/oracle-db-11gr2-installation-on-fedora-12.php

https://support.oracle.com/


DOWNLOAD AND EXTRACT THE LATEST SOFTWARE FROM ORACLE SUPPORT (TO /HOME/ORACLE/DBAIII/)


TEST AND VERIFY THE ENVIRONMENT

We begin our task by scoping out our environment. Snooping around, we find that our database SID is “ora11” installed in /u01/app/oracle/product/11.2.0/db_1. Looking through /etc/oratab showed that there are three other databases that are available.

Before digging deeper into the database, we decided to assess the Linux environment a little more, running $ df –kh to see how much disk space we’re dealing with.


A typical out-of-place Oracle install should take around 5GB in software alone! Looks like we’re going to need more disk space. We decided to tackle this right away, so we added a 20GB disk to the server (see Appendix). Once the new hard drive was available to the Production machine, we logged in as root and verified.


$ fdisk –l showed that /dev/sdb is indeed available with 21.5GB unpartitioned space.

The plan now is to ultimately extend the Logical Volume (/dev/vg_fc12ora112/lv_root) so that / mount can have more space to sustain the upgrade. To do this, we first needed to create a Physical Partition for /dev/sdb, then extend the Volume Group (vg_fc12ora112), and finally extend the Logical Volume.

We created a new Physical Volume from /dev/sdb using pvcreate and displayed it using pvdisplay.

Then we identified and extended the Volume Group with vgdisplay and vgextend.


lvdisplay gives us the Logical Volume we ultimately want to extend: dev/vg_fc12ora112/lv_root

Now we’re going to extend the Logical Volume by 20GB with lvextend -L+20G /dev/vg_fc12ora112/lv_root


We were quickly alerted with the “Insufficient free space” error. This is expected since Volume Management naturally takes a portion of the 20GB drive, leaving barely less than 20GB for us to extend our Volume Group. No matter. We were able to extend the Logical Volume by the maximum number of extents available—which was 20,622—resulting in a total 37.83GB

$ lvextend -L+20622 /dev/vg_fc12ora112/lv_root

Finally, we ran resize2fs to resize the File System and verified its success with $ df -km. As expected, /dev/mapper/vg_fc12ora112-lv_root mounted on / now has over 22GB available:

At this point, Disk space should no longer be a problem. Normally we’d do an RMAN backup of the existing database at this point, but through our test runs we found having DBUA take care of backups is more convenient.

Concluding our information gathering, we started the database and ran a few test queries just to confirm database functionality and version numbers.


Pre-UpgradeFollowing the Best Practices for Upgrading to Oracle Database 11g Release 2 – April 2012 – White Paper, there are many precautionary tasks we can complete to make the Upgrade as painless as possible.

CHECKING FOR HIDDEN PARAMETERS

It is advisable to take care of hidden parameters that may be in the source database. If any exist, find and remove them. Running the query below returned no results, so this doesn’t apply to our Production database.

SELECT name, description from SYS.V$PARAMETER WHERE name LIKE ‘\_%’ ESCAPE ‘\’

RUN THE ORACLE 11G RELEASE 2 PRE-UPGRADE TOOL

Running the utlu112i_5.sql script from support.oracle.com automatically runs a battalion of tests for invalid objects, duplicates, non-valid components, mis-configurations, space on tablespaces, obsolete/deprecated parameters, timezone files, and other warnings and cautions.

We downloaded the script to /tmp/utlu112i_5.sql, changed directories to /tmp, logged into database / as sysdba and declared spool upgrade_11.2.0.3.log to spool output to /tmp/upgrade_11.2.0.3.log. Executing the script yielded something like the following:


Notable Warnings from the outputted /tmp/upgrade_11.2.0.3.log



FIX WARNINGS FOUND FROM RUNNING THE PRE-UPGRADE TOOL

Let’s tackle the shared_pool_size and java_pool_size problems first:


Now let’s fix the Miscellaneous warnings regarding INVALID OBJECTS. Find the INVALID objects using these three queries:

select substr(comp_name,1,40) comp_name, status, substr(version,1,10) version from dba_registry order by comp_name;

select substr(object_name,1,40) object_name, substr(owner,1,15) owner, object_type from dba_objects where status='INVALID' order by owner, object_type;

select owner, object_type, count(*) from dba_objects where status='INVALID' group by owner, object_type;

From running the query we can see that there’s one PROCEDURE owned by SYS that is INVALID. To fix this, we should try to VALIDATE them using $ORACLE_HOME/rdbms/admin/utlrp.sql


This didn’t seem to do anything as OBJECTS WITH ERRORS is said to be 0, and ERRORS DURING RECOMPILATION is also returning 0. At least one of them should be 1 since we know of an INVALID object: SYS.MANAGE_SAL(). Running the three queries above confirmed our suspicions. The INVALID object was still there. We then tried to compile the procedure manually.

Since this seemed more of a PL/SQL programming error, rather than debugging the procedure, we decided to IGNORE this warning. We’ll perhaps revisit it after the upgrade. The INVALID object should appear post-upgrade when we rerun utlu112i_5.sql on 11.2.0.3.


To handle the Network ACL warning for user APEX_030200 dependent on DBMS_LDAP package, we ran the following block, as documented by Configuring Fine-Grained Access to External Network Services After Upgrading Oracle Database (http://docs.oracle.com/cd/E11882_01/server.112/e17222/afterup.htm#CEGDGAHJ)

DECLARE acl_path VARCHAR2(4000);BEGIN SELECT acl INTO acl_path FROM dba_network_acls WHERE host = 'fc12ora112.uml.edu' AND lower_port IS NULL AND upper_port IS NULL;IF DBMS_NETWORK_ACL_ADMIN.CHECK_PRIVILEGE (acl_path, 'APEX_030200','connect') IS NULL THEN DBMS_NETWORK_ACL_ADMIN.ADD_PRIVILEGE (acl_path, 'APEX_030200', TRUE, 'connect');END IF;EXCEPTION WHEN no_data_found THEN DBMS_NETWORK_ACL_ADMIN.CREATE_ACL ('ACL_name.xml', 'ACL description', 'APEX_030200', TRUE, 'connect'); DBMS_NETWORK_ACL_ADMIN.ASSIGN_ACL ('ACL_name.xml','fc12ora112.uml.edu');END;COMMIT;

We didn’t run into any TIMESTAMP WITH TIMEZONE data type warnings because 11.2.0.2 and 11.2.0.3 both use the same V14 time zone file version.

To handle the RECYCLE BIN warning, we ran

SQL> PURGE DBA_RECYCLEBIN;

In an attempt to decrease the amount of downtime when collecting statistics, we’ll perform the gathering before upgrading.

SQL> EXEC DBMS_STATS.GATHER_DICTIONARY_STATS;

According to the 11gr2_DBUA_Upgrade_Checklist, DBUA will return an error if Oracle Database Vault is ENABLED. So, we’ll check whether or not it’s running in the first place. It is not. No need to worry.

http://docs.oracle.com/cd/E11882_01/server.112/e17222/afterup.htm#CEGDGAHJ


InstallationEverything looks pretty good at this point. We’ve resolved the space issues by extending the logical volume, checked for hidden parameters, ran the pre-upgrade script and resolved many of its warnings. We’ve also run statistics gathering to hopefully speed up the upgrade process.

We’ve downloaded the Oracle 11.2.0.3 software and the expanded files are currently residing at /home/oracle/DBAIII/11203_source

Our next logical step is to install the Oracle 11.2.0.3 software to a new Oracle Home. Our source database is located at/u01/app/oracle/product/11.2.0/db_1 so we’ll follow that convention and create a new Oracle Home for the target at /u01/app/oracle/product/11.2.0/db_2

INSTALLING THE ORACLE 11.2.0.3 SOFTWARE

The Oracle Universal Installer is GUI based, so starting it from a terminal outside of X was difficult. During our testing we were able to get it to work by starting X Windows/Server with startx or xinit and running the command xhost + which allows clients to connect to X from any host. The client would then have to set an external display via DISPLAY=192.168.0.97:0.0; export DISPLAY and then run the installer (see Appendix).

The most graceful way, though, was to simply run the Oracle Universal Installer from an X session on the server itself. To do this, within the X desktop, start a terminal, and run /home/oracle/DBAIII/11203_source/database/runInstaller

The following screenshots cover our settings and general flow through the Installer:



Step 9 presented us with more warnings that weren’t found in any of our pre-upgrade scripts. Clicking “Fix & Check Again” automatically generated a fixup script for us to run.

Opening another terminal window and running the fixup script managed to fix the OS Kernel Parameters (shmmax) warning. The pdksh package warning, however, can safely be ignored because we have a version of ksh already installed in /usr/local/bin/ksh

Returning to the Oracle Universal Installer, we clicked the “Ignore All” checkbox to ignore the pdksh package warning and proceeded with the Installation.



The strange thing here is that although root.sh stated that ORACLE_HOME has now been set to /u01/app/oracle/product/11.2.0/db_2, it hasn’t seemed to reflect. We’ll manually set it up by editing /home/oracle/.bash_profile and changing the ORACLE_HOME.

Save the file using :w and quit vi (:q). Log out and start a new oracle/oracle session. Then check to see if ORACLE_HOME is reflecting correctly. It is now.

We’re ready to begin the Upgrade portion.


UpgradeNow that the Oracle 11.2.0.3 software has been installed, it is time to perform the upgrade using DBUA.

PERFORM THE UPGRADE USING DATABASE UPGRADE ASSISTANT (DBUA)

To initiate the upgrade, we’ll start by locating and running dbua from the target ORACLE_HOME.

The Database Upgrade Assistant utility should pop up in ten to twenty seconds. Click “Next” to proceed and select ‘ora11’ database as the database we want to upgrade.


Warnings similar to the ones we saw when running the pre-upgrade scripts will show up, but this was expected. Opt to continue with the upgrade.

Next, you’ll see the option to “Backup database”. Select this option. Note: During our test runs, one of the first things we did was create an RMAN backup of the database. Seeing as DBUA has backup functionality built in, we chose to use it instead for the Production upgrade. (See Appendix)

Clicking “Next” brings up a prompt asking if you want DBUA to start a listener and continue with the database upgrade. Choose “Continue”.


In Step 2, be sure to select “Do Not Move Database Files as Part of Upgrade”.


Note: We had an error that was ignored relating to the Enterprise Manager configuration: Password file may be missing or configured incorrectly. Refer to the log file at /u01/app/oracle/cfgtoollogs/dbua/ora11/upgrade1/emConfig.log. We’ll resolve this later by manually running /u01/app/oracle/product/11.2.0/db_2/bin/emca script (SOLVED. See Appendix).


When upgrade is complete, click “OK” to review the Database Upgrade Summary which can also be accessed at /u01/app/oracle/cfgtoollogs/dbua/ora11/upgrade1/UpgradeResults.html.


PatchPATCHING TO THE LATEST CRITICAL PATCH UPDATE

Now we need to apply the latest CPU that we downloaded in our first phase (14038787 - jul12_security_cpu). An extracted copy is located at /home/oracle/DBAIII/jul12_security_cpu/14038787.

Shutdown the database and stop listeners (if any are running). Change to /home/oracle/DBAIII/jul12_security_cpu/14038787, check that the correct opatch is being used ($ which opatch), and run:

$ opatch napply -skip_subset -skip_duplicate


Once all patches are applied, startup the database, start listeners and verify that the correct database version is in place.


Post-UpgradeRE-VERIFY AND SPOT-CHECK THE UPGRADE

Now that we’ve upgraded the Production database and performed the necessary security updates, re-verify that Oracle environment variables are set to point to our 11.2.0.3 installation.

Looking pretty good. Run a few queries or the utlu112i_5.sql (and perhaps also dbupgdiag.sql) script to determine success of upgrade. Notice that the warning for one INVALID OBJECT (SYS. MANAGE_SAL) is still there.


Note: SKIP time zone patch, as both 11.2.0.2 and 11.2.0.3 releases are on Version 14

Note: SKIP creating an editable init.ora from current SPFILE because DBUA took care of this for us.

CONCLUSION

At this point, our Production database has been upgraded to 11.2.0.3!

To take full advantage of this release, though, we have the option of changing the COMPATIBLE parameter—which is still left at 11.2.0.0. We’ve left this as-is since it was not changed even when we were running 11.2.0.2. Generating STATS can be run again if needed. Comparing the results of the before/after utlu112i_5.sql/dbupgdiag.sql looks almost identical.


AppendixADDING A 20GB HARD DRIVE TO THE PRODUCTION SERVER (VM ENVIRONMENT).

Shut down the Oracle Instance, if running, then shutdown the entire VM.

Now, click the Virtual Machine menu, select “Hard Disk (SCSI)” and “Hard Disk (SCSI) Settings…”.


Click “Add Device”.

Select “New Hard Disk”, under “File name:” choose “Save As…”. Proceed to name the disk file. We chose fc12ora112_2.vmdk. Set the disk size to 20.00GB (can give it a little more space to cover for volume management/partitioning stuff. Finally, click “Apply”.

Startup the VM (fc12ora112_students) and the drive should be available to the system. Log in as root and run $ fdisk -lIt shows a new hard disk /dev/sdb at 21.5 GB.


RESOLVING DISPLAY ISSUES WHEN ATTEMPTING TO START ORACLE UNIVERSAL INSTALLER FROM A TERMINAL OUTSIDE OF X (WINDOWS).

A few of us ran into issues when running /home/oracle/DBAIII/11203_source/database/runInstaller

A different solution than the used in our main report is to start X by running $ X or $ startx or $xinit on the Production server and then run $ xhost +


On the client terminal, set and export the DISPLAY environmental variable to the production server IP colon “0.0”, like below.

Running xdpyinfo checks to see if it can open the remote display and gather specs. Once xdpyinfo is successful, running ./runInstaller (Oracle Universal Installer) will also succeed and shoot the GUI to the Production display.

ORACLE ENTERPRISE MANAGER ERROR COMPLAINING ABOUT MISSING PASSWORD FILE

This one really stumped me [Minh]. I’m not sure if anyone else in my group experienced this problem, but even after resolving it, I’m not sure what particularly caused the issue (due to my super-limited understanding of how emctl, dbca, and emca interact). Looking into the emConfig.log specified, there was an entry stating “SEVERE: Password file may be missing or configured incorrectly.” (See image on next page)


We then tried to create a new password file with the password “mypass” and ran emca to configure and recreate db/repos$ orapwd file=$ORACLE_HOME/dbs/orapw$ORACLE_SID password=mypass entries=5


Note: we changed the passwords for SYS, DBSNMP, and SYSMAN because we didn’t know them… and the emca tool was specifically asking for them. (alter user <SYS|DBSNMP|SYSMAN> identified by password)

Things seemed to be okay until a new warning appeared: ORA-01017: invalid username/password; logon denied. Reading the possible reasons listed, we checked that 1521 is indeed the correct port (and listener is running—can tnsping returning OK 10ms).

$ ps –ef | grep tnslsnr also showed oracle 9564 1 0 20:33 ? 00:00:00 /u01/app/oracle/product/11.2.0/db_2/bin/tnslsnr LISTENER -inherit

We didn’t understand 3) Database service ora11.fc12ora112.uml.edu is not registered with listener. Register with database service.

Checked /etc/hosts which seemed to be set correctly.

Could it be due to an old listener from db_1 that is not configured/working correctly? We thought it was a good idea to create a new listener using netca.




After the new listener was created, we restarted lsnrctl (lsnrctl start) and tried emca again with no success. Someone on Oracle forums suggested it might be because the password file SID portion is lowercase, ie: orapwora11 instead of orapwORA11. Unfortunately, changing the password file filename also didn’t seem to work.

Back to square one. Did some more digging on Google and found another way of attack: Database Configuration Assistant (DBCA).

$ ORACLE_UNQNAME = ora11; export ORACLE_UNQNAME;$ dbca



Some progress, at last. But there are still some errors. We’ll go through them one-by-one. ORACLE_UNQNAME is all set. Now we can stop dbconsole.

Trying to run $ /u01/app/oracle/product/11.2.0/db_2/bin/emctl config emkey -repos -sysman_pwd sysmanResulted in “The Em Key could not be configured. The Em Key was not found in Management Repository.”


Around the same time we were getting this error, we continually tried to run emca. One such run gave us the response that led us closest to success, and further down the troubleshooting ladder:

Here we copied emkey.ora from the old (source 11.2.0.2 database) to new.

Then ran /u01/app/oracle/product/11.2.0/db_2/bin/emctl config emkey -emkeyfile /u01/app/oracle/product/11.2.0/db_2/fc12ora112.uml.edu_ora11/sysman/config/emkey.ora resulting in “The Em Key has been configured successfully.” Wonderful. Proceeded to copy the Em Key to repository which apparently causes the Em Key to become unsecure.


Things are looking good. We then tried to run emca again, but still no luck. This time complaining about

Sep 22, 2012 2:45:27 AM oracle.sysman.emcp.EMReposConfig invokeSEVERE: Failed to unlock all EM-related accountsSep 22, 2012 2:45:27 AM oracle.sysman.emcp.EMConfig performSEVERE: Failed to unlock all EM-related accounts

On a whim, we ran the Database Configuration Assistant (DBCA) again, and much to our surprise (and relief) it succeeded!!


Verify by stopping and starting dbconsole again.

$ emctl stop dbconsole$ emctl start dbconsole

Perfect. We’re done!! But before we forget about the ORACLE_UNQNAME, let’s set the variable in oracle user’s .bash_profile:


OPATCH FAILED WITH ERROR CODE 73 WHEN TRYING TO RUN OPATCH.

Resolving this was pretty simple. Make sure listeners are all stopped before running opatch.$ lsnrctl stop$ opatch napply -skip_subset -skip_duplicate

Documents

Oracle 11.2.0.3 Upgrade - Draft Studios - A Splash of Color · Web viewRun the Oracle 11g Release 2 Pre-Upgrade Tool Fix Warnings Found from Running the Pre-Upgrade Tool Installation