Whitepaper: Running Oracle e-Business Suite Database on Oracle Database Appliance

COLLABORATE 13 OAUG Forum

Copyright ©2013 by Maris Elsins Page 1

Running E-Business Suite Database on Oracle Database Appliance Maris Elsins, Yury Velikanov The Pythian Group Inc. Oracle Database Appliance (ODA) is a pre-configured, simple setup, good performance engineered system running an 11gR2 cluster. It is a great choice for small to medium sized DBs, and if you wish it can be used for Oracle e-Business Suite too. This paper will show you how the standard configuration of ODA can be adjusted to comply with the specific requirements of e-Business Suite without sacrificing ODA’s flexibility and supportability. The paper will also share the authors’ experience migrating, running and maintaining R12.1.3 Oracle e-Business Suite (eBS) database tier on ODA.

Pythian completed a database consolidation project last year. Four databases of versions 10.2.0.3-10.2.0.5 were upgraded to 11.2.0.3 and migrated to a single ODA. The project was especially interesting because one of the databases was an e-Business Suite R12 database, and there was no information on My Oracle Support on how to cope with the specific e-Business Suite requirements and how to preserve the supportability and ease of maintenance of ODA's technology stack. This paper is a case study of how Pythian configured the eBS database on ODA v2.2.

Why choose ODA? ODA is the smallest of Oracle's engineered systems designed for running Oracle Database, but it still brings a huge set of benefits – simple deployment as it's pre-configured, reliable as it's pre-tested, highly available as all the components are redundant, and affordable with its 50,000$ (60,000$ for ODA X3-2) list price and pay-as-you-grow licensing model.

ODA contains two database-server nodes, each configured with two Intel Xeon X5675 6-core processors, giving total of 24 cores per ODA. Each node has 96 GB of memory and 500Gb of mirrored storage for the operating system and software installations. The shared storage consists of twenty 600Gb SAS drives that can be configured using high or normal redundancy, resulting in 4Tb or 6Tb of space for storing the data and backups, and an additional four 73Gb SSD drives triple-mirrored for online redo logs. The hardware specification of the ODA makes it attractive to many medium-sized databases, including ones supporting e-Business Suite.

A new Oracle Database Appliance X3-2 version was announced in March 2013. It provides the same benefits, but the hardware is different. Each database-server node is configured with two Intel Xeon E5-2690 8-core processors, so the total number of CPU cores is 32. Each node has 256 GB of memory and 600 GB of storage for operational system and software installations. The shared storage is built of twenty 900 GB 2.5” SAS-2 drives that provide 6Tb space if triple mirroring is configured and 9Tb with double mirroring. There are four 200 GB SSD drives included for 260 Gb of space for redo logs configured in ASM high redundancy disk group configuration.

Challenges Any engineered system will have limitations, and ODA is no exception. These limitations can be divided into two categories - limitations by design and supportability limitations.

Limitations by design are enforced by the provided hardware configuration. Depending on which ODA you choose (ODA or ODA X3-2), The computing capacity is limited to a single ODA and the configuration is non-expandable - it’s not possible to interconnect two ODAs to create a larger cluster. On the otherhand, even the old version of ODA, with its 96Gb of RAM and 24 CPU cores is more than enough for most small and medium eBS installations. The limited disk space of 4Tb (6Tb from v2.4) can partially be addressed by storing backups on NFS-mounted storage over a 10Gbps interface and in case of X3-2 you have an additional option of adding a Storage Expansion to double the available capacity.

Supportability limitations - There are strict rules on how to manage the ODA. Oracle only supports patching and upgrading by using the oakcli (OracleApplianceKitClient) utility. The patches are designed for a consistent and uniformly patched system, so anything that is installed additionally



using opatch can cause issues with patching in the future. There are certain situations when Oracle Support could allow installation of a one-off patch, but it's also noted that these one-offs could be required to be uninstalled before patching ODA in the future. The Oracle Global Inventory should be kept clean and unmodified. Additional products should not be installed on ODA, as oakcli relies on the information stored in the inventory and any unexpected entries can cause issues with patching.

The first publicly available versions of ODA software were quite limited, Oracle releases new software versions every 3 to 6 months and they add new features. For example: v2.1 supported only a single 11.2.0.2 oracle home, v2.2 gave us Oracle Database version 11.2.0.3, v2.3 introduced support for multiple oracle homes and allowed running multiple databases of different versions, v2.4 gave us a choice of normal redundancy for +DATA and +RECO diskgroups (the only option before that was high redundancy) and, finally, v2.5 brought Oracle Virtualization support to ODA. An organization needs to have a roadmap on how to plan and upgrade ODA to keep it on a supportable version.

There are also “supportability” challenges from the E-Business Suite side. The specific requirements of E-Business Suite R12 are listed in the following My Oracle Support notes: "Interoperability Notes EBS 12.0 and 12.1 with Database 11gR2 [ID 1058763.1]" and "Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 12 [ID 823587.1]" (similar notes exist for 11i - 823586.1 and 881505.1). These documents list a number of requirements that collide with the supportability limitations of ODA:

• Install Oracle Database 11g Products from the 11g Examples CD • Apply additional 11.2.0.3 RDBMS patches (4247037, 9858539, 12942119, 12960302,

12985184, 13001379, 13004894, 13258936, 13366268) • Create a dedicated oracle listener for eBS database • Customized TNS_ADMIN and ORA_NLS10 settings • Specific initialization parameters

Although the list is not very long, it’s clear the default Oracle Home configuration on ODA is not suitable for running the eBS database. The next section of the paper describes how Pythian customized the configuration of ODA to be able to run the e-Business Suite database and to minimize the negative effect it could leave on the future maintenance tasks.

Preparation the ODA for eBS database The E-Business Suite database configuration described in the following sections was implemented on ODA (v1) and not ODA X3-2. But, as X3-2 introduced changes mostly to the Hardware configuration, the approach should be valid for the new ODA X3-2 too.

A Dedicated Oracle Home was created for the e-Business Suite database by cloning the RDBMS Oracle home available on ODA in /u01/app/oracle/product/11.2.0.3/dbhome_1. The new Oracle home was created with path /u01/cst/{DB_NAME}/product/11.2.0.3 – we created a new directory (cst) under /u01 to store all custom Oracle homes and we used DB_NAME in the path as we planned to create multiple homes, and each of them could have its own patch requirements. The default Oracle home was left intact to be able to run other databases in an unmodified configuration.

Additionally it was decided to create a dedicated Oracle Inventory in /u01/cst/oraInventory for all customized installations to keep The default global inventory clean and avoid the possible negative effect on future patching of the ODA. The new Oracle home was registered in the new inventory. As we had two oracle inventories instead of one, we also introduced a need to maintain /etc/oraInst.loc file manually by adding the correct inventory location depending on the type of maintenance work being done – it should point to the default inventory (/u01/app/oraInventory) for normal operations and it should point to the custom inventory for the maintenance activities affecting the eBS database Oracle home.

By creating a configuration that was completely independent from the default ODA setup, we could install the requirements of e-Business Suite database in the freshly created Oracle home without affecting the seeded configuration. The Examples CD for 11.2.0.3 patch set was downloaded from My Oracle Support (patch 10404530) and installed into the new Oracle home along with all the required one-off patches.



Please note the default grid infrastructure Oracle home was used by all the databases running on ODA, since eBS didn’t have any specific grid infrastructure patching requirements.

From a certification point of view, ODA is an Oracle Linux 5.6+ server cluster running Oracle grid infrastructure and database versions 11.2.0.2+. Both OS and database versions are certified for running Oracle e-Business Suite R12.1. It’s also important to note that despite the fact a new Oracle home was installed manually (a customization of the Oracle Appliance Kit product) all components – oracle grid infrastructure, oracle database, e-Business Suite, and even the oracle home we cloned – are still set up in a fully certified and supported configuration.

An important preparation step for going live with ODA is implementation of a monitoring framework for timely notification about issues. The best way for monitoring the ODA hardware is by configuring the Auto Service Request (ASR) support feature that is included in the ODA software bundle. ASR automatically creates Service Requests, provides diagnostic information to Oracle Support, and notifies administrators about hardware faults. ODA also supports installation of 3rd party agents to manage, monitor, backup, replicate, authenticate, or otherwise act on the database, the server, or the environment (note ID 1415773.1), giving the opportunity to handle the ODA the same way as any other server in your IT infrastructure.

It is worth mentioning that one day before our go live activities, one of the HDDs failed, but as there was triple-redundancy it had a little impact on the project. After a short rebalancing period, ODA recovered the protection level with one less HDD. A few weeks later a new HDD was added seamlessly to the running production system.

Database migration and configuration As part of the migration project to ODA we also had to upgrade the database from 10.2.0.5 to 11.2.0.3. Since the old HW was not very powerful we wanted to utilize the performance of the ODA to speed up the upgrade process. This requirement revealed another problem – how to migrate 10gR2 database to 11gR2 on ODA? The following options were considered:

• Upgrade the DB on the old hardware and then move to ODA – the option was rejected as it required too much downtime.

• Install 10gR2 Oracle home and create a physical standby on ODA, then switchover and do the upgrade on ODA – this approach would allow us to meet the downtime requirements but the problem was the ASM diskgroups on ODA were shipped with database compatibility setting of 11.2.0.2, and therefore the 10gR2 RDBMS software couldn’t use them. A workaround of recreating the ASM diskgroups with compatibility setting of 10.2 was considered, but was found too intrusive and too risky as it may break ODA supportability.

The chosen upgrade method was based on the fact that RMAN is able to restore backups taken on an older version of the database into a newer release (MOS ID 369644.1). The backup of the 10.2.0.5 database was restored using 11.2.0.3 software into ASM disk groups with database compatibility setting of 11.2.0.2. After completing the restore, the archived log apply process was scripted to keep the target database synchronized with the source. Remaining redo was applied and “alter database open resetlogs upgrade” was executed to put the database into the upgrade mode at the switchover, after which we could continue the normal upgrade process on the ODA. The pre-upgrade procedures still had to be performed on the source (10gR2) database.

Oracle e-Business Suite has requirements for specific initialization parameters listed in My Oracle Support note “Database Initialization Parameters for Oracle E-Business Suite Release 12 [ID 396009.1]”, but in addition to these it was found that a vanilla database created by “oakcli create database” command had the following non-default initialization parameters:

• _disable_interface_checking=TRUE • _ENABLE_NUMA_SUPPORT=FALSE • _FILE_SIZE_INCREASE_INCREMENT=2143289344 • _gc_policy_time=0 • _gc_undo_affinity=FALSE • _KGL_CLUSTER_LOCK_READ_MOSTLY=TRUE • _kill_diagnostics_timeout=140 • _lm_rcvr_hang_allow_time=140



• db_block_checking='FULL' • db_block_checksum='FULL' • db_lost_write_protect='TYPICAL' • filesystemio_options='setall' • parallel_adaptive_multi_user=FALSE • parallel_execution_message_size=16384 • parallel_min_servers=0 • parallel_threads_per_cpu=2 • use_large_pages='ONLY' • compatible=11.2.0.3

As the parameters had obviously been fine tuned for ODA we set them in our eBS database too.

Clusterware resources for the eBS database and the listener were created to ease the operational tasks of the database. A dedicated listener was created for the e-Business Suite database using port pool 1; it’s important to choose other port pool then 0 to avoid port conflicts with the default listener running on port 1521. The following commands were used to create the listener:

• srvctl add listener -l LISTENER_EBSDB -o $ORACLE_HOME -p 1522 • srvctl setenv listener -l LISTENER_EBSDB -T

TNS_ADMIN=$ORACLE_HOME/network/admin

The important bit for creating the database resource for an eBS database is to set the ORA_NLS10 environment parameter as required in note 1058763.1:

• srvctl add database -d EBSDB -o /u01/cst/EBSDB/product/11.2.0.3 -p +DATA/EBSDB/PARAMETERFILE/spfile.704.792145311 -a "DATA,RECO,REDO" -n EBSDB

• srvctl setenv database -d EBSDB -t "TNS_ADMIN=$ORACLE_HOME/network/admin,ORA_NLS10=$ORACLE_HOME/nls/data/9idata"

• srvctl add instance -d EBSDB -i EBSDB1 -n oda01a-net1 • srvctl add instance -d EBSDB -i EBSDB2 -n oda01a-net1

It’s also critical to note the TNS_ADMIN variable for cluster resources points to the default location in $ORACLE_HOME/network/admin, but, as AutoConfig creates node specific directories, they have to be made visible from the default path by adding the following IFILE settings on both database servers:

• echo "IFILE=${ORACLE_HOME}/network/admin/${CONTEXT_NAME}/tnsnames.ora" > $ORACLE_HOME/network/admin/tnsnames.ora

• echo "IFILE=${ORACLE_HOME}/network/admin/${CONTEXT_NAME}/listener.ora" > $ORACLE_HOME/network/admin/listener.ora

• echo "IFILE=${ORACLE_HOME}/network/admin/${CONTEXT_NAME}/sqlnet.ora" > $ORACLE_HOME/network/admin/sqlnet.ora

Impact on Maintenance of ODA and eBS database The implemented configuration of ODA v2.2 with the eBS database running in a separate oracle home has been very stable for almost half a year – the pre-configured hardware and Grid configuration seems to be working. The first installation of an ODA Patch Bundle is still in front of us. As a dedicated Oracle home and inventory have been created the impact on ODA’s patching should be minimal. There are, however, a few configuration points that are visible to oakcli utility and could affect the way it works:

• /etc/oraInst.loc – the inventory_loc setting should point to the default or the custom oracle inventory depending on the type of the maintenance being performed.

• /etc/oratab – the file lists all the databases running on the server, including the eBS database running from the customized oracle home. One should avoid running “oakcli update -patch 2.x.x.x.x” to update all components as it upgrades all databases on the ODA by default, instead oakcli options --infra, --gi and --database should be used to update the software followed by “--database <db_name>“ for each database running from the default oracle home. The custom Oracle home will have to be patched manually or it will need to be re-cloned from the upgraded default oracle home of ODA.



• Clusterware resources – the cluster resources are globally visible, so if the customized resources cause trouble they should be temporarily removed.

• Data on ASM diskgroups – it’s unlikely any of ODA Patches would be built in a way that could harm data on the ASM diskgroups, but precautions should be taken by taking a backup of the database and storing it outside the ODA.

On the other side, the ODA platform brings some restrictions for maintaining the eBS database as well:

• Patching – as a completely separate Oracle home is used there are no restrictions to apply one-off patches, however Patchsets and Patch Set Updates (PSUs) can’t be installed if the grid Infrastructure is not upgraded because of the dependencies between RDBMS and grid Oracle homes. The grid home should be upgraded only by applying ODA bundle patches.

• Security Patch Updates – based on My Oracle Support note “Patch Set Updates for Oracle Products [ID 854428.1]”, the security patches are developed only for the base release, and once a PSU has been installed, the recommended way to get future security content is to apply subsequent PSUs. The Oracle Home that was cloned for running e-Business Suite database was 11.2.0.3.2 (with a PSU applied), therefore the only way to get the security fixes is to install the latest ODA Patch bundles.

Future Perspective The future possibilities of running e-Business Suite databases on ODA depend on what features get included in the coming ODA Patch Bundles. Already we see the trend of transforming the Oracle Database Appliance into an “Oracle Database and Application Appliance” – the Patch Bundle v2.5 brings virtualization to ODA. Oracle e-Business Suite is already certified with Oracle VM 3.1.1 (see note ID 465915.1), giving us a great opportunity to place the whole e-Business Suite implementation on a single ODA box.

For the implementation described in this article (running on ODA bundle patch v2.2), an upgrade to an ODA software version supporting multiple Oracle homes should be considered. It would allow running the e-Business Suite database from an ODA-supported Oracle home by installing additional one-off patches (they would probably need to be uninstalled before ODA patching), giving us a cleaner and more easily understandable configuration with a single Oracle inventory.

Conclusions ODA is great HW for running midsized databases, but official support for the e-Business Suite database tier on ODA is not yet available. Configuring eBS on ODA requires balancing between supportability of the ODA’s configuration and implementation of all its specific requirements. Our solution was to create a dedicated Oracle home and inventory for e-Business Suite database to allow installation of additional one off patches and products from the Examples CD. The impact the chosen configuration leaves on the further maintenance of ODA and e-Business suite is low and does not reduce the benefits ODA brings to the operation of e-Business Suite. It is also expected Oracle will continue to develop the capabilities of ODA and one day perhaps they will provide complete support for e-Business Suite too.

Documents

Whitepaper: Running Oracle e-Business Suite Database on Oracle Database Appliance