Discussion SAP on Exadata 20140618

<Insert Picture Here>

SAP on Oracle Exadata

RAC

RAC

What is a Exadata for SAP?

• GRID Cluster of database nodes

• Cluster Manager CRS

• „Volume Manager“ ASM

• Operating RAC Database

• Storage in Exadata Storage Cells

• Storage cell software on Cells

• Optionally hosting SAP instances

• Controlled by CRS

• sapctl

Database Database Node Node

RAC RAC Instance Instance

SAP SAP

GRID GRID

ASM ASM

CRS CRS

Storage Cell Storage Cell

Storage Cell Storage Cell

Exadata for SAP facts 1

•

•

•

GRID cluster, RAC databases

Database files on ASM only

no shared file system offered by Exadata

(DBFS is not supported in SAP environment)

•Local Oracle Homes for GRID and for RDBMS

(exception standard SAP RAC setup)

• all Oracle software installed on Exadata using Exadata standards

• all Oracle software installed under OS environment ‘oracle’

• ASM using standard Exadata setup (different to SAP ASM setups)

• ASM in cluster mode


• SAP instances cannot run on Exadata (Exceptions: ASCS, SCS,ERS)

• SAP central services on the database nodes require an SAP Unicode Kernel

•

•

BR*Tools can be installed on the database nodes

If non-SAP and SAP databases are operated on the same

Exadata, refer to SAP OSS note 1677978 "Administration of mixed SAP/Non-SAP Environments on Exadata“

•OSS note 1634767 “Support for SAP software in Linux / Oracle

ASM/ACFS Cluster“.


• OSS note 1590515 "SAP Software and Oracle Exadata“

• Exadata models: X2-2, X2-8, X3-2, X3-8, X4-2 with Linux 5

• RDBMS 11.2.0.2 Patch 1 (OSS 1591389) or 11.2.0.3 Patch 2 (OSS 1656654)

•

•

•

•

•

•

Exadata SW version 11.2.2.3.2

No Hybrid Columnar Compression (HCC)

No SAP instance (except ASCS, SCS, ERS)

Unicode kernels only

SAP Kernel on Exadata 7.20 64-Bit, don’t use EXT kernel

Brtools supported (OSS 1627541)


• fast recovery area (FRA) may be used for flashback logs

(flashback database). No support by brtools for backups!!

Exadata for SAP prereqs 1

• Again important:

• SAP OSS note 1590515 "SAP Software and Oracle Exadata",

• SAP OSS note 1431798

"Oracle 11.2.0: Database Parameter Settings“

• All Oracle software installed on Exadata using Exadata standards, SAP general standards don’t apply on this platform

• All Oracle software installed under OS environment ‘oracle’ SAP Kernel CD (minimum 7.00 EP 3 or 7.20 EP 1 and higher)

Central services on Exadata: kernel 7.20 64bit unicode non EXT

BR*Tools software (minimum 7.20 patch 18 and higher)

Exadata for SAP prereqs 2

•

•

•

•

log_buffer not less than 128MB

Disk_asynch_io = true

Filesystemio_options = setall (even if on ASM)

ASM diskgroups +DATA (high) +RECO (normal)


Oracle Grid with RAC 11g Release 2 for SAP

Active-Active Cluster

RAC: The Cluster Database

Network

Users Centralized Management

Console

Interconnect No Single Point Of Failure High Speed

Switch or

Interconnect

Clustered Database Servers

Hub or Switch

Fabric Storage Area Network

Drive and Exploit Industry Advances in

Clustering

Mirrored Disk Subsystem

Oracle RAC Architecture

public network

VIP1 VIPn

Node n Node1

Oracle Clusterware interconnect

shared storage

Redo / Archive logs all instances

Database / Control files Managed by ASM

OCR and Voting Disks

Oracle Clusterwa

Services

cluster

Services

Listener Listener

instance 1

instance n

ASM ASM

Oracle Clusterware

Oracle Clusterwa

Operating Sys Operating Sys

What does RAC mean?

Instance

Instance

Instance

Oracle

Oracle

Oracle

SAP Instance

SAP Instance

SAP Instance

Building a HA Solution with Oracle

• •

RAC only protects the Oracle Instance

For a complete HA solution we need to protect also

• •

•

•

•

•

•

•

•

•

HW: Multiple Servers Network: Bonding

Physical Database: ASM, Mirroring, DG

SAP Enqueue server: sapctl

Application Level : DG

Rolling Upgrades and Patches

Maintenance Procedure: Online Reorg. etc

BRR: RMAN, Flashback

Migration Procedures: O2O/OOO

Planned/Unplanned Downtimes

HA

•

•

•

•

•

Oracle – RAC

SAP – Enqueue Replication

Disks – Mirroring / Standby Database

Network – Bonding, redundant networks

Logical- / User errors – Standby Database,

Flashback database Site failures – streched clusters, Standby

Database

•

Standby - Dataguard

• physical standby

• logical standby ( Not in SAP per 105047) • DG on RAC needs db_unique_name in srvctl • commands: srvctl ... –d <db_unique_name> ...

• ...

Still 2 tools: CRSCTL & SRVCTL

1) CRSCTL manages all cluster(ware) related operations

Usage: crsctl <command> <object> [<options>] command:

enable|disable|config|start|stop|relocate|replace|stat|add|delete|modify |getperm|setperm|check|set|get|unset|debug|lsmodules|query|pin|unpin

2) SRVCTL manages all Oracle resources related operations

Usage: … objects:

database|instance|service|nodeapps|vip|asm|diskgroup|listener|srvpool|se rver|scan|scan_listener|oc4j|home|filesystem|gns


Application Connect via SQL Net Setup for SAP Required DNS?

New

Single

Components in 11.2

Client Access Name (SCAN)

• Used by clients to connect to any database in the cluster

• Removes the requirement to change the client connection if cluster changes

• Load balances across the instances providing a service

• Provides failover between “moved instances”

FR

EE

Ora

cle

Gri

d In

frastr

uctu

re

Clu

ste

rSC

AN

na

me

Siebel

PSFT

RAC

DB1

RAC

DB2

RAC

One

2 Listener level providing SCAN

Application

Server

Oracle RAC Database Cluster

SCAN

Listeners

Clients

Local Listeners

2 Listener level providing SCAN

Application

Server

Oracle RAC Database Cluster SCAN

Listeners Listeners

Clients

1. client asks the scan-listener adressed by one of the 3 scan ip‘s for TNS service

scan listener responds to client with right node-listener address

client asks node listener for service

2.

3.

1

2

3

Local L

11.2 network

3

DNS connect

1

DNS 2 4

client

SCAN address

Resolved by 3 fix IP’s returned in a round robin fashion Oracle RAC cluster - GRIDA

Local listeners

SCAN listeners

corporate DNS domain

TNS-Setup / Services (11g)

• Connect to a TNS-Service via SCAN listeners which is also a CRS-Resource

• EZCONNECT

• Advantage: Services can be dynamically relocated and switched with CRS-commands on SAP instance level

• No more tnsnames.ora entries

listener.ora in GRID-Home

1) SQL*net EZconnect

New naming convention without need for tnsname.ora file

Syntax: <user>/<password>@//<IP-name>:<port>/<service> Or short: //<IP-name>:<port>/<service>

To be used in spfile: local_listener, Remote_listener

Environment : dbs_ora_tnsname

SAP profiles: dbs_ora_tnsname, dbs/ora/tnsname

IP-Name: SCAN-VIP is used only!!

IP addresses

Static (DNS) setup needs

One hostname, ora-vip per host One abap-vip,one java-vip One asinst-vip per asinst 3 Scan IP’s (one name and 3 IPs)

Single Client Access Name Network Configuration for SCAN - DNS

•

•

Requires a DNS adress entry to be used

In DNS, SCAN is a single name defined to resolve to 3 IP-addresses:

• Each cluster will have 3 SCAN-Listeners, combined with a SCAN-VIP defined as cluster resources

The SCAN VIP/LISTENER combination will failover

to another node in the cluster, if the current node fails •

Cluster Resources -------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE dherac1

ora.LISTENER_SCAN2.lsnr 1 ONLINE ONLINE dherac2

ora.LISTENER_SCAN3.lsnr 1 ONLINE ONLINE dherac1

Scan.saptest.pub

IN A 133.22.67.194

IN A 133.22.67.193

IN A 133.22.67.192

2) Tnsnames.ora

If there is any requirement for using tnsnames.ora, it‘s still possible

Must reside in $GRID_HOME/network/admin and RDBMS home/network/admin or $TNS_ADMIN

Remove TNS aliases used for 10g local_listener and remote_listener DB parameters from tnsnames.ora

TNS name resolution

Init.ora

Db_domain=‘world‘

Init.ora

Service_names=‘...‘,‘RAC001‘,‘...‘

Sqlnet.ora

NAMES.DEFAULT_DOMAIN=de.oracle.com

Connect sapr3/sap@rac001

TNS-Setup (SAP) with 11g

• Sqlnet.ora: NAMES.DIRECTORY_PATH=(EZCONNECT)

• SAP-Instance connects with sqlplus sapr3/sap@//<scan-vip>:<port>/<service>

sqlplus sapr3/sap@//saprac-scan/D04 (default port#)

• SAP-Startprofile

EZCONNECT = //saprac-scan.de.oracle.com/EDU_D04.world

SETENV_03 = dbs_ora_tnsname=$(EZCONNECT)

( .world comming from db_domain!!)

• SAP-Instanceprofile

//saprac-scan.de.oracle.com/EDU_D04.world

TNS-Setup (Listener)

• lsnrctl status:

Service "EDU_DVEBMGS00_saprac1" has 1 instance(s).

Instance "EDU001", status READY, has 1 handler(s) for this service...

• lsnrctl needs GRID environment for all listeners (node and scan)

TNS-Setup R3trans

• R3trans is needed during SAP startup to DB, i.e. at least one instance is up

It also connects with

check if

• • connect sap<sid>/pwd@<service_name>

•

•

•

•

Service to check all instances

Uses dbs_ora_tnsname

service_names=‘EDU, ...‘

R3trans -x

TNS-Setup (tnsnames.ora) – 11g

EDU.WORLD =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = scan-vip)(PORT = 1527))

(CONNECT_DATA =

(SERVICE_NAME = EDU.WORLD)

(GLOBAL_NAME = EDU.WORLD) - ERASE!!!!

) )

•

•

• •

No address list

Scan address used only

No TAF or failover entries, provided in CRS – TNS service definition srvctl add service -d <db_unique_name> -s <service_name> -r

"<preferred_instance>" -a "<failover_instances>" -P BASIC -k <net_num> -l PRIMARY -e SELECT -z <failover_retries> -w <failover_delay>

TNS for brtools •

•

•

•

Init.sap: parallel_instances

Init.sap: db_services

password files plus sysdba to system

Tnsnames entries for dedicated single instances EDU001.WORLD =

(DESCRIPTION =

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = saprac1-vip)(PORT = 1527))

)

(CONNECT_DATA =

(SERVICE_NAME = EDU001.WORLD)

)

)

• Static instance definition in listener.ora SID_LIST_LISTENER =

(SID_LIST=(SID_DESC=(SID_NAME=<SID>001)

(ORACLE_HOME=/oracle/<SID>/11202)))

TNS-Settings for 11g

Instance Profile

SAPSYSTEMNAME=EDU

SAPDBHOST=<node vip>

INSTANCE_NAME=D04

dbs/ora/tnsname=//saprac-scan.de.oracle.com/EDU_D04.world

Start Profile

EZCONNECT = //saprac-scan.de.oracle.com/EDU_D04.world

SETENV_03 = dbs_ora_tnsname=$(EZCONNECT)

Parameter Name Value

local_listener //<node vip.domain>:<port>

remote_listener //<SCAN-vip.domain>:<port>

<ORACLE_SID>.service_names (<DBSID>,<ORACLE_SID>)

db_domain World (SAP default, but can also be

removed …)

listener_networks ((name=<network-name>)

(local_listener= "//<second-ora-vip-

of-local-node>:<port>"))

SAP instance startup and TNS 1 SAP instance profile

• SAP instance profile defines settings for particular SAP on instance startup

• SAP instance requires instance specific TNS service

• SAP instance profile parameter :

dbs/ora/tnsname=<scan-ip-name>:<port>/<SAP-instance-TNS-service>

• Dbs/ora/tnsname=xd01-scan.th.oracle.com:1527/E11_D04

• Dbs/ora/tnsname=E11_D04.world (tnsnames.ora)

instance

SAP instance startup and TNS 2 OS environment

• R3trans checks database connection on SAP instance start

• only needs to know if database available, don‘t cares for DB instances

• uses static common service (<SID>.service_names) offered all DB instances

• R3trans uses OS environment variable dbs_ora_tnsnames

by

dbs_ora_tnsname=<scan-ip-name>:<port>/<DBSID-service>

•

•

•

Dbs_ora_tnsname=xd01-scan.th.oracle.com:1527/E11

Dbs_ora_tnsname=E11 (tnsnames.ora)

Problem: OS environment overwrites SAP instance profile setting dbs/ora/tnsnames, solution next slide

SAP instance startup and TNS 3 SAP start profile

• SAP start profile is able to define environment settings particular SAP instance startup

for

•

•

•

overwrites OS environment variables

requires instance specific TNS service

SETENV_03 = dbs_ora_tnsname=<scan-ip-name>:<port>/<SAP-

instance-TNS-service>

• SETENV_03=dbs_ora_tnsname=xd01- scan.th.oracle.com:1527/E11_D04

• SETENV_03=dbs_ora_tnsname=E11_D04.world (tnsnames.ora)


SAP on Oracle Exadata

SAP enqueue service, HA in Exadata

SAPCTL Agenda

•

•

•

•

•

SAP Enqueue Replication

Control Flow SAPCTL

Oracle Clusterware Framework

SAPCTL Commandline Interface

Changes in SAP Profiles / Scripts

Concept of SAP Enqueue Replication

SAP Business Transaction Oracle database transaction

The SAP Enqueue Service provides distributed lock management for SAP R/3 instances.

SAP uses an active/passive approach to enable high availability for the Enqueue Service.

There is exactly one Enqueue Service running in the SAP system. The Replication Service

run on the node on which the Enqueue Service runs in the event of a failure or planned

outage.

The Replication Service maintains a copy of the Enqueue Service´s lock table; this copy is kept in a shared memory segment during runtime.

The node that supports the Enqueue Service also hosts a virtual IP address (VIP) for communication. Whenever the active node fails or is shut down, the VIP and the Enqueue Service fail over to the next node.

What is SAP Enqueue HA?

Control flow within SAPCTL to

support

Host 1


Host 2

TCP Connection Lock

Table

Enqueue

Server

Lock Replica

Table

Lock State

Transfer

Replication Server

VIP

0. Normal Operation, CRS checks healthiness of resources on all hosts in the cluster


support

Host 1


Host 2

Crash Lock

Table

Enqueue

Server

Lock Replica

Table

Replication Server

VIP

1. CRS detects a failure of the node or in the Enqueue Server


support SAP Enqueue Replication

Host 2

Lock Replica

Table

Replication Server

VIP

2. CRS switches the VIP to the node where Replication Server is running



Host 2

Enqueue

Server

Lock Replica

Table

Replication Server

3. CRS starts a new Enqueue Server on the node supporting the VIP attaching to

Memory segment of replicator VIP



Host 2

Enqueue

Server

Lock Table Replication

Server

4. by attaching to the shared memory

segment hold by replicator, new enqueue

server inherits lock table replica, which

becomes the new lock table

VIP



Host 2

Enqueue

Server

Lock Table

5. After new enqueue server took over

lock table, replicator is stopped and

memory is now owned by new enqueue

server; SAP transactions can continue

VIP



Host N Host 2

Lock

Table

Enqueue

Server

Lock Replica

Table

Replication Server

VIP

6. CRS starts Replication Server on another available node in the cluster



Host N Host 2

Connect Lock

Table

Enqueue

Server

Lock State

Transfer Lock Replica

Table

Replication Server

VIP

7. Back to normal operation, CRS checks healthiness of resources on all hosts in the cluster

Possible Configuration

SAP Enqueue Repl. SAP Enqueue Server SAP Enqueue Repl. SAP Enqueue Repl.

listener listener

cluster database cluster database

ORACLE_HOME ORACLE_HOME

ORA_CRS_HOME ORA_CRS_HOME ORA_CRS_HOME ORA_CRS_HOME

Shared Storage

• Oracle Clusterware HA Agent for SAP is protecting SAP Enqueue Server and SAP Enqueue Replication Server It is also providing resources to the RAC database running on

nodes 1 & 2

•

The Oracle Clusterware Framework

•

•

Calls SAP HA – agent

agent will be invoked by

of three commands

Oracle Clusterware with one

•

•

•

start

stop

check

• Oracle Cluster Registry

• Stores information relating interacts and manages the

to how Oracle Clusterware application

Oracle Clusterware Framework Architecture

Framework

Please Start

Agent

‘start’

‘check’ How are you ?

Please Stop ‘stop’

How the HA Agent for SAP is called by the Framework The ‘start’ command invokes the SAP tool sapcontrol

command depending on CRS resource type to issue the start

- for startservice process

CRS resource type SAP.<SID>.<INST>.startsrv

sapcontrol -nr <instance-nr> -function StartService <SID>

- for the SAP instance

CRS resource type SAP.<SID>.<INST>.<type>

<type> qualifies instance type, e.g. abapenq, gwinst, webdisp

sapcontrol -nr <instance-nr>

-function Start

How the HA Agent for SAP is called by the Framework The ‘stop’ command invokes the SAP tool sapcontrol

command depending on CRS resource type to issue the stop

- for startservice process

CRS resource type SAP.<SID>.<INST>.startsrv

sapcontrol -nr <instance-nr> -function StopService

- for the SAP instance

CRS resource type SAP.<SID>.<INST>.<type>

<type> qualifies instance type, e.g. abapenq, gwinst,


-function Stop

webdisp

How the HA Agent for SAP is called by the Framework

The ‘check’ command invokes the SAP tool

check function

sapcontrol to issue the

- for startservice process and SAP instance

All CRS resource types defined for SAP


-function GetProcessList

No specific check for different instance type, e.g. by calls to

ensmon, msprot, dpmon, gwmon. All is covered by sapcontrol.

Overview: SAPCTL commands for with SAP R/3 Enqueue Replication

use

•

•

•

•

•

•

sapctl

sapctl

sapctl

sapctl

sapctl

sapctl

create

status

start

stop

relocate

remove

- registers resources

- shows status of resources

-

-

-

-

starts resources

stops resources

move resource to different

delete configuration

host

SAPCTL command create

• sapctl create

-sapsid EDU

-if eth0

-nm 255.255.254.0

-nodes node1,node2

-abapenq ASCS00

-abaprep ERS01

-abapvip 140.86.242.75

-abapmsport 3601

-javaenq SCS02

-javarep ERS03

-javavip 140.86.242.76

-javamsport 3902

-asinst DVEBMGS04

-asinstvip 140.86.242.77

<- name of the SAP System

<- network interface

<- netmask

<- list of nodes

<- Enqueue Service Identifier for WEB AS ABAP

<- Replication Service Identifier for WEB AS ABAP

<- VIP Address for WEB AS ABAP

<- Portnumber of ABAP message server

<- Enqueue Service Identifier for WEB AS JAVA

<- Replication Service Identifier for WEB AS JAVA

<- VIP Address for WEB AS JAVA

<- Portnumber of JAVA message server

<- Instance name SAP Central Instance

<- VIP Address for SAP Central Instance

SAPCTL command config ABAP

• sapctl config

shows list of protected SAP systems

• sapctl config abapenq –sapsid EDU

shows configuration of Central Instance

• sapctl config abaprep –sapsid EDU

shows configuration of Replication Server

• sapctl config all –sapsid EDU

shows complete configuration for SAP system

SAPCTL command status

• sapctl status abapall –sapsid EDU

shows status of ABAP Enqueue and Replication

• sapctl status javaall –sapsid EDU

shows status of JAVA Enqueue and Replication

• sapctl status all –sapsid EDU

shows status of all SAP resources

SAPCTL command start

• sapctl start abapenq –sapsid EDU

starts SAP ABAP Enqueue instance

• sapctl start abaprep –sapsid EDU

starts SAP ABAP Replication instance

• sapctl start all –sapsid EDU

starts all SAP resources

SAPCTL command stop

• sapctl stop javaenq –sapsid EDU

stops SAP JAVA Enqueue instance

• sapctl stop javarep –sapsid EDU

stops SAP JAVA Replication instance

• sapctl stop all –sapsid EDU

stops all SAP resources

SAPCTL command update

• sapctl update

-sapsid EDU <- name of the SAP System

-abapvip 140.86.242.63

-javapvip 140.86.242.63

<-

<-

<-

IP address VIP

IP address VIP

IP address VIP -asinstvip 140.86.242.63

-a node

-d node

<- node to add

<- node to delete

SAPCTL command relocate

• sapctl relocate

-sapsid EDU <- name of the SAP System

-abapenq ASCS01

-javaenq SCS02

-abaprep ERS11

-javarep ERS12 -asinst DVEBMGS00

<-

<-

<-

<- <-

ABAP Enqueue instance

JAVA Enqueue instance

ABAP Replication instance

JAVA Replication instance SAP CI central instance

-to node <- where to relocate

SAPCTL command remove

• sapctl remove all –sapsid EDU

removal of all SAP resources

- only remove of all SAP resources possible

- `root` permission required

Supported SAP instance types

ASCS

SCS ERS

DVEBMGS

DV

GW

WebDisp

Central Services ABAP

Central Services JAVA

Replication Service ABAP / JAVA

Cental Instance (e.g. For solution

Application Instance

Standalone Gateway

SAP WebDispatcher

manager)

Sapstartsrv SAP Start Service per Instance

SAPCTL installation location

sapctl uses direrctory /usr/sap/sapctl.

Perl scripts in /usr/sap/sapctl/bin

Logs in /usr/sap/sapctl/log

Copy of Perl interpreter in /usr/sap/sapctl/perl

Symbolic link to crsctl and crsctl.bin in /usr/sap/sapctl/bin

The sapctl package

sapctl

crssapactions.pl

has two perl scripts:

Command line interface

Action script called by CRS

No wrapper, no user environment, no configuration required.

No change in SAP scripts. Instance specific information is stored

In CRS resource.

SAP changes required to use SAP HA

• Instance profile Standalone Enqueue Server (ENQ + MSG)

Instance profile Enqueue Replication Server (ENR)

enque/process_location=REMOTESA

(in DEFAULT.PFL important for all dialog instances)

enque/server/replication=true

enque/server/internal_replication= true

•

•

•

Start profile Standalone Enqueue Server

Start profile Enqueue replication Server

As recommended by SAP HA documentation

Server Pool SAPCTL

SAP_SP_<SID> defined by

• The Server Pool SAP_SP_<SID> contains the list of cluster nodes where SAP resources can run on. All CRS resources for a specific

SAP instance are associated with the Server Pool

• Adding or deleting nodes can be done by modifying the Server Pool attribute SERVER_NAMES

• The Server Pool is owned by root, so root permission is required when changing attribute values

List of CRS Types used for SAPCTL

Defined by SAPCTL create command

•

•

•

•

•

•

sap.abapenq.type

sap.abaprep.type

sap.javaenq.type

sap.javarep.type

sap.startsrv.type

app.appvip.type

Already existing, defined by CRS

•

•

ora.network.type

cluster_resource

List of CRS SAPCTL

Resources used for

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

sap.<SID>.ASCS<NR>.startsrv

sap.<SID>.ASCS<NR>.abapenq

sap.<SID>.SCS<NR>.startsrv

sap.<SID>.SCS<NR>.javaenq

sap.<SID>.ERS<NR>.startsrv

sap.<SID>.ERS<NR>.abaprep

sap.<SID>.abapvip

sap.<SID>.javavip

sap.<SID>.DVEBMGS<NR>.startsrv

sap.<SID>.DVEBMGS<NR>.sapinst

sap.<SID>.DVEBMGS<NR>.instvip

sap.<SID>.DV<NR>.startsrv

sap.<SID>.DV<NR>.sapinst

sap.<SID>.DV<NR>.instvip

sap.<SID>.G<NR>.startsrv

sap.<SID>.G<NR>.sapgw

sap.<SID>.G<NR>.gwvip

sap.<SID>.W<NR>.startsrv

sap.<SID>.W<NR>.wdspinst

sap.<SID>.W<NR>.wdspvip

ora.net<x>.network

Dependencies for SAPCTL

between resources used

(A)SCS ERS

Start

Stop

Attraction

Dispersion

StartService StartService

Application vip

Dependencies for SAPCTL

between resources used

DVEBMGS,DV,GW,WebDisp

Start

Stop

StartService

Application vip

SAPCTL Log Files

• The logfile sapctl.log shows all actions performed by sapctl. All actions as well as output of system commands is recorded here.

Structure of log entries:

<host> date time | pid | trace information of executed program steps

Output of calls to SAP tool sapcontrol is also recorded

Similar log in

${ORA_CRS_HOME}/log/<hostname>/agent/crsd/scriptagent_<sidadm>

•

•

•


SAP Central Services on Exadata ASCS/SCS install

SAP Installation on DB nodes 1 Overview

• Recommended: NFS mount /sapmnt of existing SAP system can be mounted on DB nodes as usual

•

•

to provide profiles

No general need for HA NFS

Able to host multiple platform binaries

• 1. DB

Step DB node preparation using sapinst on nodes

all

•

•

•

2.

3.

4.

Step install ASCS on all nodes using sapinst

Step install ERS on all nodes using sapinst

maintain ERS profiles

SAP Installation on DB nodes 2 Overview

•

•

•

5. Step: manually test ASCS and ERS

optional: JAVA installation (SCS + another

6. Step: install sapctl

ERS)

SAP Installation on DB nodes 3 Requirements

•

•

•

SAP Kernel CD (minimum 7.00 EP 3 or 7.20 EP 1 and higher)

Sapinst support starting from version 7.0 EP 3 and 7.3 EP 1

SAP OSS note 1619343 "SAPinst for Oracle Exadata on Oracle Linux and Solaris X86“

• Download Software Provisioning Manager from SWDC Hint:

Install VNCserver on DB nodes to locally run sapinst (no need for JCE, Java Cryptography Extension policy file) Requires one virtual IP for ABAP (<ABAP-VIP>) and one for JAVA (<JAVA-VIP>), if used

•

•

1. Step – prepare node 1

• OSS 1619343 - SAPinst for Oracle Exadata and Solaris X86

• Make use of NFS for /sapmnt

on Oracle Linux

• Run sapinst on all DB nodes for Exadata Node Preparation


•

•

•

•

•

•

start DB node preparation using sapinst to

Create OS users and groups

OS environment, Oracle client

SAP kernel plus brtools

Requires SAP kernel CD and ORACLE client

Must be executed on each node

additionally


2. Step – ASCS installation 1

• run sapinst on node 1 having /sapmnt mounted all DB nodes

• start with NO SAP profile

to share on

• unmount /sapmnt on all other DB nodes before starting sapinst (local /sapmnt files will not be used, just for install)

• use virtual hostname <ABAP-VIP>

• use of environment SAPINST_USE_HOSTNAME = < ABAP-VIP >

• Bind <ABAP-VIP> locally on node for sapinst run crsctl start ip –A <ABAP-VIP>/<netmask>/eth0

crsctl stop ip –A <ABAP-VIP>/<netmask>/eth0

• Run sapinst for ASCS installation on all remaining DB nodes






3. Step – ERS installation 1

• except node 1, all other DB nodes run sapinst /sapmnt NFS mount

• start with NO SAP ERS profile

without

• NO use of environment SAPINST_USE_HOSTNAME

• Run sapinst for ERS installation on all remaining DB nodes

• Merge ERS profiles on „shared“ NFS mounted /sapmnt

• ERS for ABAP and ERS for JAVA must use different instance numbers





4. Step – maintain SAP DEFAULT.PFL

profiles 1

SAPSYSTEMNAME = <SAP-SID>

SAPGLOBALHOST = <ABAP-VIP>

SAPDBHOST = <ABAP-VIP>

rdisp/mshost = <ABAP-VIP> rdisp/msserv_internal = 39<ASCS-Instance#>

rdisp/msserv = sapms<SAP-SID>

enque/process_location = REMOTESA enque/serverhost = <ABAP-VIP>

enque/serverinst = <ASCS-Instance#>

4. Step – maintain DEFAULT.PFL

SAP profiles 2

dbms/type = ora

j2ee/dbtype j2ee/dbname

j2ee/dbhost

= =

=

ora <SAP-SID>

<JAVA-VIP>

= <JAVA-VIP> j2ee/scs/host j2ee/scs/system = <SCS-Instance#> j2ee/ms/port = 39<SCS-Instance#>

rdisp/vb_dispatching = 0

4. Step – maintain SAP profiles AS Instance profiles

SAPSYSTEM = <local-SAP-Instance#>

INSTANCE_NAME = <local-SAP-Instance-name>

SAPLOCALHOST = <local-Hostname>

rdisp/wp_no_vb = <# > 1> rdisp/wp_no_vb2 = <# > 1>

rdisp/vbname = <local-Hostname>

dbs/ora/tnsname = //<SCAN-NAME>[:<PORT>]/<SAP- SID>_<SAP-Instance-name>

5. Step – test ASCS & ERS 1

• Bind <ABAP-VIP> on node 1 crsctl start ip –A <ABAP-VIP>/<netmask>/eth0

•

•

•

Start ASCS on node 1

Check ASCS

On remaining DB nodes one by one

•

•

•

Start ERS

Check ERS

Stop ERS

• Unbind <ABAP-VIP> on node 1 crsctl start ip –A <ABAP-VIP>/<netmask>/eth0

• Repeat on next node


• Check ASCS sapcontrol –host <ABAP-VIP> -prot NI_HTTP -nr <#> -function

GetProcessList

25.04.2013 18:13:28 GetProcessList

OK

name, description, dispstatus, textstatus, starttime, elapsedtime, pid

ms.sapE11_ASCS01, , GREEN, Running, 2013 04 25 08:35:33, 9:40:57, 57023

en.sapE11_ASCS01, , GREEN, Running, 2013 04 25 08:35:33, 9:40:57, 57024


• Check ERS sapcontrol -host <host> -prot NI_HTTP -nr <#> -function GetProcessList

25.04.2013 18:14:34 GetProcessList

OK

name, description, dispstatus, textstatus, starttime, elapsedtime, pid

enrepserver, EnqueueReplicator, GREEN, Running, 2013 04 25 12:56:25,

5:21:11, 103217

6. Step – Sapctl installation

1. Download sapctl package is attached to OSS note Note 1496927 - Protection of Enqueue Replication through Oracle Clusterware

2.

3.

4.

unpack SAPCTL.tgz into a temporary directory

create 2 symbolic links for crsctl and crsctl.pl

copy perl directory from ORACLE_HOME

•

•

•

•

2 Versions: one for 10g and one for 11g

Use enclosed documentation

New directory structure: /usr/sap/sapctl/bin and

Starting/stopping/control by sapcontrol (11g)

log

https://service.sap.com/sap/support/notes/971289
































Sapctl – Definition of Resources

• Example: sapctl create -sapsid EDU -if eth0 -nm

255.255.255.0 -nodes saprac1,saprac2 -abapvip 192.162.119.145 -abapenq ASCS01 -abaprep

ERS11


BRR, CRS Utilities and trouble shooting

Backup, Restore and Recovery in RAC with or without ASM/Exadata

•

•

No differences to Single Instance

Recovery: Could be done in parallel by multiple

instances using parallel recovery using PQX

slaves of mounted instances

Archiving to shared directory or ASM or FRA

For ASM or FRA use RMAN and brtools

ASM: DB consistency check only with RMAN

•

•

•

Backup, Restore and Recovery in Exadata









Brtool ASM/Exadata support

•

•

•

•

•

•

•

•

•

•

One dedicated node, not for Exadata

Shared Oracle Home (init<SID>.sap), not for Exadata

Brtools support Exadata starting 7.20 patch

SYSTEM needs SYSOPER or SYSDBA

Static Instance definition in listener.ora

PWD file

brbackup

brarchive

RMAN, 3rd party backup (backint)

ASM: RMAN plus brtools only

18

Brtool ASM/Exadata support

• brrecover

•

•

•

Do

brrecover

brrecover

brrecover

–u

–u

–u

/

/

/

-t

-t

-t

complete

dbpit

reset

•

•

•

•

not use brrestore with ASM.

OMF format is used for datafile creation.

Can not move/rename OMF file, alias can be changed.

<sid>adm can ssh to <oracle> with no password

within and between all nodes.

Brtool configuration

•

•

•

Support from BR*Tools 7.20 level 18 and up

All GRID/RDBMS installed under oracle user.

Only

•

supported backup types

backup_dev_type = rman_util | rman_disk

rman_stage backup_dev_type =

disk disk_copy_cmd = rman |

rman_set backup_dev_type = tape

| pipe

tape_copy_cmd = rman

|

•

•

• All backup strategies with RMAN are supported

Brtool configuration

• All br* executable has to own by oracle with 6774 brarchive*

brbackup*

brconnect*

brrecover*

brrestore*

-rwsrwsr--

-rwsrwsr--

-rwsrwsr--

-rwsrwsr--

-rwsrwsr--

-rwsrwsr--

-rwsrwsr--

1

1

1

1

1

1

1

oracle

oracle

oracle

oracle

oracle

oracle

oracle

oinstall

oinstall

oinstall

oinstall

oinstall

oinstall

oinstall

7732338

7908129

9970354

8376747

2783544

May

May

May

May

May

31

31

31

31

31

16:30

16:30

16:30

16:31

16:31

10479944 May 31 16:31 brspace*

4103679 May 31 16:31 brtools*

• All directories in SAP data home /oracle/<SID> should owned by oracle:oinstall and have g+w permission.

new sapprof contains init.ora and init.sap for brtools

brtools should initiate from <sid>adm account.

be

•

•

Brtools configuration

• oracle user account should contain brtool environment variables.

OPS$oracle account should be created. • SQL>

SQL>

SQL>

connect / as sysdba

create user ops$oracle identified externally;

grant sapdba to ops$oracle;


• Use ASM alias for spfile and controlfile

<asm_disk_group>/<db_name>/spfile<db_name>.ora

<asm_disk_group>/<db_name>/cntrl<db_name>.dbf

Use ASM alias for archive destination

log_archive_dest_1 = "LOCATION=+ARCH/EDU/oraarch“

•

SQL>

SQL>

SQL>

connect / as sysdba

alter diskgroup ARCH add directory '+ARCH/EDU';

alter diskgroup ARCH add directory '+ARCH/EDU/oraarch';


• Need

•

•

•

Special init.sap parameters for Exadata:

loc_ora_homes = yes

asm_root_dir = ASM

asm_ora_sid = (<db_inst1>:<asm_inst1>,

<db_inst2>:<asm_inst2>, ...)

asm_ora_home = (<db_inst1>:<asm_home1>,

<db_inst2>:<asm_home2>, ...)

•

Backup/Restore of ASM Data

•

•

•

Normal RMAN backups

Split Mirror backups

Special cases:

• Loss of Oracle cluster registry and

MOS note 1062983.1

voting files

• ASM metadata restore, MOS note 451900.1

Documents

Discussion SAP on Exadata 20140618