Sun ZFS Storage Appliance 7000 Systems

Configuring Oracle Solaris Cluster Software With Oracle RAC/CRS and Sun ZFS Storage Appliance 7000 SystemsJune 2011

By J. Randriam

This document describes the installation steps for a 3-node Sun Cluster 3.2 01/09 or Oracle Solaris Cluster 3.3 (or later) configuration running with Oracle Real Application Clusters (RAC) / Cluster Ready Services (CRS) 11.1.0.7 and a Sun ZFS Storage 7120 System.Note: A Sun ZFS Storage 7120 was used in this configuration, but the information in this document should be applicable to the entire Sun Storage 7000 Unified Storage series of products.

This document covers the following topics:

IntroductionInitial Preparation on the Sun ZFS Storage 7120 Using the CLICreating NFS Filesystems Using the Sun ZFS Storage 7120 CLIAccessing the NFS Filesystems From the Cluster NodesAccessing the iSCSI LUNs From the Cluster NodesAdding a Quorum DeviceConfiguring Oracle RAC / CRS in the ClusterAuthor-Recommended Resources

IntroductionThis document makes the following assumptions:

The Oracle Solaris Cluster software has already been installed on the nodes.A Sun ZFS Storage 7120 System is connected to the same subnet as the cluster nodes and is reachable (that is, "ping-able") from each of the nodes.

In the following figure, the cluster nodes are paborn1, paborn2, and paborn3, and they are running the Oracle Solaris 10 10/09 Operating System. The Sun ZFS Storage 7120 name is pandroid4, and this system is running the 2010.Q3.3.1 software.

Page 1 of 9How to Configure Oracle Solaris Cluster with oracle RAC/CRS and Sun ZFS Storage Appliance 7000 Systems

4/30/2012http://www.oracle.com/technetwork/articles/servers-storage-admin/solclusterrac-410456.html?printOnly=1

In this configuration, a Sun ZFS Storage 7120 is employed for two purposes:

To provide NFS filesystems for the CRS Oracle Cluster Registry (OCR) and voting files, as well as for RAC database filesTo provide some iSCSI LUNs that will be used as quorum devices (or, more precisely, as software quorum devices)

Initial Preparation on the Sun ZFS Storage 7120 Using the CLIIn the Sun ZFS Storage 7120, filesystems and LUNs are referred to generically as shares. These shares are created within the context of a project. The same project can contain a mix of filesystems and LUNs. The creation of a new project is straightforward.To perform the following steps, connect via ssh to the Sun Storage 7210 to access the Command Line Interface (CLI):1. Create a project (crs-project is used for this example):

pandroid4:> shares pandroid4:shares> project crs-project pandroid4:shares crs-project (uncommitted)> commit

The newly created project can then be used to create the NFS filesystems and the iSCSI LUNs that will be accessed by the Sun Cluster. 2. Create an iSCSI target:

Note: in this example, the target's IQN is automatically generated by the system.

pandroid4:> configuration san targets iscsi pandroid4:configuration san targets iscsi> create pandroid4:configuration san targets iscsi target (uncommitted)> set alias="Tgt4RAC" alias = Tgt4RAC (uncommitted) pandroid4:configuration san targets iscsi target (uncommitted)> set auth=none auth = none (uncommitted) pandroid4:configuration san targets iscsi target (uncommitted)> set interfaces=nge0 interfaces = nge0 (uncommitted) pandroid4:configuration san targets iscsi target (uncommitted)> commit pandroid4:configuration san targets iscsi> list ...

TARGET ALIAS target-000 Tgt4RAC | +-> IQN iqn.1986-03.com.sun:02:57a9b940-6df5-c127-8175-fd49d7c9fa37 ...

The created target - more specifically the target's IQN - will be referenced later when accessing the iSCSI LUNs from the Cluster nodes.

3. Create an iSCSI target group (Grp4RAC in this example), and include the previously created target in that group:

pandroid4:configuration san targets iscsi groups> create pandroid4:configuration san targets iscsi group (uncommitted)> set name="Grp4RAC" name = Grp4RAC (uncommitted) pandroid4:configuration san targets iscsi group (uncommitted)> set targets=iqn.1986-03.com.sun:02:57a9b940-6df5-c127-8175-fd49d7c9fa37 targets = iqn.1986-03.com.sun:02:57a9b940-6df5-c127-8175-fd49d7c9fa37 (uncommitted) pandroid4:configuration san targets iscsi group (uncommitted)> commit pandroid4:configuration san targets iscsi groups> list



...

group-002 Grp4RAC | +-> TARGETS iqn.1986-03.com.sun:02:57a9b940-6df5-c127-8175-fd49d7c9fa37 ...

This target group will be referenced later when the iSCSI LUNs are created (see step 6 below) 4. Create the iSCSI initiators:

a. on the host side, determine the IQN of each initiator (i.e each node) :

root@paborn1 # iscsiadm list initiator-node Initiator node name: iqn.1986-03.com.sun:01:00144f971d46.paborn1 create pandroid4:configuration san initiators iscsi initiator (uncommitted)> set alias="paborn1" alias = paborn1 (uncommitted) pandroid4:configuration san initiators iscsi initiator (uncommitted)> set initiator=iqn.1986-03.com.sun:01:00144f971d46.paborn1 initiator = iqn.1986-03.com.sun:01:00144f971d46.paborn1 (uncommitted) pandroid4:configuration san initiators iscsi initiator (uncommitted)> commit pandroid4:configuration san initiators iscsi> list NAME ALIAS initiator-000 paborn1 | +-> INITIATOR iqn.1986-03.com.sun:01:00144f971d46.paborn1 ...

Repeat Step a and b for each node.

5. Create an initiator group (rabornGrp in this example), and include the previously created initiators - referenced by their IQN - in that group:

pandroid4:configuration san initiators iscsi groups> create pandroid4:configuration san initiators iscsi group (uncommitted)> set name=abornGrp name = abornGrp (uncommitted) pandroid4:configuration san initiators iscsi group (uncommitted)> set initiators=iqn.1986-03.com.sun:01:00144f971d46.paborn1 ,iqn.1986-03.com.sun:01:00144fac24ac.paborn2



,iqn.1986-03.com.sun:01:00144f97ed34.paborn3 initiators = iqn.1986-03.com.sun:01:00144f971d46.paborn1,iqn.1986-03.com.sun:01:00144fac24ac.paborn2,iqn.1986-03.com.sun:01:00144f97ed34.paborn3 pandroid4:configuration san initiators iscsi group (uncommitted)> commit pandroid4:configuration san initiators iscsi groups> list GROUP NAME group-001 abornGrp | +-> INITIATORS iqn.1986-03.com.sun:01:00144f971d46.paborn1 iqn.1986-03.com.sun:01:00144fac24ac.paborn2 iqn.1986-03.com.sun:01:00144f97ed34.paborn3 ...

This initiator group will be referenced later when the iSCSI LUNs are created (see step 6 below). For a given LUN, it will define which initiators (i.e nodes) are allowed to access it. 6. Create the iSCSI LUNs:

a. Select the project that was created in Step 1.

pandroid4:shares> select crs-project

b. Create LUNs (crslunA in this example) in that project, and associate them with the target group and the initiator group previously created:

pandroid4:shares crs-project> lun crslunA pandroid4:shares crs-project/crslunA (uncommitted)> set volsize=20G volsize = 20G (uncommitted) pandroid4:shares crs-project/crslunA (uncommitted)> set targetgroup=Grp4RAC targetgroup = Grp4RAC (uncommitted) pandroid4:shares crs-project/crslunA (uncommitted)> set initiatorgroup=abornGrp initiatorgroup = abornGrp (uncommitted) pandroid4:shares crs-project/crslunA (uncommitted)> commit pandroid4:shares crs-project> list LUNs: NAME SIZE GUID crslunA 20G 600144F0FB360BAD00004C73C7E90001 ...

Repeat step b. for each LUN, as needed.

At this point, the iSCSI LUNs you created can be acquired as iSCSI targets by the cluster nodes (the iSCSI initiators).

Creating NFS Filesystems Using the Sun ZFS Storage 7120 CLI1. Select the project where the filesystem will be created ( crs-project in this example)

pandroid4:shares> select crs-project

2. Create a filesystem (nfs-for-crs in this example) :



pandroid4:shares crs-project> filesystem nfs-for-crs pandroid4:shares crs-project/nfs-for-crs (uncommitted)> set mountpoint=/export/crs mountpoint = /export/crs (uncommitted) pandroid4:shares crs-project/nfs-for-crs (uncommitted)> set root_user=root root_user = root (uncommitted) pandroid4:shares crs-project/nfs-for-crs (uncommitted)> set sharenfs="anon=0" sharenfs = anon=0 (uncommitted) pandroid4:shares crs-project/nfs-for-crs (uncommitted)> set root_permissions=744 root_permissions = 744 (uncommitted) pandroid4:shares crs-project/nfs-for-crs (uncommitted)> commit pandroid4:shares crs-project> list Filesystems: NAME SIZE MOUNTPOINT nfs-for-crs 79.6K /export/crs ...

Repeat these steps for each filesystem, as needed.

Accessing the NFS Filesystems From the Cluster NodesAt this point, the filesystems you just created are usable and can be accessed and mounted from any node.As a quick verification, you can try the following command on a node using the IP address of the Sun ZFS Storage 7120 (10.6.204.78 for this example):

# cd /net/10.6.204.78/export/crs # mount -v|grep crs 10.6.204.78:/export/crs on /net/10.6.204.78/export/crs type nfs remote/read/write/nosetuid/nodevices/intr/retrans=10/retry=3/xattr/dev=5580006 on Mon Oct 5 18:30:32 2009

Accessing the iSCSI LUNs From the Cluster NodesTo make the created LUNs visible and usable from the nodes, execute the following additional steps on each node:

1. Enable the static discovery method for iSCSI:

# iscsiadm modify discovery -s enable # iscsiadm list discovery Discovery: Static: enabled

3. To make sure the dev and devices tree gets updated with the LUNs newly added, run the following command:

# devfsadm -i iscsi

4. Now the LUNs should be visible. You can use the iscsiadm list target command to see the new LUNs and their associated device name (cXtYdN):

# iscsiadm list target -S Target: iqn.1986-03.com.sun:02:57a9b940-6df5-c127-8175-fd49d7c9fa37 Alias: pool-0/local/crs-project/crslunA TPGT: 1 ISID: 4000002a0000 Connections: 1 LUN: 0 Vendor: SUN Product: SOLARIS OS Device Name: /dev/rdsk/c6t600144F04AC9EB700000144FA6E77400d0s24AC9EB700000144FA6E77400d0s2 ...

5. If necessary, use the format command to change the label, the partitioning, or both, as needed.

6. Once you're done adding all the iSCSI LUNs you created in the Sun ZFS Storage 7120 on each node, update the Sun Cluster DID name space with the newly added devices. From one node, run this command:

# cldevice populate Configuring DID devices did instance 24 created. did subpath paborn1:/dev/rdsk/c6t600144F04AC9EB700000144FA6E77400d0 created for instance 24. Configuring the /dev/global directory (global devices) obtaining access to all attached disks #

7. Check that all odes are seeing the newly added LUNs:

# cldevice status ...

/dev/did/rdsk/d24 paborn1 Ok

Adding a Quorum DeviceYou can pick one of the iSCSI LUNs to be used as a quorum device.

For example:

# clquorum add d24 # clquorum show d24 === Quorum Devices === Quorum Device Name: d24 Enabled: yes Votes: 2 Global Name: /dev/did/rdsk/d24s2 Type: shared_disk Access Mode: scsi3> Hosts (enabled): paborn2, paborn3, paborn1

Configuring Oracle RAC / CRS in the Cluster1. Edit the /etc/vfstab file on each node.

For this example, two NFS filesystems have been created in the Sun ZFS Storage 7120 for Oracle RAC/CRS: /export/oracrs and /export/oradb. (See the Creating NFS Filesystems From the Sun ZFS Storage 7120 GUI for information on how to create NFS filesystems.)

a. Add the following vfstab entries. Note the options so the NFS filesystems that were created in the Sun ZFS Storage 7120 get mounted automatically at boot time.

# vi /etc/vfstab ...

10.6.204.78:/export/crs - /data/crs nfs 2 yes rw,bg,forcedirectio,wsize=32768,rsize=32768,hard,noac,nointr,proto=tcp,vers=3 10.6.204.78:/export/oradb - /data/db nfs 2 yes rw,bg,forcedirectio,wsize=32768,rsize=32768,hard,noac,nointr,proto=tcp,vers=3

b. For now, manually mount these filesystems:

# mount /data/crs # mount /data/db

2. Change the ownership of the filesystems to oracle/dba:

# chown oracle:dba /data/crs # chown oracle:dba /data/db

3. Configure the RAC Framework Resource Group.

From one node, run the following commands:



Note: Adjust the nodelist to match your configuration, and also adjust Desired_primaries and Maximum_primaries as well.

# clrg create -S rac-framework-rg # clrs create -t SUNW.rac_framework:4 -g rac-framework-rg rac-framework-rs # clrs create -t SUNW.rac_udlm -g rac-framework-rg -p resource_dependencies=rac-framework-rs rac-udlm-rs # clrg online -emM rac-framework-rg

4. Install Oracle CRS and the Oracle binaries, and then create the database.

Note: This document doesn't describe how to perform this in detail; it provides only an outline of what needs to be done for this task.

a. Install the CRS OCR and the VOTE devices on the NFS filesystems that were previously created on the Sun ZFS Storage 7120 and mounted on the nodes (mounted on /data/crs).

Note: A minimum of five devices of 256MB each need to be created, and the owner must be changed to oracle:dba.

One way to accomplish this is to use the mkfile command:

# mkfile 256m ocrdev1 # chown oracle:dba ocrdev1 # ls -l ocrdev1 -rw------T 1 oracle dba 268435456 Nov 2 16:56 ocrdev1

Alternatively, you can use iSCSI LUNs as CRS configuration devices, and you can create slices on the iSCSI LUN using the format command, as you do on a physical disk. Please refer to Sun Cluster release documentation for Oracle RAC configuration for details.

b. Install Oracle CRS and the rdbms binaries on local disks. For this example, we'll use CRS_HOME=/install/oracle/crs and ORACLE_HOME=/install/oracle/10g.

c. Add crs_framework resource to rac-framework-rg after CRS binary installation completion.

# clrs create -t SUNW.crs_framework -g rac-framework-rg \ -p resource_dependencies=rac-framework-rs crs-framework-rs \ # clrs enable crs-framework-rs

d. Create Oracle listeners.

e. Create the Oracle RAC database. (We'll refer to this as testdb in this example.)5. Create and configure a scalable_rac_server_proxy resource group (rac-proxy-rg in this example). note: Adjust the variables to your configuration.

# clrt register SUNW.scalable_rac_server_proxy # clrg create -S -p rg_affinities=++rac-framework-rg rac-proxy-rg # clrc create -t SUNW.scalable_rac_server_proxy -g rac-proxy-rg \ -p Resource_dependencies=rac-framework-rs \ -p CRS_HOME=/install/oracle/crs -p DB_NAME=testdb \ -p ORACLE_HOME=/install/oracle/10g \ -p ORACLE_SID{paborn1}=testdb1 \ -p ORACLE_SID{paborn2}=testdb2 \



-p ORACLE_SID{paborn3}=testdb3 \ -p resource_dependencies_offline_restart=crs-framework-rs rac-proxy-rs

#clrg online -emM rac-proxy-rg

Author-Recommended ResourcesOracle Solaris Cluster on Oracle Technology NetworkOracle Solaris Cluster AdministrationOracle Solaris Cluster Data Service for Oracle Real Application Clusters GuideSun ZFS Storage 7000 System Administration GuideNote: Online help for the Sun Storage 7000 series products isavailable by clicking the Help button in the GUI

Revision 1, 06/10/2011



Documents

Sun ZFS Storage Appliance 7000 Systems