Virtualized 11gR2 Linux RAC on a Virtual Box Host Ahbaid Gaffoor Amazon.com/A9.com

Virtualized 11gR2 Linux RAC on a

Virtual Box Host

Ahbaid GaffoorAmazon.com/A9.com

Overview1. Single Node Configuration2. Shared Storage3. ASM 4. Cloning Nodes5. Installing Oracle RAC6. Verification

2

1. Single Node Configuration1. Required Packages2. Memory Configuration3. Network Configuration4. Kernel Configuration5. User Limits6. Security7. NTP8. User and Group Creation9. VBox Guest Additions

3

[root@n1 setup]# more packages.txtkernel-headers-2.6.18-194.el5.x86_64.rpmlibgomp-4.4.0-6.el5.x86_64.rpmbinutils-2.17.50.0.6-14.el5.x86_64.rpmcompat-libstdc++-33-3.2.3-61.x86_64.rpmelfutils-libelf-0.137-3.el5.x86_64.rpmelfutils-libelf-devel-0.137-3.el5.x86_64.rpmelfutils-libelf-devel-static-0.137-3.el5.x86_64.rpmgcc-4.1.2-48.el5.x86_64.rpmgcc-c++-4.1.2-48.el5.x86_64.rpmglibc-2.5-49.x86_64.rpmglibc-common-2.5-49.x86_64.rpmglibc-devel-2.5-49.x86_64.rpmglibc-headers-2.5-49.x86_64.rpmksh-20100202-1.el5.x86_64.rpmlibaio-0.3.106-5.x86_64.rpmlibaio-devel-0.3.106-5.x86_64.rpmlibgcc-4.1.2-48.el5.x86_64.rpmlibstdc++-4.1.2-48.el5.x86_64.rpmlibstdc++-devel-4.1.2-48.el5.x86_64.rpmmake-3.81-3.el5.x86_64.rpmsysstat-7.0.2-3.el5.x86_64.rpmunixODBC-2.2.11-7.1.x86_64.rpmunixODBC-devel-2.2.11-7.1.x86_64.rpm

Packages - List

1.1

REQ

UIR

ED P

ACKA

GES

[root@n1 setup]# more install_packages.shumount /dev/cdom 1>/dev/null 2>&1mount /dev/cdrom /media 1>/dev/null 2>&1export CD=/media/Server

for p in `cat packages.txt`; do echo "# $CD/$p" rpm -Uvh --nodeps $CD/$pdone

Packages – Install Script

[root@n1 ~]# df -h -t tmpfsFilesystem Size Used Avail Use% Mounted ontmpfs 2.0G 0 2.0G 0% /dev/shm

Automatic Memory Management and /dev/shm

Size /dev/shm for MEMORY_TARGET shared memory allocation of 3G[root@n1 ~]# umount tmpfs

[root@n1 ~]# df -h -t tmpfsFilesystem Size Used Avail Use% Mounted on

[root@n1 ~]# mount -t tmpfs shmfs -o size=3000m /dev/shm

[root@n1 ~]# df -h -t tmpfsFilesystem Size Used Avail Use% Mounted onshmfs 3.0G 0 3.0G 0% /dev/shm

[root@n1 ~]# cat /etc/fstab/dev/VolGroup00/LogVol00 / ext3 defaults 1 1LABEL=/boot /boot ext3 defaults 1 2#tmpfs /dev/shm tmpfs defaults 0 0tmpfs /dev/shm tmpfs size=3000m 0 0devpts /dev/pts devpts gid=5,mode=620 0 0sysfs /sys sysfs defaults 0 0proc /proc proc defaults 0 0/dev/VolGroup00/LogVol01 swap swap defaults 0 0

Edit /etc/fstab for persistence

1.2

MEM

ORY

CO

NFI

GU

RATI

ON

n1 199.1.1.61n2 199.1.1.62

Public IPs

VIP IPs (<= 11gR2)n1-vip 199.1.1.161n2-vip 199.1.1.162

Private IPsn1 10.0.0.61n2 10.0.0.62

SCAN (Single Client Access Name) IPs (>= 11gR2)n-scan 199.1.1.151n-scan 199.1.1.152

1.3

NET

WO

RK C

ON

FIG

URA

TIO

N

[root@n1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0# Intel Corporation 82540EM Gigabit Ethernet ControllerDEVICE=eth0BOOTPROTO=staticBROADCAST=199.1.1.255HWADDR= 08:00:27:0F:D3:AFIPADDR=199.1.1.61NETMASK=255.255.255.0NETWORK=199.1.1.0ONBOOT=yes

eth0 private interconnect configuration

n1 (& n2) /etc/hosts file:# Do not remove the following line, or various programs# that require network functionality will fail.127.0.0.1 localhost.localdomain localhost::1 localhost6.localdomain6 localhost6

# n1,n2 public interfaces199.1.1.61 n1.ilmtech.com n1199.1.1.62 n2.ilmtech.com n2

# n1,n2 internal interfaces10.0.0.61 n1-int.ilmtech.com n1-int10.0.0.62 n2-int.ilmtech.com n2-int

# n1,n2 vip interfaces199.1.1.161 n1-vip.ilmtech.com n1-vip199.1.1.162 n2-vip.ilmtech.com n2-vip

1.3

NET

WO

RK C

ON

FIG

URA

TIO

N

[root@n1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2# Intel Corporation 82540EM Gigabit Ethernet ControllerDEVICE=eth2BOOTPROTO=staticBROADCAST=10.0.0.255HWADDR=08:00:27:78:3E:63IPADDR=10.0.0.61NETMASK=255.255.255.0NETWORK=10.0.0.0ONBOOT=yes

eth2 private interconnect configuration

[root@n1 ~]# /etc/init.d/network startBringing up loopback interface: [ OK ]Bringing up interface eth0: [ OK ]Bringing up interface eth2: [ OK ]

[root@n1 ~]# ifconfig eth2eth2 Link encap:Ethernet HWaddr 08:00:27:78:3E:63 inet addr:10.0.0.61 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe78:3e63/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:115 errors:0 dropped:0 overruns:0 frame:0 TX packets:35 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:15079 (14.7 KiB) TX bytes:5792 (5.6 KiB)

[root@n1 ~]# ping -c 1 10.0.0.61PING 10.0.0.61 (10.0.0.61) 56(84) bytes of data.64 bytes from 10.0.0.61: icmp_seq=1 ttl=64 time=0.020 ms

--- 10.0.0.61 ping statistics ---1 packets transmitted, 1 received, 0% packet loss, time 0msrtt min/avg/max/mdev = 0.020/0.020/0.020/0.000 ms

1.3

NET

WO

RK C

ON

FIG

URA

TIO

N

Edit /etc/sysctl.conf# Oracle 11gR2 configuration# http://download.oracle.com/docs/cd/E11882_01/install.112/e16768/toc.htm#BHCCADGD

# concurrent outstanding requestsfs.aio-max-nr = 1048576

# filesfs.file-max = 6815744

# Shared Memorykernel.shmall = 2097152kernel.shmmax = 536870912kernel.shmmni = 4096

# Semaphores semmsl semmns semopm semmnikernel.sem = 250 32000 100 128

# Networknet.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 262144net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048586

# Huge Pages for Oracle group 501 (oracle user group)vm.hugetlb_shm_group=501 1.

4 KE

RNEL

CO

NFI

GU

RATI

ON

Apply/etc/sysctl.conf changes[root@n1 ~]# /sbin/sysctl -pnet.ipv4.ip_forward = 0net.ipv4.conf.default.rp_filter = 1net.ipv4.conf.default.accept_source_route = 0kernel.sysrq = 0kernel.core_uses_pid = 1net.ipv4.tcp_syncookies = 1kernel.msgmnb = 65536kernel.msgmax = 65536kernel.shmall = 4294967296fs.aio-max-nr = 1048576fs.file-max = 6815744kernel.shmall = 2097152kernel.shmmax = 536870912kernel.shmmni = 4096kernel.sem = 250 32000 100 128net.ipv4.ip_local_port_range = 9000 65500net.core.rmem_default = 262144net.core.rmem_max = 4194304net.core.wmem_default = 262144net.core.wmem_max = 1048586vm.hugetlb_shm_group = 501

1.4

KERN

EL C

ON

FIG

URA

TIO

N

Edit /etc/security/limits.conf# oracle limitsoracle soft nproc 2047oracle hard nproc 16384oracle soft nofile 1024oracle hard nofile 65536oracle soft stack 10240

1.5

USE

R LI

MIT

S

Edit /etc/pam.d/login# oraclesession required pam_limits.so

1.6

SECU

RITY

Edit /etc/selinux/config# SELINUX= can take one of these three values:# enforcing - SELinux security policy is enforced.# permissive - SELinux prints warnings instead of enforcing.# disabled - SELinux is fully disabled.SELINUX=disabled

Edit /etc/sysconfig/ntpd# -x sets the the slew/step threshold to 600 secondsOPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid“

1.7

NTP

Restart ntpd[root@n1 ~]# /etc/init.d/ntpd restartShutting down ntpd: [ OK ]ntpd: Synchronizing with time server: [ OK ]Starting ntpd: [ OK ]

Create dba and oinstall groups[root@n1 ~]# cat /etc/sysctl.conf | grep ^vm.hugetlb_shm_groupvm.hugetlb_shm_group=501[root@n1 ~]# groupadd -g 501 dba[root@n1 ~]# groupadd -g 502 oinstall

1.8

USE

R AN

D G

ROU

P CR

EATI

ON

Create oracle user[root@n1 oracle]# mkdir -p /u01/app

[root@n1 oracle]# useradd -m -d /u01/app/oracle -u 501 -g oinstall -G dba -s /bin/bash oracle

[root@n1 oracle]# passwd oracleChanging password for user oracle.New UNIX password:Retype new UNIX password:passwd: all authentication tokens updated successfully.

Edit /etc/sudoers# Allow oracle user full sudo accessoracle ALL=(ALL) ALL

Create grid and db directories (as oracle)[root@n1 oracle]# su - oracle[oracle@n1 ~]$ mkdir -p /u01/app/grid/11.2[oracle@n1 ~]$ mkdir -p /u01/app/oracle/product/db/11.2

Edit ~oracle/.bash_profile on n1 to include# Oracle Common Settingsexport ORACLE_BASE=/u01/app/oracleexport ORACLE_HOME=$ORACLE_BASE/product/db/11.2export ORACLE_BIN=$ORACLE_HOME/binexport ORACLE_UNQNAME=n

export GRID_HOME=/u01/app/grid/11.2export GRID_BIN=$GRID_HOME/bin

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib:$LD_LIBRARY_PATHexport CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$CLASSPATHexport PATH=$ORACLE_BIN:$PATH

export TMP=/tmpexport TMPDIR=$TMP

# Oracle host n1 settingsexport ORACLE_HOSTNAME=n1.ilmtech.comexport ORACLE_SID=n1

1.8

USE

R AN

D G

ROU

P CR

EATI

ON

Mount VirtualBox ISO image from HostD:\Program Files\Sun\VirtualBox\VBoxGuestAdditions.iso

Attach using Host Window: Menu -> Devices -> CD/DVD Devices -> More CD/DVD Images…

Mount as CD

1.9

Vbox

Gue

st A

dditi

onsMount and run VirtualBox Guest Additions from Guest

[root@n1 ~]# df -h | grep media/dev/hdc 33M 33M 0 100% /media/VBOXADDITIONS_3.2.8_64453

[root@n1 ~]# cd /media/VBOXADDITIONS_3.2.8_64453/

[root@n1 VBOXADDITIONS_3.2.8_64453]# sh ./VBoxLinuxAdditions-amd64.run

Why?• Mouse Pointer Integration• Improved Video Resolution• Seamless windows• Time Synchronization• Shared Clipboard• Automated Logins

2. Shared Storage1. Shared Disk Creation 2. Attaching Shared Disks3. Partitioning Shared Disks

17

VBoxManage.exeD:\Program Files\Sun\VirtualBox\VBoxManage.exeShared Disks new in 3.2.8 - http://forums.virtualbox.org/viewtopic.php?f=1&t=33517

2.1

SHAR

ED D

ISK

CREA

TIO

N

Create Shared DisksVBoxManage.exe createhd --filename rac-n-asm01-vdi.disk –size 10240 --format VDI --variant Fixed --type shareable –remember..VBoxManage.exe createhd --filename rac-n-asm05-vdi.disk –size 10240 --format VDI --variant Fixed --type shareable –remember

C:\>VBoxManage.exe createhd --filename Q:\VirtualBox\SSD\rac\n\shared\rac-n-asm01-vdi.disk --size 10240 --format VDI --variant Fixed --type shareable --rememberOracle VM VirtualBox Command Line Management Interface Version 3.2.8(C) 2005-2010 Oracle CorporationAll rights reserved.

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%Disk image created. UUID: d18e5c34-6604-464f-a495-e9aaa925697c

http://forums.virtualbox.org/viewtopic.php?f=1&t=33517

2.2

ATTA

CHIN

G S

HAR

ED D

ISKS

Attach Shared Disks to n1VBoxManage.exe storageattach n1 --storagectl “SATA Controller” --port 1 --device 0 --type hdd --medium rac-n-asm01-vdi.disk..VBoxManage.exe storageattach n1 --storagectl “SATA Controller” --port 5 --device 0 --type hdd --medium rac-n-asm05-vdi.disk

C:\>VBoxManage.exe storageattach n1 --storagectl "SATA Controller" --port 1 --device 0 --type hdd --medium Q:\VirtualBox\SSD\rac\n\shared\rac-n-asm01-vdi.diskOracle VM VirtualBox Command Line Management Interface Version 3.2.8(C) 2005-2010 Oracle CorporationAll rights reserved.

Find scsi Devices (SATA Controller)[root@n1 scsi]# cat /proc/scsi/scsiAttached devices:Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: ATA Model: VBOX HARDDISK Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 05Host: scsi1 Channel: 00 Id: 00 Lun: 00 Vendor: ATA Model: VBOX HARDDISK Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 05Host: scsi2 Channel: 00 Id: 00 Lun: 00 Vendor: ATA Model: VBOX HARDDISK Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 05Host: scsi3 Channel: 00 Id: 00 Lun: 00 Vendor: ATA Model: VBOX HARDDISK Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 05Host: scsi4 Channel: 00 Id: 00 Lun: 00 Vendor: ATA Model: VBOX HARDDISK Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 05Host: scsi5 Channel: 00 Id: 00 Lun: 00 Vendor: ATA Model: VBOX HARDDISK Rev: 1.0 Type: Direct-Access ANSI SCSI revision: 05

2.3

SHAR

ED D

ISK

PART

ITIO

NIN

G

scsi0

[root@n1 scsi]# fdisk -l /dev/sda

Disk /dev/sda: 42.9 GB, 42949672960 bytes255 heads, 63 sectors/track, 5221 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System/dev/sda1 * 1 13 104391 83 Linux/dev/sda2 14 5221 41833260 8e Linux LVM

2.3

SHAR

ED D

ISK

PART

ITIO

NIN

G

Create Partitions on /dev/sdb (scsi1) .. /dev/sdf (scsi5)[root@n1 scsi]# parted /dev/sdb mklabel gptInformation: Don't forget to update /etc/fstab, if necessary.

[root@n1 scsi]# parted /dev/sdb printModel: ATA VBOX HARDDISK (scsi)Disk /dev/sdb: 10.7GBSector size (logical/physical): 512B/512BPartition Table: gpt

Number Start End Size File system Name Flags

Information: Don't forget to update /etc/fstab, if necessary.

[root@n1 scsi]# parted /dev/sdb mkpart primary 0 10700Information: Don't forget to update /etc/fstab, if necessary.

[root@n1 scsi]# parted /dev/sdd printModel: ATA VBOX HARDDISK (scsi)Disk /dev/sdb: 10.7GBSector size (logical/physical): 512B/512BPartition Table: gpt

Number Start End Size File system Name Flags 1 17.4kB 10.7GB 10.7GB primary

Information: Don't forget to update /etc/fstab, if necessary.

10.7GB

parted takes size in MB, 10.7GB is 10700MB

3. ASM1. ASM Lib Installation2. ASM Lib Configuration3. ASM Disks

22

ASM Lib InstallationKernel Version[root@n1 ~]# echo ùname -r`.ùname -i`2.6.18-194.el5.x86_64

ASM Lib files included with OELR5U5 under $CD/Server[root@n1 ~]# ls /media/Server/oracleasm*/media/Server/oracleasm-2.6.18-194.el5-2.0.5-1.el5.x86_64.rpm/media/Server/oracleasm-2.6.18-194.el5debug-2.0.5-1.el5.x86_64.rpm/media/Server/oracleasm-2.6.18-194.el5xen-2.0.5-1.el5.x86_64.rpm/media/Server/oracleasm-support-2.1.3-1.el5.x86_64.rpm

Oracle ASM lib available at: http://www.oracle.com/technetwork/topics/linux/downloads/rhel5-084877.htmloracleasmlib-2.0.4-1.el5.x86_64.rpm

[root@n1 ASM]# rpm -ivh oracleasm-support-2.1.3-1.el5.x86_64.rpmwarning: oracleasm-support-2.1.3-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159Preparing... ########################################### [100%] 1:oracleasm-support ########################################### [100%]

[root@n1 ASM]# rpm -ivh oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpmwarning: oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159Preparing... ########################################### [100%] 1:oracleasm-2.6.18-164.el########################################### [100%]

[root@n1 ASM]# rpm -ivh oracleasmlib-2.0.4-1.el5.x86_64.rpmwarning: oracleasmlib-2.0.4-1.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159Preparing... ########################################### [100%] 1:oracleasmlib ########################################### [100%]

3.1

ASM

Lib

INST

ALLA

TIO

N

http://www.oracle.com/technetwork/topics/linux/downloads/rhel5-084877.html

ASM Lib Configuration[root@n1 ~]# /usr/sbin/oracleasm configure -iConfiguring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM librarydriver. The following questions will determine whether the driver isloaded on boot and what permissions it will have. The current valueswill be shown in brackets ('[]'). Hitting <ENTER> without typing ananswer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracleDefault group to own the driver interface []: dbaStart Oracle ASM library driver on boot (y/n) [n]: yScan for Oracle ASM disks on boot (y/n) [y]: yWriting Oracle ASM library driver configuration: done

3.2

ASM

Lib

CO

NFI

GU

RATI

ON

Load oracleasm module[root@n1 ~]# /usr/sbin/oracleasm initLoading module "oracleasm": oracleasmMounting ASMlib driver filesystem: /dev/oracleasm

Verify oracleasm module[root@n1 ~]# /usr/sbin/oracleasm statusChecking if ASM is loaded: yesChecking if /dev/oracleasm is mounted: yes

Mark ASM Disk Header /dev/sdb .. /dev/sdf[root@n1 ~]# which oracleasm/usr/sbin/oracleasm

[root@n1 ~]# oracleasm createdisk DISK1 /dev/sdb1Writing disk header: doneInstantiating disk: done

[root@n1 ~]# oracleasm createdisk DISK2 /dev/sdc1Writing disk header: doneInstantiating disk: done

[root@n1 ~]# oracleasm createdisk DISK3 /dev/sdd1Writing disk header: doneInstantiating disk: done

[root@n1 ~]# oracleasm createdisk DISK4 /dev/sde1Writing disk header: doneInstantiating disk: done

[root@n1 ~]# oracleasm createdisk DISK5 /dev/sdf1Writing disk header: doneInstantiating disk: done 3.

3 AS

M D

ISKS

List ASM Disks[root@n1 ~]# oracleasm listdisksDISK1DISK2DISK3DISK4DISK5

Oracle ASM and Devices• ASM Disks can show up with different device paths• Can use udev to control device naming• Oracle ASM scans for disk header tag of “ORCLDISK”• Oracle ASM scans for disk header label of “DISKn”•Another node can mount disks on different devices and have ASM detect drives correctly

3.3

ASM

DIS

KS

ASM Disk Header (tag and label)[root@n1 ~]# dd if=/dev/sdb1 bs=128 count=1 | od --format=a1+0 records in1+0 records out128 bytes (128 B) copied, 0.011425 seconds, 11.2 kB/s0000000 nul nul nul nul nul nul nul nul nul nul nul nul u I S K0000020 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul0000040 O R C L D I S K D I S K 1 nul nul nul0000060 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul*0000200

Tag: ORCLDISK

Label: DISK1

Non ASM Disk Header[root@n1 ~]# dd if=/dev/sda1 bs=128 count=1 | od --format=a1+0 records in1+0 records out128 bytes (128 B) copied, 1.3e-05 seconds, 9.8 MB/s0000000 nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul nul*0000200

Oracle ASM Disk Devices[root@n1 ~]# cd /dev/oracleasm/disks

[root@n1 disks]# ls -altotal 0drwxr-xr-x 1 root root 0 Sep 15 17:54 .drwxr-xr-x 4 root root 0 Sep 15 17:54 ..brw-rw---- 1 oracle dba 8, 17 Sep 15 18:02 DISK1brw-rw---- 1 oracle dba 8, 33 Sep 15 18:02 DISK2brw-rw---- 1 oracle dba 8, 49 Sep 15 18:02 DISK3brw-rw---- 1 oracle dba 8, 65 Sep 15 18:02 DISK4brw-rw---- 1 oracle dba 8, 81 Sep 15 18:02 DISK5

3.3

ASM

DIS

KSMajor device number 8:

http://www.kernel.org/pub/linux/docs/device-list/devices.txt

8 block SCSI disk devices (0-15) 0 = /dev/sda First SCSI disk whole disk 16 = /dev/sdb Second SCSI disk whole disk 32 = /dev/sdc Third SCSI disk whole disk ... 240 = /dev/sdp Sixteenth SCSI disk whole disk

http://www.kernel.org/pub/linux/docs/device-list/devices.txt

4. Cloning Nodes1. Cloning n1 (non-ASM) Disks2. Create n2 Guest3. Change Hostname4. Change Network Addresses5. Attach Shared Storage6. Verify ASM Disks on n1 & n2

28

Shutdown n1[root@n1 ~]# shutdown –h now

4.1

CLO

NIN

G n

1 (n

on-A

SM) D

ISKS

Clone n1 non-ASM hard disk n1-disk0.vdi to n2-disk0.vdiVBoxManage.exe clonehd <sourcefile> <targetfile> –remember

C:\>VBoxManage.exe clonehd "X:\Virtual Machines\virtualbox\11gRAC\n1-disk0.vdi" "X:\Virtual Machines\virtualbox\11gRAC\n2-disk0.vdi" --rememberOracle VM VirtualBox Command Line Management Interface Version 3.2.8(C) 2005-2010 Oracle CorporationAll rights reserved.

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%Clone hard disk created in format 'VDI'. UUID: 0b80e4d8-e8f7-4ccb-96fb-3d5769d45b53

Create n2 Guest• Same as n2• Use cloned hard disk n2-disk0.vdi• Same networking interfaces eth0, eth1: bridged and eth2 : host only

4.2

CREA

TE n

2 G

UES

T (u

sing

clo

ned

disk

)

Create n2 Guest

4.2

CREA

TE n

2 G

UES

T (u

sing

clo

ned

disk

)

Create n2 Guest

4.2

CREA

TE n

2 G

UES

T (u

sing

clo

ned

disk

)

4.4

CHAN

GE

NET

WO

RK A

DD

RESS

ES

Edit ifcfg-eth0 network script[root@n1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0# Intel Corporation 82540EM Gigabit Ethernet ControllerDEVICE=eth0BOOTPROTO=staticBROADCAST=199.1.1.255HWADDR=08:00:27:09:0f:7dIPADDR=199.1.1.62NETMASK=255.255.255.0NETWORK=199.1.1.0ONBOOT=yes

Edit ifcfg-eth2 network script[root@n1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth2# Intel Corporation 82540EM Gigabit Ethernet ControllerDEVICE=eth2BOOTPROTO=staticBROADCAST=10.0.0.255HWADDR=08:00:27:1b:07:eeIPADDR=10.0.0.62NETMASK=255.255.255.0NETWORK=10.0.0.0ONBOOT=yes

Reboot n2:[root@n1 ~]# shutdown –r now

Hostname now n2:[root@n2 ~]# hostnamen2.ilmtech.com

4.5

ATTA

CH S

HAR

ED S

TORA

GE

(to

n2)



4.5

ATTA

CH S

HAR

ED S

TORA

GE

(to

n2)



Verify Oracle ASM Disk Devices on n1 and n2:[root@n1 ~]# oracleasm listdisksDISK1DISK2DISK3DISK4DISK5

4.6

VERI

FY A

SM D

ISKS

[root@n2 ~]# oracleasm listdisksDISK1DISK2DISK3DISK4DISK5

5. Installing Oracle RAC1. Environment2. Cluster Verification3. Grid Infrastructure4. Database Binaries5. Database Creation – DBCA6. Verification

37

5.1

Envi

ronm

ent

Environment Variables[oracle@n1 ~]$ set | grep ÔRACLEORACLE_BASE=/u01/app/oracleORACLE_BIN=/u01/app/oracle/product/db/11.2/binORACLE_HOME=/u01/app/oracle/product/db/11.2ORACLE_HOSTNAME=n1.ilmtech.comORACLE_SID=n1ORACLE_UNQNAME=n

[oracle@n2 ~]$ set | grep ÔRACLEORACLE_BASE=/u01/app/oracleORACLE_BIN=/u01/app/oracle/product/db/11.2/binORACLE_HOME=/u01/app/oracle/product/db/11.2ORACLE_HOSTNAME=n2.ilmtech.comORACLE_SID=n2ORACLE_UNQNAME=n

SSH equivalence[oracle@n1 ~]$ ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (//u01/app/oracle/.ssh/id_rsa):Created directory '/u01/app/oracle/.ssh'.Enter passphrase (empty for no passphrase):Enter same passphrase again:Your identification has been saved in /u01/app/oracle/.ssh/id_rsa.Your public key has been saved in /u01/app/oracle/.ssh/id_rsa.pub.The key fingerprint is:fb:01:95:10:de:55:fd:9d:ae:cf:c7:ed:9a:e6:01:59 [email protected]

[oracle@n2 ~]$ cd ~/.ssh/[oracle@n2 .ssh]$ cp id_rsa.pub authorized_keys[oracle@n2 .ssh]$ tar -cvf s.tar *authorized_keysid_rsaid_rsa.pub

# scp s.tar to n1, create ~oracle/.ssh with same privs, untar s.tar there

5.2

Clus

ter V

erifi

catio

n

Cluster Verification – runcluvfy.sh (Grid Infrastructure Install Directory)[oracle@n1 grid]$ pwd/u01/app/oracle/installer/grid

[oracle@n1 grid]$ ./runcluvfy.sh stage -pre crsinst -n n1,n2 -verbose 1>runcluvfy.log 2>&1

[oracle@n1 grid]$ cat runcluvfy.log | grep -i failed n2 missing glibc-devel-2.5 (i386) failed n1 missing glibc-devel-2.5 (i386) failedResult: Package existence check failed for "glibc-devel-2.5 (i386)"

[oracle@n1 grid]$ sudo rpm -qa | grep glibc-devel[sudo] password for oracle:glibc-devel-2.5-49

[oracle@n1 grid]$ ssh root@n2 rpm -qa | grep glibc-develglibc-devel-2.5-49

[oracle@n1 grid]$ cat runcluvfy.log | grep -i ^CheckingChecking node reachability...Checking user equivalence...Checking node connectivity...Checking hosts config file...Checking for multiple users with UID value 0Checking Core file name pattern consistency...Checking to make sure user "oracle" is not in "root" groupChecking daemon liveness...Checking NTP daemon command line for slewing option "-x"Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option "-x"Checking on nodes "[n2, n1]"...…..(Full listing in Appendix A)…..

Cluster Verification – runcluvfy.sh (some checks)

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n

Grid Infrastrucutre - Installation

[oracle@n1 grid]$ pwd/u01/app/oracle/installer/grid

[oracle@n1 grid]$ ./runInstaller

• Run from Grid Infrastructure Installer directory• Both n1 and n2 are up• ssh equivalence in place (runcluvfy.sh)

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• “Install and Configure Grid Infrastructure for a Cluster”

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• “Typical Installation”

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• SCAN Name: n-scan (DNS entry)• Can use /etc/hosts but ignore errors on n-scan lookups

[oracle@n1 grid]$ dig +short n-scan.ilmtech.com199.1.1.151199.1.1.152

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Add node n2

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Set SCAN Name: “n-scan”

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Specify Public and Private network interfaces

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Oracle Base for Grid Infrastructure• Grid Home• Cluster Registry Type – ASM• Password – Complex! (Oracl3ASM in my example)• OSASM Group - dba

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Redundancy – External• Candidate Disks (all ASM disks)

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Grid Inventory Location

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Prerequisite Checks (Ignoring glibc-devel-2.5)

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Review Summary

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Installing….

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Installing….

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Root scripts on n1 and n2

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Root scripts on n1 and n2• /u01/app/oraInventory/orainstRoot.sh

[root@n1 ~]# /u01/app/oraInventory/orainstRoot.shChanging permissions of /u01/app/oraInventory.Adding read,write permissions for group.Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.The execution of the script is complete.

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Root scripts on n1 and n2• /u01/app/11.2.0/grid/root.sh

[root@n1 ~]# /u01/app/11.2.0/grid/root.shRunning Oracle 11g root.sh script...

The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]: Copying dbhome to /usr/local/bin ... Copying oraenv to /usr/local/bin ... Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...Entries will be added to the /etc/oratab file as needed byDatabase Configuration Assistant when a database is createdFinished running generic part of root.sh script.……..(Full listing is in Appendix B)……..

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• After running scripts, installation continues

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Installation Successful

5.3

Grid

Infr

astr

uctu

re In

stal

latio

n


• Installation Done

5.4

Dat

abas

e Bi

narie

s

Database Binaries

[oracle@n1 grid]$ pwd/u01/app/oracle/installer/database

[oracle@n1 grid]$ ./runInstaller

• Run from Database Installer directory• Both n1 and n2 are up• ssh equivalence in place

• Uncheck Oracle Support Security Updates• Accept warning

5.4

Dat

abas

e Bi

narie

s

Database Binaries

• Choose “Create and configure a database”

5.4

Dat

abas

e Bi

narie

s

Database Binaries

• Choose “Server Class”

5.4

Dat

abas

e Bi

narie

s

Database Binaries

• Choose “Real Application Clusters database installation”• Ensure both nodes n1 & n2 are checked

5.4

Dat

abas

e Bi

narie

s

Database Binaries

• Choose “Typical install”

5.4

Dat

abas

e Bi

narie

s

Database Binaries

• Storage – ASM• Password Complexity (Oracl3ASM used in example)• Global database nane – n.<domain>

5.4

Dat

abas

e Bi

narie

s

Database Binaries

• Prerequisite Checks• Ignoring glibc-devel-2.5 (verified it’s installed)

5.4

Dat

abas

e Bi

narie

s

Database Binaries

• Summary

5.4

Dat

abas

e Bi

narie

s

Database Binaries

• Installing…..

5.4

Dat

abas

e Bi

narie

s

Database Binaries

• Installing…..

5.4

Dat

abas

e Bi

narie

s

Database Binaries

Database Binaries

• Done (skipped DBCA)

5.4

Dat

abas

e Bi

narie

s

5.5

Dat

abas

e Cr

eatio

n

Database Creation - DBCA

[oracle@n1 11.2]$ pwd/u01/app/oracle/product/db/11.2

[oracle@n1 11.2]$ ./dbca

• Run from Database Home• Both n1 and n2 are up• ssh equivalence in place

5.5

Dat

abas

e Cr

eatio

n


• Choose “Oracle Real Application Clusters database”

5.5

Dat

abas

e Cr

eatio

n


• “Create a Database”

5.5

Dat

abas

e Cr

eatio

n


• “General Purpose”

5.5

Dat

abas

e Cr

eatio

n


• Cluster database configuration

5.5

Dat

abas

e Cr

eatio

n


• No Enterprise Manager

5.5

Dat

abas

e Cr

eatio

n


• Passwords same • Oracl3ASM to be consistent in this example

5.5

Dat

abas

e Cr

eatio

n


• Storage Type ASM• Use OMF in +DATA (ASM)

5.5

Dat

abas

e Cr

eatio

n


• Use Flash Recovery• Use +DATA (ASM) for Flash Recovery Area

5.5

Dat

abas

e Cr

eatio

n


• Skip Sample Schemas• No Custom Scripts

5.5

Dat

abas

e Cr

eatio

n


• Memory, Sizing, Char Sets, Connection Mode• Accept Defaults (Guest VM has enough RAM allocated)

5.5

Dat

abas

e Cr

eatio

n


• Storage in +DATA (ASM)

5.5

Dat

abas

e Cr

eatio

n


• Create Database• Save and review the scripts

5.5

Dat

abas

e Cr

eatio

n


• Scripts generating• Ok on HTML dialog that comes next

5.5

Dat

abas

e Cr

eatio

n


• Installing

5.5

Dat

abas

e Cr

eatio

n


• Installing…..

5.5

Dat

abas

e Cr

eatio

n


• Done

5.6

Verifi

catio

n

Verification

• Instances• Database

[oracle@n1 ~]$ ps -ef|grep pmon|grep -v greporacle 3569 1 0 04:20 ? 00:00:00 asm_pmon_+ASM1oracle 6185 1 0 04:34 ? 00:00:00 ora_pmon_n1

[oracle@n1 ~]$ srvctl config databasen

[oracle@n2 ~]$ ps -ef|grep pmon|grep -v greporacle 3722 1 0 04:20 ? 00:00:00 asm_pmon_+ASM2oracle 6047 1 0 04:35 ? 00:00:00 ora_pmon_n2

[oracle@n2 ~]$ srvctl config databasen

[oracle@n2 ~]$ srvctl config database –d nDatabase unique name: nDatabase name: nOracle home: /u01/app/oracle/product/db/11.2Oracle user: oracleSpfile: +DATA/n/spfilen.oraDomain: ilmtech.comStart options: openStop options: immediateDatabase role: PRIMARYManagement policy: AUTOMATICServer pools: nDatabase instances: n1,n2Disk Groups: DATAServices:Database is administrator managed

5.6

Verifi

catio

n

Verification

• Query instances[oracle@n2 ~]$ sqlplus "/ as sysdba"

SQL*Plus: Release 11.2.0.1.0 Production on Thu Nov 11 05:04:32 2010

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to:Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit ProductionWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,Data Mining and Real Application Testing options

SQL> select * from v$active_instances;

INST_NUMBER INST_NAME----------- ------------------------------------------------------------ 1 n1.ilmtech.com:n1 2 n2.ilmtech.com:n2

References

• Web:• http://www.oracle-base.com/articles/rac/ArticlesRac.php#11gR2• http://tahiti.oracle.com documentation

• Books:• Linux Recipes for Oracle DBAs by Kuhn, Kim and Lopez (Apress)• Oracle Automatic Storage Management by Vengurlekar, Vallath,

Long (Oracle Press)• DNS and Bind by Liu and Albitz (O’Reilly)

Refe

renc

es

http://tahiti.oracle.com/

Appe

ndix

A :

runc

luvf

y.lo

g

./runcluvfy.sh stage -pre crsinst -n n1,n2 -verbose 1>runcluvfy.log 2>&1

runcluvfy.log

Appe

ndix

B :

root

.log

[root@n1 ~]# /u01/app/11.2.0/grid/root.sh

root.log

Documents

Virtualized 11gR2 Linux RAC on a Virtual Box Host Ahbaid Gaffoor Amazon.com/A9.com