Upload
dangbao
View
250
Download
0
Embed Size (px)
Citation preview
Building Oracle Grid with Oracle
VM on Blade Servers and iSCSI
Storage
Kai Yu
Dell Oracle Solutions Engineering
• Kai Yu
– 15 years with Oracle technology, specialized in
Oracle Grid/RAC, Oracle Apps and Oracle VM
– Oracle ACE Director, Oracle papers author and
frequent presenter at OOW and Collaborate
– IOUG Oracle RAC SIG president
• Dell Oracle Solutions Engineering Lab
– Dell’s Oracle Technology solutions based on Dell
PE servers and Dell storages.
– The solutions stack: servers, storage, network,
OS, virtualization, Oracle RAC, Oracle Apps
About Author
2
Introduction to Oracle Grid
Grid Reference Configuration POC Project
Grid Implementation
Consolidate Applications on Grid
QA
3
Agenda
• Enterprise Grid Computing– Group independent hardware and software components in to a
single resource pool
– Can be provisioned, distributed and redistributed on demand
– Standardization, Virtualization, Automation
• Benefit of Grid Computing– Quickly meet the changing of business needs
– Dynamically allocate the resources for changing workloads
– Resource consolidation and sharing reduce the IT cost
– Providing Predictable service levels
• Key Technologies for Oracle Grid– Sever clustering: Real Applications Cluster (RAC)
– Server Virtualization: Oracle VM
– Storage Virtualization: Automatic Storage Management (ASM)
– Management and Provisioning: Oracle Enterprise Manager4
Introduction to Oracle Grid
5
Introduction to Oracle Grid
• Oracle Real Application Clusters– Multiple instances running on many hosts interconnected by
high speed private network share a single database
– Oracle Cache Fusion Technology for node synchronization
– All Instances can execute transactions simultaneously on the
single database in the shared storage
– Provide High Availability for the applications
– Scalability, special useful for commodity hardware.
• Automatic Storage Management (ASM)
– An integrated file system and volume manager
– Spread data evenly on disks; automatically rebalance
– Easy management of file system
– Provide shared storage for RAC nodes
• Oracle VM Technology
– Virtual Server: Xen Hypervisor, management domain dom0
Linux kernel with support of devices, IO, networking, etc.
– Virtual Server Pool: an autonomous region of VM servers
– VM: guest OS with applications running on domU
7
Introduction to Oracle Grid
• Enterprise Grid vs Cloud Computing– Dynamic shared infrastructure as the basis of Cloud computing
– Provide the key building blocks for cloud computing
– Help IT build private cloud and cloud-like infrastructure
• Engineering solution on Grid Computing – Provide customers a solution that builds cloud like services
• Complete stack of software and hardware platform
• Provide building, deploying and managing solutions
• Ensure customers the enterprise grade configuration
– Engineering Solution through the Oracle Grid Proof of Concept
project
9
Introduction to Oracle Grid
• Grid Reference POC Project – Engineering solution for Grid Architecture and implementation
Design/implement an enterprise grade Grid infrastructure
Dynamically provision multi-tier applications in a single Grid
Based on commodity hardware and SAN storage
• Grid Architecture– Physical Grid on clustering: 8 + nodes Oracle 11g RAC
– Virtual Grid: a set of virtual machines running on Oracle VM
servers based on Dell Blades
– Grid control as the unified management
• Grid Hardware Platform– Commodity Hardware: Dell 11g Blade Servers:
– Shared SAN storage: Dell EqualLogic iSCSI storage1
0
Grid Architecture Design
Grid Implementation
1
4
• Implementation Tasks– Configure Grid Control Management Infrastructure
– Configure EqualLogic Shared storage
– Configure Physical Grid
– Configure Virtual Grid
• Grid Control Management Configuration– Oracle Enterprise Linux OEL 4.7 64 bit
– Oracle Enterprise Manager Grid control 10.2.0.3
– Grid control 10.2.0.5 upgrade
– Enable Virtual Management Pack Restart OMS server
Grid Implementation
1
6
• EqualLogic shared storage configuration– Storage volumes for the Physical Grid
– Storage volumes for the Virtual Grid
Volume Size Raid Used for OS Mapping
blade_crs 2GB 10 OCR/Votingdisk Yes
blade_data1 100GB 10 Data for DB1 ASM diskgroup1
blade_data2 100GB 10 Data for DB2 ASM diskgroup2
blade_data3 150GB 10 Data for DB3 ASM diskgroup3
blade_data5 150GB 10 Data for DB4 ASM diskgroup5
Volume Size Raid Used for OS Mapping
blade_data4 400GB 10 VM Repositories /OVS
blade_data6 500GB 10 VM Repositories
/OVS/9A87460A7EDE4
3EE92201B8B7989DBA
6
vmcor1-5 1GB each 10 OCR/vogtingdisk 2XOCRs/3Xvotingdisks
vmracdb1 50GB 10 Data for RAC DB ASM diskgroup1
vmracdb2 50GB 10 Data for RAC DB ASM diskgroup2
Grid Implementation
1
8
• Physical Grid Configuration : 8-node 11g RAC
– Network configuration
– iSCSI storage connections
• Configure host access to storage: eth2 and eth3, /etc/iscsi/iscsid.config
• iSCSI interfaces: i-eth2 on eth2, i-eth3 on eth3:
iscsiadm -m iface -I i-eth2 --op=new
iscsiadm -m iface -I i-eth2 --op=update -n iface.hwaddress -v <hwadd>
iscsiadm -m discovery -t st -p 10.16.7.100 --interface=i-eth2 --interface=i-eth3
iscsiadm -m node -p 10.16.7.100 --interface i-eth3 --login
iscsiadm -m node -p 10.16.7.100 --interface i-eth2 –login
Check Partitions : more /proc/partitions:
Network Interface IO modules Connections IP address
eth0 A1 Public Network 155.16.9.71-78
eth2 B1 iSCSI connection 10.16.7.241-255(odd)
eth3 B2 iSCSI connection 10.16.7.240-254(odd)
eth4,eth5 C1,C2 Bonded to bond0 192.168.9.71-78
VIP Virtual IP for 11g RAC 155.16.9.171-178
Grid Implementation
1
9
• Find iSCSI ID: /sbin/scsi_id -gus /block/sdd
• Linux Device Mapper to establish mutipath devices to storage alias:
/etc/multipath.conf .
multipaths {
multipath {
wwid 36090a0287065124c45a2941c7fc76340
alias ocr_crsp14
}
…….
}
service multipathd restart
– Check multipath devices in /dev/mapper:
ocr-crsp1, data1, data2, data3, data5
Make partitions: fdisk
Grid Implementation
2
0
– Use block devices for 11g Clusterware and RAC
databases: OCRs, Votingdisks and ASM diskgroups.
– Configure 11g RAC database Infrastructure
• 11g RAC clusterware configuration:
private interconnect : bond0
OCRs and votingdisks on mutipath devices
• 11g ASM configuration:
ASM instances provide the storage virtualization
• 11g RAC software to provide the database services
• Grid control agent 10.2.0.5 to connect to OMS server
Grid Implementation
2
1
– Consolidate Databases
• Sizing the DB: Storage, # of DB instances, which nodes to run
• Provision the storage/ASM diskgroup
• Create the database
• For some ERP applications, convert the installed DB to RAC
Grid Implementation
2
2
– Scale out the physical Grid Infrastructure• Add more databases to the Grid
• Increase capability of the existing databases
• Scale out the storage:
– add additional EqualLogic array
– expand existing or create new volumes
– make new volume accessible to the servers
– add new ASM diskgroup or add disk to the existing diskgroup or resize the disk of an diskgroup
• Add new node to the cluster
• Dynamically move the database to less busy node:
Grid Implementation
2
3
• Implementation Tasks Overview
– Virtual servers Installation
– Virtual server network and storage configuration.
– Connect VM servers to the Grid Control
– Create guest VM using VM template
– Configure the VMs using VM Management Pack with Oracle Enterprise Manager
Virtual Server Installation
– Prepare local disk and enable virtualization on BIOS
– Install Oracle VM server
– Change Dom0 memory : /boot/grub/menu.lst:
edit line: kernel /xen-64bit.gz dom0_mem=1024M
– Ensure VM agent working: #service ovs-agent status
Grid Implementation
2
5
– Configure shared storages on dom0 • OVM repositories : blade_data4: 400GB,
• Shared disks for Oracle RAC on VMs
• Configure iSCSI and multipath devices on dom0
• /dev/mapper/ovs_data4p1
• Put OVM repository on blade_data4p1 (OVM 2.1)
mkfs.ocfs2 -b 4k -C 64k -L ovs /dev/mapper/ovs_data4p1
unmount /OVS, moun new OVS: mount -a -t ocfs2
edit /etc/fstab to include the new /OVS partiotion
create directories : iso_pool, proxy, running_pool,
seed_pool, sharedDisk under /OVS
Grid Implementation
2
6
Put OVM repository on blade_data4p1 (OVM 2.2)
service iptables stop; service o2cb configure,
mkfs.ocfs2 -Tdatafiles -N8 /dev/mapper/p1ovsdisk1p1
remove the local /OVS: /opt/ovs-agent-2.3/utils/repos.py -l
[ * ] 9e7012d5-69ba-4ac2-a58d-3400cd044efc => /dev/sda3
/opt/ovs-agent-2.3/utils/repos.py -d 9e7012d5-69ba-4ac2-a58d-3400cd044efc
create new repository on shared devices;
/opt/ovs-agent-2.3/utils/repos.py -n /dev/mapper/p1ovsdisk1p1[ NEW ] 567d581b-ec4a-4da8-94df-841dc0c5a390 => /dev/mapper/p1ovsdisk1p1
create the newly created repository the cluster root repository:
/opt/ovs-agent-2.3/utils/repos.py -r 567d581b-ec4a-4da8-94df-841dc0c5a390[ R ] 567d581b-ec4a-4da8-94df-841dc0c5a390 => /dev/mapper/p1ovsdisk1p1
Reboot all the VM servers
Grid Implementation
2
7
– VM Management tools:
• Oracle VM Manager
• Enterprise Manager: VM Management Pack
– Connect the VM servers to Enterprise Manager
Pre-requisites Steps: oracle user in oinstall group, oracle sshuser equivalence between dome0 and OMS server, Oracle user sudo privileges : /etc/sudoers
Grid Implementation
3
0
• Create Guest VMs using VM template
– Virtual Machine as node of Virtual Grid
– Create guest VM using VM templates
– Import a VM template: download template; Discover VM
template
Grid Implementation
3
2
• Prepare storage for VMs: virtual disks:From the Images files in repository:
• Local disk:
• Shared disk
• Disks as resources for VM:
Grid Implementation
3
4
• Disks shown in VM as the virtual disk partitions:
• Attached the storage to the guest VM a) vm.cfg: disk = [''file:/OVS/sharedDisk/racdb.img,xvdc,w!', b) vm.cfg: disk = ['‘phy: /dev/mapper/vmracdbp1, xvdc,w!',
• Virtual network interface through Xen bridgesvm.cfg:vif = ['bridge=xenbr0,mac=00:16:3E:11:8E:CE,type=netfront','bridge=xenbr1,mac=00:16:3E:50:63:25,type=netfront',]xenbr0 eth0 in the guest VMxenbr1 eth1 in the guest VM
3
6
Consolidate Applications
• Consolidate enterprise applications on the
Grid– Applications and middleware on the virtual Grid
• VMs Applications Application VM templates->New VMs
• Application template VMs Modification New application
template
• Import: existing VM: Copy VM image Import VM image
– Deploy database service on the physical Grid
• Provisiong Storage Attach storage to NodesASM
diskgroupDatabase
create Schema->connect to Database
– Deploy DEV/Test Application suite on the virtual Grid
• Multi-tier nodes are on the VMs
• Fast deployment based on templates
3
7
Consolidate Applications
• Grid Architecture to host applications
Oracle VM
Apps
1
Virtual Grid
Grid Structure and Applications
EqualLogic SAN
storage
Apps
2
ebsap
p1
Big_5
2vm2
nvmr
ac4
ebs2a
pp2
ebs2d
b1
11g RAC
Bann
1
Bann
2Banner RAC
DB
EBS R12 Test
Suite on VMs
Physical Grid
EBS RAC DB
EBS R12 Apps
Tier on VMs
11g RAC Databases Grid
Other Database
Other Database
VM server Pool
4
0
• Dell Grid POC: Oracle Grid on Dell Blades and EqualLogic
• Oracle Enterprise Manager Grid control for management
• Provides the infrastructure to consolidate the enterprise
applications and databases
• Acknowledgement: David Mar, Roger Lopez (Dell Oracle
Solutions Engineering), John Tao(Dell IT), Rajat
Nigam(Oracle)
• Related Dell Oracle Whitepapers:1) Building an Oracle® Grid with Oracle VM on Dell ™ Blade servers
and Dell EqualLogic iSCSI storage
2) Implementing Oracle® E-Business Suite in a Scalable and Reliable
Virtual System Architecture
3) Efficient Deployment of Oracle® Real Application Clusters (RAC) in a
Virtualized Environment Using Enterprise Manager Provisioning Pack
SUMMARY
References
4
1
1. Best Practices of Deployment of Oracle® ASM with Dell EqualLogic PS iSCSI Storage System, Dell White Paper
2 . Oracle Press Release: Oracle® Enterprise Manager Extends Management to Oracle VM Server Virtualization
3. Oracle® Enterprise Manager Concepts 10g Release 5 (10.2.0.5)Part Number B31949-10
4. Oracle Enterprise Manager Grid Control ReadMe for Linux x86-6410g Release 5 (10.2.0.5) April 2009
5. How to Enable Oracle VM Management Pack in EM Grid Control 10.2.0.5, Metalink Note #: 781879.1
6. Oracle VM: Converting from a Local to Shared OVS Repository, Metalink note # 756968.1
7. How to Add Shared Repositories to Oracle VM Pools with Multiple Servers, Metalink Note # 869430.1
8. Implementing Oracle Grid: A successful Customer Case Study, Kai Yu, Dan Brint, IOUG Select Journal Volume 16, Number 2, Second Quarter 2009
9. Deploying Oracle VM Release 2.1 on Dell PowewrEdge servers and Dell/EMC storage Dell white paper
10. Dell Reference Configuration Deploying Oracle® Database on Dell™ EqualLogic™ PS5000XV iSCSI Storage A Dell Technical White Paper
11. Technical Best Practices for Virtualization & RAC – Oracle RAC SIG webseminar slides, Michael Timpanaro-Perota & Daniel Dibbets
12. Kai Yu’s Oracle Blog