Session ID 2853 1

Presented by: Brent Maynard, Myke WattsMarshall University

April 9, 2013

Session ID 2853

Banner Virtualization

How Virtualization adds flexibility to our production environment, and enables new opportunities

Session ID 2853 2

Session Rules of Etiquette

• Please turn off your cell phone/pager

• If you must leave the session early, please do so as discreetly as possible

• Please avoid side conversation during the session

Thank you for your cooperation!

Session ID 2853 3

Introduction

• In this session, we hope to show you the benefits we have gained by moving from a physical 32bit architecture to a virtualized 64bit architecture.

Session ID 2853 4

Agenda

• Where we started• Why a virtualized Environment• 64bit benefits• Migration – Source code and database• Infrastructure • Cloning• Performance, Reliability, and Disaster

Planning

Session ID 2853 5

Where we started

.

Session ID 2853 6

Where we are – all virtualized

• Two transactional database servers• Two ODS/EDW database servers• Four Banner instances• Four Middleware servers (INB, SSB, etc.)• High availability• Increased performance• Business continuity

Session ID 2853 7

Where we are – all virtualized

vSphere 10 node Cluster

Session ID 2853 8

Where we started

SSB INB

BANNER

BANAUX

BANNER

BANPROD

BANNER

BANSTAGE

PHYSICAL (32 bit)

ODS/EDW ODS/EDW

warehouse warehouse

COGNOSARGOS

COGNOSARGOS

Session ID 2853 9

Background – 32bit Banner

• Redhat Enterprise RHEL5 32 bit• Dell R710 dual socket quad-core with

12GBytes memory• Dell Equallogic 6510x storage array• 1.5 TBytes allocated storage• Oracle 10g Database and Application

Services• Typically we have ~250 concurrent Banner

Forms users

Session ID 2853 10

Background - continued

• Due to multi-core expansion over the past few years, we decided to virtualize our INB/SSB Oracle Application Services because of Oracle OAS licensing constraints 18 months ago.

VMware ESX hostDual socket, quad core

MUSSBDell dual

socket dual core

MUINBDell dual

socket dual core

Before

Vmware Client

MUSSB4 cpus

Now

Vmware Client

MUINB4 cpus

Session ID 2853 11

Background - continued

• We have migrated to the 11g Oracle Fusion infrastructure• Hopefully by the time of this presentation, Java 7 support.

OFM Classic DomainINB/SSB/DWproduction

OFM Basic DomainBEP/WF/BRM

production

OFM Classic DomainINB/SSB/DW

preprod

OFM Basic DomainBEP/WF/BRM

preprod

ESX Host

Session ID 2853 12

Why Server Virtualization?

.

Session ID 2853 13

Why did Marshall University Virtualize?

• $$$$$$– Reduced server expenditures– Reduced power requirements– Reduced cooling requirements– Service availability– Flexibility

Session ID 2853 14

Why did Marshall University Virtualize?

• At Marshall, the majority of our core services are now running in the ESX environment.– Exchange– Sharepoint– Web (www.marshall.edu)– Banner (Workflow, BRM, DegreeWorks,

ODS/EDW)

Session ID 2853 15

64 bit Benefits

.

Session ID 2853 16

Why 64bit?

• RHEL5 removed highmem constructs that RHEL4 had allowed us to use to grant Oracle large chunks of memory.

• RHEL5 32bit caused us some issues when our pool space was consumed during heavy use periods.

• With 64bit linux and Oracle, we can now grant Oracle large chunks of memory (7.5 Gbytes) .

Session ID 2853 17

Migration from 32bit to 64bit

.

Session ID 2853 18

Migration

• Install Oracle 11gR2 64 bit DB server• makefiles• Cobol• library paths• rsync

Session ID 2853 19

Migration – 11g database

• Install Oracle 11gR2 Enterprise Edition• Allow Oracle Support to apply the latest

patch set• Configure to connect to Grid Control• Confirm the latest CPU patch was applied• Allow the installer to create a test

database• Install Oracle 11g Client (full client)

Session ID 2853 20

Migration – makefiles

• Download the Banner 8 makefiles from Ellucian and choose the file that is specific to your OS

• FAQ CMS-1361 Banner Example Makefiles

Session ID 2853 21

Migration – Cobol

• The Pro*Cobol pre-compiler is not included with the Oracle 11g Enterprise Edition install. It is included with the 11g Client and must be installed separately

• $COBOL=/opt/FJSVcbl• $COBOL=/opt/FJSVcbl64

Session ID 2853 22

Migration – rsync

• To move the file system from the 32bit environment to the 64 bit environment, we used the linux rsync command.

• It takes approximately 4 hours to sync the file system.

• rsync benefits– Maintains file dates– Maintains file permissions and ownership

• File system ACL’s do NOT migrate, so those will need to be applied manually if you have any set.

Session ID 2853 23

Migration – Database

• Create new production database (empty)• Oracle export/import• Duration• Post import cleanup• BAN_ roles replaced by USR_ roles• Database ACL’s (workflow, smtp, http)

Session ID 2853 24

Migration – Database

• Use DBCA to create a new Banner database (general purpose)

• Give it the same DB name as the previous Banner database

• Configure the new database to connect to Enterprise Manager Grid Control

• Configure a listener

• Set appropriate locations for datafiles, logfiles, control files

• Choose the correct Unicode character set AL32UTF8.

Session ID 2853 25

Migration – Database

• The Oracle 11g Grid Control does not support database cloning from 10g 32bit to 11g 64bit.

• Use either the Oracle import/export or datapump utilities.

• DegreeWorks issues– Even though the export was successful with no errors, the

subsequent import has missing DegreeWorks objects.

– We had to manually create the DDL to recreate the missing DegreeWorks objects.

Session ID 2853 26

Migration – Database

• For our instance, it took approximately 11 hours to export the database, copy the export file to the new server, and import the file into the new database.

• (Fee Assessment Audit Table)

Session ID 2853 27

Migration – Database

• Oracle 10.2.0.5 and 11g no longer allows password protected roles such as BAN_DEFAULT_M to be granted as default roles

• BAN_DEFAULT roles are replaced by USR_DEFAULT roles

• Users in your current database with BAN_DEFAULT roles will have to be granted the corresponding USR_DEFAULT role

Session ID 2853 28

Migration – Database

• Ellucian supplies three scripts to aid in the role conversion

gcnvconnectrole.sql

gcnvconnectrole_ban_default_m.sql

gcnvconnectrole_ban_default_q.sql

• These scripts will create the new roles and apply them to users either by adding the new role or replacing the existing role.

• Ellucian FAQ’s:1-XFTOI5

1-H4TMPD

Session ID 2853 29

Migration – Database

• Oracle 11g requires ACL’s to be added for DegreeWorks, Workflow, and any custom apps that perform HTTP or SMTP calls from the database. FAQ# 1-648TQ0

• ACL_and_Banner_11g_database.pdf

Session ID 2853 30

Infrastructure Environment

.

Session ID 2853 31

Infrastructure

• We have an excellent infrastructure team!• VMware templates have been created.• Flexible storage solutions using the Dell

Equallogic storage array.• OPSCODE Chef provisioning services

– Server configuration (iptables, users, httpd, profiles, groups, etc.)

– Configuration management

• A 64bit RHEL5/6 server can be provisioned, configured, and booted in around 15 minutes.

Session ID 2853 32

Infrastructure – Grid control

• We use Oracle 11g Grid Control to manage our Oracle database infrastructure.– Backups– Alert Monitoring– Database Cloning– Performance Monitoring– Regular Maintenance

Session ID 2853 33

Infrastructure – Grid control

Session ID 2853 34

Infrastructure – Grid control

• For our Banner Environment we have multiple instances running.– BANPROD– BANAUX1 Clone of production for Research Corporation testing– BANAUX2 Clone of production for University testing.– BANSTAGE Staging environment for new releases

• BANAUX1 and BANAUX2 share a common code tree

Session ID 2853 35

VMware – current versions

• vSphere Software– Vmware vCenter Server v5– Vmware ESXi v5 Hypervisor

• vSphere Features– vSphere HA – clustering ability to hot migrate VM

from one physical server to another– vSphere DRS – ability to balance performance

across cluster

• vCenter Site Recovery Manager

Session ID 2853 36

VMware Hardware Architecture

• 10 x Dell r710 Aggregate Specs– 108 CPU (287 GHz, mix of Nahalem and Westmere) – 1.03 TB 1333Mhz Memory

• 48 TB iSCSI storage (3 tiers)– Equallogic 6010x Hybrid SSD/SAS 15k 10GBe (8 x SSD, 8 x 15k SAS)– Equallogic 6510x 10k 10Gbe (48 x 10k SAS)– Equallogic 6510e 7.2l 10Gbe (48 x 7.2k SATA)– 4 x Dell 8024f SFP+

• Networking (per server)– 2 x 1Gbe – VMkernel & vMotion– 6 x 1Gbe – Virtual machine data networking– 2 x 10Gbe – iSCSI network storage

Session ID 2853 37

VMware Hardware Architecture

Session ID 2853 38

VMware Hardware Architecture

Session ID 2853 39

VMware Hardware Architecture

Session ID 2853 40

VMware Hardware Architecture

Session ID 2853 41

Cloning

.

Session ID 2853 42

VMware and SAN cloning

• With VMware, we can now easily clone– Servers– Environments

• With Equallogic, we can clone data LUNs

Session ID 2853 43

VMware cloning

Production Isolated vLan

Session ID 2853 44

Database Cloning – Grid Control

Session ID 2853 45

PerformanceReliability

Disaster Recovery

.

Session ID 2853 46

VMware cluster

• VMware server farm– No difference in performance compared to standalone server– IO throughput is superb with multiple 10GBytes i-scsi

adapters– vMotion and dynamic load balancing among cluster members– Cloud services for DR contingency

• We are talking with cloud services vendors to pilot a DR hot-site clone of our production environment.

• Sister Institutions within the state of WV we share a 10Gbit fiber ring.

– Rack swap– VMware Site Recovery Manager– Equallogic SYNCREP (byte level SAN replication)

Session ID 2853 47

Summary

• Server virtualization is working very well for Marshall University.

• We have reduced infrastructure costs,• Gained a higher level of service for

customers.• Increased flexibility to respond to customer

needs (multiple Banner instances),• And created a viable disaster recovery

solution.

Session ID 2853 48

Questions & Answers

?

Session ID 2853 49

Thank You!

• Brent Maynard – brent.maynard@marshall.edu– (304) 696-2591

• Myke Watts – iwatts@marshall.edu– (304) 696-3921

• Presentation available• http://webpages.marshall.edu/~maynard169/live2853.pptx

.  

© 2013 Ellucian. All rights reserved.