Jack Garvey

One of five regional water management districts for the state of Florida

Four broad areas of responsibility:Water supply

Flood protectionWater quality

Natural systems management

Where we where at the last User group meeting:• 16 ESX servers in Brooksville• 4 ESX Servers in Tampa• 2 ESX Servers in Bartow• 2 ESX Servers in Sarasota• 20 750 gig luns presented from San to Brooksville, 4 to Bartow,

3 to Sarasota and 4 to Tampa. • Each service office has its own San. Tampa, Bartow, and

Sarasota are ISCSI• 2 Vranger Servers in Brooksville and 1 at each service officeCont..

Where we are (cont):• Approximately 268 VM servers and about 75 VM workstations• We have 9 clusters of Esx Servers across all service offices• The version of ESX server is now 3.5.0 Update 1 • Vranger has been updated to version 3.2.4.2 and is now

working very well with Data Domain• Vranger has been loaded and tested on a Vm server with

version 3.2.5.6 So far has tested well getting same results as physical server.

• HA and DRS are now configured for all ESX servers• Update manager has been installed and configured

Where we are:• 19 ESX servers in Brooksville• 4 ESX Servers in Tampa• 2 ESX Servers in Bartow• 2 ESX Servers in Sarasota• Updated to 3.5.0 Release 3 (testing release 4)• 25 750 gig luns presented from San to Brooksville, 5 to Bartow,

4 to Sarasota and 4 to Tampa. • Approximately 321 VM servers and about 112 VM workstations• Have moved Vranger over to Virtual servers and is working well• Update manger is working as planned and upgrades are now a

breeze. (cont.)

How we got to where we are:• New projects requiring more servers, Production, Development

and acceptance• Many Ad Hoc machines being put up by Administrators that are

now being overseen and needing justification• Update manager was installed for virtually no downtime when

pushing updates to the ESX servers• Data Domain was acquired to cut down on the amount of space

and time needed to complete backups. As everything else grows, space requirements are now almost exceeding our Data Domain Storage

Data Domain Data Flow OverviewIn this configuration there is no need for a physical server for

Vranger.

Using Data Domain::• Had to set up a VMkernel port on each ESX server to

communicate with Vranger. Must be in same subnet as Data Domain and ESX servers.

• Directory structure had to be setup on Data Domain.• In Virtual Console you added the Data Domain share as an

NFS (Network File System) storage device.• CIFS (Common Internet File System) is used to be able to view

the directory structure in windows. This is used for file level restores from Vranger.

• File level restores are now a matter of a few minutes as opposed to 20 – 40 minutes before.

.

On bkvdd01:Total files: 5,703; bytes/storage_used: 16.9 Original Bytes: 15,487,321,427,962 Globally Compressed: 1,580,860,214,777 Locally Compressed: 911,374,049,611 Meta-data: 5,225,289,304

On bkvdd02:Total files: 8,930; bytes/storage_used: 17.7 Original Bytes: 16,384,582,995,066 Globally Compressed: 1,496,833,113,939 Locally Compressed: 921,967,281,592 Meta-data: 5,022,873,216

• The use of Power Shell to manage, deploy and automate tasks within the VMware Infrastructure with a simple to use command line interface

• Powershell is a command line interface and scripting language from Microsoft. It lets you perform administrative tasks by execution of cmdlets (pronounced Commandlets)

• VMware has introduced VMware Infrastructure Toolkit, which works along with Microsoft's Powershell

• Several components are required in order to use Powershell with VMware

1. Microsoft Powershell and .NET framework

2. VMware Infrastructure Toolkit for Windows 1.5

3. (Optional) PowerGui from Quest(a Powershell Gui interface)

4. Also VMware Vitoolkit powerpack

cont.

• PowerGui is a free application that adds a GUI and script editor to PowerShell to make it easier for you to develop code and use PowerShell. The GUI allows you to use PowerShell without having to know how to code. The code is automatically generated when you perform certain actions. It also comes with a great script editor that you can use to write scripts with. You can extend PowerGui by adding PowerPacks, which are add-on modules written specifically for certain applications and purposes such as VMware.

• There already many scripts written to automate many things in VMware, such as adding columns to your Virtual Center view for specific information about your VM’s. In the next slide you will see where I added a number of snapshots column, and the size of the snapshots column.

cont.

• You can also create reports, I have created one that lists all of my snapshots and the dates of creation all done through PowerGui as seen here:

cont.

• Next slide shows what PowerGui looks like and some of the options that can be used with it, also included with PowerGui is power scripting a tool for editing and writing your own scripts.

• There are many good tutorials out on the web as I had this up an working in about a half an hour and was creating scripts the same day. For anyone interested I can send the links that I had used which were very helpful. Also the links to all of the software mentioned which is all free.

cont.

Lessons Learned:• We went through a major IP readdressing here and found it was

best to put the ESX servers in maintenance mode, change all addresses: DNS, Service Console, etc.. Then reboot the server. Once rebooted we found that some servers needed to be taken out of the cluster and then re-added as there were issues migrating servers back and forth. VMware support had said that it had to do with the security cert.

• When ESX servers go in Isolation mode, (when no connection is detected from the server) HA default is to power off VM’s and move them. Network Engineers were troubleshooting a Cisco router, pulled a blade and an ESX server lost connection. It went into Isolation mode and powered down some Production servers and brought them up on another server. We have since-

Cont.

Lessons Learned (cont)

• (cont) changed the setting to leave the VM’s powered on in Isolation mode. This is not advisable if using ISCSI as it will not lock the drive and can cause issues if VM server is migrated.

• EVC (Enhanced VMotion capability) We now have an EVC cluster up and working. 3 same servers and 1 different all seems to be working with no problems. When using HP servers there is a setting in the bios that needs to be changed in order for it to work. The setting is under the advanced CPU options in the bios. You need to enable the No Execute Memory option. This will allow EVC to work This is not documented anywhere that we had seen.

Where we are going:• Virtualizing as many servers as possible. We have about 14

more physical servers we can virtualize.• Evaluating a monitoring and reporting tool for VMware.• Creating less clusters and adding ESX servers with the new

CPU compatibility in 3.5.0 Release 2 (EVC).• Getting all ESX Server up to version 3.5.0 Release 4.

• Questions and Comments:• Question for the User Group: Is anyone using EVC and if so

how big of a cluster do you have? We have been looking for best practice, VMware could not answer, only saying not to use more than 32 ESX servers.