VIRTEXCPOWSTO

A DELL

End-t

Dell

Septe

RTUALICHANGWEREDORAGE

L TECHNICAL

to-End So

Product G

ember 20

ZING MGE 20DGE™

E

L WHITE PA

lutions En

Group

10

MICRO10 ON SERVE

APER

ngineering

OSOFTN DELLERS AN

g

T L ND

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

© 2010 Dell Inc. All rights reserved. Reproduction of this material in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell.

Dell, the DELL logo, and the DELL badge, PowerConnect, and PowerVault are trademarks of Dell Inc. Symantec and the SYMANTEC logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the US and other countries. Microsoft, Windows, Windows Server, and Active Directory are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell Inc. disclaims any proprietary interest in trademarks and trade names other than its own.

September 2010

Contents A DELL TECHNICAL WHITE PAPER ................................................................................. 1 

1. Introduction ................................................................................................... 3 

2. Overview....................................................................................................... 4 

2.1  Hardware Components .............................................................................. 4 

3. Deployment Best Practices for Virtualized Exchange Server 2010 ................................... 7 

4. Reference Architecture for 4000 Mailbox Virtualized deployments .................................. 9 

4.1  Performance Study ................................................................................... 14 

5. Summary ..................................................................................................... 17 

6. Reference and Additional Information .................................................................. 18 

1. Introduction Enterprises are seeking ways to streamline IT budgets without affecting operations and limiting

features to end users. Microsoft Exchange® 2010 introduces new performance enhancements,

which can significantly reduce storage requirements, and extends high-availability features.

Exchange Server 2010 is officially supported by Microsoft on virtualized infrastructures and if

designed correctly, can further reduce the total cost of ownership and maximize return on

investment. In this white paper, we design and evaluate a 4000 mailbox Exchange 2010

deployment on Dell’s Virtualization Business Ready Configuration (BRC).

Virtualization Business Ready Configuration’s (BRC) from Dell are tools that help customers to

minimize complexity during planning, ordering, deployment and installation of their virtual

infrastructure. Dell offers multiple BRC’s that provides the customer a choice between multiple

server and storage platforms, and for their virtualization infrastructure software. The BRC is

optimized for running virtualized workloads and provide optimal redundancy, scalability and

manageability. This document covers an approach to virtualize Exchange 2010 on the Dell

Virtualization BRC for a Small Medium Business. The approach used in this document is hypervisor

agnostic, and therefore the configuration can be used for hypervisors from Microsoft, VMware or

any third party vendor that is validated by Microsoft to be used for Exchange Virtualization.

This white paper describes using the Business Ready Configuration for Virtualization to implement

an Exchange Server 2010 messaging infrastructure on a VMware® vSphere™ environment. It

documents the Exchange 2010 architecture, server and storage configurations, virtual machine

sizing and best practices to enable an efficient virtualized Exchange 2010 messaging

infrastructure. The best practices presented are the result of installing, sizing and evaluating

Exchange Server 2010 components on a Virtualization BRC at the End-to-End Solutions Engineering

Lab. Experimental results from the testing form the basis for sizing and performance guidance

presented in this white paper to architect an optimal Exchange 2010 solution

2. Overview Dell offers multiple BRC’s that provide the customer a choice between multiple server, storage

and virtualization platforms. Customers use the BRC’s from Dell for quick and efficient ordering,

deployment and installation of their virtual infrastructure. The BRC used to host the virtualized

Exchange 2010 building block presented in this document is available at

http://content.dell.com/us/en/enterprise/d/virtualization/BusinessReadyDataCenter.aspx. This

BRC combines the Dell PowerEdge™ M-Series blade servers and EqualLogic™ PS6000 storage area

network (SAN) with Cisco® networking switches and VMware® vSphere™ 4 technology to help

achieve an automated and efficient data center that can be used to host Exchange 2010 Server

roles along with other virtualized workloads.

2.1 Hardware Components

Figure 1: Typical vSphere 4 Architecture with PS series SAN

The PowerEdge M610 is a two-socket blade server designed with energy-optimized options

supporting the new Intel® Xeon® 5600 series processors, DDR3 memory and advanced embedded

management capability.

P o we r E d g eM1 0 0 0 e

9 1 01 2

1 1 1 23 4

1 3 1 45 6

1 5 1 67 8

0

1

00

1

0 0

1

00

1

0 0

1

00

1

0 0

1

00

1

0

0

1

00

1

0 0

1

00

1

0 0

1

00

1

0 0

1

00

1

0

VMware vSphere 4

Storage Fabric

Network

VM VM VM VM

Pow

erEdge M

1000e M

610 blade servers

EqualLog

ic PS

6000 S

eries SA

N

The Dell EqualLogic PS6000 is an iSCSI storage device from Dell designed for deployment in

enterprise business environments. It consists of four 1Gb Ethernet network ports per controller for

data, faster processors, 2 GB of cache per controller, support for RAID 6, increased drive capacity,

and a new monitoring application, SAN HQ, at no additional cost. For more information on Dell

EqualLogic storage, see www.dell.com/equallogic.

In the following table, we provide a summary specification of the pre-configured solution. In the

rest of the document, we will discuss reference architectures and best practices for deploying and

using these solutions. The solution ID can be used to order or view the complete virtualization

solution to deploy the reference architecture.

Table 1: Solution Specification

Hardware

Solution ID 1131816.1

Blade Chassis (1) M1000e

ESX Blade Server (6) x M610

Storage Device (4) PS6000XV

Management Blade Server (1) x M610

Chassis Configuration

I/O module for A1, C1, A2, C2 Cisco M 3130G

I/O module for B1, B2 Cisco M 3130G with (2) 1000BAS-T SFP modules

Stacking cables 6

Management Redundant Chassis Management Controllers (CMC)

KVM Integrated Avocent keyboard, video, and mouse (iKVM) switch

ESX Blade Server Configuration

Blade Server Model M610

Processor 2 x Intel Xeon X5660, 2.8Ghz, 12M Cache,Turbo, HT, 1333MHz Memory

Memory 96 GB (12 x 8 GB, DDR3)

Add-in Controllers (2) Broadcom 5709 Dual Port GbE I/O Card for M-Series Blades

Local storage and controller Diskless configuration

Hypervisor VMware ESXi 4.0

Storage Configuration

Storage Device (4) PS6000XV

Drives 16 X 600GB, 15K SAS

Storage Capacity 9.6 Terabyte capacity per Device

Management Blade Server

Blade Server Model (1) M610

Processor (2) x Intel Xeon (Nehalem) E5520, 2.26Ghz, 8M Cache

Memory 8 GB

Network Controller (2) Broadcom 5709 Dual Port GbE I/O Card for M-Series Blades

Virtualization Management S/W VMware vCenter 4.0, 3Yr Upgrade Subscription

Software and Services

Services 3 Year ProSupport for IT and Mission Critical 4HR 7x24 Onsite Pack

EqualLogic PS Array Installation M1000e Blade Chassis On-site installation service

3. Deployment Best Practices for Virtualized Exchange Server 2010

Microsoft started supporting Exchange Server on Virtualized hardware with Exchange Server 2007.

Exchange Server 2010 is supported as well, assuming certain conditions are fulfilled. In terms of

hypervisors, Windows 2008 with Hyper-V Technology, Microsoft Hyper-V Server and any third party

hypervisor (such as VMware and Citrix offerings) that are validated under Window’s SVVP (Server

Virtualization Validation Program) are supported. The guest VM running the Exchange Server 2010

roles has to be deployed on Windows Server 2008 SP2 or R2 to be supported. All server roles are

supported in a virtualized environment, except for Unified Messaging role, which has to be on a

physical server.

Other important configuration or deployment scenarios that to keep in mind –

Dynamic VHD disks are not supported; neither are snapshots, differencing or delta

mechanisms

Exchange 2010 HA features like Database Availability Groups (DAGs) cannot be combined

with hypervisor based clustering solutions.

Overprovisioning of virtual processor is not supported for a ratio greater than 2:1

Minimum size of the fixed disk for the Exchange 2010 operating system virtual disk has to

be 15Gb in addition to page file size (equal to the memory allocated for the VM).

In a supported configuration, server based applications cannot be run on the physical root

machine of a server that is hosting the Exchange 2010 roles inside virtual machines. It is

okay to run management software like anti-virus, backup software etc on the root

partition.

Operating System disk images for the virtual machines have to be stored on serparate

storage array or disks from the Exchange database or log storage.

Sizing the Exchange Server Mailbox Roles/VMs Exchange Server 2010 is not virtualization aware and therefore the majority of the design

principles used for a physical deployment will translate to the virtual deployment design. Virtual

machines are sized similar to physical servers, depending on the Exchange server role that is being

hosted on the virtual machine. The number of cores and memory per virtual machine will depend

on the message profiles and mailbox sizes. The following guidelines are recommended by

Microsoft when deploying Exchange 2010 on a Hyper-V environment to size virtual machines

hosting Exchange roles–

The Hub and CAS server roles are combined in a single VM. This VM is assigned 4 virtual

CPUs and 8GB of memory. The number of multi-role Hub/CAS deployed is the same as the

number of Mailbox VMs deployed, i.e. the ratio of Hub/CAS VMs to Mailbox VMs is 1:1.

The Mailbox server role is assigned four virtual CPUs. The approximate maximum number

of mailboxes recommended per Mailbox VM is 3000 to 4000 depending on the user profile

and concurrency. The memory assigned to each Mailbox VM is 4GB (for OS) + memory

needed for mailbox database cache. The requirements per user for mailbox database

cache size ranges from 3MB-9MB (light profile to very heavy profile).

In the design presented in this paper, we use a multi-role Exchange VM which combines the

HUB/CAS/Mailbox roles in a single machine. The goal of this design is to use a part of the

resources available in the Virtualization infrastructure, and share the hardware with other

virtualized applications and workloads.

The number of drives required for database and logs is calculated using the same

principles as a physical deployment. The Microsoft storage calculator is used to determine

the storage design in terms of the number of drives and RAID type for the solution. For

internal or direct attach storage configurations SCSI pass-thru is recommended where the

volumes are presented directly to the Mailbox VMs for database and log volumes. For iSCSI

SAN, the initiator is configured on the host server disk is again presented to the guest VM

as a pass-thru disk to database and log storage.

4. Reference Architecture for 4000 Mailbox Virtualized deployments

The Agile Consolidated model from Dell for Exchange Server 2010 deployments are targeted for

the virtualized datacenters that have already invested heavily in virtualization1. These

datacenters are already running their Tier-2 and Tier-3 applications on VMs and looking at moving

their Tier-1 apps on to VMs as well. The storage infrastructure for this class of deployment is on

Dell SAN products such as Dell EqualLogic™ PS arrays and Dell/EMC CX4 and AX4 arrays.

A sample reference architecture for the Agile Consolidated Dell model using internal storage is

shown in Error! Reference source not found.. This architecture is designed for a 4000 mailbox

with an I/O profile of 150 messages per day and mailbox sizes of up to 2GB. Table 2 lists the input

variables that were used for this configuration.

Table 2 - Input Requirements for 4000 User Configuration

Input Detail

Number of mailboxes 4,000

I/O Profile 150 Messages/Day

Size of Mailbox Up to 2GB

Average Message Size 75KB

Number of Copies 2-copy DAG (1 active, 1 passive)

The high-availability for mailboxes is provided using built-in Exchange resiliency features thru the

Database Availabiliy Group (DAG). The 4000 mailboxes are evenly distributed between four

Exchange VMs, hosted on two separate physical servers. Two Database Availability Groups (DAGs)

are created across the 4 VMs so that the mailbox copy is hosted on a separate physical server. The

hypervisors are not part of a cluster since virtualization features are not used in this design to

provide high-availability. The hypervisors not hosting Exchange, can be part of a cluster and

applications running on them can be load balanced thru the virtualization software.

Each physical server hosts a multi-role Mailbox/Hub/CAS server VM. Error! Reference source not

found. details the host and VM configurations to support 4000 users. Full provisioning of resources

is configured so that any server can handle a scenario where all the passive copies become active

in the event of failure happening on a server or individual databases. High Availbility in the local

1 http://www.dell.com/downloads/global/solutions/security/exchange_2010.pdf

site is pr

applicatio

Figure 3

deployme

Virtual M

copies, l

performa

used for

images an

ovided by Ex

on and serve

shows the c

ent is install

Machine. Dell

logs, and th

ance and pro

storing the E

nd storage fo

xchange Data

er layers as w

Figure

consolidated

led. Two Pow

EqualLogic

he VM OS im

vide resilien

Exchange da

or other VMs

abase Availa

well as the st

2 - Agile Con

hardware in

werEdge M6

iSCSI SAN st

mages. The

cy against d

atabase and

.

ability Group

torage and da

nsolidated Mo

nfrastructure

10 servers a

torage is use

arrays are

isk failures a

log files in a

ps (DAGs) wh

atabase laye

odel on VMw

e on which t

are used to

ed to host th

configured

at the storag

addition to o

hich provide

ers.

ware

the 4,000 use

host the mu

e Exchange

using RAID

ge level. Mul

other volume

protection a

er Exchange

ulti-role Exch

Mailbox data

D-10 to max

ltiple volume

es to hold th

at the

2010

hange

abase

ximize

es are

he VM

Figure 3 shows the layout of the Exchange 2010 server roles and distribution of virtual machines

on the servers. The 4,000 users are distributed across four Exchange VMs, each host running two

Mailbox VMs. In the event of a server failure, Exchange DAG will activate the passive copy on

working server to restore service to the failed mailboxes.

Configuration details for the server, storage and virtual machines are shown in

Table 3. Each PowerEdge M610 is configured with two quad-core CPUs and 96 GB of memory so it

can handle additional load in case of server failure on the other host. Each Mailbox VM instance is

configured with 4v CPUs and 48GB of memory and handles 1000 active mailbox users and 1000

passive users. The hosts are connected to the iSCSI SAN using the software iSCSI initiator in the

host hypervisor.

Figure 3 - Exchange 2010 Reference Architecture for 4000 Users

Table 3 – Server and Storage Configurations

Solution Output Detail

Host Server Configuration 2 PowerEdge M610

2 x Quad-Core CPUs 6.4 GT/s

96 GB DDR4 Memory

Mailbox/HUB/CAS VM 2VM’s per Host

4 vCPU, Memory per VM: 48GB

1000 Mailboxes Active and 1000 Mailboxes Passive per Virtual Machine

Storage Configuration 4 x EqualLogic PS6000XV iSCSI Enclosures

16 x 3.5” 600 GB 15K RPM SAS disks per enclosure

RAID-10 across the entire array

Active Copy (first copy)

Passive Copy (second copy)

Exchange VM1

Exchange VM2

Exchange VM3

Exchange VM4

DB3 DB4

DB8 DB7

DB1 DB2

DB6 DB5

DB1 DB2

DB5 DB6

DB3 DB4

DB7 DB8

MB VM1 MB VM2 MB VM3 MB VM4

DAG2

DAG1

Database Configuration

Databases 4 database per Mailbox VM (data and logs combined per database)

Total 16 databases (8 active and 8 passive databases)

Mailboxes 500 Mailboxes per DB

Up to 2GB capacity per Mailbox

4.1 Performance Study A smaller configuration of the reference architecture presented was tested and evaluated in the

Dell enterprise solutions lab to validate the architecture and gather experimental data. Figure 4

illustrates the configuration that was tested - VMware ESX Server v4.1 is installed on both Dell

PowerEdge M610 servers. Each host is running a single Guest VM installed with Microsoft Windows

Server 2008 R2 Enterprise Edition. Exchange 2010 roles are installed on the guest virtual machines

and host 1000 active and 1000 passive mailboxes each. The mailbox VM is assigned four virtual

CPUs and 24GB of memory. It was configured with 2 vNICs, one connected to the public server

network and the other was a private network configured for the DAG (Database Availability

Group). The Exchange 2010 database and logs are hosted on volumes created on the externally

connected EqualLogic PS6000XV array.

Figure 4 - Performance Study Configuration and Setup

Figure 5 shows the average SendMail and 95th Percentile Send Mail latencies measured when

simulating a 1000 user mailbox deployment on each virtual machine. The 95th percentile latencies

are well within the 1-second threshold recommended by Microsoft for adequate end user

performance. The SendMail latencies are still within the threshold when a Database Availability

Group (DAG) is created across the two Exchange VMs and they host the passive copy of the

databases.

Figure 5 - Microsoft LoadGen Send Mail Latencies

Figure 6 illustrates the CPU and Memory usage for the ESX host and Virtual Machines when running

2000 mailbox LoadGen simulation on the standalone configuration and DAG configuration. As

expected there is a minor increase in both CPU and memory usage for the DAG configuration as

the Exchange server is hosting the passive copy in addition to the active mailboxes.

Figure 6 - CPU and Memory Usage for ESX Server and Exchange VM

The database and log performance metrics are shown in the following figures. In the stand alone

configuration there are no log reads being performed. When a DAG is established, we see log

reads on both Exchange VMs as an artifact of the DAG functionality. As expected the DB and log

I/O’s are significantly higher in a DAG configuration due to the maintaining a database copy. The

DB read latencies are observed to increase in the DAG configuration as well, but are within the

recommended thresholds by Microsoft for acceptable performance. The latencies presented in the

70 69 71 77

223 218

245 249

0

50

100

150

200

250

300

VM1 1K Mailboxes

VM2 1K Mailboxes

VM1 2K Mailboxes

(1K Active + 1K Passive)

VM2 2K Mailboxes

(1K Active + 1K Passive)

Late

ncy

(m

s)

Send Mail Latency 95th Percentile SendMail Latency

3 2 3 4

119

1315

25537 25376 25950 25922

18948 1877721029 20658

0

5000

10000

15000

20000

25000

30000

0

2

4

6

8

10

12

14

16

VM1 1K Mailboxes

VM2 1K Mailboxes

VM1 2K Mailboxes

(1K Active + 1K Passive)

VM2 2K Mailboxes

(1K Active + 1K Passive)

Me

mo

ry C

on

sum

pti

on

(M

byt

es)

CP

U U

tiliz

atio

n (

%)

Physical Server CPU Utilization VM CPU Utilization Memory Usage  ‐ Physical Server Memory Usage  ‐ VM

DAG

DAG

DAG configuration is a sum of the attached and recovery latencies – the attached read latency is

still within 10ms which is half of the recommended threshold by Microsoft for optimal end user

Exchange performance.

Figure 7 - Database Throughput Performance

Figure 8 - Database and Log Latency

Figure 9 shows how the ESX host and Exchange VM are affected when a DB failiure occurs on one of the Mailbox VMs. We simulate a failure on VM2 and this makes the copy on VM1 active, and it is therefore running 2000 active mailboxes. This was done to validate that the VM resources were adequate to handle a failure scenario at the database or host level. The chart plots the impact on resource usage over normal operation when the VM is running 1K active and 1K passive mailboxes.

11

4

42

14

47 47

87 86

0 02.9 2.7

67 6973

68

0

10

20

30

40

50

60

70

80

90

100

VM1 1K Mailboxes

VM2 1K Mailboxes

VM1 2K Mailboxes

(1K Active + 1K Passive)

VM2 2K Mailboxes

(1K Active + 1K Passive)

DB Reads/Sec DB Writes/SecLog Reads/Sec Log Writes/Sec

8.5 9.0

13.3

21.1

2.11.2

2.0 1.50 0

2.6 2.41.01 0.71 0.78 0.82

0.0

5.0

10.0

15.0

20.0

25.0

VM1 1K Mailboxes

VM2 1K Mailboxes

VM1 2K Mailboxes

(1K Active + 1K Passive)

VM2 2K Mailboxes

(1K Active + 1K Passive)

Late

ncy

(ms)

DB Read Latency DB Write Latency

Log Read Latency Log Write Latency

DAG

DAG

Figure 9 - Performance Impact when simulating a database faliure

At the Exchange application level we observe an increase in the RPC latency, operations/sec and messages being queued. This is expected behavior since all mailbox operations that was being handled by the two VMs are now being handled by a single VM. One VM is now hosting 2K active mailboxes, as compared to 1K active and 1K passive mailboxes. Hosting passive mailboxes does not incur RPC or Exchange related load on the VM, but has in impact on the storage subsystem. Since the VM was sized adequately to handle this addition load of 1K active mailboxes, there is only a minor increase in the resource utilization and end user performance is not impacted as verified by the SendMail latencies in the two scenarios. The DB read and write latencies were verified to be less than 10 ms which is well within the recommended threshold by Microsoft, proving that we have enough resources allocated to the VM to handle a database failure.

5. Summary This paper presented a reference architecture for deploying Exchange Server 2010 on virtualized Dell infrastructure. A four thousand user architecture was presented using the agile consolidated model from Dell. The high-availability for mailboxes was provided using built-in Exchange resiliency features thru the Database Availabiliy Group (DAG). The performance of the proposed architecture was evaluated using a scaled-down version and test results showed that the configuration was able to handle a 2000 mailbox deployment efficiently. The test setup can be scaled up to handle 4000 users without adding more blades since the CPU and memory usage was very minimal. Adding additional EqualLogic arrays result in a linear performance increase and will handle the I/O requirements for 4000 mailboxes. This setup can be shared with other virtualized applications, and scaled to accommodate future growth for the organization.

2.3% 1.8% 0.0%5.4%

26.1%

63.4%

23.4%

91.7% 92.1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ESX Host CPU Utilization

VM CPU Utilization 

Memory Usage ESX Host

Memory Usage Virtual Machine

MS ExchIS RPC Avg Latency

MS ExchIS RPC Operations/sec

MS ExchIS Mailbox Messages 

Queued

MS ExchIS Mailbox Messages 

Delivered/Sec

MS ExchIS Mailbox Messages 

Submitted/Sec

Pe

rfo

rman

ce I

mp

act

Relative Performance of VM with 2K active Mailboxes vs. VM with 1K Active + 1K Passive Mailboxes

6. Reference and Additional Information

Dell Exchange 2010 Sizing Guide http://www.dell.com/downloads/global/solutions/Introduction_to_Exchange_2010_Sizing.pdf Microsoft’s Exchange 2010 Mailbox Server Role Requirements Calculator http://msexchangeteam.com/archive/2009/11/09/453117.aspx

Dell Architecture Models for Exchange 2010 http://www.dell.com/downloads/global/solutions/security/exchange_2010.pdf VMware vSphere™ Reference Architecture for Small Medium Business http://i.dell.com/sites/content/business/solutions/engineering-docs/en/Documents/VMware-vSphere-Reference-Architecture-SMB.pdf

Documentation for VMware vSphere 4 for Dell PowerEdge Servers http://support.dell.com/support/edocs/software/eslvmwre/VS/VS.htm VMware vSphere 4 iSCSI SAN Configuration Guide http://www.vmware.com/pdf/vsphere4/r40/vsp_40_iscsi_san_cfg.pdf VMware Network Concepts http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf Configuring VMware vSphere Software iSCSI with Dell EqualLogic PS Series Storage http://www.equallogic.com/resourcecenter/assetview.aspx?id=8453