How the Microsoft Information Technology organization designed the corporate Exchange Server 2007 environmentPublished: November 2007Exchange Server 2007 Design and Architecture at Microsoft

AgendaSolution overviewReasons for Microsoft IT to use Exchange Server 2007Environment prior to Exchange Server 2007Planning and design processArchitecture and design decisionsDeployment planningBest practices

The costs and general limitations associated with the platforms and technologies used in the Exchange Server2003 environment prevented Microsoft IT from efficiently meeting emerging messaging and business needs. Solution OverviewWith Exchange server 2007 Microsoft IT created new opportunities to drive down costs and system complexities, increase security, and deploy new features not available in previous versions of Exchange Server.Increased reliability.Larger mailbox sizes.Reduced total cost of ownership (TCO).Increased protection against spam.Reduced topology complexities.improved regulatory complianceEnhanced remote access and mobility options.

Reasons for Microsoft IT to use Exchange Server 2007Increase employee productivityIncrease operational efficiency Decrease security risksDecrease costsOur mission is to deliver value by enabling people with innovative and reliable information technology solutions that seamlessly integrate with, and improve, how people work..Jim DuBoisGeneral Manager, MSITMicrosoft Corporation

Environment Prior to Exchange Server2007

Environment Prior to Exchange Server2007 (Directory Infrastructure)Multiple forests for various legal and business requirements 70% of resources in Corporate forest over 1 million objects9 domains in corporate forest based on geography202 sites in hub and spoke topologyDedicated Exchange site in Redmond

Environment Prior to Exchange Server2007 (Directory Topology)

Environment Prior to Exchange Server2007 (Messaging Topology)Centralized administration from RedmondFour administrative groups (North America, Dublin, Singapore, and Sao Paulo)Routing topology correspond to WAN linksRouting group connectors between routing groups with default option Four central bridgehead servers in North America as remote bridgehead servers in the RGC configuration Inbound Internet mail messages through two redundant locations

Environment Prior to Exchange Server2007 (Messaging Topology)

Routing groupMailbox servers (clustered)Public-folder serversBridge-head serversFront-end serversGateway serversSpecial purposeRG_REDMOND-EXCHANGE2158603RG_DUBLIN62220RG_SINGAPORE52220RG_SAO PAULO1200RG_REDMOND PERIMETER000030RG_SILICON VALLEY PERIMETER000030

Planning and Design Process

Architecture and Design DecisionsAdministration and permissions modelMessage routing topologyServer architectures and designsMailbox storage designBackup and recoveryClient access server topologyUnified messagingInternet mail connectivityMicrosoft IT is our first and best customer. Almost two years prior to RTM, Microsoft IT began with pre-release production deployments to help us build an excellent product. The close relationship with Microsoft IT is so vital to our culture of quality and customer satisfaction that we do not ship products or service packs until Microsoft IT signs off on the enterprise readiness. We shipped Exchange Server2007 on December 7, 2006, with the confidence and proof in hand that the product delivers on its potential to help customers build reliable enterprise-class messaging environments while reducing total cost of ownership.Terry MyersonGeneral ManagerExchange Server Product GroupMicrosoft Corporation

Administration and Permissions ModelSecurity Principles and GuidelinesExclusive Microsoft IT ManagementCentralized System AdministrationDefault Permissions Mode Formal Approval Process Permissions Review

Administration and Permissions Model (Approval Processes)

Message Routing TopologyNetwork Infrastructure and Site ConsolidationDedicated Exchange Sites in the Active Directory TopologyOptimized Message Transfer Between Hub Transport serversConnectivity to Remote SMTP domainsIncreased Message Routing securityCoexistence with Exchange Server 2003

Message Routing Topology (Network Infrastructure and Site Consolidation)Physical network -> IP routing topology -> Active Directory site topologyPrevious consolidation With Exchange 2003 made planning easierMany benefits of consolidated datacentersUncomplicated messaging topology Best possible Hub Transport server utilizationReduced chance of server communication issues

Message Routing Topology (Dedicated Exchange Sites in the Active Directory Topology)

Message Routing Topology (Optimized Message Transfer Between Hub Transport servers)

Message Routing Topology (Increased Message Routing Security)Messaging traffic encryption and lab environment exceptionTechnologies usedIPSec Transport layer (TLS) Restricted access to SMTP submission pointsForefront Security on Hub Transport and Edge Transport

Message Routing Topology (Coexistence with Exchange Server 2003)Special routing group where Active Directory site topology defines the message routing topology

Message Routing Topology (Coexistence with Exchange Server 2003)

Routing group connectorLocal bridgeheadsRemote bridgeheadsFrom RG_REDMOND to EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR)Any local server can send mail over this connector. This enables all Exchange 2003 servers to transfer messages directly to the Hub Transport servers without involving Exchange 2003 bridgeheads. All Hub Transport servers located in ADSITE_REDMOND-EXCHANGEFrom EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) to RG_REDMONDAll Hub Transport servers located in ADSITE_REDMOND-EXCHANGE.All Hub Transport servers located in RG_REDMONDFrom RG_DUBLIN to EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR)Any local server can send mail over this connector.All Hub Transport servers located in ADSITE_DUBLINFrom EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) to RG_DUBLINAll Hub Transport servers located in ADSITE_DUBLIN.The public-folder servers in RG_DUBLIN, which also function as bridgehead serversFrom RG_SINGAPORE to EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR)Any local server can send mail over this connector.All Hub Transport servers located in ADSITE_SINGAPOREFrom EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) to the Singapore routing groupAll Hub Transport servers located in ADSITE_SINGAPORE.The public-folder servers in RG_SINGAPORE, which also function as bridgehead servers

Server Architectures and DesignsFlexible and Scalable Messaging Infrastructure Multiple-Role and Single-Role Server Designs Scaling Up Server Designs

Server Architectures and Designs (Flexible and Scalable Messaging Infrastructure)

Server Architectures and Designs (Multiple-Role and Single-Role Server Designs)

Server roleRed-mondSilicon ValleyDublinSinga-poreSao PauloTechnologyMailbox31015151Microsoft Windows Clustering and CCR. Network interface card (NIC) teaming by using NICs connected to different switchesEdge Transport33220Domain Name System (DNS) round robin and Mail Exchanger (MX) records with same cost values. Multiple Hub Transport servers as bridgeheads in Send Connector configurationHub Transport80331Automatic load balancing through Mail Submission Service. Edge Subscriptions for Hub/Edge connectivity.Client Access16064Web Publishing Load Balancing (WPLB) on Microsoft Internet Security and Acceleration (ISA) Server 2006. Microsoft Network Load Balancing (NLB) internally.Unified Messaging7022Automatic round robin load balancing between Unified Messaging servers. Multiple voice over IP (VoIP) gateways per dial plan.

Server Architectures and Designs (Scaling Up Server Designs)New scaled-up Mailbox designs after initial rollout Up to 6000 users with 500 MB mailboxesQuad-core Intel Xeon with 16 GB RAM to eliminate bottleneck

Mailbox Storage DesignEliminating Storage as the Single Point of failureReducing Storage Costs and Configuration Complexities Optimizing the Storage Design for Reliability and Recoverability Standardizing the Storage Design

Mailbox Storage Design (Eliminating Storage as the Single Point of Failure)CCR configuration with cluster nodes and the file-share witness in the same Active Directory site

Mailbox Storage Design (Optimizing the Storage Design for Reliability and Recoverability)CCR still requires reliability and recoverability provisions at storage and server levelsMicrosoft IT uses these strategiesRAIDSeparate transaction logs from database filesNo circular logging on Mailbox serversConfigure multiple storage groups per Mailbox server

Mailbox Storage DesignStandardizing the Storage Design

Mailbox Storage Design6000-user mailbox server with two USBBs per cluster node

Backup and Recovery Performing VSS-Based Backups on Passive Node Eliminating Backups to Tape Optimizing Backup Cycles According to SLAs

Backup and Recovery (Performing VSS-Based Backups on Passive Node)Software VSS backups on passive node with DPM

Backup and Recovery (Eliminating Backups to Tape)14 days of online database backups

Backup and Recovery (Optimizing Backup Cycles According to SLAs)New 500 MB and 2 GB quotas would overtax existing backup processesWeekly full, daily incrementalSeven storage groups on each LUN

Storage groupMonTueWedThuFriSatSunSG 1FullIncIncIncIncIncIncSG 2IncFullIncIncIncIncIncSG 3IncIncFullIncIncIncIncSG 4IncIncIncFullIncIncIncSG 5IncIncIncIncFullIncIncSG 6IncIncIncIncIncFullIncSG 7IncIncIncIncIncIncFull

Client Access Server Topology Preserving Existing Namespaces for Mobile Access to Messaging Data Increasing Security Based on ISA Server2006 Providing Load Balancing and Fault Tolerance for External Client Connections Providing Load Balancing and Fault Tolerance for Internal Client Connections Optimizing Offline Address Book Distribution Enabling Cross-Forest Availability Lookups

Client Access Server Topology (Preserving Existing Namespaces for Mobile Access to Messaging Data)60,000 Outlook Web Access unique users per month and 30,000 ActiveSync sessionsExisting Multiple URL namespaces to distribute load that need to be preserved with Exchange 2007Deploy Client Access servers, verify, then migrate usersEach Active Directory site with Mailbox servers must also include Client Access serversRedirect Office Outlook Web Access users to Client Access servers that are local to the users Mailbox server via ExternalURL propertyClient Access servers act as proxy servers for local Client Access servers (Exchange ActiveSync, Exchange Web Services)

Client Access Server Topology (Increasing Security Based on ISA Server2006) Stateful inspection and application-layer filtering Blocks any traffic that appears out of context, such as requests to initiate a connection on an established session SSL bridging process enables ISA Server2006 to filter invalid data packets before the traffic reaches the Client Access serversExternally trusted SSL certificates for both external and internal traffic

Client Access Server Topology (Providing Load Balancing and Fault Tolerance for External Client Connections)

Client Access Server Topology (Providing Load Balancing and Fault Tolerance for Internal Client Connections)

Client Access Server Topology (Optimizing Offline Address Book Distribution)

Client Access Server Topology (Enabling Cross-Forest Availability Lookups)

Unified Messaging (Topology)

Unified Messaging (Redundancy and Load Balancing)

Unified Messaging (Security)Many possible security issues: SIP Proxy impersonation, session hijacking, sniffing, etcSecure protocols such as MTLS can mitigate riskTrusted LANs, VLANs, and other methods of segmentationIPSecGeneral practices such as strong password

Unified Messaging (Feature and User Considerations)Some settings and features with default values, some customizedNeed to customize dial plans, VoIP gateway partners, hunt groups, mailbox policies, etc.Need to inform users of changes and provide documentation for self-serviceMicrosoft created custom e-mail templatesCustom intranet site with documentation for usage and user self-service

Internet Mail Connectivity Inbound and Outbound Message Transfer Redundancy and Load Balancing Increasing Perimeter Network Security Server Hardening Optimizing Spam and Virus Scanning Optimizing Outbound Message Transfer

Internet Mail Connectivity (Inbound and Outbound Message Transfer)

Internet Mail Connectivity (Redundancy and Load Balancing) Multiple Hub Transport servers with Edge Transport serversAll Hub Transport servers transfer outbound messages to local Edge Transport serversEdge Transport servers can transfer inbound messages to Hub Transport serversFor inbound messages, DNS round-robin and MX records with preference value of 10Edge Transport servers in Europe and North America

Internet Mail Connectivity (Increasing Perimeter Network Security)

Internet Mail Connectivity (Server Hardening) PortsServicesFile SharesAccountsSecurity updates

Internet Mail Connectivity (Optimizing Spam and Virus Scanning) Connection-filtering configurationIP block-list, IP allow-list providers, and Sender Reputation Level Recipient-filtering configuration Content-filtering configurationStore SCL: 5Reject SCL:7No delete or quarantine SCLAttachment-filtering configuration with Forefront Security

Internet Mail Connectivity (Optimizing Outbound Message Transfer) Built-in protection on SMTP connectors, including header firewall, tarpitting, backpressure, etcOne receive connector that faces the Internet and one send connector for transferring incoming e-mail to Hub Transport serversOne receive connector faces Hub Transport servers for outbound messagesThree send connectors for relaying outbound messages to Internet hosts

Deployment PlanningIntroducing Exchange Server 2007 into the Corporate Production EnvironmentVerifying the Successful Integration of Exchange Server 2007Fully Deploying Client Access Servers in North AmericaFully Deploying Hub Transport Servers in North AmericaDeploying Mailbox Servers in North AmericaIntroducing Edge Transport Servers in North AmericaDeploying Forefront Security for Exchange Server 2007Deploying Exchange Server 2007 in Regional Data CentersSwitching the Messaging Backbone to Exchange Server 2007Completing the Transition to Exchange Server 2007

Deployment Planning (Fully Deploying Client Access Servers in North America)

Deployment Planning (Fully Deploying Servers in North America)Hub Transport role including SMTP connectorsMailbox role and user migration at least 16,000 mailboxes before other deployment tasksEdge Transport coexistence and replacementForefront Security

Planning and Design Best PracticesClearly define goalsDesign for production in mindDesign for peak load daysTest in lab environmentIdentify key risksDevelop rollback and mitigation procedures

Server Design Best PracticesUse multiple-core processors and design storage based on both capacity and I/O performanceUse VSS-based backupEliminate single points of failure

Deployment Best PracticesEstablish flexible and scalable messaging infrastructureCarefully plan URL namespacesManage permissions through security groupsUse fewest permissions necessaryUse Forefront and multiple layers of protectionPlace Edge Transport servers in a perimeter networkUse ISA Server2006 to publish Client Access servers

SummaryMessaging environment hosts 130,000-plus mailboxes with 500 MB and 2 GB quotas in 4 datacenters on 62 Mailbox servers25 Client Access, 15 Hub Transport, 10 Edge, and 11 Unified Messaging serversMany cost and reductions with transition to Exchange Server 2007Migration from SAN to DAS storage with Exchange Server 2007USBBs enable scaling up Mailbox serversEliminated single points of failureIncreased security and better filtering

For More InformationAdditional content on Microsoft IT deployments and best practices can be found on http://www.microsoft.comMicrosoft IT Showcase Webcasts http://www.microsoft.com/howmicrosoftdoesitwebcastsMicrosoft TechNet http://www.microsoft.com/technet/itshowcase

This document is provided for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.

2006 Microsoft Corporation. All rights reserved. This technical white paper presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Microsoft Press, Active Directory, ActiveSync, Forefront, Outlook, Windows, and Windows Server are either registered are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.

*AbstractBy replacing all Exchange Server 2003 servers with servers running Exchange Server 2007, Microsoft IT created new opportunities to drive down costs and system complexities while at the same time increasing security and deploying new features not available in previous versions of Exchange Server.IntroductionMicrosoft Information Technology (Microsoft IT) maintains a complex Microsoft Exchange Server environment consisting of several geographic locations and multiple Active Directory forests. There are 16 data centers, four of which host Exchange Mailbox servers, to support more than 515 office locations in 102 countries with 121,000 users, including managers, employees, contractors, business partners, and vendors. Site and server consolidation conducted with Microsoft Exchange Server2003 and new deployment features available in Microsoft Exchange Server2007 in combination with proven planning, design, and deployment methodologies enabled Microsoft IT to transition this environment to Exchange Server2007 in less than eight months. Microsoft IT decommissioned the last Mailbox servers running Exchange Server2003 in the corporate Active Directory forest shortly after Microsoft released the new Exchange Server release to manufacturing (RTM) version on December 7, 2006.Today we will cover the Exchange Server2007 architectures, designs, and technologies that Microsoft IT chose for the corporate environment and the strategies, procedures, successes, and practical experiences that Microsoft IT gained during the planning and design phase. In addition to common planning and design tasks typical for many Exchange Server deployment projects, such as server design, high-availability implementation, and capacity planning, transitioning a complex messaging environment to run on Exchange Server2007 also entails specific planning considerations regarding directory integration, routing topology, Internet connectivity, client access technologies, and unified messaging (UM).The most important benefits Microsoft IT achieved with the production rollout of Exchange Server2007 included a substantial reduction of hardware, storage, and backup costs while maintaining 99.99 percent availability of messaging services. New features, such as cluster continuous replication (CCR), enabled Microsoft IT to reduce single points of failure in the messaging environment, increase Mailbox server resilience from storage failures, and eliminate tape backups to reduce costs. Exchange Server2007 also enabled Microsoft IT to overcome the scalability limitations of the 32-bit platform and increase mailbox quotas in the corporate production environment from 200 megabytes (MB) to 500 MB and 2 gigabytes (GB). Microsoft IT also lowered maintenance overhead and associated costs by replacing third-party unified messaging systems with Exchange Server2007based Unified Messaging servers. Other important benefits included increased fault tolerance and simplified server maintenance through load-balanced server configurations for middle-tier services such as Client Access, Hub Transport, and Unified Messaging server roles. Microsoft IT also increased messaging protection through Microsoft Forefront Security for Exchange Server installed on all Hub Transport and Edge Transport servers.This presentation contains information for business and technical decision makers who are planning to deploy Exchange Server2007. We assume the audience is already familiar with the concepts of the Windows Server2003 operating system, Active Directory, and previous versions of Exchange Server. A high-level understanding of the new features and technologies included in Exchange Server2007 is also helpful. Detailed product information is available in the Microsoft Exchange Server2007 Technical Library at http://www.microsoft.com/technet/prodtechnol/exchange/2007/library/default.mspx.*AgendaSolution overviewReasons for Microsoft IT to use Exchange Server 2007Environment prior to Exchange Server 2007Planning and design processArchitecture and design decisionsDeployment planningBest practices

*SituationWith Exchange Server 2003, Microsoft IT streamlined the messaging environment through server and site consolidation. Server clusters based on Windows Clustering and a highly available, shared storage solution based on storage area network (SAN) technology helped to ensure 99.99 percent availability. However, the costs and general limitations associated with the platforms and technologies used in the Exchange Server 2003 environment prevented Microsoft IT from efficiently meeting emerging messaging and business needs. SolutionBy replacing all Exchange Server 2003 servers with servers running Exchange Server 2007, Microsoft IT created new opportunities to drive down costs and system complexities while at the same time increasing security and deploying new features not available in previous versions of Exchange Server.Benefits Increased reliability through new high-availability technologies, such as Cluster Continuous Replication. Larger mailbox sizes through Mailbox servers equipped with cost-efficient storage solutions. Reduced total cost of ownership (TCO) through cost-efficient storage solutions and elimination of tape backups. Further reduced TCO by replacing third-party unified messaging systems with Unified Messaging servers in the Exchange organization. Increased protection against attacks, spam, and malicious messages through Edge Transport servers. Reduced topology complexities and improved regulatory compliance through Hub Transport servers. Enhanced remote access and mobility options through Client Access servers.

*Reasons for Microsoft IT to use Exchange Server 2007A global survey of 1,400 chief information officers (CIOs) conducted by Gartner Executive Programs in 2006 indicated the focus of IT is increasingly shifting from cost-cutting to improving productivity, performance, and competitiveness. Following a decade of unrestrained growth in IT and approximately five years of consolidation and cutbacks thereafter, IT departments are now in a better position to refocus their strategies on future-oriented business goals. After a period of cost-cutting through server and site consolidation, Microsoft IT used the deployment of Exchange Server2007 as an opportunity to shift gears and focus on implementing solutions that help improve productivity and competitiveness, streamline system administration, and increase messaging protection beyond the levels already possible with Exchange Server2003 Service Pack2. For the deployment of Exchange Server2007, Microsoft IT defined the following key objectives: Increase employee productivityThis included increasing mailbox quotas by as much as 1,000 percent, from 200MB to 500MB and 2 GB, and deploying new productivity features available in Exchange Server2007, such as unified messaging, which enables users to receive all messages in their mailboxes, including e-mail, voice mail, and fax messages. In addition to desktop and portable clients, users can use standard telephones to access these messages. Increasing employee productivity also included deploying Microsoft Office Outlook2007 as the primary messaging client so users can benefit from new and advanced information management features such as instant search, managed folders, and more. Increase operational efficiencyThis included reducing administrative overhead associated with maintaining the messaging environment through features that are directly available in Exchange Server2007, such as Exchange Management Shell. Based on the Microsoft .NET Framework, Exchange Management Shell enables Microsoft IT to create custom scripts that facilitate mundane deployment tasks, such as applying a consistent set of configuration settings per server role across multiple servers. Decrease security risksThis included deploying Edge Transport servers and Forefront Security for Exchange Server in the perimeter network to increase security and messaging protection and to reduce the number of legitimate messages incorrectly identified as spam (false positives). This also included encrypting all internal server-to-server message traffic using Transport Layer Security (TLS) to help protect confidentiality for messages in transit. Decrease costsThis included redesigning server architectures and backup solutions for high availability to meet challenging SLAs. In redesigning server architectures, Microsoft IT heavily focused on incorporating the features directly available in Exchange Server2007, replacing expensive SAN storage with a more cost-efficient DAS solution for CCR-based Mailbox server clusters, and eliminating tape backups. All of these considerations resulted in significant cost savings. From backup changes alone, Microsoft IT realized a cost savings of approximately 5 million per year.*Environment Prior to Exchange Server2007 The Microsoft IT deployment options and design decisions for the transition to Exchange Server2007 heavily depended on the characteristics of the existing network, Active Directory, and the messaging environment. Among other things, it was important to perform the transition from Exchange Server2003 to Exchange Server2007 without service interruptions or data loss. Like any organization operating a large messaging environment, Microsoft IT also had to take a phase of coexistence into account, because it was not possible to transition the entire environment in one gigantic step. To understand the Microsoft IT design decisions in detail, it is necessary to review the environment in which Microsoft IT performed the transition.The figure illustrates the locations of the data centers that contain Mailbox servers in the corporate production environment and the overall wide area network (WAN) connections between them. Concerning the WAN backbone, it is important to note Microsoft IT deliberately designed the network links to exceed capacity requirements. Only 10 percent of the theoretically available network bandwidth is dedicated to messaging traffic. The vast majority of the bandwidth is for non-messaging purposes to support the Microsoft product development teams.For regional connectivity, Microsoft IT relies on a mix of Internet-based and privately owned/leased connections, as follows: Regional data centers and main campusThe main campus and regional data centers are connected together in a privately owned WAN based on frame relay, asynchronous transfer mode (ATM), clear channel ATM links, and SONET links. Office buildings with standard or high availability requirementsOffice buildings connect to regional data centers over fiber-optic network links with up to eight wavelength division multiplexing (WDM) channels per fiber pair. Regional offices with up to 150 employeesRegional offices use a persistent broadband connection or leased line to a local Internet service provider (ISP) and then access their regional data centers through a transparent virtual private network over Internet Protocol security (VPN/IPSec) tunnels. Mobile usersThese use a dial-up or non-persistent broadband connection to a local ISP, and then access their mailboxes through VPN/IPsec tunnels, or by using Microsoft Exchange ActiveSync, remote procedure call (RPC) over Hypertext Transfer Protocol (RPC over HTTP, also known as Outlook Anywhere), or Microsoft Office Outlook Web Access over secure HTTP (HTTPS) connections.*Environment Prior to Exchange Server2007 (Directory Infrastructure)Microsoft IT has implemented an Active Directory environment with multiple forests. Each forest provides a clear separation of resources and establishes strict boundaries for effective security isolation. At Microsoft, some forests exist for legal reasons, others correspond to business divisions within the company, and yet others are dedicated to product development groups for early pre-release deployments and testing without interfering with the rest of the corporate environment. For example, by maintaining separate product development forests, Microsoft IT can prevent uncontrolled Active Directory schema changes in the Corporate Forest.The most important forests at Microsoft have the following purposes: CorporateSeventy percent of the resources used at Microsoft reside in this forest. The Corporate Forest includes approximately 140 domain controllers. The Active Directory database is 11 GB in size, with over 1 million directory objects, including users, groups, organizational units, workstations, servers, domain controller accounts, and printer objects. Corporate StagingMicrosoft IT uses this forest to stage software images, gather performance metrics, and create deployment documentation. Exchange DevelopmentMicrosoft uses this forest for running pre-release Exchange Server versions in a limited production environment. Extranet and Test ExtranetMicrosoft IT has implemented this forest to provide business partners with access to corporate resources. There are approximately 30,000 user accounts in this forest. The Test Extranet forest enables the Extranet Technology team to test new solutions for partner collaboration without interfering with the Extranet Forest. MSN and MSNBC MSN is an online content provider through Internet portals such as msn.com. Microsoft IT manages this forest jointly with the MSN technology team.MSNBC is a news service and a joint venture between Microsoft and NBC Universal News. Legal reasons require Microsoft to maintain a separate forest for MSNBC. Microsoft IT manages this forest jointly with the MSNBC technology team.Windows DeploymentMicrosoft IT created this forest to launch pilot projects during the Windows Server2003 deployment phase as a pre-staging environment prior to deployment and feature configuration in the Corporate Forest. It is a limited production forest. Windows LegacyThis forest is used as a test environment for compatibility testing of previous Windows Server versions with Exchange Server (specifically Microsoft Windows2000 service pack testing).Microsoft IT implemented nine domains in the Corporate Forest, separated into geographic regions. At the time of the production rollout, all domains in the Corporate Forest operated at the Windows Server2003 functional level and contained between seven and 30 domain controllers. Overall, the corporate production environment (that is, the Corporate Forest) includes 202 Active Directory sites in a hub-and-spoke topology that closely mirrors the network infrastructure. The authoritative source of IP address and subnet information necessary for the Active Directory site definitions is an infrastructure database that the Microsoft IT network team maintains.*Environment Prior to Exchange Server2007 (Directory Topology)A highly granular site and subnet topology provides advantages in terms of Exchange Server communication. Although Exchange Server2003 relies primarily on routing groups to describe the physical network topology, communication with Active Directory and client referral logic is site-aware. The Directory Service Access (DSAccess) component prefers to communicate with domain controllers and global catalog servers in the local Active Directory site. Through an Active Directory site topology that closely mirrors the physical network infrastructure, Microsoft IT confines DSAccess communication to local network segments. The figure shows the Active Directory sites in the Corporate Forest that are relevant for the Exchange Server organization.As shown in the figure, there are four sites with Mailbox servers and a fifth site with Internet mail gateway servers running Exchange Server2003. The remaining Active Directory sites, ADSITE_REDMOND and ADSITE_NORTH CAROLINA, contain infrastructure servers and domain controllers to handle authentication requests from client workstations and LOB applications, but no Exchange servers. Although these sites are not very relevant for Exchange Server2003, they influence the future Exchange Server2007 design. With respect to the Exchange Server2007 design, it is important to note that all site links are bidirectional and IP-based.The Active Directory site topology at Microsoft mirrors the network layout of the corporate production environment, with ADSITE_REDMOND as the hub site in a hub-and-spoke arrangement of sites and site links. In contrast, the routing topology of Exchange Server2003, defined through a hub-and-spoke topology of routing groups and routing group connectors, used a routing group containing the servers that physically resided in the ADSITE_REDMOND-EXCHANGE site. This difference is significant for the design of Exchange Server2007 message routing, It is also worth mentioning that the Active Directory site named ADSITE_REDMOND-EXCHANGE contains only Exchange servers and domain controllers configured as global catalog servers. Microsoft created this dedicated site design during the Exchange2000 Server time frame to provide its Exchange2000 and 2003 servers with exclusive access to highly available Active Directory servers, shielded from client authentication and other application traffic. Microsoft IT continues to use the dedicated Exchange site in its Exchange Server2007 environment for the following reasons: Performance assessmentsExclusive Active Directory servers provide an opportunity to gather targeted performance data. Based on this data, Microsoft IT and developers can assess the impact of Exchange Server versions and service packs on domain controllers in a genuine large-scale production environment. Windows Server2008 domain controllersMicrosoft IT maintained Windows Server2008 Beta domain controllers in the corporate production environment. Microsoft IT decided to separate the Windows Server2008 deployment from the messaging environment by using Active Directory sites without Exchange servers, such as ADSITE_REDMOND. *Environment Prior to Exchange Server2007 (Messaging Topology)Microsoft IT uses a centralized administration model for Exchange servers. Although there are four administrative groups (North America, Dublin, Singapore, and Sao Paulo) in the legacy Exchange Server topology, Microsoft IT did not use custom permissions based on administrative groups. All Exchange Server administrators are located in Redmond and perform system administration and remote monitoring. The regional data centers are only responsible for hardware maintenance.The Exchange Server2003 routing topology followed a hub/spoke architecture that corresponded to the WAN links. The data centers in Redmond, Dublin, Singapore, and Sao Paulo corresponded to routing groups with Mailbox servers. Two additional routing groups existed with gateway servers dedicated to external communication: RG_REDMOND PERIMETER and RG_SILICON VALLEY PERIMETER. Silicon Valley provided Internet mail redundancy for the messaging environment.Between the routing groups, Microsoft IT used routing group connectors (RGCs) with the default option Any local server can send mail over this connector. Accordingly, all Exchange servers were able to transfer messages to adjacent routing groups directly. Using only one physical path to each routing group and all Exchange servers as local bridgeheads eliminated the need to communicate link state changes within and across routing groups. It also enabled Microsoft IT to keep the number of dedicated bridgehead servers at a minimum.To implement the hub/spoke topology, Microsoft IT deployed four central bridgehead servers in North America, which Microsoft IT specified as remote bridgehead servers in the RGC configuration for the regions. Exchange servers in RG_DUBLIN, RG_SINGAPORE, and RG_SAO PAULO that wanted to transfer messages to other routing groups could do so through one of these bridgehead servers. In this way, the bridgehead servers created a bifurcation point for messages addressed to recipients in multiple locations. By splitting messages into multiple copies (bifurcation) at the latest possible point (the bridgeheads), Microsoft IT preserved network bandwidth on WAN links.The four central bridgehead servers also performed message routing for communication with external entities, such as Internet destinations, partner domains, and Exchange organizations in other forests. For communication with Exchange organizations in other forests, Microsoft IT configured Simple Mail Transfer Protocol (SMTP) connectors directly on the bridgeheads. Messages addressed to partners or Internet recipients, on the other hand, reached their destinations through further bridgehead servers (for antivirus scanning) and Internet mail gateways.The Internet mail gateways in RG_DUBLIN and RG_SINGAPORE only handled outbound Internet mail for their local routing groups, whereas the Internet mail gateways in RG_REDMOND PERIMETER and RG_SILICON VALLEY PERIMETER were responsible for outbound and inbound Internet mail transfer. All inbound Internet mail messages reached Microsoft through two redundant locations in North America, which was the most efficient configuration for Microsoft because it has the flat e-mail domain namespace (@microsoft.com) and the majority of mailboxes are located in its Redmond location. The Internet mail gateways performed a series of anti-spam and other filtering checks (for example, block-list, sender, and recipient filtering), before routing the messages to internal bridgehead servers for virus scanning. Microsoft IT opted not to co-deploy antivirus solutions on the 32-bit Internet mail gateway servers in order to be able to apply anti-spam filtering first and avoid the overhead associated with virus scanning. *Environment Prior to Exchange Server2007 (Messaging Topology)Microsoft IT assigned a dedicated role to most servers in the Exchange Server 2003 organization, which allowed a more precise hardware configuration for each server to reflect high performance and scalability demands. An exception to this rule was the Mailbox server in Sao Paulo. Due to moderate workload, Microsoft IT was able to consolidate server roles in this location. The server in Sao Paulo acted as a Mailbox server, public-folder server, and bridgehead server. Moderate workload also enabled Microsoft IT to assign multiple roles to the public-folder servers in Dublin and Singapore. These servers assumed the responsibilities of public-folder servers and bridgehead servers for the regions.The table lists the number of servers per role that Microsoft IT deployed in each routing group (March 31, 2006).Because of server and site consolidation during the Exchange Server2003 time frame, Microsoft IT deployed almost all Exchange Server2003 Mailbox servers as clustered systems with a maximum of 4,000 mailboxes per Exchange Virtual Server (EVS). Through this configuration, Microsoft IT achieved 99.99 percent server availability

*Planning and Design ProcessThe figure illustrates how Microsoft IT aligned the Exchange Server2007 design and deployment processes from assessment and scoping through full production rollout. The individual activities correspond to the phases and milestones outlined in the Microsoft Solutions Framework (MSF) Process Model. Assessment and Scoping In extensive planning sessions, product managers, service managers, the Exchange Systems Management team, Tier 2 Support team, Helpdesk, and Messaging Engineering team collaborated in virtual project teams to identify business and technical requirements and translated these requirements into proposals to the Exchange Server Product group. Deployment Planning Exercises Within Microsoft IT, the Messaging Engineering team is responsible for creating the architectures and designs of all Exchange-related technologies. At a stage when the actual product was not available, messaging engineers began their work with planning exercises based on product development plans. For example, the Exchange Messaging team decided to use CCR to eliminate single points of failure in the Mailbox server configuration and DAS to drive down storage costs while at the same time increasing mailbox quotas up to a factor of 10. Engineering Lab The Messaging Engineering team maintains a lab environment that simulates the corporate production environment in terms of technologies, topology, product versions, and interoperability scenarios, but without production users. Testing in the Engineering Lab enables the messaging engineers to ensure that the conceptual and functional designs scale to the requirements and scope of the deployment project. Pre-Release Production Deployments Microsoft IT maintains a pre-release infrastructure, which is a limited production environment for running pre-release versions of server products. Pre-release production deployments begin prior to the alpha phase and continue through the beta and release candidate (RC) stages until Microsoft releases the product to manufacturing. Pre-release production deployments enable the developers to determine the enterprise readiness of the software, identify issues that might otherwise not be found prior to RTM, and collect valuable user feedback. Technology Adoption Program The Exchange Server2007 Technology Adoption Program (TAP) started during the pre-alpha stage in April 2005. TAP is a special Microsoft initiative, available by invitation only, to obtain real-world feedback from Microsoft partners and customers. Pilot Projects An important task of the Messaging Engineering team is to document all designs, which the messaging engineers pass as specifications to the technical leads in the Systems Management team for acceptance and implementation. The messaging engineers also assist the technical leads during pilot projects and server installations and help develop a set of build documents and checklists to provide operators with detailed deployment instructions for the full-scale production rollout. Production Rollout The server designs that the Messaging Engineering team creates include detailed hardware specifications for each server type, which the Infrastructure Management team and the Data Center Operations group at Microsoft IT use to coordinate the procurement and installation of server hardware in the data centers. The Data Center Operations group builds the servers, installs the operating systems, and joins the new servers to the appropriate forest before the Exchange Systems Management team takes over for the deployment of Exchange Server2007 and related components. *Architecture and Design DecisionsOne of the most important objectives that Microsoft defined for the Exchange Server2007 production rollout was to finish the transition no later than the official RTM date of the product. Microsoft IT committed to perform the rollout at full scale by using the Beta 2 release to demonstrate the enterprise readiness of Exchange Server2007 to Microsoft customers. However, the Beta 2 release was not perfect, and Microsoft IT did not know all of the products performance parameters yet. For these reasons, Microsoft IT deliberately created initial designs for the Client Access, Hub Transport, Edge Transport, Mailbox, and Unified Messaging server roles that exceeded actual production requirements. To maximize the ROI of server hardware and storage technology, Microsoft IT began to optimize the designs after the completion of the initial rollout.

*Administration and Permissions ModelTo prevent fraud, protect assets, and ensure that messaging resources are used as intended, Microsoft IT implemented a strict administrative design according to the principle of fewest privileges as well as formal approval processes for granting administrative rights. The Messaging Engineering team used the following principles and guidelines in the administrative design for Exchange Server2007: Always use groups to assign rights Use the default permissions model Grant the least permission necessary Implement approval processesThe administrative model of Exchange Server2007 relies on Active Directory forests to define security boundaries. Within a single forest, there is no security isolation, because forest owners and enterprise administrators can always gain access to all resources in any domain. Accordingly, Microsoft IT grants enterprise administrator and top-level domain administrator rights in the Corporate Forest only on a temporary basis and enforces very strict approval processes.Exchange Server2007 supports centralized and decentralized administration by means of four administrator roles: Exchange Organization Administrators Exchange Server Administrators Exchange Recipient Administrators Exchange View-Only AdministratorsThe Exchange Server2007 permissions model is straightforward and flexible because it promotes the use of universal security groups for permissions management that closely correspond to the administrative roles listed in the previous section. The Exchange Setup /PrepareAD process creates these universal security groups in the root domain with a forest-wide scope. The groups are located in the Microsoft Exchange Security Groups container. They are globally available for permission assignments to Exchange Server resources in any domain, and they can contain users and groups from any domain in the forest.One important feature of the default configuration is that there is no Exchange Server Administrators group because that is not how Microsoft IT applies permissions for the Exchange Server Administrator role. Instead, Microsoft IT applies the permissions directly on the object in the configuration partition. Another important fact is that Exchange Organization Administrators have permissions to modify recipient attributes through membership in the Exchange Recipient Administrators group. Microsoft IT considered removing the Exchange Organization Administrators from the Exchange Recipient Administrators group but found no compelling reason to deviate from the default model during the initial deployment. The default permissions do not grant Exchange Organization Administrators rights to create, modify, or delete objects within the Active Directory domain-naming context, which would require at least Account Operators privileges. The authoritative source of user account information is a separate LOB application, maintained by the HR department and synchronized with Active Directory through MIIS2003.*Administration and Permissions Model (Approval Processes)The domain topology that Microsoft IT implemented in the Corporate Forest features an empty root domain to support strict management processes for schema updates and for granting forest-wide administrative rights and privileges. Only a very limited number of IT managers have administrative permissions in the root domain. Because the Exchange Setup /PrepareAD process creates the universal security groups of Exchange Server2007 in the root domain, Microsoft IT implicitly gained the ability to enforce formal approval processes for Exchange Server administration as well.Restricting access to the universal security groups of Exchange Server2007 in the root domain does not prevent Microsoft IT from delegating approval processes to individual teams and groups that are ultimately responsible for the corporate and Exchange Server environment, such as the Legal and Corporate Affairs (LCA) team and the Exchange Messaging team. To delegate approval processes, Microsoft IT added security groups from child domains to the universal security groups in the root domain. Team managers with permissions to change group membership information in child domains can use these security groups to delegate administrative rights.The figure shows the principle of delegating approval processes to individual teams and groups through security groups. As a best practice, Microsoft IT always uses security groups (in child domains) as opposed to individual user accounts to assign administrative permissions. The membership information of security groups in the root domain rarely changes.One interesting aspect of the Exchange Server2007 default permissions model is that when running setup with the /PrepareLegacyExchangePermissions option, it begins the automatic process of changing the permissions model of the Exchange Server2003 environment for coexistence with Exchange Server2007. Microsoft IT completes this process by running the setup with the /PrepareAD option, which finishes the changes to property sets and security groups. While applying new Exchange Server2007 permissions, Microsoft IT took the opportunity to review and clean up permission assignments from the Exchange Server2003 environment.Microsoft IT used security groups to grant administrative permissions to Exchange Server2003 resources at the organization and administrative group levels. Yet, some members of these groups no longer need these permissions. By using new security groups in the child domains, added to the Exchange Organization Administrators group in the root domain, Microsoft IT effectively devised the following strategy to reevaluate all existing Exchange Server administrators: Current Exchange Server administratorsIndividuals who need administrative rights to manage Exchange Server2007 resources must request them from the Systems Management team. Following each individual approval, the Systems Management team adds the corresponding user account to the new security groups to grant Exchange Organization Administrators permissions. Former Exchange Server administratorsIndividuals who no longer need administrative permissions automatically lose these rights as Microsoft IT decommissions Exchange Server2003 resources and corresponding administrator groups.*Message Routing TopologyAgile companies that heavily rely on e-mail in practically all areas of the business, such as Microsoft, cannot tolerate messages that take hours to reach their final destinations. Information must travel fast, reliably, predictably, and securely. For Microsoft, this means 99 percent of all messages within the corporate production environment must reach their final destination in 90 seconds worldwide. Of course, this SLA does not apply to messages that leave the Microsoft environment, because Microsoft IT cannot guarantee message delivery across external systems. Microsoft established this mail-delivery SLA during the Exchange Server2003 time frame, and it was equally important during the design of Exchange Server2007. Another important design goal was to increase security in the messaging backbone by means of access restrictions to messaging connectors, data encryption through TLS, and messaging antivirus protection based on Forefront Security for Exchange Server.Microsoft IT had to consider the following new Exchange Server2007 routing features in the design of the message routing topology: Active Directory sitesComparable to routing groups in Exchange Server2003, Active Directory sites define a boundary for Hub Transport servers to deliver messages directly to Mailbox servers, distribution group expansion servers, connector servers, and Edge Transport servers subscribed to the local site. For destinations in remote Active Directory sites, the Hub Transport server in the local site must relay the messages to a Hub Transport server in the remote site for further delivery. At least one Hub Transport server must exist in every Active Directory site with mailbox servers. IP site linksComparable to routing group connectors in Exchange Server2003, IP site links define logical paths between Active Directory sites, which Hub Transport servers use to perform message routing calculations. It is important to note that Active Directory supports IP and SMTP site links, whereas Exchange Server2007 ignores SMTP site links in the routing topology. If a Hub Transport server resides in a site connected only by an SMTP site link, routing errors will occur. It is necessary to replace the SMTP site link with an IP-based site link. Least-cost routing pathsEach IP site link is associated with a cost value that Exchange Server2007 uses to calculate the message transfer paths across the routing topology. If multiple IP site links exist between two Active Directory sites with Hub Transport servers, Exchange Server2007 transfers the message along the path with the least combined cost to the ultimate destination. Next hop selectionIf the least-cost routing path to the ultimate destination includes multiple Active Directory sites and IP site links, Exchange Server2007 attempts to relay the messages to an Active Directory site as close as possible to the ultimate destination when delivery to that destination fails. Queue at the point of failure and backoffIf message delivery to the next hop fails (for example, because there was no Hub Transport server available in the target site), Exchange Server2007 attempts to deliver the messages to an interim site along the least-cost routing path. This backoff mechanism starts with the Active Directory site directly adjacent to the unavailable next hop, and then backs off, site by site, along the least-cost routing path until a connection is established or no further site exists. Delayed fan-outIf a message has multiple recipients in destinations that share part of or the entire least-cost routing path, Exchange Server2007 transfers a single message copy to the point in the routing path where a fork occurs. At the fork, Exchange Server2007 splits the message into separate copies. The bifurcation at the latest possible point to preserve network bandwidth is called delayed fan-out.*Message Routing Topology (Network Infrastructure and Site Consolidation)An important aspect for planning message routing in an Exchange Server2007 environment is the physical network topology. The physical network determines the IP routing topology, which directly influences the Active Directory site topology, which in turn determines the message routing topology of Exchange Server2007. By using the existing Active Directory site infrastructure for message routing purposes, Exchange Server2007 takes advantage of an optimal network configuration with little need for adjustments.For example, Microsoft IT did not need to perform any network optimization to accommodate Exchange Server2007, although the Exchange Server2003 site and server consolidation that Microsoft IT conducted at a global scale three years ago benefited Microsoft IT in the Exchange Server2007 design. Microsoft IT only had four main Active Directory sites with Exchange Mailbox servers and reliable WAN links with sufficient net-available bandwidth to take into consideration for Exchange Server2007 topology and routing planning. The Exchange Server2003 site consolidation greatly reduced the complexity of the transition exercise for Microsoft IT.The fact that Microsoft IT only had to consider four main locations and Active Directory sites that already mapped to an optimized network infrastructure, led to the following benefits:Uncomplicated messaging topologyNo additional configuration was necessary to establish a functional Exchange Server2007 routing topologyalthough opportunities to increase the efficiency of message transfer still existed. However, managing only four sites that house Mailbox servers and optimizing message flow by means of Exchange Serverspecific IP site links was a straightforward undertaking for Microsoft IT.Best possible Hub Transport server utilizationMailbox servers submit messages for routing and transport to a Hub Transport server in the local Active Directory site. Depending on the location of the recipients, the Hub Transport server then transfers the messages to a Mailbox server within the local site, to a Hub Transport server in another Active Directory site, or through a messaging connector to another external destination. The result is that message transfer cannot work if there is no Hub Transport server in the local site of the Mailbox server. In addition, this implies that a small number of Active Directory sites with a large number of Mailbox servers provide a better Hub Transport server utilization than a large number of Active Directory sites with a small number of Mailbox servers. For example, Microsoft IT deployed three Hub Transport servers for 15 Mailbox servers in ADSITE_DUBLIN to perform load balancing and provide fault tolerance. If the Mailbox servers in Dublin resided in two Active Directory sites instead, Microsoft IT would have had to deploy one additional Hub Transport server, because each site would have required a minimum of two Hub Transport servers to achieve load balancing and fault tolerance.Reduced chance of server communication issuesFor purposes of message routing, client access, and unified messaging, it is important to deploy Hub Transport servers, Client Access servers, and Unified Messaging servers in each Active Directory site that contains Mailbox servers. Active Directory sites correspond to one or more IP subnets that represent areas with reliable, high-speed IP connectivity. Yet, it is possible to leave gaps or specify overlapping IP subnets in the Active Directory site topology, which can cause communication issues if Exchange Server2007 cannot correctly determine its site membership. Keeping the Active Directory site topology straightforward helps Microsoft IT avoid these types of issues. *Message Routing Topology (Dedicated Exchange Sites in the Active Directory Topology)Although dedicated Active Directory sites for Exchange Server may be used for domain controller/global catalog isolation and dedication of that infrastructure to Exchange servers for reliability or performance reasons, in the context of Exchange Server2007 transport, dedicated Exchange sites can complicate the routing topology. They deviate from the classic definition of an Active Directory site as areas with reliable, high-speed, low-latency IP connectivity. Because dedicated Exchange sites generally do not correspond to the physical network layout, they can lead to a message routing topology that does not use the physical network links as efficiently as possible. For example, Microsoft IT maintains a dedicated Exchange Active Directory site in addition to a central hub Active Directory site in Redmond. In the Active Directory replication topology, this dedicated Exchange site is a tail site. ADSITE_REDMOND is the Active Directory hub site that is used as the Active Directory replication focal point, yet this site does not contain any Hub Transport servers. Accordingly, at this point Exchange Server2007 cannot use ADSITE_REDMOND as a hub site for message routing purposes and by default interprets the Microsoft IT Exchange organization as a full-mesh topology where Exchange2007 Hub servers in each region connect to each other via a single SMTP hop. This does not match the Active Directory replication and network topology. Exchange Server2007 uses the full-mesh topology in this concrete scenario, because along the IP site links all message transfer paths appear to be direct.The figure illustrates this situation by placing the Active Directory site topology and default message routing topology on top of each other. The routing topology depicted in the figure works because Exchange Server2007 can transfer messages directly between the sites with Hub Transport servers (such as ADSITE_SINGAPORE to ADSITE_DUBLIN), yet all messages travel through the Redmond location according to the physical network layout. In this topology, all bifurcation of messages, sent to recipients in multiple sites, takes place at the source sites and not at the latest possible point along the physical path, which would be Redmond. For example, if a user in Sao Paulo sends a single message to recipients in the sites ADSITE_REDMOND-EXCHANGE, ADSITE_DUBLIN, and ADSITE_SINGAPORE, the source Hub Transport server in Sao Paulo establishes three separate SMTP connections, one SMTP connection to Hub Transport servers in each remote site, to transfer three message copies. Hence, the same message travels three times over the network from Sao Paulo to Redmond. Microsoft IT could avoid this by eliminating the dedicated Exchange site ADSITE_REDMOND-EXCHANGE and moving all Exchange servers to ADSITE_REDMOND. The Hub Transport servers in ADSITE_REDMOND would then be in the transfer path between ADSITE_SAO PAULO, ADSITE_DUBLIN, and ADSITE_SINGAPORE, which would mean that Exchange Server2007 could delay bifurcation until messages to recipients in multiple sites reach ADSITE_REDMOND. In this situation, the source server would only need to transfer one message copy to Redmond, the message routing topology would follow the physical network layout, and Microsoft IT would not have to take any extra configuration or optimization steps.Based on such considerations, it is a logical conclusion that the design of an ideal Exchange Server2007 environment takes the implications of dedicated Exchange Active Directory sites into account. On one hand, it is beneficial to keep the message routing topology straightforward and the complexities associated with maintaining and troubleshooting message transfer minimal. On the other hand, Microsoft IT had to weigh the benefits of eliminating the dedicated Redmond Exchange site ADSITE_REDMOND-EXCHANGE against the impact of such an undertaking on the overall deployment project in terms of costs, resources, and timelines. *Message Routing Topology (Optimized Message Transfer Between Hub Transport servers)Although Exchange Server2007 generated a functioning message routing topology without any extra design work, Microsoft IT decided to review the routing topology based on business and technical requirements to drive further optimizations. Key factors that influenced the optimization decision included the 90 seconds, 99 percent of the time mail-delivery SLA and the desire to save network bandwidth on WAN links by increasing the efficiency of message transfer. Important reasons that compelled Microsoft IT to optimize the Exchange Server2007 message transfer topology include the following: Efficient message flowAt Microsoft, 99 percent of the messages must reach their recipients within 90 seconds or less. Preserved WAN bandwidthThe corporate production environment handles more than 6 million internal messages daily. Although most message traffic stays in the local site or has Redmond headquarters as the destination, optimized message flow can help to preserve WAN bandwidth for all messages with recipients in multiple remote Active Directory sites.Having made the decision to optimize message routing, Microsoft IT augmented the Active Directory site link topology in order to take advantage of the Exchange Hub Transport servers in ADSITE_REDMOND-EXCHANGE. To achieve efficient message flow and preserve WAN bandwidth, it was necessary to place ADSITE_REDMOND-EXCHANGE in the routing path between ADSITE_DUBLIN, ADSITE_SAO PAULO, and ADSITE_SINGAPORE by creating additional Active Directory site IP links. This approach ensured that Exchange Server2007 could bifurcate messages traveling between regions closer to their destination. By configuring the ExchangeCost attribute on Active Directory site links, which Exchange Server2007 adds to the Active Directory site link definition, Microsoft IT was able to perform the message flow optimization without affecting the Active Directory replication topology. The ExchangeCost attribute is only relevant for Exchange Server2007 message routing decisions between sites, not Active Directory replication. To establish a hub/spoke topology between all sites with Exchange servers, Microsoft IT created three additional Active Directory IP site links (see the figure).Microsoft IT specified a Cost value of 999 (highest across the Active Directory topology) for these new IP site links so that Active Directory does not use these site links for directory replication.Using the Set-AdSiteLink cmdlet, Microsoft IT assigned an ExchangeCost value of 10 to the new Exchange-specific site links. This value is significantly lower than the Cost value of all other Active Directory site links, so that Exchange Server2007 uses the Exchange-specific site links for message routing path discovery.The figure illustrates how the Exchange-specific site links change the message routing topology. The dedicated Exchange site ADSITE_REDMOND-EXCHANGE in North America now acts as a fork in the routing path to the sites in ADSITE_DUBLIN, ADSITE_SAO PAULO, and ADSITE_SINGAPORE.Based on the Exchange-specific Active Directory/IP site link topology, Exchange Server2007 routes messages in the Corporate Forest as follows: messages to a single destination,messages to an unavailable destination, and messages to recipients in multiple sites.*Message Routing Topology (Increased Message Routing Security)To ensure compliance with legal and regulatory requirements, Microsoft IT encrypts most messaging traffic in the corporate production environment. The only exceptions are internal destinations without user mailboxes, such as lab and test environments. Microsoft IT uses the following encryption technologies to prevent unauthorized access to information during message transmission: IP securityIPSec encrypts data communication and prevents unauthorized access to resources in the corporate production environment. Because IPSec works at the IP layer, it can help secure communication between servers without relying on the application to support the encryption natively. Microsoft IT extensively used IPSec encryption in its Exchange Server2003 environment to help secure internal SMTP transactions and continues its use today in scenarios where SMTP servers and applications do not support native TLS-based SMTP encryption. Although IPSec technology offers strong encryption controls, managing custom IPSec policies can be quite cumbersome. With the transition to Exchange Server2007, Microsoft IT was able to accomplish most of the transport encryption by using native Exchange Server product features. Transport Layer SecurityExchange Server2007 supports TLS right out of the box. Hub Transport servers use TLS to encrypt all message traffic within the Exchange Server2007 environment and rely on opportunistic TLS encryption for communication with remote destinations, such as Hub Transport servers in other Microsoft ITmanaged forests. Edge Transport servers also support TLS and domain security to establish security-enhanced message transfer paths to business partners over the Internet. Native support for SMTP TLS on Hub Transport and Edge Transport servers enabled Microsoft IT to eliminate the dependency on complex IPSec policies for encryption of internal and external messages in transit.In addition to encrypting messaging traffic internally, Microsoft IT also protects its internal messaging environment by restricting access to inbound SMTP submission points. This helps Microsoft IT minimize mail spoofing and ensure that unauthorized SMTP mail submissions from rogue internal clients and applications do not affect corporate e-mail communications. To accomplish this goal, Microsoft IT removes all default Receive connectors on the Hub Transport servers and configures custom Receive connectors by using the New-ReceiveConnector cmdlet to accept only authenticated SMTP connections from other Hub Transport servers and Edge Transport servers in the environment. To meet the needs of internal SMTP applications and clients, Microsoft IT established a separate SMTP gateway infrastructure based on Exchange Server2007 Hub Transport servers that enforces mail submission access controls, filtering, and other security checks.Furthermore, Microsoft IT deployed Forefront Security for Exchange Server on all Hub Transport and Edge Transport servers to implement messaging protection against viruses and other malicious e-mail content at multiple layers in the Exchange Server2007 infrastructure. Despite the fact that internal messages and messages from the Internet might pass through multiple Hub Transport and Edge Transport servers, performance-intensive antivirus scanning is performed only once. Forefront Security adds a security-enhanced antivirus header to each scanned message, so further Hub Transport or Edge Transport servers do not need to scan the same message a second time. This avoids processing overhead while maintaining an effective level of antivirus protection for all inbound, outbound, and internal e-mail messages.*Message Routing Topology (Coexistence with Exchange Server 2003)Coexistence with Exchange Server2003 introduces a special connectivity scenario that Microsoft IT had to consider during the planning phase. Exchange Server2007 integrates into the existing routing topology through a special routing group, called EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR), which has the following limitations and restrictions for compatibility reasons: All computers running Exchange Server2007 must be members of EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR)Within this routing group, Exchange2007 servers use the Active Directory site topology for message routing. Computers running previous versions of Exchange Server cannot be members of EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR)Exchange Server2003 is unaware of the various Exchange2007 server roles and cannot recognize server-specific communication requirements. Therefore, Exchange Server2003 cannot coexist in the same routing group with Exchange Server2007.One important implication of these restrictions is that EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) differs from the classical Exchange Server routing group in scope and definition. EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) is global in scope and does not define a network region with reliable connectivity. Rather, EXCHANGE ROUTING GROUP (DWBGZMFD01QNBJR) defines a parallel hemisphere in which the Active Directory site topology defines the message routing topology, as illustrated in the figure.Microsoft IT started the transition of the Corporate Forest messaging environment from Exchange Server2003 to Exchange Server2007 in the Redmond location. With the introduction of Exchange Server2007 Mailbox servers, Hub Transport servers, and other server roles in that location, Microsoft IT needed to establish message transfer between the legacy Exchange Server2003 and new Exchange Server2007 environment and maintain this messaging connectivity for the entire period of coexistence. For that reason, Microsoft IT decided to connect the Exchange Server2007specific routing group first to the Exchange Server2003 routing group RG_REDMOND in North America by using routing group connectors. Later, during the transition phase, as the Exchange Server2007 infrastructure expanded into the regions, Microsoft IT created routing group connectors between the Exchange Server2007specific routing group and the regional Exchange Server2003 routing groups. Shortly after this stage, Microsoft IT raised the cost of the legacy routing group connectors between regional routing groups in the Exchange Server2003 environment, to route all messaging traffic between regions through the Exchange Server2007 messaging backbone and take advantage of native Exchange Server2007 routing features.*Message Routing Topology (Coexistence with Exchange Server 2003)Microsoft IT performed the following steps to establish messaging connectivity between Exchange Server 2003 and Exchange Server 2007 in the corporate production environment:During the first Exchange Server 2007 installation, Microsoft IT selected a bridgehead server from RG_REDMOND to establish the initial routing group connection.To grant all Exchange 2003 servers the necessary send and receive permissions on Hub Transport servers, Microsoft IT made sure that the ExchangeLegacyInterop universal security group in the root domain included the Exchange Domain Servers global security groups from the child domains. When using Exchange Server 2007 tools to create the routing group connectors, the specified Exchange Server 2003 servers are automatically added to the ExchangeLegacyInterop group to grant the legacy servers the required permissions to send mail to and receive mail from Exchange Server 2007 Hub Transport servers.For fault tolerance and load balancing, Microsoft IT installed additional Hub Transport servers in ADSITE_REDMOND-EXCHANGE and adjusted the bridgehead server configuration of the initial routing group connection, as summarized in Table 2.Following the deployment of Exchange Server 2007 in Dublin and Singapore, Microsoft IT established additional RGCs to optimize message flow and facilitate decommissioning of legacy routing groups. The Sao Paulo location did not require an additional RGC, because Microsoft IT transitioned all recipients in this location in a single step. Because Microsoft IT used routing group connectors in a straightforward hub/spoke topology with the default setting of Any local server can send mail over this connector, it was not necessary for Microsoft IT to suppress link state updates on Exchange 2003 servers by specifying the SuppressStateChanges Registry parameter in preparation of the deployment of additional routing group connectors. Microsoft IT recommends that all customers suppress link state updates regardless of routing group connector configuration. *Server Architectures and DesignsSome important business requirements that Microsoft IT addressed in the server architectures and designs for Exchange Server2007 revolved around the goals of eliminating the performance and scalability issues of the 32-bit platform and establishing a flexible messaging infrastructure to support growing mailboxes and a larger number of clients. The product group helped Microsoft IT achieve the first of these goals by optimizing Exchange Server2007 for 64-bit server hardware, specifically x64 processors. By exploiting the advantages of dedicated Exchange2007 server roles in combination with load-balanced, fault-tolerant server configurations, the Exchange Messaging team was also able to maintain SLAs with 99.99 percent availability of messaging services.*Server Architectures and Designs (Flexible and Scalable Messaging Infrastructure)Microsoft IT heavily focused on single-role server deployments in almost all regions of the messaging environment. A server role is a logical unit that groups a selected set of server features and components together to perform a specific messaging function. Although the single-role server design increases the hardware footprint in the data center, it also increases the flexibility and scalability of the messaging environment. The figure illustrates an Exchange Server2007 architecture based on single-role server designs.Exchange Server2007 supports the following five separate server roles to perform the tasks of an enterprise messaging system: Client Access serversSupport Post Office Protocol3 (POP3) and Internet Message Access Protocol4 (IMAP4) clients, as well as Exchange ActiveSync, Office Outlook Web Access, and Outlook Anywhere and new Outlook2007 client functions. Edge Transport serversHandle message traffic to and from the Internet and run spam filters. Microsoft IT also installs Forefront Security for Exchange Server on all Edge Transport servers for virus scanning. Hub Transport serversPerform the internal message transfer, distribution list expansions, and message conversions between Internet mail and Exchange Server message formats. At Microsoft, all Hub Transport servers also run Forefront Security for Exchange Server for virus scanning. Mailbox serversMaintain mailbox store databases and provide Office Outlook clients and Client Access servers with access to the data. Unified Messaging serversIntegrate voice and fax with e-mail messaging and run Outlook Voice Access.*Server Architectures and Designs (Multiple-Role and Single-Role Server Designs)With the exception of the Edge Transport server role, Exchange Server2007 supports multiple-role server deployments. The Client Access server role, Hub Transport server role, Mailbox server role, and Unified Messaging server role can coexist on the same computer in any combination. Placing several roles on a single computer is advantageous for small Exchange Server deployments. The multiple-role approach provides the benefits of a reduced server footprint and can help to minimize the hardware costs. For example, Microsoft IT deployed a multiple-role server in Sao Paulo for the Hub Transport server role, Client Access server role, and Unified Messaging server role to use hardware resources efficiently. Similar to the Exchange Server2003 design, Microsoft IT consolidated server roles in this location due to moderate workload. However, the Mailbox server in Sao Paulo is a single-role server according to the requirements for CCR. Microsoft IT based its decisions to combine Exchange server roles on the same hardware or separate them between dedicated servers on capacity, performance, and availability demands. Mailbox servers are prime examples of systems with high capacity, performance, and availability requirements at Microsoft. Accordingly, Microsoft IT deployed the Mailbox servers in all regions in a single-role design, which enabled Microsoft IT to eliminate single points of failure in the Mailbox server configuration by using CCR. In Redmond, Dublin, and Singapore, Microsoft IT also used the single-role design for the remaining server roles, because these regions include a large number of users and multiple Mailbox servers. Single-role server deployments provide Microsoft IT with the following benefits: Optimized server hardware and software componentsDuring the initial production deployment in 2006, Microsoft IT used the following hardware per server role: Client AccessTwo dual-core AMD Opteron, 2.2 gigahertz (GHz), with 4 GB of memory. Edge TransportTwo dual-core AMD Opteron, 2.2 GHz, with 8 GB of memory. Hub TransportTwo dual-core AMD Opteron, 2.2 GHz, with 8 GB of memory. Mailbox (2,000 users with 500-MB quotas)Two dual-core Intel Xeon, 2.66 GHz, with 12 GB of memory. Mailbox (2,400 users with 2-GB quotas)Two dual-core Intel Xeon, 3.0 GHz, with 16 GB of memory. Mailbox server (3,600 users with 2-GB quotas)Four dual-core AMD Opteron, 2.6 GHz, with 24 GB of memory. Unified MessagingOne dual-core AMD Opteron, 2.2 GHz, with 4 GB of memory. Flexible systems scaling approachThe single-role server deployment enables Microsoft IT to design server hardware more accurately according to specific tasks and increase the capacity of the messaging environment selectively according to specific demands and changing trends, as illustrated in the figure from the previous slide. Structured system administration and maintenanceAt Microsoft IT, several groups deploy, manage, and maintain the messaging environment. These groups collaborate closely while individual system engineers, program managers, and service managers specialize in specific areas of expertise, closely related to server roles. For example, different system engineers designed the message routing topology and the Mailbox server configuration. Role-specific load balancing and fault toleranceDifferent server roles support different techniques and architectures for load balancing and fault tolerance. For example, if multiple Hub Transport servers exist in the same Active Directory site, Exchange Server2007 balances the message traffic automatically between these servers, whereas Mailbox servers are not load-balanced in the same way. The table shows the number of servers and the technologies per server role that Microsoft IT uses in the corporate production environment to implement load balancing and fault tolerance. *Server Architectures and Designs (Scaling Up Server Designs)Following the successful completion of the initial production rollout, Microsoft IT began to develop new, scaled-up Mailbox server designs in order to increase the density of the user mailboxes per server, consolidate resources, and reduce maintenance overhead. The new designs take advantage of quad-core processors and scale up to 6,000 users with 500-MB mailboxes per Mailbox server. In the initial design, scaling out with additional Mailbox servers was the only reasonable option for Microsoft IT to handle the increased demand for mailbox sizes of 500 MB and 2 GB without jeopardizing SLAs. The new designs enabled Microsoft IT to consolidate all the 2,000-user Mailbox servers used during the original beta deployment by a factor of three. Because each Mailbox server corresponds to a two-node CCR cluster, Microsoft IT was able to repurpose more than 60 recently purchased enterprise servers.To scale up to 6,000 mailboxes per server, Microsoft IT capitalized on the newest processor technologies at the time of deployment, such as the quad-core Intel Xeon Processor X5355. This enabled the elimination of the processor bottleneck on Microsoft IT Mailbox servers that existed on the dual-core CPU systems used during initial deployment and prevented achieving that scale. The X5355-compatible server model that Microsoft IT selected offered eight slots for Fully Buffered Dual Inline Memory Modules (FB-DIMM). This implies that with a maximum module size of 4 GB, the server architecture accommodates up to 32 GB of memory, which corresponds to a memory configuration at the upper end of product recommendations (2 GB + 2 MB to 5 MB/mailbox). However, at the time Microsoft IT developed this server design, 4 GB DIMMs did not offer an attractive memory capacity/price ratio. To remain cost-efficient, Microsoft IT decided to use 2-GB memory modules instead and designed the storage solution according to the I/O requirements that result from having less memory per user on the server.In comparison to the initial Mailbox servers, the new design allocated 60 percent less memory per user. Among other things, this means that even though the Extensible Storage Engine (ESE) of Exchange Server2007 can cache any amount, the limited physical memory available means that the ESE can only cache approximately 2 MB of messaging data per user. This has a direct impact on the amount of I/O operations, because with less memory to cache data, the storage engine must go to disk more often to reload frequently used data. After consolidation to the new server platform, the same users will have a higher I/O profile. To meet the increased requirements in terms of I/O operations per second (IOPS), Microsoft IT optimized the design of the storage subsystem.*Mailbox Storage DesignThe mailbox is one of the very few components in an Exchange Server2007 organization that cannot be load-balanced across multiple servers. Each individual mailbox is unique and can only reside in one mailbox database on one active Mailbox server. It follows that the mailbox store is one of the most critical Exchange Server components that directly affect the availability of messaging services. With previous versions of Exchange Server, Microsoft IT relied on SAN solutions to provide the necessary configuration for its mailbox clusters. SAN provided a higher level of availability due to the architecture, and enabled Microsoft IT to achieve the number of disks required for I/O throughput and scalability. Mailbox servers clustered by using Windows Clustering and SAN-based storage enabled Microsoft IT to achieve 99.99 percent availability with Exchange Server2003, yet the shared storage solution was a single point of failure that was expensive and required specialized skills to optimize and maintain the configuration. Additionally, the mailbox databases on disks remained single points of failure.To break through the old limitations, Microsoft IT defined the following storage design requirements for Exchange Server2007: Continue to maintain 99.99 percent availability at the service level. Increase Mailbox server resilience by removing single points of failure in the storage subsystem and its components. Reduce storage infrastructure costs and increase mailbox quotas from 200 MB to 500MB and 2 GB depending on the type of mailboxes. Increase deleted items retention from three days to 14 days to enable users to recover deleted mail items within a larger time window without the necessity of restoring from backup.The new Mailbox server design with CCR required Microsoft IT to double the storage and the storage arrays in order to remove the data and storage points of failure. Microsoft IT looked into the cost of deploying Mailbox servers with SAN technology and determined that CCR with SAN-based storage would require redundant SAN environments, which were cost prohibitive. Instead, Microsoft IT decided to use direct attached storage, which met the requirements, reduced costs, and reduced operational complexity for Microsoft IT.With direct attached storage, Microsoft IT eliminated the need to use SAN, Internet SCSI (iSCSI), or other shared storage technologies in the cluster configuration. Every cluster node can use its own direct attached storage subsystem to maintain a separate copy of mailbox databases, which also helps to improve the failover behavior, because a server failure due to hardware or software problems that affect a database on the first node is unlikely to affect the recovery operation on the second node. By removing single points of failure that existed in the Exchange Server2003 architecture and by using improved failover behavior in the Mailbox server configuration, Microsoft IT felt confident in making the switch from SAN to direct access storage (DAS) storage as the basis of its server architectures in the Exchange Server2007based messaging environment. *Mailbox Storage Design (Eliminating Storage as the Single Point of Failure)Exchange Server2007 supports two different clustering techniques for Mailbox server configurations: single copy cluster (SCC) and cluster continuous replication (CCR). Both types rely on Windows Clustering, yet only CCR provides redundancy at the storage level by replicating mailbox data from one server to another at the storage group level through a mechanism commonly known as asynchronous log shipping. SCC is comparable to an Exchange Server2003 clustering configuration where all cluster nodes use a shared storage subsystem for quorum resource, mailbox databases, and transaction logs. Because only CCR eliminates the mailbox store as a single point of failure, and the fact that CCR technology does not impose any specialized hardware or storage requirements, Microsoft IT decided to use this technology as the core of its Mailbox server designs to increase Mailbox server resilience from storage-level failures.To use server hardware efficiently while providing the required redundancy level through CCR, Microsoft IT implemented two-node Majority Node Set (MNS) server clusters with file-share witness. Each cluster node stores a copy of the MNS quorum on the local system drive and keeps it synchronized with the other node. The file-share witness feature enables the use of a file share that is external to the cluster as an additional vote to determine the status of the cluster in a two-node MNS quorum cluster deployment. This helps to avoid an occurrence of network partition within the cluster, also known as split-brain syndrome. Split-brain syndrome occurs when all networks designated to carry internal cluster communications fail, and nodes cannot receive heartbeat signals from each other. To enable the file-share witness feature, Microsoft IT specifies a file share on a Hub Transport server in the MNSFileShare property of the MNS resource configuration.The figure illustrates the Microsoft IT CCR configuration. Both CCR cluster nodes and the file-share witness must exist in the same Active Directory site. Microsoft IT did not deploy geographically dispersed clusters during the initial production rollout of Exchange Server2007 to avoid the complexities of building IP subnets and Active Directory sites that span multiple geographic locations, which would be a requirement for distributed Windows Server2003 cluster nodes.As depicted in the figure, Microsoft IT uses three network adapters in each cluster node. The first two adapters connect the cluster node in a NIC teaming configuration through separate Gigabit Ethernet switches to the public network