Vincent YimPremier Field EngineerMicrosoft Services

Troubleshooting Hybrid Mailflow

MNGIN301

AgendaRefresher/Overview of Hybrid RoutingMailflow OptionsEOP in HybridReview tools to assist in mail flow troubleshootingIssuesOther fun stuffQuestions

Refresher/Overview of Hybrid Routing2 Distinct Exchange organizationsHCW creates connectors in each Exchange org. # of connectors vary based on Exchange versionSecure Mail

On-premises Organization

Exchange Online

Protection

Inbound from Office 365

Outbound to On-premises

Inbound from On-premises

Outbound to Office 365*

Exchange Online

Refresher/Overview of Hybrid RoutingAll messages that are sent between on-premises and ExO are sent over a secure connection using TLS• The Hybrid Configuration wizard creates a dedicated send connector on-premises

scoped to the coexistence domain (tenant.mail.microsoftonline.com)• An outbound connector in EOP is also created and is scoped to the default SMTP

domain (contoso.com)

Each organization is configured to treat messages sent from the other organization as internal• This allows messages to bypass anti-spam settings and other services

The TLS connection for on-prem server must be a minimum of Exchange 2010 SP1Any other SMTP end point accepting the messages will cause the required headers to be lost which will impact secure mail functionality

Refresher/Overview of Hybrid RoutingE-mail domain sharing

Both orgs will accept “contoso.com” authoritative

How do we prevent mail loops?Actually, it’s all about how addressing works

Requires a coexistence domain for “Backboning” mailflow

Refresher/Overview of Hybrid RoutingCoexistence Domain• Based off of the Microsoft Online Default Routing Domain• The coexistence domain is a domain created for each Office 365 tenant

in the format of <your tenant>.mail.onmicrosoft.com domain• For example, if your Default Routing domain is “tenant.onmicrosoft.com”

then your coexistence domain would be “tenant.mail.onmicrosoft.com”• Created when you activate DirSync in your Office 365 tenant• AutoDiscover and MX records created automatically for this domain• Provides the backbone of all coexistence features• Added as an on-premises email address policy when the HCW is run• Mailboxes moved to Exchange Online will have the coexistence domain

stamped on their user object as a target address

Demo

DirsyncStates Pre/Post Migration

MailflowOptions

10

On-Premises Organization

External User

Exchange

Exchange Online

Exchange Online Protection

Inte

rnet

Third Party Email

Security System

“Chris”Cloud

Mailbox

“David”On-premises

Mailbox

Secure MailEncrypted & Authenticated Mail Flow

MX resolves to on-

premises gateway

MX is switched to Exchange

Online Protection

Outbound Exchange

Online traffic is delivered

direct

You can choose to

route outbound on-

premises mail via EOP

Mail Flow OptionsIn addition to choosing how inbound messages are routed, you can also choose how outbound messages sent from Exchange Online recipients are routed. The following describes the available options:• Centralized mail control: This option routes outbound messages sent from the

Exchange Online users through on-premises• This enables you to apply compliance rules to these messages that must be applied

to all of your recipients, regardless of whether they're located in Exchange Online or on-premises

• Decentralized mail control: This option routes outbound messages sent from Exchange Online directly to the InternetUse this option, if you do not need to apply any on-premises policies or other

processing to messages that are sent from recipients in the Exchange Online

MailflowOptions

12Exchange Online

Exchange Online Protection

On-Premises Organization

Exchange

Third Party Email

Security System

External User

Inte

rnet

“Chris”Cloud

Mailbox

“David”On-premises

Mailbox

Secure MailEncrypted & Authenticated Mail Flow

MX resolves to on-

premises gateway

All email in and out of the

Exchange Online tenant must go via on-premises

MX is switched to Exchange

Online Protection

EOPWhen you create inbound/outbound connectors in Exchange Online Admin Center, these are sitting at the edge (EOP)

SPAM Filtering Bypassed

Review Tools for TroubleshootingDelivery reportsEnd user can run. Eliminates some helpdesk callsSomewhat useless to Admin

Message TraceLoopsNDRsMessages dropped due to virusExport to CSV

Use the protocol logSet to verbose

Review Tools for TroubleshootingAnalyze HeadersExRCA has Message Header AnalyzerOWA MHA App

Telnet(your Exchange server might be using IP that's been blacklisted by SPAMHAUS or one of other RBL services in use by EOP)

DLP policy ruleHits found through message traceOr EACOr (delayed) Mail Protection Reports for Exchange

Demo

Mail Protection Reports for Exchange

Other Fun stuff• Testing and Tracing Malware Filters• Create a file called EICAR.txt with the following text: X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*

• Attach EICAR.TXT to a new mail message, and send it through the service. • Confirm your antimalware filter settings have taken affect (policy changes can take up

to an hour to replicate across datacenters)• This “EICAR” test attachment will cause the message to be treated as malicious

antivirus/antimalware engines

Other Fun stuff• Testing and Tracing Content Filter• A GTUBE message should always be detected as spam by the content filter, and the actions that are performed upon the message should match your configured settings. Include the following GTUBE text in a mail message on a single line, without any spaces or line breaks:

• XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

Other fun stuffOn prem senders to internet recipients will get SPAM filteringDemo

Other fun stuffOutbound SPAM filterWhy did the on-prem message route through high risk delivery pool?Outbound spam filtering is needed because malicious programmers and their malware are out there taking over computers inside corporate networks every day. This means that users in your organization can be sending large amounts of outbound spam without your knowledge

IssuesRunning a Hybrid server from home?

ISPs using dynamic IP ranges will connect, but sessions will then be dropped by EOP.

"454 4.7.5 Certificate validation failure." CRL check from hybrid server

SMTP fixup/mailguard220 *************************************************************************************************************** The above is a tell-tale sign that mailguard is enabled on a firewall appliance (most likely Cisco PIX), and it prevents either side from seeing the STARTTLS verb.Cannot perform secure mail flow without StartTLS verb

IssuesChanging datacenter IP ranges? Quite possibly need to re-run HCW if datacenter IP changesWith Exchange 2010 HCW, point-in-time list is copied

IssuesWith Exchange 2010 HCW, you may need to adjust the EHLO response guessed by HCW 

Issues Missing header?X-MS-Exhange-Organization-AuthAs = Internal or AnonymousIf anonymous, your message took another path

© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.