Exchange Server 2007 Trouble

8/13/2019 Exchange Server 2007 Trouble

1/55

Advanced TroubleshootingStrategies For Microsoft

Exchange Server 2007

Scott SchnollPrincipal Technical Writer

Exchange Server Product Group

Microsoft Corporation


2/55

Agenda

Troubleshoot Methodology

Exchange Troubleshooting Tools

Diagnostic Logging in Exchange Area-specific Troubleshooting

Setup

Performance Transport


3/55

Troubleshooting Methodology


4/55

Troubleshooting Methodology

Tools

Built

-

in tools

Operating

system tools

Advanced tools

Notification,

corrective

action, trend

analysis

Monitoring

Start with

a baseline

Without one,

you have no

comparisons

With one, you

can spot

problems

Knowledge

How

components

work

How components

interact

How components

depend on other

elements


5/55



6/55


Monitoring

Message Tracking

Queue Viewer

Routing Log Viewer

Performance Monitor

Troubleshooting

Best Practices Analyzer

Database Troubleshooter

Mail Flow Troubleshooter

Performance Troubleshooter


7/55

Exchange TroubleshootingTools

General Cmdlets

Test-SystemHealth

Test-ServiceHealth

Client Access Cmdlets

Test-MAPIConnectivity

Test-ActiveSyncConnectivity

Test-IMAPConnectivity

Test-POPConnectivity

Test-OWAConnectivity

Test-UMConnectivity

Test-WebServicesConnectivityTest-OutlookWebServices


8/55

Exchange TroubleshootingTools

CI and CR Cmdlets

Test-ExchangeSearch

Test-ReplicationHealth

Transport Cmdlets

Test-MailFlow

Test-SenderID

Test-IPBlockListProvider

Test-IPAllowListProvider

Test-EdgeSynchronization


9/55

Diagnostic Logging In Exchange


10/55


Exchange logging quite extensive

Starts with Setup

Continues through life of Exchange server

Transport Logs

Message Tracking Logs

Protocol Logs (SMTP)

Agent Logs

Connectivity Logs Routing Logs

Pipeline Tracing Logs


11/55


Mailbox Logs

Messaging Records Management Logs

Cluster Logs

Client Access Logs

Protocol Logs (POP3, IMPA4)

IIS Logs

General Logs

Event Logs

Certificate Logs


12/55


Get-EventLogLevel

Set-EventLogLevel -Level

Logging Level Description

Lowest Only critical events, error events, and events with a logging level

of zero are logged; default level for all processes exceptMSExchange ADAccess\Topology and MSExchangeADAccess\Validation

Low Events with a logging level of 1 or lower are logged; default levelfor MSExchange ADAccess\Topology and MSExchangeADAccess\Validation

Medium Events with a logging level of 3 or lower are logged.

Maximum Events with a logging level of 5 or lower are logged.

Expert Events with a logging level of 7 or lower are logged.


13/55


Best Practices

Be aware of impact to monitoring/eventlog collection agents

Set EventLogLevel back to original levelwhen finished troubleshooting

Using wildcards

Asterisks are only for EventSource part of syntax

Get-EventLogLevel MSExchangeIS\9000*\*

Get-EventLogLevel MSExchangeIS\9000 Private\*

Research events at Errors and Events MessageCenter
http://go.microsoft.com/fwlink/?LinkId=34258http://go.microsoft.com/fwlink/?LinkId=34258http://go.microsoft.com/fwlink/?LinkId=34258http://go.microsoft.com/fwlink/?LinkId=34258


14/55

Troubleshooting Exchange Setup


15/55


Use Setup logs to troubleshoot errors thatoccur during setup or block installationGet-SetupLog.ps1C:\ExchangeSetupLogs\ExchangeSetup.log error tree

Get-SetupLog tree:$false error:$false | Where {

$_.status eq "Error" } | select datetime, depth,description, status | Out-HTML | Out-IELog name and path Description

\ExchangeSetupLogs\ExchangeSetup.log

Tracks progress of every task performed duringSetup; contains details on pre-req checks,installation progress, and config changes made bySetup

\ExchangeSetupLogs\ExchangeSetup.msilog

Windows Installer log file that contains details onextraction of Exchange code from installer file(ExchangeServer.msi)


16/55


ExchangeSetup.log is most relevant/usefulwhen troubleshooting

Several documented resolutions for Setupfailures at http://technet.microsoft.com/en-us/library/bb232206(EXCHG.80).aspx

Task levels denoted by [X]

[0]Begin main run of a particular task

[1]High level run of a specific task

[2]Subset of a particular task
http://technet.microsoft.com/en-us/library/bb232206(EXCHG.80).aspxhttp://technet.microsoft.com/en-us/library/bb232206(EXCHG.80).aspxhttp://technet.microsoft.com/en-us/library/bb232206(EXCHG.80).aspxhttp://technet.microsoft.com/en-us/library/bb232206(EXCHG.80).aspxhttp://technet.microsoft.com/en-us/library/bb232206(EXCHG.80).aspx


17/55


[1/27/2008 3:46:26 PM] [0]**********************************************

[1/27/2008 3:46:26 PM] [0] Starting Microsoft Exchange 2007 Setup[1/27/2008 3:46:26 PM] [0] **********************************************

...

[1/27/2008 4:11:12 PM] [0] End of Setup

[1/27/2008 4:11:12 PM] [0] **********************************************

[1/27/2008 4:11:57 PM] [1] Executing '$RoleTargetVersion ="8.1.240.06"', handleError = False

[1/27/2008 4:11:57 PM] [2]Launching sub-task '$error.Clear();

$RoleFqdnOrName = exmbx1.contoso.com"'.

[1/27/2008 3:52:31 PM] [0] ExSetupUIwas started with the followingcommand: '-mode:install -sourcedir:D:\amd64 /FromSetup'.


18/55


In which phase of Setup did failure occur?

Bootstrap phasedisplays canopener and pre-reqlinks for .NET Framework 2.0, MicrosoftManagement Console 3.0, and WindowsPowerShell 1.0

File copy phasecopies core install files to%TEMP%\ExchangeServerSetup and sets BestPractices Analyzer XML file into culture-specific

folder (e.g., EN for English) Setup wizard phasewalks admin through GUI-

based setup (license agreement, error reporting,paths, type, roles, etc.)


19/55


In which phase of Setup did failure occur?

Readiness check phaseuses Best PracticesAnalyzer engine and XML (Test-SetupHealth)

rules file to verify system and organizationalreadiness for selected install type

Installation phasedeletes temporary files andproceeds with Org and domain prep (if notalready done) and installation and configurationof specified role(s)


20/55


Recovering from Failed Setup

Setup creates Watermark entry in registry toresume at point of failureHKLM\Software\Microsoft\Exchange\v8.0\\

The value for Watermark can be mapped to aninstall task in a *.PS1 file in\ExchangeSetupLogs

If a Watermark is present, note for which role,

then run the following to resume and completeinstallation:Setup.com /roles:


21/55

Troubleshooting

Exchange 2007 Performance


22/55

Troubleshooting Exchange2007 Performance Significant changes in architecture change

the ways in which you troubleshootand what you troubleshoot

Scoping How many servers affected? Which servers are affected?

What are the current queue states?

Are queues growing?Are performance counters spiking?

Are external dependencies healthy?


23/55

Troubleshooting Exchange 2007Performance

Consider the performance impact of

Antivirus (file system and Exchange-based)

Backup applicationsArchiving and compliance, including MRM

Monitoring agents and tools

Desktop tools that integrate with Outlook


24/55


Isolate cause of resource issues using Windows Task Manager

Performance Monitor

Process Monitor

Network Monitor

Exchange Profile Analyzer

Event Viewer

Performance Troubleshooting Analyzer

Watch out for renamed objects in SP1 Exchange Database object renamed

to MSExchange Database


25/55


Check for counter values over thresholdsObject \ Counter Description ThresholdProcessor\% Processor Time(_Total)

Percentage of time the processor isrunning non-idle threads

90% (peak)75% (ongoing)

System\Processor QueueLength

Number of threads in processorqueue

2

Network Interface\BytesTotal/sec

Rate at which network adapter isprocessing data bytes

6-7 MB/sec (100 MBps)60-70 MB/sec (1000 MBps)

Network Interface\PacketsOutbound Errors

Number of outbound packets thatcould not be transmitted due to errors

0

LogicalDisk\Avg. Disk sec/Read Average time of a read of data fromdisk

50 ms (logs, peak)20 ms (logs, ongoing)

LogicalDisk\Avg. Disk sec/Read Average time of a read of data fromdisk

50 ms (database, peak)20 ms (database, ongoing)

LogicalDisk\Avg. Disk sec/Write Average time of a write of data to disk 50 ms (logs, peak)10 ms (logs, ongoing)

LogicalDisk\Avg. Disk sec/ReadLogicalDisk\Avg. Disk sec/Write

Average time of reads/writes on disk 10 ms (TEMP/TMP, Pagefiledisk, SMTP queue disk


26/55


Check for counter values over thresholdsObject \ Counter Description Threshold

MSExchangeIS\RPC Averaged Latency RPC latency averaged for last 1024packets

25 ms

MSExchangeIS\RPC Requests Number of client requests being processedby IS

30

MSExchange ADAccess DomainControllers\Long running LDAPoperations/Min

Number of LDAP operations on DC thattook longer than 15 seconds/Min 50

MSExchange Database\Version bucketsallocated (Information Store instance)

Number of version buckets (16K chunks ofversion store) allocated

1,800

MSExchangeTransport Queues\Largest

Delivery Queue Length

Number of messages in largest delivery

queue 200

MSExchange Database ==>Instances\Log Bytes Write/sec

Rate at which bytes are written to log512,000

.NET CLR Memory\% Time in GC Percentage of elapsed time spent ingarbage collection since last garbagecollection cycle

10 %


27/55


Performance Analyzer Log (PAL)http://www.codeplex.com/pal

Generate HTML reports from performancemonitor counter log file (.blg file)

Uses XML configuration files that parse themost important counters for Exchange

performance issues and issues alerts whenthresholds are exceeded for those counters

T bl h i E h 200
http://www.codeplex.com/palhttp://www.codeplex.com/pal


28/55


Windows Server 2008 (and Vista)include new TCP auto-tuning features

Not all network devices (routers, switches, firewalls,

etc.) support these features, and some can actuallymake things much slower Cisco PIX 500 Series Firewall, Cisco PIX 10000 Firewall,

Cisco PIX Classic Firewall, Cisco IOS Firewall, SonicwallFirewall, Check Point Firewall, some NG R55 routers,

some Netgear routers

Disable auto-tuning on Windows 2008/Vista: netsh interface tcp set global autotuninglevel=disabled


29/55

Troubleshooting

Exchange 2007 Transport

T bl h ti T l


30/55

Troubleshooting Tools

ExTRA: Exchange Troubleshooting Assistant Internal/External DSN received

Issues with Queue (size, status)

Message Tracking Lost Messages

Routing Log Viewer (SP1) Routing and Topology issues

Advanced: ETW Tracing, Pipeline Tracing Typically as part of a CSS escalation

E TRA B i


31/55

ExTRA Basics

A sibling tool to the Microsoft ExchangeServer Best Practices Analyzer (ExBPA)

Union of troubleshooting tools and otherrelated functionality

ExPTA: Exchange Performance TroubleshootingAnalyzer

ExDRA: Exchange Disaster Recovery Analyzer ExMFA: Exchange Mail Flow Analyzer

E TRA P i it


32/55

ExTRA Prerequisites

ExTRA 1.1 (Downlevel version) .NET Framework version 1.1

IIS Common Files (to allow remote metabase access)

ExTRA 2007 (in Toolbox)

Installed with Exchange Management Tools

IIS Common Files

Fix for SmtpClient issue in .NET 2.0 SP1

For both versions Need sufficient credentials to gather data from both Active

Directory and Exchange servers

S t B d A l i


33/55

Symptom-Based Analysis


34/55

Symptom-Based AnalysisChoose the right symptom

Symptom Choose this when you see: Troubleshooting includes

NDR User gets an NDRDSN code is known

DSN-based analysis

DNS check

Message tracking for specific DSN

Inbound Messages not arriving from theInternet

Intra-org messages not arriving

Network Test (DNS, Firewall)

SMTP configurationSending test mail

Outbound Messages not going out to theInternet

Search message and track

Queue Messages are stuck in one ofthe queues on a server

Analysis based on the type of queues(remote delivery, directory lookup, local

delivery)MailboxSubmission

Messages not going to fromMailbox to Hub Transport

MAPI connectivity check

Hub Transport health check

EdgeSync Edge Subscription not working Configuration check Network Test (DNS, Firewall)

Active Directory Application Mode (ADAM)checks

R t C A l i


35/55

Root Cause Analysis

Choice of correct symptom is critical to success

High-level symptom validation is performed infirst step of analysis

Server operating state and configuration are

collected, additional steps executed whenvariance from known good condition found

Branching to new steps continues untilroot cause identified

Not all root causes currently identified, but mostcommon ones are covered

Web updates for ExTRA will fill gaps over time

M T ki


36/55

Message Tracking

Message Tracking tool in the ExchangeManagement Console Toolbox

Based on ExTRA

Constructs cmdlet filters used by Get-MessageTrackingLog

Basic server-to-server tracking

PowerShell scripts can relate eventstogetherto track messages end-to-end

M T ki L


37/55

Message Tracking Log Enabled by default

Default valuesMessageTrackingLogEnabled: True

MessageTrackingLogMaxAge: 30

MessageTrackingLogMaxDirectorySize: 250 MB

MessageTrackingLogMaxFileSize: 10 MB

MessageTrackingLogSubjectLoggingEnabled: True

EventIDdescribes tracking event action BADMAIL, DEFER, DELIVER, DSN, EXPAND, FAIL,

POISONMESSAGE, RECEIVE, REDIRECT, RESOLVE, SEND,

SUBMIT, TRANSFER

Sourcedescribes component involved ADMIN, AGENT, DSN, GATEWAY, PICKUP, ROUTING, SMTP,

STOREDRIVER

M T ki L dl t


38/55

Message Tracking Log: cmdlet

Get-MessageTrackingLog

EventID (Receive,Send,Deliver,Fail,etc)Get-MessageTrackingLog -EventID fail -Server exht1

Time Range (start, end)Get-MessageTrackingLog -start 03/01/2008 09:00 AM

-end 03/01/2008 09:30 AM Sender Address

Get-MessageTrackingLog -Sender [email protected]

MSExchangeTransportLogSearch service on

server performs search and server-side filtering FAIL event for every NDR the server generates

RecipientStatus field displays reason FAIL occurred

Message Tracking Log: Event


39/55

Message Tracking Log: EventTimestamp : 3/16/2008 2:50:03 PM

ClientIp :

ClientHostname : exht1

ServerIp :ServerHostname : exmbx1

SourceContext :

ConnectorId :

Source : STOREDRIVER

EventId : DELIVER

InternalMessageId : 36308614

MessageId :

Recipients : {[email protected]}

RecipientStatus : {}

TotalBytes : 15682

RecipientCount : 1

RelatedRecipientAddress :Reference :

MessageSubject : Troubleshooting Decks

Sender : [email protected]

ReturnPath : [email protected]

MessageInfo : 3/16/2008 2:51:59 PM

Ro ting Log Vie er


40/55

Routing Log Viewer

Introduced in Service Pack 1 Equivalent to Winroute

Displays routing table

Provides comparison of topology at twopoints in time, identifies differences

Useful in determining transport topology

Route to remote Active Directory Site Route to connector with external address space

Routing Log Viewer: Backoff Path


41/55

Routing Log Viewer: Backoff Path

Routing Log Viewer:


42/55

Routing Log Viewer:Comparing Logs

Event Tracing For Windows


43/55

Event Tracing For Windows(ETW)

ExTRA Trace Control enables ETW traces StartRunExtra.exe Select a task

Select Trace Control

Trace components useful in diagnosing transport issues

Transport StoreDriver

AD Driver

Data.Storage

Common scenarios defined that enable correct

components/tags Filtering reduces the number of events logged in trace session,

but must know sender or recipient before reproduction of issue

ETW: Configure Trace File


44/55

ETW: Configure Trace File

ETW: Types Components Tags


45/55

ETW: Types, Components, Tags

ETW: Set Tags Manually


46/55

ETW: Set Tags Manually(optional)

ETW: Set Tags Manually


47/55

ETW: Set Tags Manually(optional)

Pipeline Tracing


48/55

Pipeline Tracing

Used to capture copies of messages before/after agent

execution

Configuration (both parameters mandatory) PipelineTracingPath:

PipelineTracingSenderAddress: SMTP address

Enable Pipeline TracingSet-TransportServer -PipeLineTracingEnabled:$TRUE

Warning: one or more copies of every message matchingPipelineTracingSenderAddresswill be saved in

PipelineTracingPath Entire message content logged to disk, so set appropriate

ACL on folder specified in PipelineTracingPath

Pipeline Tracing: Example


49/55

Pipeline Tracing: Example Enable:

Set-TransportServer EXHUB1PipelineTracingEnabled:$TruePipelineTracingPath:C:\TracePipelineTracingSenderAddress:[email protected]

Monitor Trace Folder: \MessageSnapshots\

Contains original message, plus pipeline tracing for routingand SMTP receive

\RulesTracking Disable:

Set-TransportServer EXHUB1PipelineTracingEnabled:$False

Pipeline Tracing: Directory


50/55

Pipeline Tracing: Directory

Key Takeaways


51/55

Key Takeaways

Knowledge of how components interact anddepend on one another is critical to successof troubleshooting

Exchange Server 2007 includes built-in

instrumentation that provides rich diagnosticinformation for troubleshooting purposes

A variety of tools from Windows Server andExchange Server can provide workflow steps

around the troubleshooting process

Resources


52/55

Resources

Troubleshooting OWA 2007 Publishing Ruleson ISA Server 2006

Troubleshooting Outlook RPC dialog boxes

Configuration tips and commontroubleshooting steps for multiple forestdeployment of Autodiscover service

Want To Be An Expert?
http://blogs.technet.com/isablog/archive/2008/04/29/troubleshooting-owa-2007-publishing-rules-on-isa-server-2006.aspxhttp://blogs.technet.com/isablog/archive/2008/04/29/troubleshooting-owa-2007-publishing-rules-on-isa-server-2006.aspxhttp://msexchangeteam.com/archive/2008/05/02/448809.aspxhttp://msexchangeteam.com/archive/2008/02/13/448127.aspxhttp://msexchangeteam.com/archive/2008/02/13/448127.aspxhttp://msexchangeteam.com/archive/2008/02/13/448127.aspxhttp://msexchangeteam.com/archive/2008/02/13/448127.aspxhttp://msexchangeteam.com/archive/2008/02/13/448127.aspxhttp://msexchangeteam.com/archive/2008/02/13/448127.aspxhttp://msexchangeteam.com/archive/2008/05/02/448809.aspxhttp://blogs.technet.com/isablog/archive/2008/04/29/troubleshooting-owa-2007-publishing-rules-on-isa-server-2006.aspxhttp://blogs.technet.com/isablog/archive/2008/04/29/troubleshooting-owa-2007-publishing-rules-on-isa-server-2006.aspx


53/55

Want To Be An Expert?

Get in depth and up to

date technical resourcesfrom TechNet Leverage the variety

of Webcasts andVirtual Labs available

Be part of the ExchangeProduct Dialogue

Join the ExchangeCommunity

http://technet.microsoft.com/exchange/

Track Resources
http://technet.microsoft.com/exchange/http://technet.microsoft.com/exchange/


54/55

Track Resources

Exchange Team Blog (You Had Me at EHLO)http://msexchangeteam.com

Exchange Server TechCenterhttp://technet.microsoft.com/exchange

Exchange Newsgroupsmicrosoft.public.exchange*

Exchange Forumshttp://forums.microsoft.com/TechNet/default.aspx?ForumGroupID=235&SiteID=

17
http://msexchangeteam.com/http://technet.microsoft.com/exchangehttp://forums.microsoft.com/TechNet/default.aspx?ForumGroupID=235&SiteID=17http://forums.microsoft.com/TechNet/default.aspx?ForumGroupID=235&SiteID=17http://forums.microsoft.com/TechNet/default.aspx?ForumGroupID=235&SiteID=17http://forums.microsoft.com/TechNet/default.aspx?ForumGroupID=235&SiteID=17http://forums.microsoft.com/TechNet/default.aspx?ForumGroupID=235&SiteID=17http://technet.microsoft.com/exchangehttp://technet.microsoft.com/exchangehttp://msexchangeteam.com/http://msexchangeteam.com/


55/55

2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market

Documents

Exchange Server 2007 Trouble