Windows Crash Dump Analysis

Daniel PearsonDavid Solomon Expert Seminars

AgendaUnderstanding Windows CrashesAnalyzing Windows CrashesIntroducing Driver VerifierPerforming Manual AnalysisAdvanced Debugging Techniques

Daniel Pearson7 years working at MicrosoftSenior Escalation Lead in the Windows base operating system teamLead in the Mobile Internet sustained engineering team

3 years at Digital Equipment CorporationSupporting Intel and Alpha systems running Windows NT

Instructor with David SolomonDavid, co–author of the Windows Internals series

Why analyze a Windows crash?

Understanding Windows Crashes

Why Does Windows CrashThe result of an unhandled exceptionAn event that occurs that requires the execution of code outside the normal flow of controlCan be initiated by either software or hardware during execution

A device driver detects an unrecoverable conditionA device driver or part of the operating system can explicitly crash the system if it detects an inconsistency that would corrupt the system if it continued to run

The result of a hardware failureHardware conditions such as a nonmaskable interrupt or faulty memory, CPUs, disks, etc.

Third party device drivers

70%

Microsoft code

Causes of Windows Crashes

Hardware

Too corrupt

13%11%

6%

What Happens During a CrashKeBugCheckEx, the Windows kernel API is calledAccepts a bugcheck code that indicates the reason for the crash and four parameters that supply additional information, such as the address of the detected corruption

KeBugCheckEx( IN ULONG BugCheckCode, IN ULONG_PTR BugCheckParameter1, IN ULONG_PTR BugCheckParameter2, IN ULONG_PTR BugCheckParameter3, IN ULONG_PTR BugCheckParameter4 );

Inside of KeBugCheckExDisables all interruptsDisables interrupts and raises the interrupt request level of the CPU to highPrevents other devices from interrupting the bugcheck code

Freezes all CPUs and notifies any registered driversWill attempt to freeze other CPUs whilst giving them a chance to flush their caches of dataThe system will callback into any drivers that have registered to be notified when a crash occurs

Writes a crash dump to disk and restartsCrash dump information is written to disk and the system restarted only if configured to do so

The Windows Stop Screen

The Windows 8 Stop Screen

Bugcheck CodesDocumented in the Windows Driver KitThe Windows Driver Kit currently documents over 250 unique bugcheck codes

Reference included with the Debugging ToolsContains descriptions of common bugcheck codes, the parameters passed to them and describes techniques to diagnose kernel faults or additional troubleshooting steps that can be taken

Viewable using the kernel debuggerPossible to display specific information about bugcheck codes using the !analyze commande.g. !analyze –show e2

Memory Dump TypesSmall memory dumpRecords the smallest set of useful information including the list of currently loaded kernel modulesA small memory dump is created in addition to either the kernel or complete option being selected

Kernel memory dumpRecords only kernel memory, which speeds up the process of writing a crash dump and reduces sizeThe default option on both Windows 7 and Windows Server 2008 R2 is a kernel memory dump

Complete memory dumpRecords the entire contents of system memory to disk, not shown on systems > 2 GB of memory

Analyzing Windows Crashes

Analyzing a Crash DumpUse any one of the Microsoft kernel debuggersThe Debugging Tools for Windows contains two debuggers that can be used to analyze crash dumps, kd, a command line tool and WinDbg, a Windows–based debugger

Configure the debugger to point to symbols_NT_SYMBOL_PATH=srv*C:\SYMBOLS*http://msdl.microsoft.com/download/symbols

Troubleshoot symbol loading errors with !sym noisyTurns on additional tracing information inside of symsrv.dll, the symbol server DLL

Cache

Automated AnalysisThe debugger performs basic crash analysisDisplays the stop code and parameter information while attempting to identify the offending driverPrompts the user to run !analyze –v to get more detailed debugging information

The result of executing the !analyze command!analyze uses the bugcheck code, its parameters and a set of heuristics to determine what component was the most likely cause of the bugcheck or system crash

Can be disabled if desiredThe environment variable DBGENG_NO_BUGCHECK_ANALYSIS=1 disables automated analysis

Demo

Memory CorruptionRegisters, small areas of extremely fast storageUsually measured by the number of bits they holde.g. a “32–bit” register or a “64–bit” register

x86 architecture provides 16 basic program registersEAX, EBX, ECX, EDX, EDI, ESI, EBP, ESP CS, DS, SS, ES, FS, GS EFLAGS EIP

x64 adds an additional 8 general–purpose registersRAX, RBX, RCX, RDX, RDI, RSI, RBP, RSP, R8–R15

General–purpose Segment

Using Special PoolRegisters, small areas of extremely fast storageUsually measured by the number of bits they holde.g. a “32–bit” register or a “64–bit” register

x86 architecture provides 16 basic program registersEAX, EBX, ECX, EDX, EDI, ESI, EBP, ESP CS, DS, SS, ES, FS, GS EFLAGS EIP

x64 adds an additional 8 general–purpose registersRAX, RBX, RCX, RDX, RDI, RSI, RBP, RSP, R8–R15

General–purpose Segment

Demo

Introducing Driver Verifier

Driver VerifierUseful for identifying code defects in driversMonitors drivers to detect incorrect function calls or actions that might corrupt the systemPerforms more thorough checks on device drivers as well as simulating different failures

Included as part of the operating systemSupport for Driver Verifier is built in as part of the Windows kernel and executive

Required for Windows logo certificationThe requirements for the Windows logo program state that a driver must not fail while running under Driver Verifier

Configuring Driver VerifierConfigurable using the Driver Verifier toolEnables the user to choose which tests are run against selected drivers and the systemConfiguration information is written to the HKEY_LOCAL_MACHINE registry hive

Contains standard settings for common defectsAllows the user to view data collected from the different verification tests

Support for using a command line interfaceUseful in an automated environment or enabling Driver Verifier on a remote system

Demo

Performing Manual Analysis

Manual Analysis!analyze doesn’t always offer resultsIt might not tell you anything useful or you want to know in more detail what was happening

Several useful commands and techniquesVerify the time of the crash using .time, a short uptime value can indicate frequent problemsCheck the stack on each CPU, !cpuinfo will display all the CPUs, use ~s and k to display each stack

Additional manual analysis techniquesLook at virtual memory usage using !vm, make sure pools are not depleted or contain errorsDisplay loaded drivers with lm n t, make sure you recognize all drivers and they’re up to date

Demo

Advanced Debugging Techniques

Attaching a Kernel DebuggerSupport for attaching a kernel debuggerSupported transports include a serial, USB or IEEE 1394 cable, named pipes or a network connectionNetwork support is to be included with Windows 8 and requires a supported network adapter

The system must be started in debugging modeConfigurable by modifying the boot configuration database, e.g. bcdedit /debug on

Required for debugging initialization failuresWhen the system is unable to write a crash dump or the crash occurs early during initialization

Debugging Hung SystemsPossible for systems to become unresponsiveKeyboard and mouse frozen, the LED for the Num Lock key doesn’t change state and the cursor doesn’t react when you attempt to move the mouse on the screen

Instant system lockupUsually the cause of a kernel synchronization deadlock, a device driver executing an infinite loop at a high interrupt request level or a thread stuck at a very high priority level

Slow grinding to a haltUsually the cause of resource depletion, noticeable when items drawn blank or objects disappear

Initiating a Manual CrashUsing a PS/2 keyboardRequires setting the HKLM\SYSTEM\CurrentControlSet\Services\i8042prt\Parameters\CrashOnCtrlScroll registry key

Using a built in NMI buttonRequires setting the HKLM\SYSTEM\CurrentControlSet\Control\CrashControl\NMICrashDump registry key plus the use of specialized hardware

Using the kernel debuggerBreak in with the debugger and execute the .crash command

Demo

Further InformationWindows Internals, 5th EditionRussinovich, Mark, and David Solomon. 2009. Microsoft Press

Memory Dump, Software Trace, Debugging, Malware and Intelligence Analysis Portalhttp://dumpanalysis.org

Advanced Windows Debugging and Troubleshootinghttp://blogs.msdn.com/ntdebugging

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.