Download pdf - The Evolution of Windows Crash Dump Research · PDF fileThe Evolution of Windows Crash Dump ... Driver Name On Disk Driver Base Name in Memory Purpose diskdump.sys dump_diskdump

© 2013 CrowdStrike, Inc. All rights reserved.

The Evolution of Windows Crash Dump

Research

Aaron LeMasters

Brucon 2014


Introduction

• Crash dump infrastructure primer– How it is used by the operating system

• Manipulating the crash dump infrastructure– Bypass technique (April 2012)

–Windows XP through Windows 8.1– Bypass technique made easier (January 2013)

–Windows 8 through 8.1– Dump filter side-channel technique (May 2013)

–Windows 8 through 8.1– What’s new in Windows 8.1 (July 2014)


Crash Dump Infrastructure Primer


Basic terminology• Port driver – abstraction interface provided by OS, hides

underlying protocol details from class driver

• Miniport driver – manufacturer-supplied driver to interface with hardware (Host Bus Adapter/HBA); linked against port driver for specific transport technology

• Class driver – a driver that abstracts the underlying technology of a category of devices that share similar qualities (e.g., cdrom.sys)

• Normal I/O path – the route an I/O request takes during regular system operation

• Crash dump I/O path – the route the kernel uses to write a crash dump file to disk during a crash


Two Paths


Why two paths?

• When a bugcheck occurs, a pristine path is needed

to write the dump file, in case a third-party driver in

the normal I/O path caused it.

• What differs between the two paths?Normal I/O Path Crash Dump I/O Path

Primary drivers Many, layered Modified port and

miniport

Filter drivers Many, layered Crash dump filters only

Controlled by I/O manager Kernel or crashdmp.sys

Documented? Yes *cough*


The Crash Dump Stack• A “stack” of drivers providing an “I/O path” to a mass storage

device, consisting of:– A dump port driver– A dump miniport driver– One or more crash dump filter drivers– Crashdmp.sys

• Initialized in two phases:– System startup/page file creation (pre-initialization)– System crash (post-initialization)

• Used when:– A bug check occurs– The system is about to hibernate


Dump Stack ComponentsDriver Name On Disk Driver Base Name in Memory Purpose

diskdump.sys dump_diskdump SCSI/Storport dump port driver with required

exports from scsiport.sys and storport.sys. This

driver is unloaded.

dumpata.sys dump_dumpata IDE/ATA dump port driver with required

ataport.sys exports. This driver is unloaded.

scsiport.sys dump_scsiport The final SCSI/Storport dump port driver.

ataport.sys dump_ataport The final IDE/ATA dump port driver.

atapi.sys dump_atapi An older, generic ATAPI miniport driver

provided by the OS for IDE/ATA drives

vmscsi.sys dump_vmscsi The miniport driver provided by VMWare for

SCSI drives.

LSI_SAS.sys dump_LSI_SAS The miniport driver provided by LSI

Corporation for serial-attached storage drives.

dumpfve.sys dump_dumpfve Windows full volume encryption crash dump

filter driver


Pre- and Post-Initialization


Pre-initialization: crashdmp.sys

• KiInitializeKernel() IoInitSystem()

OR NtCreatePagingFile()– IoInitializeCrashDump(HANDLE PageFile)

–IopLoadCrashDumpDriver() – loads crashdmp.sys

–Crashdmp!DriverEntry() – fills crash dump call table

• Entry point called with two arguments:– Name of the arc boot device– Pointer to a global crashdmp callback table


Pre-initialization: crashdmp.sys (cont’d)Table Offset* Windows 7 Value Windows 8 RP Value

0x0 1 1

0x4 1 4

0x8 CrashdmpInitialize

0xC CrashdmpLoadDumpStack

0x10 CrashdmpInitDumpStack

0x14 CrashdmpFreeDumpStack

0x18 CrashdmpDisable

0x1C CrashdmpNotify

0x20 CrashdmpWrite

0x24 CrashdmpUpdatePhysicalRange

0x28 CrashdmpResumeCapable

0x2C CrashdmpGetTransferSizes

0x30 CrashdmpLogStatusData

0x34 CrashdmpReady

*Pointer width is 8 bytes on 64-bit systems


Pre-initialization: Dump port and filters

• KiInitializeKernel() IoInitSystem()

OR NtCreatePagingFile()– IoInitializeCrashDump()

–Crashdmp!CrashDmpInitialize():

–Crashdmp!CrashdmpLoadDumpStack():

– Crashdump!QueryPortDriver()

– Crashdmp!LoadPortDriver()

– Crashdmp!LoadFilterDrivers()

– Crashdmp!InitializeFilterDrivers()



(cont’d)

• Crashdmp!CrashdmpInitialize():

– References boot device’s FILE_OBJECT

– Read registry for crash dump settings

–Stored in local crashdump control block

– Initializes crashdump control block

– Registers a KbCallbackSecondaryDumpData bugcheckcallback

– Calls CrashdmpLoadDumpStack()…



(cont’d)

• Crashdmp!CrashdmpLoadDumpStack():– Queries port driver capabilities

(IOCTL_SCSI_GET_DUMP_POINTERS,IOCTL_SCSI_GET_DRIVE_GEOMETRY_EX, etc)

– Locates all crash dump drivers: port, miniport and crash dump filter drivers

–Copies them into memory with “dump_” prefix– No DRIVER_OBJECT or DEVICE_OBJECT!

–Port driver will be one of dump_scsiport, dump_ataport or dump_storport; on disk either diskdump.sys(scsi/storport) or dumpata.sys (ataport)

–Miniport can be named anything


Crash Dump Environment

• The normal I/O path is completely disabled

• All CPU’s are disabled except the one that the current

thread is executing on

• The single, active CPU becomes single-threaded with

HIGH_LEVEL IRQL and is uninterruptible

• I/O sent through the crash dump stack is synchronous

• For IDE controllers, only the channel containing the

device with the paging file is enabled


Post-initialization: Call Entry Points

• KeBugCheck2() IoWriteCrashDump():– Calls the ninth entry in the CrashDmpCallTable table, CrashDmpWrite():–CrashdmpInitDumpStack():

–StartFilterDrivers() – calls the DumpStart callback of each crash dump filter driver

–InitializeDumpDriver() – calls the dump driver entry point; calls the DiskDumpOpen callback provided by the dump port driver; fills dump context structure with callback pointers CrashdmpWriteRoutine, CrashdmpWritePendingRoutine, CrashdmpFinishRoutine


Post-initialization: Dump port callbacks• KeBugCheck2() IoWriteCrashDump():

–CrashDmpWrite():– CrashdmpInitDumpStack():

– DumpWrite() – creates dump file based on configuration:– FillDumpHeader()

– Calls one of:• WriteFullDump()

• WriteKernelDump()

• WriteMiniDump()

– InvokeSecondaryDumpCallbacks() - Invokes all BugCheckSecondaryDumpDataCallback callbacks to allow drivers to append data to the completed crash dump file.

– InvokeDumpCallbacks() - Invokes all BugCheckDumpIoCallback callbacks, informing them crash dump is complete.


Bypass Technique

April 2012


Overview

• Platforms: Windows XP - 7

• Target: Dump port driver (eg, diskdump.sys)

• Goal: Use crash dump I/O path to defeat MBR rootkits

• Technique: Force a path that can only WRITE to disk to

also READ from disk using SRB’s and a few hacks to

bypass normal OS usage of the stack and communicate

directly with the port driver

• First published at Syscan ([5] and [6]) in April 2012


Motivation: TDL4 bootkit

• Modifies the miniport’s device object and driver object

– DRIVER_OBJECT.DriverStartIo hooked

– DEVICE_OBJECT.DriverObject hooked

– Lowest attached device is unlinked from the miniport device by hooking DEVICE_OBJECT.NextDevice

• Monitors for read or write attempts to the boot sector

and its hidden file system

– If boot sector, return original, clean MBR

– If hidden file system, return zeroes


The Goal


The Technique


Identify crash dump drivers• Walk loaded module list

• Single out dump drivers easily via “dump_” prefix

• Port driver will be one of dump_scsiport, dump_storport or dump_ataport

• Miniport driver name:– open a handle to the class driver’s device object, walk attached

devices to the lowest one– Vista+ DUMP_POINTERS_EX.DriverList

• Once drivers have been found, call their entry points with the appropriate arguments– Dump port: DriverEntry(NULL, DumpInit*)– Dump miniport: DriverEntry(NULL, NULL)


Get boot device information• IOCTL to get hardware register mapping and port

configuration information (IOCTL_SCSI_GET_DUMP_POINTERS); data returned in DUMP_POINTERS or DUMP_POINTERS_EX structure

• IOCTL to get boot device location such as Target id, path Id, and Lun (IOCTL_GET_SCSI_ADDRESS); data returned in a SCSI_ADDRESS structure

• Resulting information is stored in the DUMP_INITIALIZATION_CONTEXT structure before calling the dump port driver’s DriverEntry


Find StartIo or DispatchCrb

• How do we send I/O requests?– There is no device object for dump port driver

• Use internal functions–StartIo (SCSI) – accepts a single

SCSI_REQUEST_BLOCK (SRB) as argument–DispatchCrb (IDE) – accepts a single argument, a

channel extension structure

• Find functions by scanning dump port driver’s image’s text section for “magic bytes”


Find dump port device extension• Can’t simply call the internal functions, more initialization required

• Port/miniport share a device extension structure that must be properly initialized– An internal variable of the port driver– Most critical fields are filled in by dump port driver/miniport driver’s DriverEntry routines

• Two methods to find Device Extension– Call a dump port function that leaks a pointer to it– Use DumpInit.MemoryBlock + 16

– Dump port’s DriverEntry()assigns Device Extension to DumpInit->MemoryBlock+16, which we control and pass as second argument to DriverEntry() when we call it

• Both methods work for all transports all the way through Windows 8


Send SRB Mimic DiskDumpWrite()

– Allocate an MDL at offset 0xD0 (0x118 x64) into the device extension structure –MDL describes the SRB.DataBuffer

• SRB:– SRB.Function - SRB_FUNCTION_EXECUTE_SCSI– SRB.PathId, SRB.TargetId, SRB.Lun – set to corresponding fields in SCSI_ADDRESS

– SRB.CdbLength - 10 for 10-byte SCSI-2 command– SRB.SrbFlags - specify flags for a read operation– SRB.DataTransferLength - 512– SRB.DataBuffer - allocate 512 bytes NonPagedPool – result stored here– SRB.Cdb – the SCSI-2 command descriptor block (cdb)

– Describes the location on disk to read– Call StartIo()


Bypass Technique Made EasierJanuary 2013


Overview

• Platforms: Windows 8 – 8.1

• Target: Dump port driver (eg, diskdump.sys)

• Goal: Read or write any location on mass storage device

• Technique: Use new optimizations and features in Windows 8 dump port driver, including new read capability, to make “bypass” much more stable

• First published at Bsides Jackson in November 2012 [4] and Crowdstrike blog in January 2013 [3]


Why a new read capability?

• Supports new logging and hybrid boot features

introduced in Windows 8

• This change represents the first time Microsoft

leverages the crash dump stack for purposes other

than writing a crash file


What was made easier

• A single new internal function, DiskDumpIoIssue

can be called for read.– No longer a need to call multiple internal functions

– No longer necessary to craft SRB’s for read

• No need to poll/wait on result or retry failed

requests


Technique

• Locate and initialize the crash dump miniport driver

• Locate, patch and initialize the crash dump port driver

• Locate the dump port driver’s new DiskDumpIoIssue()

• Disable the normal I/O path

• Call DiskDumpIoIssue() with a disk offset and an MDL that describes the buffer to store the result.

• Enable the normal I/O path

• Unpatch the dump port driver


Democrashdd


Dump filter side channel technique

(aka “Dump Filter Hijack”)May 2013


Overview

• Platforms: Windows 8 – 8.1

• Target: crashdmp.sys

• Goal: Read or write any location on mass storage

device

• Technique: As a crash filter driver, abuse new crash

dump stack logging capabilities.

• First reported at No Such Con Paris in May 2013 [2],

basis for SOURCE Boston CTF challenge that spring


What is dump stack logging?• Undocumented; new in Windows 8

• Enabled via registry:– CrashControl\EnableLogFile = 1– CrashControl\DumpLogLevel = 0xfffffffd

• Writes basic diagnostic information to C:\DumpStack.log.tmp

– Created when dump stack is initialized, just after filters are loaded

– Keeps exclusive handle opened, stored in GlobalContext

– Stores the file’s disk runs/mappings in GlobalContext


Crash Dump Filter Drivers

• Each crash dump filter driver is passed

FILTER_CONTEXT structure

• One field, GlobalContext, is a pointer to an

undocumented book-keeping structure used by

crashdmp.sys– Inside this structure is a handle to the global log file

read/written by crashdmp.sys


Disk Run/Mapping Pair

• A series of volume-relative {size,offset} pairs that describe the location of a file– Maintained by the file system (e.g., NTFS)

• Documentation of FSCTL_QUERY_RETRIEVAL_POINTERS:struct {

LONGLONG SectorLengthInBytes;

LONGLONG StartingLogicalOffsetInBytes;

} MappingPair;


Disk Run/Mapping Pair (cont’d)

• Since there can be multiple runs for a file, GlobalContext stores them in a structure :typedef struct _LOG_DISK_RUNS {

ULONG Count;

PMAPPING_PAIR Array;

} LOG_DISK_RUNS, *PLOG_DISK_RUNS;

• Disk runs are used to build SRBs in internal I/O routines

• NB: The disk runs are retrieved through the normal I/O

path.


Simple Abuse Example• As a crash dump filter driver, we have indirect access to GlobalContext through our FilterInitializationpointer

• Simply retrieve the disk runs of some other file, say, ntoskrnl.exe

– Might need FSCTL_GET* instead of FSCTL_QUERY*

• Overwrite GlobalContext->LogFileDiskRuns

• Cause system hibernation or crash– As dump/hiberfil.sys is written and crashdmp.sys logs its actions,

it’s actually writing to ntoskrnl.exe not dumpstack.log.tmp

• Your kernel gets trashed!


But we can do better…

• To achieve arbitrary read/write, locate and use crashdmp.sys internal logging functions:–WriteLogDataToDisk() – Iterates over disk runs

stored in global context structure, building an MDL for each request and using internal I/O functions (which call into dump port driver) to write the data to disk using the crash I/O path

–ReadLogDataFromDisk() – Same as WriteLogDataToDisk(), except it uses read functions to read the log data from disk


Restrictions• These functions can only be called during crash or

hibernation, because:

1. They implicitly assume the restrictions of a crash dump environment – most notably that the normal I/O path is disabled

2. Required crashdmp.sys internal functions are not initialized until they are needed (at crash or hibernation time).

– Calling either of the logging functions outside of a crash/hibernate context will result in a null pointer dereference.


Two-stage Hijack

• In addition to those restrictions, because the

normal I/O path is needed to retrieve the disk runs,

the dump logging feature must be hijacked in two

stages:– Pre-crash/hibernate: Retrieve target file disk runs

– Post-crash/hibernate: Implant disk runs, call read/write functions


Two-stage Hijack (cont’d)


Advanced Abuse Example: Patch a driver

1. Retrieve disk runs of the driver

2. (Hibernation or crash occurs)

3. Implant the file layout in the global context structure

4. Call ReadLogDataFromDisk() to retrieve the driver’s contents

5. Modify the contents held in the resulting buffer in memory

6. Call WriteLogDataToDisk() to overwrite the driver with the modified contents

7. Restore original log file layout so that crashdmp.sys does not overwrite target file when it finalizes the log file


Implementation Protips

• Pre-crash/hibernate staging can be done any time after filter driver is loaded

• Post-crash/hibernate work is best done in Dump_Finish callback– At this point, crashdmp.sys has finished any logging and no

longer using internal log file position info

• FILTER_CONTEXT and GlobalContext can be modified without concern for synchronization– They are allocated at kernel initialization and left alone until

they are needed during a crash/hibernation


What’s new in Windows 8.1July 2014


Surprisingly absent..

• No significant changes to crashdmp.sys or dump

port driver in Windows 8.1!

• A few interesting changes in the kernel itself,

though…


“CrashAPI”

• New “CrashAPI” added to the kernel

– 57 new functions comprise this internal API

– Some features exposed via new NtSystemDebugControlparameter 37 to create kernel dump from user mode

• Optional flag to this parameter “UseDumpStorageStack”

– This represents the second time Microsoft has shown intention to use the crash path outside of its normal use

• See the tool and source code I released, LiveDump.exe


Offline dumps

• HKLM\System\CurrentControlSet\Control\CrashControl– EnableOfflineDumps – if set to 1, AttemptOffline is read– AttemptOffline – if set to 1, allows the dump file to be

written to a PCI debugging device if the kernel supports it.

• New power manager function appears related but not fully implemented -PoFxRegisterCrashdumpDevice


Dump capsules

• Minimal support

• Appears to be placeholder feature

• Related to PCI debugging


DemoLiveDump

DmpExt


Wrap-up


Source code and tools

1. CrashDD – Tool like unix-style ‘dd’ to read/write using crash path (available upon request)

2. DmpFlt – A crash dump filter driver– https://code.google.com/p/livedump/

3. LiveDump – Generates full/kernel dumps using NtSystemDebugControl

– https://code.google.com/p/livedump/

• DmpExt – Windbg extension, will be posted to google code shortly

https://code.google.com/p/livedump/

https://code.google.com/p/livedump/


Resources1. Latest source code, whitepapers and presentations are always available at

http://crashdmp.wordpress.com

2. Crashdmp-ster Diving the Windows 8 Crash Dump Stack (5/16/2013) – Presentation at No Such Con, Paris, introducing the crash dump filter driver side-channel method.

3. I/O You Own: Windows 8 Update (1/9/2013) – A blog post discussing the new features in the Windows 8 crash dump stack, as well as an overview of a new technique to use the stack outside the operating system.

4. BSides Jackson: I/O You Own: Regaining Control of Your Disk in the Presence of Bootkits(11/10/2012) – Slide deck of my updated presentation for BSidesJackson. Covered previously-published but revised material that included an overview of Windows 8 crash dump stack changes.

5. SyScan Singapore 2012: I/O You Own: Regaining Control of Your Disk in the Presence of Bootkits(4/26/2012) – Original research presentation at SyScan 2012.

6. SyScan 2012 Preview – I/O You Own: Regaining Control of Your Disk in the Presence of Bootkits(4/23/2012) – A blog post introducing the first release of this research, presented at Syscan 2012.

http://crashdmp.wordpress.com/

http://blog.crowdstrike.com/2013/01/io-you-own-windows-8-update.html

http://blog.crowdstrike.com/2013/01/io-you-own-windows-8-update.html

http://www.securitybsides.com/w/page/53447313/BSidesJackson

http://www.slideshare.net/CrowdStrike/io-you-own-regaining-control-of-your-disk-in-the-presence-of-bootkits

http://www.syscan.org/index.php/archive/view/year/2012/city/sg/pg/program

https://www.mandiant.com/blog/syscan-2012-preview-io-regaining-control-disk-presence-bootkits/


Thank You!

@lilhoser

[email protected]


Greetz to Alex Ionescu

mailto:[email protected]
