Encyclopedia of Crash Dump Analysis Patterns · Encyclopedia of Crash Dump Analysis Patterns ... A CIP catalog record for this book is available from the British Library. ISBN-13:

Encyclopedia of Crash Dump Analysis Patterns

Detecting Abnormal Software Structure and Behavior in Computer Memory

Dmitry Vostokov Software Diagnostics Institute

OpenTask

2 |

Published by OpenTask, Republic of Ireland

Copyright © 2015 by Dmitry Vostokov

Copyright © 2015 by Software Diagnostics Institute

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any

form or by any means, without the prior written permission of the publisher.

You must not circulate this book in any other binding or cover, and you must impose the same condition on any

acquirer.

OpenTask books are available through booksellers and distributors worldwide. For further information or

comments send requests to [email protected].

Product and company names mentioned in this book may be trademarks of their owners.

A CIP catalog record for this book is available from the British Library.

ISBN-13: 978-1-906717-21-6 (Paperback)

First printing, 2015

Version 1.0

mailto:[email protected]

| 3

Summary of Contents

Summary of Contents 3

Detailed Table of Contents 17

Preface 41

A 43

Abridged Dump 43

Accidental Lock 47

Activation Context 54

Active Thread 57

Activity Resonance 59

Affine Thread 61

Annotated Disassembly 64

B 65

Blocked DPC 65

Blocked Queue 66

Blocked Thread 69

Blocking File 82

Blocking Module 85

Broken Link 86

Busy System 88

C 97

C++ Exception 97

4 |

Caller-n-Callee 99

Changed Environment 102

Cloud Environment 106

CLR Thread 108

Coincidental Error Code 112

Coincidental Frames 114

Coincidental Symbolic Information 118

Corrupt Dump 124

Corrupt Structure 126

Coupled Machines 128

Coupled Modules 129

Coupled Processes 130

Crash Signature 136

Crash Signature Invariant 138

Crashed Process 139

Critical Section Corruption 140

Critical Stack Trace 148

Custom Exception Handler 149

D 154

Data Alignment 154

Data Contents Locality 155

Data Correlation 160

| 5

Deadlock 162

Debugger Bug 200

Debugger Omission 201

Design Value 202

Deviant Module 203

Deviant Token 210

Dialog Box 211

Directing Module 215

Disconnected Network Adapter 216

Disk Packet Buildup 218

Dispatch Level Spin 221

Distributed Spike 224

Distributed Wait Chain 232

Divide by Zero 234

Double Free 238

Double IRP Completion 258

Driver Device Collection 260

Dry Weight 261

Dual Stack Trace 262

Duplicate Extension 263

Duplicated Module 267

Dynamic Memory Corruption 272

6 |

E 292

Early Crash Dump 292

Effect Component 295

Embedded Comments 301

Empty Stack Trace 302

Environment Hint 306

Error Reporting Fault 307

Exception Module 310

Exception Stack Trace 312

Execution Residue 314

F 330

Fake Module 330

False Effective Address 334

False Function Parameters 335

False Positive Dump 338

Fat Process Dump 340

Fault Context 341

First Fault Stack Trace 342

FPU Exception 343

Frame Pointer Omission 345

Frozen Process 349

G 353

| 7

Ghost Thread 353

Glued Stack Trace 355

H 358

Handle Leak 358

Handle Limit 359

Handled Exception 365

Hardware Activity 374

Hardware Error 378

Hidden Call 386

Hidden Exception 391

Hidden IRP 397

Hidden Module 398

Hidden Parameter 400

Hidden Process 402

High Contention 404

Historical Information 415

Hooked Functions 416

Hooked Modules 422

Hooking Level 424

I 427

Incomplete Stack Trace 427

Incomplete Session 428

8 |

Inconsistent Dump 430

Incorrect Stack Trace 431

Incorrect Symbolic Information 437

Injected Symbols 442

Inline Function Optimization 445

Instrumentation Information 449

Instrumentation Side Effect 453

Insufficient Memory 456

Invalid Exception Information 502

Invalid Handle 506

Invalid Parameter 518

Invalid Pointer 521

IRP Distribution Anomaly 523

J 525

JIT Code 525

L 528

Last Error Collection 528

Last Object 530

Late Crash Dump 531

Lateral Damage 532

Least Common Frame 533

Livelock 535

| 9

Local Buffer Overflow 537

Lost Opportunity 540

M 542

Main Thread 542

Managed Code Exception 545

Managed Stack Trace 552

Manual Dump 553

Memory Fluctuation 562

Memory Leak 564

Message Box 588

Message Hooks 591

Missing Component 594

Missing Process 608

Missing Thread 609

Mixed Exception 614

Module Collection 619

Module Hint 622

Module Product Process 624

Module Variable 625

Module Variety 627

Multiple Exceptions 630

N 640

10 |

Namespace 640

Nested Exceptions 641

Nested Offender 648

Network Packet Buildup 651

No Component Symbols 652

No Current Thread 655

No Data Types 657

No Process Dumps 658

No System Dumps 659

Not My Version 660

NULL Pointer 662

O 668

OMAP Code Optimization 668

One-Thread Process 672

Optimized Code 674

Optimized VM Layout 676

Origin Module 678

Out-of-Module Pointer 680

Overaged System 681

P 682

Packed Code 682

Paged Out Data 685

| 11

Paratext 687

Pass Through Function 689

Passive System Thread 691

Passive Thread 695

Past Stack Trace 702

Patched Code 704

Pervasive System 705

Platform-Specific Debugger 706

Pleiades 708

Pre-Obfuscation Residue 709

Problem Exception Handler 710

Problem Module 712

Problem Vocabulary 713

Process Factory 714

Punctuated Memory Leak 719

Q 723

Quiet Dump 723

R 724

Random Object 724

Raw Pointer 727

Reduced Symbolic Information 728

Reference Leak 729

12 |

Regular Data 732

RIP Stack Trace 733

Rough Stack Trace 735

S 738

Same Vendor 738

Screwbolt Wait Chain 739

Self-Diagnosis 740

Self-Dump 745

Semantic Split 747

Semantic Structure 754

Shared Buffer Overwrite 758

Shared Structure 766

Small Value 767

Software Exception 769

Special Process 770

Special Stack Trace 775

Special Thread 776

Spike Interval 777

Spiking Thread 778

Stack Overflow 787

Stack Trace 808

Stack Trace Change 822

| 13

Stack Trace Collection 823

Stack Trace Set 839

Step Dumps 842

Stored Exception 843

String Hint 844

String Parameter 846

Suspended Thread 848

Swarm of Shared Locks 850

System Object 855

T 858

Tampered Dump 858

Technology-Specific Subtrace 871

Template Module 879

Thread Age 883

Thread Cluster 885

Thread Starvation 886

Top Module 892

Translated Exception 893

Truncated Dump 894

Truncated Stack Trace 897

U 898

Ubiquitous Component 898

14 |

Unknown Component 913

Unloaded Module 917

Unrecognizable Symbolic Information 921

Unsynchronized Dumps 926

V 927

Value Adding Process 927

Value Deviation 928

Value References 932

Version-Specific Extension 933

Virtualized Process 937

Virtualized System 945

W 951

Wait Chain 951

Waiting Thread Time 1001

Well-Tested Function 1010

Well-Tested Module 1011

Wild Code 1012

Wild Pointer 1014

Y 1016

Young System 1016

Z 1018

Zombie Processes 1018

| 15

Appendix A 1025

Reference Stack Traces 1025

Appendix B 1026

.NET / CLR / Managed Space Patterns 1026

Contention Patterns 1027

Deadlock and Livelock Patterns 1028

DLL Link Patterns 1029

Dynamic Memory Corruption Patterns 1030

Executive Resource Patterns 1031

Exception Patterns 1032

Falsity and Coincidence Patterns 1033

Hooksware Patterns 1034

Insufficient Memory Patterns 1036

Meta-Memory Dump Patterns 1037

Module Patterns 1038

Optimization Patterns 1039

Process Patterns 1040

RPC, LPC and ALPC Patterns 1041

Stack Overflow Patterns 1042

Stack Trace Patterns 1043

Symbol Patterns 1044

Thread Patterns 1045

16 |

Wait Chain Patterns 1046

Appendix C 1047

Crash Dump Analysis Checklist 1047

Index 1051

| 17

Detailed Table of Contents

Summary of Contents 3

Detailed Table of Contents 17

Preface 41

A 43

Abridged Dump 43

Accidental Lock 47

Activation Context 54

Active Thread 57

Mac OS X 57

Activity Resonance 59

Affine Thread 61

Annotated Disassembly 64

JIT .NET Code 64

B 65

Blocked DPC 65

Blocked Queue 66

LPC/ALPC 66

Comments 68

Blocked Thread 69

Hardware 69

Software 71

18 |

Comments 79

Timeout 81

Blocking File 82

Blocking Module 85

Comments 85

Broken Link 86

Busy System 88

C 97

C++ Exception 97

Windows 97

Comments 97

Mac OS X 98

Caller-n-Callee 99

Changed Environment 102

Comments 105

Cloud Environment 106

CLR Thread 108

Comments 111

Coincidental Error Code 112

Coincidental Frames 114

Coincidental Symbolic Information 118

Windows 118

| 19

Mac OS X 122

Corrupt Dump 124

Comments 125

Corrupt Structure 126

Coupled Machines 128

Coupled Modules 129

Coupled Processes 130

Semantics 130

Strong 131

Comments 132

Weak 133

Crash Signature 136

Crash Signature Invariant 138

Crashed Process 139

Critical Section Corruption 140

Critical Stack Trace 148

Custom Exception Handler 149

Kernel Space 149

User Space 151

D 154

Data Alignment 154

Page Boundary 154

20 |

Data Contents Locality 155

Data Correlation 160

Function Parameters 160

Deadlock 162

Critical Sections 162

Comments 169

Executive Resources 174

LPC 178

Managed Space 183

Mixed Objects 186

Kernel Space 186

User Space 191

Comments 198

Self 199

Comments 199

Debugger Bug 200

Debugger Omission 201

Design Value 202

Deviant Module 203

Comments 209

Deviant Token 210

Dialog Box 211

Directing Module 215

| 21

Disconnected Network Adapter 216

Disk Packet Buildup 218

Dispatch Level Spin 221

Distributed Spike 224

Comments 231

Distributed Wait Chain 232

Divide by Zero 234

Kernel Mode 234

User Mode 236

Windows 236

Mac OS X 237

Double Free 238

Kernel Pool 238

Comments 241

Process Heap 246

Windows 246

Comments 255

Mac OS X 257

Double IRP Completion 258

Driver Device Collection 260

Dry Weight 261

Dual Stack Trace 262

Duplicate Extension 263

22 |

Comments 266

Duplicated Module 267

Comments 271

Dynamic Memory Corruption 272

Kernel Pool 272

Comments 278

Managed Heap 282

Process Heap 285

Windows 285

Comments 286

Mac OS X 290

E 292

Early Crash Dump 292

Effect Component 295

Embedded Comments 301

Empty Stack Trace 302

Comments 305

Environment Hint 306

Error Reporting Fault 307

Exception Module 310

Exception Stack Trace 312

Comments 313

Execution Residue 314

| 23

Mac OS X 314

Managed Space 316

Comments 317

Unmanaged Space 318

Comments 329

F 330

Fake Module 330

False Effective Address 334

False Function Parameters 335

False Positive Dump 338

Fat Process Dump 340

Fault Context 341

First Fault Stack Trace 342

FPU Exception 343

Frame Pointer Omission 345

Frozen Process 349

G 353

Ghost Thread 353

Glued Stack Trace 355

H 358

Handle Leak 358

Handle Limit 359

24 |

GDI 359

Handled Exception 365

.NET CLR 365

Kernel Space 370

User Space 371

Comments 373

Hardware Activity 374

Hardware Error 378

Comments 383

Hidden Call 386

Hidden Exception 391

Kernel Space 391

User Space 392

Hidden IRP 397

Hidden Module 398

Comments 399

Hidden Parameter 400

Hidden Process 402

High Contention 404

.NET CLR Monitors 404



Comments 411

| 25

Processors 412

Historical Information 415

Comments 415

Hooked Functions 416

Kernel Space 416

Comments 419

User Space 420

Hooked Modules 422

Comments 423

Hooking Level 424

I 427

Incomplete Stack Trace 427

GDB 427

Incomplete Session 428

Comments 429

Inconsistent Dump 430

Comments 430

Incorrect Stack Trace 431

Comments 436

Incorrect Symbolic Information 437

Injected Symbols 442

Inline Function Optimization 445

26 |

Managed Code 445

Unmanaged Code 447

Instrumentation Information 449

Instrumentation Side Effect 453

Comments 455

Insufficient Memory 456

Committed Memory 456

Control Blocks 458

Handle Leak 459

Comments 463

Kernel Pool 468

Comments 476

Module Fragmentation 477

Comments 484

Physical Memory 485

PTE 488

Comments 489

Region 490

Reserved Virtual Memory 492

Session Pool 495

Stack Trace Database 496

Invalid Exception Information 502

Invalid Handle 506

General 506

| 27

Comments 509

Managed Space 510

Invalid Parameter 518

Process Heap 518

Invalid Pointer 521

General 521

IRP Distribution Anomaly 523

J 525

JIT Code 525

.NET 525

Comments 527

L 528

Last Error Collection 528

Last Object 530

Late Crash Dump 531

Lateral Damage 532

Comments 532

Least Common Frame 533

Livelock 535

Local Buffer Overflow 537

Mac OS X 537

Windows 539

28 |

Lost Opportunity 540

M 542

Main Thread 542

Managed Code Exception 545

Managed Stack Trace 552

Manual Dump 553

Kernel 553

Comments 555

Process 558

Comments 561

Memory Fluctuation 562

Process Heap 562

Memory Leak 564

.NET Heap 564

Comments 570

I/O Completion Packets 571

Page Tables 572

Process Heap 578

Comments 584

Regions 585

Message Box 588

Comments 590

Message Hooks 591

| 29

Missing Component 594

General 594

Static Linkage 598

User Mode 598

Missing Process 608

Comments 608

Missing Thread 609

Comments 613

Mixed Exception 614

Comments 618

Module Collection 619

General 619

Predicate 621

Module Hint 622

Comments 623

Module Product Process 624

Module Variable 625

Module Variety 627

Multiple Exceptions 630

Windows 630

Kernel Mode 630

Managed Space 635

User Mode 636

30 |

Mac OS X 638

N 640

Namespace 640

Nested Exceptions 641

Managed Code 641

Unmanaged Code 644

Nested Offender 648

Network Packet Buildup 651

No Component Symbols 652

No Current Thread 655

No Data Types 657

No Process Dumps 658

No System Dumps 659

Comments 659

Not My Version 660

Hardware 660

Software 661

NULL Pointer 662

Windows 662

Code 662

Data 664

Mac OS X 665

Code 665

| 31

Data 667

O 668

OMAP Code Optimization 668

One-Thread Process 672

Optimized Code 674

Comments 675

Optimized VM Layout 676

Origin Module 678

Out-of-Module Pointer 680

Overaged System 681

Comments 681

P 682

Packed Code 682

Paged Out Data 685

Paratext 687

Mac OS X 687

Comments 688

Pass Through Function 689

Comments 690

Passive System Thread 691

Kernel Space 691

Passive Thread 695

32 |

User Space 695

Comments 701

Past Stack Trace 702

Patched Code 704

Pervasive System 705

Platform-Specific Debugger 706

Pleiades 708

Pre-Obfuscation Residue 709

Problem Exception Handler 710

Comments 711

Problem Module 712

Problem Vocabulary 713

Process Factory 714

Punctuated Memory Leak 719

Q 723

Quiet Dump 723

R 724

Random Object 724

Raw Pointer 727

Reduced Symbolic Information 728

Reference Leak 729

Regular Data 732

| 33

RIP Stack Trace 733

Rough Stack Trace 735

S 738

Same Vendor 738

Screwbolt Wait Chain 739

Self-Diagnosis 740

Kernel Mode 740

Comments 740

Registry 741

User Mode 743

Comments 744

Self-Dump 745

Semantic Split 747

Semantic Structure 754

PID.TID 754

Comments 757

Shared Buffer Overwrite 758

Windows 758

Mac OS X 762

Shared Structure 766

Small Value 767

Comments 768

Software Exception 769

34 |

Special Process 770

Comments 774

Special Stack Trace 775

Comments 775

Special Thread 776

.NET CLR 776

Spike Interval 777

Spiking Thread 778

Windows 778

Comments 783

Mac OS X 785

Stack Overflow 787

Windows 787

Kernel Mode 787

Comments 795

Software Implementation 797

User Mode 799

Comments 802

Mac OS X 804

Stack Trace 808

Windows 808

Database 808

File System Filters 813

| 35

General 815

I/O Request 819

Mac OS X 821

Stack Trace Change 822

Stack Trace Collection 823

I/O Requests 823

Managed Space 827

Predicate 830

Unmanaged Space 831

Comments 838

Stack Trace Set 839

Step Dumps 842

Stored Exception 843

String Hint 844

String Parameter 846

Suspended Thread 848

Swarm of Shared Locks 850

System Object 855

T 858

Tampered Dump 858

Technology-Specific Subtrace 871

COM Interface Invocation 871

Dynamic Memory 874

36 |

JIT .NET Code 876

Template Module 879

Thread Age 883

Thread Cluster 885

Thread Starvation 886

Normal Priority 886

Realtime Priority 888

Top Module 892

Translated Exception 893

Truncated Dump 894

Windows 894

Mac OS X 896

Truncated Stack Trace 897

Comments 897

U 898

Ubiquitous Component 898

Kernel Space 898

User Space 901

Unknown Component 913

Unloaded Module 917

Unrecognizable Symbolic Information 921

Unsynchronized Dumps 926

| 37

V 927

Value Adding Process 927

Value Deviation 928

Stack Trace 928

Value References 932

Comments 932

Version-Specific Extension 933

Virtualized Process 937

WOW64 937

Comments 944

Virtualized System 945

W 951

Wait Chain 951

CLR Monitors 951



General 959

Comments 963

LPC/ALPC 964

Modules 970

Mutex Objects 971

Named Pipes 973

Process Objects 975

38 |

Pushlocks 980

RPC 982

RTL_RESOURCE 986

Thread Objects 992

Window Messaging 996

Waiting Thread Time 1001

Kernel Dumps 1001

Comments 1006

User Dumps 1008

Comments 1009

Well-Tested Function 1010

Well-Tested Module 1011

Wild Code 1012

Wild Pointer 1014

Y 1016

Young System 1016

Z 1018

Zombie Processes 1018

Comments 1024

Appendix A 1025

Reference Stack Traces 1025

Appendix B 1026

.NET / CLR / Managed Space Patterns 1026

| 39

Contention Patterns 1027

Deadlock and Livelock Patterns 1028

DLL Link Patterns 1029

Dynamic Memory Corruption Patterns 1030

Executive Resource Patterns 1031

Exception Patterns 1032

Falsity and Coincidence Patterns 1033

Hooksware Patterns 1034

Insufficient Memory Patterns 1036

Meta-Memory Dump Patterns 1037

Module Patterns 1038

Optimization Patterns 1039

Process Patterns 1040

RPC, LPC and ALPC Patterns 1041

Stack Overflow Patterns 1042

Stack Trace Patterns 1043

Symbol Patterns 1044

Thread Patterns 1045

Wait Chain Patterns 1046

Appendix C 1047

Crash Dump Analysis Checklist 1047

Index 1051

Documents

Encyclopedia of Crash Dump Analysis Patterns · Encyclopedia of Crash Dump Analysis Patterns ... A CIP catalog record for this book is available from the British Library. ISBN-13: