Art into Science 2017 - Investigation Theory: A Cognitive Approach

Investigation Theory A Cognitive Approach

Chris Sanders

Chris Sanders (@chrissanders88)

Analyst @ FireEye Founder @ Rural Tech Fund PhD Researcher GSE # 64 BBQ Pit Master Author:

Practical Packet Analysis Applied NSM Investigation Theory Course

Symptoms of a Cognitive Crisis1. Demand for expertise greatly

outweights supply2. Most information cannot be trusted or

validated3. Inability to mobilize and tackle big

systemic issues

Ethnography of the SOC

“An analyst’s job is highly dynamic and requires dealing with constantly evolving threats. Doing the job is more art than science. Ad hoc, on-the-job training for new analysts is the norm."

Sundaramurthy, S. C., McHugh, J., Ou, X., Rajagopalan, S. R., & Wesch, M. (2014). An anthropological approach to studying CSIRTs. Network, 100, 2.

Ethnography of the SOC

“The profession [security] is so nascent that the how-tos have not been fully realized even by the people who have the knowledge…the process required to connect the dots is unclear even to analysts.

Sundaramurthy, S. C., McHugh, J., Ou, X., Rajagopalan, S. R., & Wesch, M. (2014). An anthropological approach to studying CSIRTs. Network, 100, 2.

Symptoms of a Cognitive Crisis1. Demand for expertise greatly

outweights supply2. Most information cannot be trusted or

validated3. Inability to mobilize and tackle big

systemic issues

The Cognitive Revolution1. Understand the

processes used to draw conclusions

2. Develop repeatable methods and techniques

3. Build and advocate training that teaches practitioners how to think

What separates novice and

expert analysts?

Mapping the Investigation Sample:

Novice and expert analysts Methodology:

30+ case studies Stimulated recall interviews Focus on individual investigations of

varying types Perform key phrase analysis – analyze

results

Key Phrase Mapping Dual Process Theory

Intuition: Implicit, unconscious, fast Reflection: Explicit, controlled, slow

IntuitionExperimentation

RestructuringImaginationIncubation

MetacognitionEvaluation

Goal SettingMaking Plans

ReflectionAnalytically

Viewing DataRule-Based Reasoning

Considering Alternatives

Results

Expe

rimen

tation

Restruc

turing

Imag

inatio

n

Incub

ation

Evalu

ation

Goal S

etting

Making

Plan

s

Viewing

Data

Consid

ering

Alter

nativ

es

Novices Experts

Intuition Metacognition Reflection

Analyzing the Flow of

the Investigation

Investigations as Mental Labyrinths

The investigation is the core construct of information security.

How do we study them when everyone has a different toolset? Follow the Data!

Alert

OSINTReputation

File Hash

Sandbox Behaviors

AV Detections

(VT)

Imphash More File Hashes

Friendly Host

Network PCAP

Host

Windows Logs

Security Log

System Log

App LogRegistry

File SystemHostile

Host NetworkPCAP

Flow

Studying the Investigation Process

Studying the Investigation Process

What data did analysts look at first?

72%16%

12%

Observed

PCAP FlowOSINT

Data Suggests: Analysts prefer a higher context data set…

…even if other data sets are available …even if lower context data sets can lead to a resolution.

Did the first move affect analysis speed?

Data Suggests: While PCAP provides richer context, it may slow down the

investigation if that’s where you start Starting with a lower context data source can increase

speed when working with higher context data

PCAP Flow OSINT

16

10 9

Avg Time to Close

What happens when Bro data replaces PCAP?

46%

25%

29%

Observed (Bro)

Bro Flow OSINT

72%

16%

12%

Observed (PCAP)

PCAP Flow OSINT

What happens when Bro data replaces PCAP?

PCAP Flow OSINT

16

10 9

Avg Time to Close (PCAP)

Bro Flow OSINT

10 10 11

Avg Time to Close (Bro)

Data Suggests: Better organization of high context data

sources can yield improvements in analysts performance

What data sources were viewed most and least frequently?

Data Suggests: Network data is used more frequently than host data…

…even when host data can be used exclusively to resolve. …even when easy access is provided to host sources.

Revisting data is more prevalent on higher context data sources

PCAP Flo

wOSIN

T

Host FS

OS Log

s

Memory

Data Sources Viewed Data Sources Revisited

PCAP84%

Flow11%

OSINT5%

How many steps were taken to make a disposition judgement?

Data Suggests: At some point, the number of data sources you

investigate impacts the speed of the investigation Understanding where data exists and when to use it can

impact analysis speed

6-10 11-15 16-20 21-250

5

10

15

6

129

3

Number of Steps

6-10 11-15 16-20 21-2505

1015202530

9 12 14

24

Avg Time to Close

Did analysts investigate friendly or hostile systems first?

9%

91%

Observed

Friendly Hostile

Data Suggests: Analysts are more compelled to investigate unknown external

threats than internal systems Analysts don’t fully understand their own techniques

41%59%

Friendly

Friendly Hostile

Thank You!

Mail: chris@chrissanders.orgTwitter: @chrissanders88

Blog: chrissanders.orgTraining:

chrissanders.org/training