Using Event Correlation Technologies h Njemanze

8/6/2019 Using Event Correlation Technologies h Njemanze

1/42

Stop The Insanity: Using EventCorrelation Technologies, Tools, andTechniques to Extract MeaningfulInformation from Data Overload

Hugh NjemanzeCTO and Founder

May 2006

2005 ArcSi ht Confidential


2/42

2005 ArcSight Confidential 2

Agenda

What is the Problem?

What is Correlation?

How to Think about the Process

Correlation Technologies, Tools and Techniques

Benefits of Visual Representations


3/42


What is the Problem?


4/42


What is the Problem?Complexity of the Security Infrastructure

Flood of unread data/logs

Islands of defense

Massive false positives

Heterogeneous devices

Inefficient and Ineffective

AntiVirusAntiVirusDatabases

FirewallsFirewallsFirewallsFirewallsFirewallsFirewalls

IntrusionDetectionSystems






Intrusion

DetectionSystems

IntrusionDetectionSystemsIntrusionDetectionSystems

Hosts




IntrusionDetectionSystemsNetwork

Equipment

Applications

ApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplications

Sign-OnSign-OnSign-On

DirectoryServices


5/42


Deal with a Flood of Diverse Data

Events from many sensors

NIDS, HIDS, firewalls, anti-virus, more

Application logs, phone logs, moreUnderstanding the protected network

Vulnerability assessment scanners

Configuration management databases

Understanding ofvulnerabilities

CVEOASIS


6/42 2005 ArcSight Confidential 6

The Needle in the Haystack

Raised Alerts

Case Workflow

Raw events

NormalAudit trail

Failed attacks

False alarms

Pre-attacksAttack

formation

Verified

breachesPolicy

violations

Identifiedvulnerabilities

Misuse

Potentialbreaches

Tens of millionsper day

Millionsper day Less than

1 millionper month

A few thousandper month



A Single Integrated Solutionis Required for ESM

AntiVirusAntiVirusDatabases

ArcSight Monitoring, Visualization, and Reporting

ArcSight Real-time Analysis, Correlation, and Workflow

ArcSight Event Collectors

FirewallsFirewallsFirewallsFirewallsFirewallsFirewalls







Intrusion

DetectionSystems

IntrusionDetectionSystemsIntrusionDetectionSystems

Hosts




IntrusionDetectionSystemsNetwork

Equipment

Sign-OnSign-OnSign-On

DirectoryServices

Applications

ApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplicationsApplications

ArcSightConsoleTM




A relation existing betweenphenomena or things which tendto vary, be associated, or occurtogether in a way not expectedon the basis of chance alone.

Merriam-Webster Dictionary



Also, Perhaps, Inference

The reasoning involved in drawing aconclusion or making a logical judgmenton the basis of circumstantial evidence

and prior conclusions rather than on thebasis of direct observation.

Princeton Universitys WordNet



Highlight Changes in Behavior

Changes in the typical event flow may indicate

An ongoing attack

Denial of service: the source is deadCompromise: the source is behaving atypically

New patterns of behavior may indicate

The presence of malware

An insider threat

Introduction of new software or devices



Escalation: Sounding the Alarm

Generate notifications

Email, page, pop-up

Open a caseTrouble tickets, incident tracking

Create alarms

Tracking events



How to Think About theProcess



Process

Intelligence

Collection, normalization and aggregation

Risk-based prioritization with vulnerability and asset information

Correlation across event sources

Rule-based correlation

Statistical Correlation

Advanced analysis



Event Normalization and Categorization

Jun 01 2005 00:00:12: %PIX-3-106011: Deny inbound (No xlate) udp src

outside:10.50.215.97/6346 dst outside:204.110.228.254/6346

Jun 01 2005 00:00:12: %PIX-6-305011: Built dynamic TCP translation from

isp:10.50.107.51/1967 to outside:204.110.228.254/62013

Jun 01 2005 00:00:12: %PIX-6-302013: Built outbound TCP connection

2044303174 for outside:213.189.13.17/80 (213.189.13.17/80) to

isp:10.50.107.51/1967 (204.110.228.254/62013)

Jun 02 2005 12:16:03: %PIX-6-106015: Deny TCP (no connection) from

10.50.215.102/15605 to 204.110.227.16/443 flags FIN ACK on interface

outside

Sample Raw Pix Events:

Jun 02 2005 12:16:03: %PIX-6-106015:

Deny TCP (no connection) from

10.50.215.102/15605 to 204.110.227.16/443

flags FIN ACK on interface outside

Categorization:Normalization:


16/42


Diverse Data Sources: Event Normalization

Comparing apples to apples

Many vendors

Many types of sensorsLots of overlap

Normalization

Common schema: info in the same place

Categorization: describing the event

Values: single domain


17/42


Diverse Data Sources: Event Normalization

Aggregation: easier to establish equivalence

Rules can be written once and applied to all sensorsof a given type

Simplifies log review when multiple brands of sensorare present

Speeds training of new personnelEasier to understand events


18/42


Risk-based Prioritization

WindowsSystems

Unix/Linux/

AIX/Solaris

SecurityDevice

SecurityDevice

Mainframe& Apps

SecurityDevice

Agents

Event

Manager

PrioritizedEvent

VulnerabilityScanner

Agents

AssetInformation

Model ConfidenceHas asset been

scanned for open portsand vulnerabilities?

RelevanceAre ports open on asset?

Is it vulnerable?

SeverityIs there a history withthis attacker or target

(active lists)?

Asset CriticalityHow important is this

asset to the business?

Agent SeverityMapping of reporting

device severity toArcSight severity


19/42


Event Correlation

Most overused and least well-defined concept in ESM.

Combine multiple events through predefined rules

Or analyze statisticalproperties of event streams

Across devices

Heavily utilizing event categorization

Helps eliminate false positives

Correlation is not prioritization! Can use priorities of individual events


20/42


Rule-based Correlation


Multiple failed loginson Windows systems

Multiple failed loginson UNIX systems

5 or more failed

loginsin a minute

from same source

Attempted Brute

Force Attack

Attempted Brute

Force Attack


21/42


Rule-based Correlation


Attempted BruteForce Attack + Successful

Login

Successful

login to Windows system

Attempted Brute

Force Attack


22/42


Statistical Correlation

Analyze statisticalproperties of event streams

?

50% increase

in traffic per port

and machine

Traffic per port going to 10.0.0.2

False positives reduction:

Correlate against other event streams

Restrict to only monitor specific systems and specific type of traffic

8


23/42


Many Correlation/Inference Techniques

Model-based

Assets

ThreatsHeuristic

Pattern

Formula

Mathematical

AnomalyCovariant

Slide 23


24/42

j8 I think the following slides can be used as some of the voice over from the previous 4-6 slides?jkyte, 10/11/2005


25/42


Model-Based Reasoning

Checking the protected network

Does the device exist?

Applications presentOperating systems

Vulnerabilities exposed

Business significance

Extensible via active lists

Attackers: suspicious, recon, hostileDevices: scanned, attacked, compromised


26/42


Heuristic: Formula-Based

SeverityWhat is the attack potential?

Model Confidence and RelevanceCould it work?

Asset Criticality

How valuable is the target?

Priority

Which incident should be worked first?


27/42


Mathematical

Statistical data monitors

Moving average

Statistics

Correlation

Pattern discovery

Covariant occurrence ofindividual events

Statistics data monitors spot gross changes in theevent flow

More attacks against certain ports, networks

Sudden drop in events from a service

Discovery spots behaviors on the protected network

New exploits

Returning exploits: that virus is back!


28/42


How: Correlation

Technologies, Tools andTechniques

Traditional Approach Log Files and Events


29/42


Traditional Approach Log Files and Events

A Visual Approach


30/42


ppSituational Awareness - Instant Awareness


31/42


Why a Visual Approach Helps

Reduce analysis and response times Quickly visualize thousands of events

Make better decisions

Situational awareness

Visualize status of business posture

Visual display of most important propertiesBe more efficient

Facilitate communication

Use graphs to communicate with other teams Graphs are easier to understand than textual events

A picture tells more than a thousandlog lines


32/42


Three Aspects of Visual Security Analysis

Situational Awareness

What is happening in a specific business area(e.g., compliance monitoring)

ific network What is happening on a spec What are certain servers doing

Real-Time Monitoring and Incident Response

Capture important activities and take action Event Workflow

Collaboration

Forensic Investigation Selecting arbitrary set of events for investigation

Understanding big picture

Analyzing relationships


33/42


Responding: Monitoring and Reporting

Live monitoring

Channels

DashboardsReporting


34/42


Situational Awareness Event Graph Dashboard


35/42


Real-time Monitoring Detect Activity


36/42


Visual Detection

Scan Events

Firewall Blocks

Scanning activity is displayed

Vi l I i i


37/42


Visual Investigation

D fi N C l ti R l d Filt


38/42


Define New Correlation Rules and Filters

Assign for further analysis if More than 20 firewall drops

froman external machine

toan internal machine

1. Rule

Internal machines on white-list

connecting toactive directory servers

2. Filter

3. Open a ticketfor Operations to

quarantine and clean infected machines

F i A l i


39/42


Forensic Analysis

Failed Logins High ratio of failed logins

Forensic Analysis


40/42


Forensic Analysis

Attacks targeting internal systemsAttacks

Revenue Generating Systems

Summing Up


41/42


Summing Up

Effective correlation enables codifying and leveragingdomain expertise to automate finding the needles in thehaystack of security logs, alerts and events

Visualization techniques provide a very intuitive way forhuman analysts to quickly spot patterns and activity thatwould otherwise be buried in logged data

Gathering all the data in one place to start with providesa vantage point from which to apply the tools andtechniques described above


42/42

Q & A

Email to: [email protected] 2005 ArcSi ht Confidential

Documents

Using Event Correlation Technologies h Njemanze