1

Fast Portscan Detection Using Sequential Hypothesis Testing

Jaeyeon Jung, Vern Paxson, Arthur W. Berger, Hari Balakrishnan

MIT CSAIL, Berkeley ICSI

2

Outline

IntroductionPrevious Detection TechniquesData AnalysisThreshold Random WalkEvaluationFuture WorksConclusion

3

Outline

IntroductionPrevious Detection TechniquesData AnalysisThreshold Random WalkEvaluationFuture WorksConclusion

4

Port Scanning

Network reconnaissance techniqueHacker will scan hosts for vulnerable ports as potential avenues of attacks

Difficult to detectHard to differentiate between benign scans and scans with malicious intentStealth scans can occur very slowlySome scans are legitimate

Search engine spidersSSH, P2P applications

5

Outline

IntroductionPrevious Detection TechniquesData AnalysisThreshold Random WalkEvaluationFuture WorksConclusion

6

Limit Distinct Connections

Network Security Monitor (NSM)Connect to more than 15 destination IP within a given time windowDrawback: window size, error flag

SnortPack-oriented:

malformed packets used for stealth scanning

Connection-oriented:touch more than X ports or Y IPs within Z seconds

7

Limit Failed Connections

Bro NIDSMalicious connections will have a high ratio of failed connection attemptsSensitive to services on specific ports

Robertson et al.Using a similar threshold methodPerformance varies based on threshold value

8

Probabilistic Model

Leckie et al.Derive an access probability distributionCompare probabilities to decide scanner or notDrawback: many false positiveLacks: confidence levels , a proiri probability

SPICEDetect low traffic stealthy scansAssign anomaly scoresRun-time processing, too complicated

9

Outline

IntroductionPrevious Detection TechniquesData AnalysisThreshold Random WalkEvaluationFuture WorksConclusion

10

Date Set

Traffic from two sites: LBL & ICSIResearch laboratories with minimal firewallingLBL: 6,000 hosts, sparse host density 4.4%ICSI: 200 hosts, dense host density 42%

Anonymized TCP logs from Bro NIDSEach log entry lists a timestampRegard IP as active if it generates a response(either a successful or rejected connection)

11

Summary of Datasets

HTTP wormsAttempt to spreading either Code Red or Nimda HTTP worm

Other_badPackets to TCP port 135, 139, 445, 1433Blocked by the firewalls

Known_bad: scanners, HTTP worms, other_bad

12

Outline

IntroductionPrevious Detection TechniquesData AnalysisThreshold Random WalkEvaluationFuture WorksConclusion

13

Observation

Scansusually come from one hostMake lots of failed connection attempts and few successful connection attemptsShould be detected quicklyBound prob of missed detection and false positive

Model

14

Sequential Hypothesis Testing

Two Hypotheses: H1 (scanner) & H0 (benign host)

Four OutcomesDetection:False Negative:False Positive:Normal:

D

D

F

F

P1- PP1- P

1 1

0 1

1 0

0 0

select H when H prob : select H when H prob : select H when H prob : select H when H prob :

15

Likelihood Ratio

Likelihood

Compare with thresholdsUpper threshold : accept H1, infected conclusionLower threshold : accept H0, benign conclusion

: inconclusive

( )n Y

1

00 1( )n <

16

Flow diagram

17

Threshold Random Walk

TRWDecision-making process corresponds to a random walk with two thresholds

Specify two parameters:Typical value:

Estimate two parameters:: probability benign connection is successful: probability scanner connection is successful

and F DP P = 0.01 & = 0.99

and0 1 01

18

Threshold Bounds

Thresholds in terms of and or andF DP P

( )false negative

( )false positive

19

Number of Observation N

Log likelihood ratio

Conditional Expectation for Xi

E[ ]=E[ ]E[ ]N 1 2 N N iS = X + X + + X S X NL

20

Conditional Expectation for SN

Conditional Expectation for N: E[ ]E[ ]=E[ ]

N

i

SNX

( )0 1fucntion of , , ,

21

E[N|H1] vs. other parameters ( )is fixed to 0.99

22

Limitations

Some cases of dependenceIf a scanner probes N inactive servers alternating with N active servers, TRW will oscillate

Dependence leads to positive correlationSuccesses are more likely to be followed by another success or likewise for failure

Dependence invalidates i.i.d. assumptionUsing a joint probability distribution, complicated

23

Outline

IntroductionPrevious Detection TechniquesData AnalysisThreshold Random WalkEvaluationFuture WorksConclusion

24

Trace-driven Simulation

Use the datasets from the two labsConnection information: source IP s, destination IP d, connection statusTCP SYN will result in an established (ACK), rejected (RST), or unanswered connection

Maintains 3 variables for each remote host sDs: distinct IP addresses previously connected toSs: decision states: PENDING, H0, or H1Ls: likelihood ratio

25

Algorithm

For each line in datasetSkip if not PENDINGDetermine if connection is successfulCheck whether d is already belongs to Ds If so, proceed to the next lineUpdate Ds and LsIf , set Ss to H1If , set Ss to H0

1sL

0sL

26

Simulation Results

27

Compare with Bro and Snort

1

# true positiveefficiency =# H

# true positiveeffectiveness =# all scanners

28

Outline

IntroductionPrevious Detection TechniquesData AnalysisThreshold Random WalkEvaluationFuture WorksConclusion

29

Potential Improvement

Leveraging Additional InformationFactor in the specific service (e.g. HTTP)Distinguish between unanswered and rejectedConsider time duration that local host inactiveConsider rate of connection attemptsIntroduce correlation

Managing StateRequire large amounts of states for trackingAttacker keeps going until system crashes

30

How to RespondWhat to do when a scanners is detected?Is it worth blocking?

Evasion and GamingAddress spoofing: white lists, honeypotsEvasion: insert legitimate connections in scan

Distributed ScansScans originating from more than one sourceDifficult to fix in this framework

31

Outline

IntroductionPrevious Detection TechniquesData AnalysisThreshold Random WalkEvaluationFuture WorksConclusion

32

Conclusion

Threshold Random WalkExtremely fast portscan detection algorithmbased on ratio of failed/successful connectionsUnderpinning: Sequential Hypothesis Testing

PerformanceRequire smaller numbers of connections (4 or 5)Quick decision, highly accurate, few false positiveConceptually simple

Fast Portscan Detection Using Sequential Hypothesis TestingOutlineOutlinePort ScanningOutlineLimit Distinct ConnectionsLimit Failed ConnectionsProbabilistic ModelOutlineDate SetSummary of DatasetsOutlineObservationSequential Hypothesis TestingLikelihood RatioFlow diagramThreshold Random WalkThreshold BoundsNumber of Observation N 20 21LimitationsOutlineTrace-driven SimulationAlgorithmSimulation ResultsCompare with Bro and SnortOutlinePotential Improvement 30OutlineConclusion