WhatisDataScience?

“Raw”data“Ac0onable”

data

“NeedleDetector”

“DataScience(DS)istheextrac0onofknowledgefromlargevolumesofdata…”

hBps://en.wikipedia.org/wiki/Data_science

FrederickD.Pearce

What’sa“Good”DSenvironment?

“Raw”Data

“Ac0onable”Data

Analy8cs“Kitchen”

Fresh/Local/Organic “Right”Tools Read/WriteRecipes

Python?ELK?Splunk?Log*?

FrederickD.Pearce

FrederickD.PearceDataScienceProcessSchema0cbyFarcasteratEnglishWikipedia,CCBY-SA3.0,

hBps://commons.wikimedia.org/w/index.php?curid=40129394

What’sinaDS“BlackBox”?

PeriodicModel“Recipe”

“Raw”Data

PeriodicModelParameters

PeriodicModel“Recipe”

Average(μReq)≈???[Requests]Freq.(fReq)≈???[Hertz]Amp.(ΑReq)≈???[Requests]

OpenDNSpythontool

“Cleaner” “Inverter”

hBp://system.opendns.com

FrederickD.Pearce

PM:Fast-FourierTransform

hBps://en.wikipedia.org/wiki/Fast_Fourier_transform

xn

0me(tn)

|Xk|

frequency(fk)

FFT

FFT-1**

*

**

*

**

* *** * * *

*

•  Data(xn)decomposedintodiscretefrequencycomponents•  Xkiscomplexvalued,withΑk=|Xk|andφk=tan-1(Xk)•  It’sfast!DFTisO(N2)whileFFTisO(N*log(N))

€

Xk ≡ xn * exp− i(2π

kN)n

n=0

N −1

∑ k = 0,...,N −1FFT:

fNyquist

FrederickD.Pearce

PM:So,what’stheanswer?

Mean(μReq)≈75.4[Requests*109],Freq.(fReq)≈1.64*10-6[Hertz],Amp.(ΑReq)≈8.2[Requests*109],Phase(φReq)≈175[degrees]

(fReq,ΑReq)

FrederickD.Pearce

Applica0onto“Log”DataDevelopPython-based“securityrecipes”thatcombinesearchandanaly0cscapabili0esofPythonandELK*

*Possibletointegratewithothercodingtoolstoo(e.g.R)

FFTof“idle”VMlogdatacollectedwithBRO(haps://www.bro.org/)

f=1/24hr

FrederickD.Pearce