79
Splunking the Endpoint Simon O’Brien Sales Engineer/Security SME, Splunk [email protected]

SplunkSummit 2015 - Splunking the Endpoint

  • Upload
    splunk

  • View
    499

  • Download
    4

Embed Size (px)

Citation preview

Page 1: SplunkSummit 2015 - Splunking the Endpoint

Splunking the  Endpoint

Simon  O’BrienSales  Engineer/Security  SME,  Splunk

[email protected]

Page 2: SplunkSummit 2015 - Splunking the Endpoint

2

DisclaimerDuring  the  course  of  this  presentation,  we  may  make  forward  looking  statements  regarding  future  

events  or  the  expected  performance  of  the  company.  We  caution  you  that  such  statements  reflect  our  current  expectations  and  estimates  based  on  factors  currently  known  to  us  and  that  actual  events  or  results  could  differ  materially.  For  important  factors  that  may  cause  actual  results  to  differ  from  those  

contained  in  our  forward-­‐looking  statements,  please  review  our  filings  with  the  SEC.  The  forward-­‐looking  statements  made  in  the  this  presentation  are  being  made  as  of  the  time  and  date  of  its  live  presentation.  If  reviewed  after  its  live  presentation,  this  presentation  may  not  contain  current  or  accurate  information.  

We  do  not  assume  any  obligation  to  update  any  forward  looking  statements  we  may  make.  

In  addition,  any  information  about  our  roadmap  outlines  our  general  product  direction  and  is  subject  to  change  at  any  time  without  notice.   It  is  for  informational  purposes  only  and  shall  not,  be  incorporated  into  any  contract  or  other  commitment.  Splunk  undertakes  no  obligation  either  to  develop  the  features  

or  functionality  described  or  to  include  any  such  feature  or  functionality   in  a  future  release.

Page 3: SplunkSummit 2015 - Splunking the Endpoint

3

About  Me…

1.5  Years

Page 4: SplunkSummit 2015 - Splunking the Endpoint

DEMO,  PART  I

Page 5: SplunkSummit 2015 - Splunking the Endpoint

5

Do  you  know  these  men?

Page 6: SplunkSummit 2015 - Splunking the Endpoint

6

Session  Goals

• Understand  why  you  should  Splunk  the  endpoint

• Believe  that  the  Universal  Forwarder  is  awesome

• Learn  about  customer  success• Get  some  artifacts  you  can  use• Bring  home  what  you  can  do  today

Page 7: SplunkSummit 2015 - Splunking the Endpoint

7

WHY?1. It  is  relatively  inexpensive  to  Splunk  

your  endpoints,  and  it  will  improve  your  security  posture.

2. VISIBILITY!  You  will  have  more  complete  information in  the  case  of  breach.

3. The  information  from  your  endpoints  maps  well  to security  guidance,  including  the  CSC  20  and  the  ASD  top  35.

Page 8: SplunkSummit 2015 - Splunking the Endpoint

8

You  may  have  heard…

Endpoint/Server  Vulnerabilities Endpoint-­‐Based  Malware

Page 9: SplunkSummit 2015 - Splunking the Endpoint

9

So  these  happened   in  2014/2015…

Endpoint/Server  Vulnerabilities Endpoint-­‐Based  Malware….the  endpoints?

Could  we  be  more  secure  if  we

Page 10: SplunkSummit 2015 - Splunking the Endpoint

10

Executive  Summary:

YES!(so  do  that)

Page 11: SplunkSummit 2015 - Splunking the Endpoint

THANKYOU!

Page 12: SplunkSummit 2015 - Splunking the Endpoint

12

The  Endpoint  is  important!

Closest  to  humans Versatile

Underprotected Data-­‐rich

Page 13: SplunkSummit 2015 - Splunking the Endpoint

13

The  Endpoint  is  important!

Closest  to  humans Versatile

Underprotected Data-­‐rich

70%of  successful  breaches  start  on  the  endpoint*

*IDC  study  2014

Page 14: SplunkSummit 2015 - Splunking the Endpoint

14

The  UF:  It’s  more  than  you  think

Logs

….your  endpoints.

The  Universal  Forwarder  allows  you  to

Page 15: SplunkSummit 2015 - Splunking the Endpoint

15

The  UF:  It’s  more  than  you  think

Logs

Page 16: SplunkSummit 2015 - Splunking the Endpoint

16

The  UF:  It’s  more  than  you  think

Scripts

Perfmon

Wire  Data

Logs

Process/Apps/FIM

Registry

Sysmon

Page 17: SplunkSummit 2015 - Splunking the Endpoint

17

Splunk  Universal  Forwarder  for  ETD*!• “Free”• Lightweight• Secure• Runs  on  many  versions  of  

Windows  &  *NIX  &  OSX• Flexible• Centrally  configurable• SCALE!

*Endpoint  Threat  Detection  (Response?)

Page 18: SplunkSummit 2015 - Splunking the Endpoint

18

What  about  the  “Response”?

VISIBILITYreactivity

(for  now)

Page 19: SplunkSummit 2015 - Splunking the Endpoint

19

Splunk  Forwarder  for  ETD*!• “Free”• Lightweight• Secure• Runs  on  many  versions  of  

Windows  &  *NIX  &  OSX• Flexible• Centrally  configurable• SCALE!

*Endpoint  Threat  Detection  (Response?)

Come  on.  Is  anyone  using  the  Universal  Forwarder  in  this  way?

YES.

Page 20: SplunkSummit 2015 - Splunking the Endpoint

20

Use  Case  1:  Large  Internet  Company

…x  (Many  indexers)

on  prem dmz

Int.  forwarders

ds

install

config

internet

UFx10,000!

Individual  certs

• Windows  event   logs• OSX  /var/log/*• Carbon  Black  output• Crash  logs  for  IT  Ops• Custom  script  for  apps  

installed• UNIX  TA  (upon  request)• Windows  TA  (upon  request)• Additional  granularity  for  

execs  and  their  admins• Moving  to  Splunk  Cloud

search

!

Page 21: SplunkSummit 2015 - Splunking the Endpoint

21

Central  Control  with  Deployment  Server

One  (Linux)  DS  =  10,000  endpoints!

Page 22: SplunkSummit 2015 - Splunking the Endpoint

Proxy  Logs

22

Additional  ways  to  gather  endpoint  data

Integrity  Management

NG  Endpoint  Protection

WhitelistingLook  for  apps  

on  splunkbase!

Page 23: SplunkSummit 2015 - Splunking the Endpoint

23

Back  to  these  breaches…

Endpoint-­‐Based  Malware

Registry  Entries

System  Event  Logs

New  Services

New  Files

Comms/Running   Proc

Security  Event  Logs

Known  Vulns/Apps

Page 24: SplunkSummit 2015 - Splunking the Endpoint

24

Let’s  map  these  to  the  capabilities  of  the  UF…Registry  Entries

System  Event  Logs

New  Services

New  Files

Comms/Running   Proc

Security  Event  Logs

Known  Vulns/Apps

Page 25: SplunkSummit 2015 - Splunking the Endpoint

25

We  configure  the  forwarder  to  give  us  data  of  interest

Registry  Entries

System  Event  Logs

Security  Event  Logs

New  Services

New  Files

Comms/Running   Proc

WinRegMon

WinEventLog:  System  and  WinHostMon

WinEventLog:  Security  +  Auditing

Scripted  Inputs

WinEventLog:  System

WinEventLog:  Security

TA-­‐Microsoft-­‐SysmonStream,  WinHostMon

Windows  Update

Monitor:  WindowsUpdate.log

Known  Vulns/Apps

Scripted  Inputs  or  WinHostMon

Configuration  examples?  See  

demo  &  appendix

Page 26: SplunkSummit 2015 - Splunking the Endpoint

26

What  could  we  look  for?• ANY  new  Windows  services• Registry  being  written  to  where  it  should  not• Users  that  shouldn’t  be  used• Unusual/unapproved  processes  being   launched  and  their   connections/hashes• Unusual/unapproved  ports/connections  in  use• Unapproved  USB  devices  being  inserted• New  files   in  places   they  should  not  be  (Windows\System32…)• Files   that  look  like  one  thing  but  are  really  another• New  drive  letters  being  mapped• Lack  of  recent  Windows  updates• Versions  of  software  known  to  be  vulnerable• …and  more

INSTANT,  GRANULAR  DATA  ABOUT  COMMON  BEHAVIOR  OF  WINDOWS  MALWARE!

Page 27: SplunkSummit 2015 - Splunking the Endpoint

DEMO,  PART  II

Page 28: SplunkSummit 2015 - Splunking the Endpoint

28

Use  Case  2:  UF  for  ATM  Security  +  Fraud• Bank  uses  ATMs  that  are  Windows-­‐based• Each  ATM  has  a  UF  installed,   securely  sending  data  to  

intermediate  forwarder  on  prem and  then  up  to  Splunk  Cloud

• Data  retrieved  from  custom  ATM  logs  – can  understand  what’s  going  on  within  1-­‐2  seconds

• Customer  reps  can  see  what  the  problem   is  easily• Understand  baseline  – when  are  ATMs  popular?  

Handle  the  cash  levels• Understand  fraud  – has  someone  stolen  a  card  +  PIN  

and  hitting  ATMs  in  close  clusters?  “Superman”  correlation

• Conversion  Opp:  know  that  a  3rd-­‐party  bank  customer  hits  a  bank  ATM  every  Friday  for  $200

Regional  Bank  in  NE,  US

Page 29: SplunkSummit 2015 - Splunking the Endpoint

29

How  about  inventory  +  vulnerabilities?

Page 30: SplunkSummit 2015 - Splunking the Endpoint

30

How  about  inventory  +  vulnerabilities?

Page 31: SplunkSummit 2015 - Splunking the Endpoint

31

Two  ways  to  get  installed  apps,  there  are  more…  

Scripted  Input  from  Windows  TA  or  WinHostMon

Microsoft  Sysmon

Page 32: SplunkSummit 2015 - Splunking the Endpoint

32

What  versions  of  what  exist  on  my  network?

Scripted  Input  from  Windows  TA  or  WinHostMon

Do  I  have  known  vulnerable  software  on  endpoints?

Page 33: SplunkSummit 2015 - Splunking the Endpoint

33

Hash  data  from  apps

Microsoft  Sysmon

Correlate  hash  with  threat  intel

Page 34: SplunkSummit 2015 - Splunking the Endpoint

34

Windows  Update  data

Page 35: SplunkSummit 2015 - Splunking the Endpoint

35

Windows  Update  Data  (two  sourcetypes)

Monitor:  WindowsUpdate.log

Monitor:  WinEventLog:System

Page 36: SplunkSummit 2015 - Splunking the Endpoint

36

Windows  Port  Data

Scripted  input  from  Windows  TA  or  WinHostMon

Page 37: SplunkSummit 2015 - Splunking the Endpoint

37

Windows  Port  DataPID  data=easy  correlation  to  

process  responsible

Or  use  sysmon…

Page 38: SplunkSummit 2015 - Splunking the Endpoint

38

Endpoint  info  critical  to  CSC  (SANS)  201  &  2:  Log  hardware  info,  running  procs/svcs3:  Scripted  inputs  to  check  for  config issues4:  Evaluate  processes/services  for  vulns5:  Look  for  malicious  new  services/processes  11:  Look  for  malicious  ports/protocols  12:  Look  for  local  use  of  priv accounts14:  Gather  windows  events/*NIX  logs16:  Evaluate  use  of  screensaver  locks17:  Identify  lapses  in  local  encryption

You  could  do  all  of  that  with  the  Universal  Forwarder.

Similar  mappings  to  ASD  35…

Page 39: SplunkSummit 2015 - Splunking the Endpoint

39

Threat  Intelligence,  you  say?

File  names  and  hashes

Expired/bogus  certs

Known   Bad  IP

Processes/Services

Page 40: SplunkSummit 2015 - Splunking the Endpoint

40

Endpoint  vulns can  be  found  if  you  google what  to  look  for…

Page 41: SplunkSummit 2015 - Splunking the Endpoint

41

Remember  this?

shellshock

• Publicly  announced  on  24/9/2014.• One  Vulnerability  Management  vendor  had  a  plugin  

on  25/9.  That’s  pretty  good!• Others  followed  on  26/9  and  29/9 – not  so  good.• These  require authenticated scans.  

Page 42: SplunkSummit 2015 - Splunking the Endpoint

42

Remember  this?

shellshock

• Publicly  announced  on  9/24/2014.• One  Vulnerability  Management  vendor  had  a  plugin  

on  9/25.  That’s  pretty  good!• Others  followed  on  9/26  and  9/29 – not  so  good.• These  require authenticated scans.  make  this  process  more  timely?

Could

Page 43: SplunkSummit 2015 - Splunking the Endpoint

43

The  Universal  Forwarder  as  self-­‐help  guru

That  UF  sure  does  a  lot  by  

itself!

Page 44: SplunkSummit 2015 - Splunking the Endpoint

44

The  Universal  Forwarder  as  self-­‐help  guru• If  you  had  the  Splunk  UF  on  all  of  your  production  

*NIX  servers…• You  could  very  quickly  program  them  to  find  

shellshock  (or  ghost,  or  poodle,  or  heartbleed).• You  avoid  Vulnerability  Management  Vendor  Lag• You  could  then  report  on  remediation  efforts  over  

time.• And the  data  ingest  would  be  very  small.

Page 45: SplunkSummit 2015 - Splunking the Endpoint

45

5  Step  Vulnerability  Tracking  Strategy1. On  day  one,  become  aware  of  vulnerability2. Google  “how  to  detect  $vulnerability$”3. Adopt  code  via  script  (shell,  batch,  etc)  and  place  into  your  Splunk  deployment  server4. Forwarders  run  code  and  deliver  results  into  Splunk   indexers5. Report  on  the  results

A  good  step  by  step

Page 46: SplunkSummit 2015 - Splunking the Endpoint

46

Use  Case  3:  UF  for  Shellshock  Tracking

“We  wrote  it  on  the  same  day  and  ran  it  – it  was  really  fundamental  to  our  defense.”  – Mark  Graff,  NASDAQ

Shellshock  on  20,000  Linux,  Solaris,  AIX  servers  tracked  in  Splunk

(Large  payment  processing  company)

Page 47: SplunkSummit 2015 - Splunking the Endpoint

47

How  about  wire  data?• Technology  Add-­‐on  or  TA  (Splunk_TA_stream)• Provides  a  new  Data  Input  called  “Wire  Data”– passively  captures  traffic  using  a  modular  input

– C++  executable  called  “Stream  Forwarder”  (streamfwd)

• Captures  application  layer  (level  7)  attributes• Automatically  decrypts  SSL/TLS  traffic  using  RSA  keys

Turn  the  UF  into  a  little  

network  sniffer

Page 48: SplunkSummit 2015 - Splunking the Endpoint

48

Stream  Protocols/Platforms  Supported• UDP• TCP• HTTP

• IMAP• MySQL  (login/cmd/query)

• Oracle  (TNS)• PostgreSQL• Sybase/SQL  Server  (TDS)  

• FTP• SMB• NFS• POP3• SMTP

• LDAP/AD• SIP• XMPP• AMQP• MAPI• IRC

Supports  Windows  7  (64-­‐bit),  Windows  2008  R2  (64  bit),  Linux  (32-­‐bit/64-­‐bit)   and  Mac  OSX  (64-­‐bit)

• DNS• DHCP• RADIUS• Diameter• BitTorrent• SMPP

Page 49: SplunkSummit 2015 - Splunking the Endpoint

49

How  much  data?

TA-­‐microsoft-­‐sysmon

Splunk_TA_windows

“a  typical  day  at  the  office…”

Nice  try,  O’Brien!

All  this  endpoint  Splunking will  blow  up  my  license…

Page 50: SplunkSummit 2015 - Splunking the Endpoint

50

How  much  data?

TA-­‐microsoft-­‐sysmon

Splunk_TA_windows

“a  typical  day  at  the  office…”

Page 51: SplunkSummit 2015 - Splunking the Endpoint

51

How  much  data?

A  12  hour  day.  Even  in  

marketing!

Page 52: SplunkSummit 2015 - Splunking the Endpoint

52

How  much  data?

12  hours  of  standard  event  logs  =  5.5  MB.  Nice!

Page 53: SplunkSummit 2015 - Splunking the Endpoint

53

How  much  data?

Hmm.  Lot  more  events…

Page 54: SplunkSummit 2015 - Splunking the Endpoint

54

How  much  data?

12  hours  of  Sysmon logs  =  241  MB.  Oh  crap.  There  goes  my  Splunk Summit  talk…!!

Page 55: SplunkSummit 2015 - Splunking the Endpoint

55

How  much  data?

Lots  of  red….let’s  take  

that  out.

Page 56: SplunkSummit 2015 - Splunking the Endpoint

56

How  much  data?

That’s  more  like  it.  16MB  of  Sysmon,  5.5MB  of  Windows  events  =  21.5MB  per  endpoint.

Coverage  for  1,000 Windows  endpoints?  21.5GB ingest,  per  day.  

Page 57: SplunkSummit 2015 - Splunking the Endpoint

57

Sysmon with  network/image  filtering?

• Start/Stop   of  all  processes• Process  names  &  full  command  line  args• Parent/child  relationships  (GUIDs)  between  processes• Session  IDs• Hash  and  user  data  for  all  processes• Filenames  that  have  their  create  times  updated• Driver/DLL  loads  with  hash  data

• Network  communication  per  process  (TCP  and  UDP)  including  IP  address,  size,  port  data• Ability  to  map  communication  back  to  process  GUID  and  session  ID

You  still  get…

You  lose…

You  retain  far  more  function  than  you  lose.

Page 58: SplunkSummit 2015 - Splunking the Endpoint

58

So  you  can  still  do…

I  surfed  a  whole  lot  in  Chrome  

today…listened  to  some  tunes,  too!

Page 59: SplunkSummit 2015 - Splunking the Endpoint

59

And  also…I  really  DID  work  on  that  300  slide  powerpoint before  lunch,  I  swear!

Page 60: SplunkSummit 2015 - Splunking the Endpoint

60

In  Sum1. If  you’re  not  Splunking  the  data  from  your  various  endpoints  today,  

you  should  be.  2. The  Splunk  Universal  Forwarder  is  a  super-­‐powerful  tool  to  use  on  

your  endpoints,  free  to  install,  scales  well,  can  be  centrally  configured,  and  data  volumes  are  quite  reasonable.  

3. For  Windows,  event  data  is  critical.  Sysmondata  is  great  too,  and  free  to  install.

4. Other  customers  from  many  verticals  are  having  continued  success  with  the  data  they  can  gather  from  endpoints.

Page 61: SplunkSummit 2015 - Splunking the Endpoint

FINAL  QUESTIONS?

Page 62: SplunkSummit 2015 - Splunking the Endpoint

THANKYOU!

[email protected]

Page 63: SplunkSummit 2015 - Splunking the Endpoint

APPENDIX

[email protected]

Page 64: SplunkSummit 2015 - Splunking the Endpoint

SYSMONDETAILS

[email protected]

Page 65: SplunkSummit 2015 - Splunking the Endpoint

65

Sysmon Info• Blog  post  from  November,  2014• App  available  on  Splunkbase,  works  with  current  (3.1)  version  of  Sysmon:  

• Forwarder  6.2+  needed  to  get  XML  formatted  Sysmon data  (a  good  idea,  cuts  down  on  size)

Page 66: SplunkSummit 2015 - Splunking the Endpoint

66

Sysmon Filters• This  works  for  Sysmon3.1+

• Add  what  you  need• If  you  actually  want  Image  and  Network  data,  add  those  stanzas

• Email  [email protected] for  links  to  example  files!

Filter  out  all  the  Splunk  activity

Page 67: SplunkSummit 2015 - Splunking the Endpoint

67

Sysmon Config List• sysmon –c  with  no  filename  will  dump  config

Image  and  Network  disabled

Page 68: SplunkSummit 2015 - Splunking the Endpoint

68

Sysmon Config Load• sysmon –c  with  filename  will  load  config

• No  restart  needed• Ignore  errors• Run  as  admin  (or  script  as  admin)

Page 69: SplunkSummit 2015 - Splunking the Endpoint

Hash  Analysis  with  Sysmon

[email protected]

Page 70: SplunkSummit 2015 - Splunking the Endpoint
Page 71: SplunkSummit 2015 - Splunking the Endpoint
Page 72: SplunkSummit 2015 - Splunking the Endpoint
Page 73: SplunkSummit 2015 - Splunking the Endpoint
Page 74: SplunkSummit 2015 - Splunking the Endpoint

Windows  registry  monitoring

[email protected]

Page 75: SplunkSummit 2015 - Splunking the Endpoint

75

Registry  Monitoring  config• Simple  examples  shown  here

• Email  [email protected]  an  extensive  registry  monitoring  configbased  on  Autoruns

Page 76: SplunkSummit 2015 - Splunking the Endpoint

76

PLACEHOLDER:  WinregWill  have  link  and  other  info  here  detailing  how  to  do  windows  registry  with  sample  config of  400+  registry  keys  to  monitor.

If  you  monitor  the  right  reg key  you  can  find  new  USB  insertions.

Page 77: SplunkSummit 2015 - Splunking the Endpoint

77

Registry  Results• USB  inserted  with  BlackPOS malware

• Malware  executed  –these  are  the  registry  changes  logged

Page 78: SplunkSummit 2015 - Splunking the Endpoint

winhostmon

[email protected]

Page 79: SplunkSummit 2015 - Splunking the Endpoint

79

WinHostMon• Get  hardware  details,  services,  processes,  apps,  etc…

• Built  right  into  the  forwarder,  no  scripts  needed