DETECTING ADVANCED THREATS WITH SYSMON, WEF

AND ELASTICSEARCH

JOSH LEWIS | ROOT9B05.25.15

DETECTING ADVANCED THREATS WITH SYSMON, WEF, AND ELASTICSEARCH

WHY EVENT LOGS?

From an advanced threat detection perspective, most analysts are relatively blind at the host level until they receive network telemetry or a security agent alert (Anti-Virus/HIPS). Based on my experience, network telemetry data is typically collected at network egress points and Anti-Virus/HIPS is poor at detecting pivot and memory based attacks. One approach to overcome these obstacles is to increase visibility at the host level and create indicators of compromise that can trigger forensic investigation by alerting on specific event logs. Event logs are built natively into most operating systems and can immediately send valuable artifacts to a log collector prior to the attacker having the ability to modify the integrity of the system.

Event logs are not a silver bullet. However, event logs can provide a tremendous amount of host telemetry data that can aid in the detection of an advanced adversary. Very few organizations collect the right event logs from relevant devices, and even fewer organizations are able to action these event logs. This article is designed to showcase a proof of concept architecture for detecting indicators of compromise through event logs.

WHAT CAN WE DETECT?

Examples of advanced threat detection that this architecture can detect include, but are not limited to:

» Outbound web connections not originating from Internet Explorer/Firefox/Chrome

» The presence of a hash across other machines on the network

» Lateral movement (file shares, remote desktop) within and between subnets

For a detailed listing of what can be detected, read the section on “overview of what can be detected”

HOW IS THIS ARCHITECTURE DIFFERENT FROM TRADITIONAL SIEM’S OR OTHER SECURITY TOOLS?

Based on my experience, most SIEM’s are:

» Very expensive

» Have overwhelming alerts that are not actionable

» Due to excessive cost, have limited data inputs and a fractional view of the environment

This architecture provides:

» Scalable, elastic, parallel processing architecture

» Open source (free), not limited by events-per-second licensing

» Dynamic collection of indicators that can be turned off/on based on threat conditions

» Targeted, hyper-actionable indicators

» Additional host telemetry data for networks, running processes/binaries

ARCHITECTURE

Three design principles were taken into consideration during the development of this architecture:

1. Most Security Information and Event Management (SIEM) analysts are already overwhelmed with a mountain of non-actionable logs. Log analysis architectures strive to collect relevant/actionable information that can be used as an indicator of compromise.

2. Most SIEM’s are at capacity from a storage and bandwidth perspective; it is not be feasible for an organization to turn on client side logging and forward all logs without a significant infrastructure upgrade. This architecture is designed for quick, actionable and scalable deployments using open source (FREE) tools.

3. The log collection and analysis tools should be easy to use and require little training/develop to gain instant return on investment and innovation.

ARCHITECTURE OVERVIEW

DETECTING INDICATORS OF COMPROMISE THROUGH EVENT LOGS CONCEPTUAL ARCHITECTURE

WINDOWS EVENT FORWARDING (WEF)

WEF BENEFITS

» The WEF architecture forwards events in the native event log format. This is helpful because the event log format uses XML to cleanly structure data into different fields that will be helpful when querying in ElasticSearch. Alternatively, using the syslog protocol will flatten all data structure, requiring parsers to be built for each event type.

» WEF utilizes group policy, so all clients that join a network or change OUs will automatically begin to participate in the log forwarding architecture. This is helpful to ensure completeness of coverage from a logging perspective.

» WEF does not require an agent to be installed. The lack of an agent requirement is helpful as some organizations will be hesitant to install another agent on each box (e.g. event-to-sys) and ensuring completeness of agent deployment can be challenging as clients are constantly changing

» WEF encrypts all data between the log collector and the clients by default

› If using HTTP: WEF will use the Microsoft Negotiate security support provider (SSP) in workgroup environments or the Microsoft Kerberos SSP in domain environments

› If using HTTPS: WEF uses a Secure Sockets Layer (SSL) certificate which can be specified during setup

WEF SETUP

1. Stand up a dedicated Windows Server 2008 R2 or later server to be used as an event collector. A dedicated server is suggested to handle the event log load from each of the clients.

2. On the log collector server, run the following commands

a. winrm qc

b. wecutil qc

3. Open the event viewer and create new event subscriptions that the clients will use to conditionally forward important security events

a. Click “Create New” from the actions pane

b. **Change the destination log to “Forwarded Events”

c. Click the “source computer initiated” subscription option and add “YourDomain\Domain Computers” group

d. Click on configure events to collect and define the query parameters. Note that you can copy and paste the XML from some of the sample queries that I have already created. Reference appendix A.

FIGURE 1: BUILDING NEW SUBSCRIPTIONS IN EVENT VIEWER

WHAT DATA TO COLLECT

THE FOLLOWING TABLE PROVIDES A LIST OF ACTIONABLE EVENT LOG IDS TO COLLECT. THIS TABLE WAS DERIVED FROM THE NSA ARTICLE REFERENCED IN THE APPENDIX AND MODIFIED BASED ON OPERATIONAL EXPERIENCE. LEVERAGE THIS DATA TO CREATE EVENT LOG SUBSCRIPTIONS. APPENDIX A ALSO PROVIDES A SCRIPT WITH PRE-BUILT XML QUERIES TO CREATE EVENT SUBSCRIPTIONS.

CATEGORY EVENT IDApplication whitelisting AppLocker Block 8003, 8004Application whitelisting AppLocker Warning 8006, 8007Application whitelisting SRP Block 865, 866, 867, 868, 882Application crashes App Hang 1002Application crashes Appcrash 1001Application crashes EMET 2System or Service Failures Windows Service Fails or Crashes 7022, 7023, 7024, 7026, 7031,

7032, 7034Windows firewall Firewall Rule Add 2004Windows firewall Firewall Rule Change 2005Windows firewall Firewall Rules Deleted 2006, 2033Anti-forensics Event Log was Cleared 104Anti-forensics Audit Log was Cleared 1102Anti-forensics Event log service stopped 6005Software and Service Installation New Kernel Filter Driver 6

Kernel Driver Signing Detect an invalid image hash of a file

5038

Kernel Driver Signing Detected an invalid page hash of an image file

6281

Kernel Driver Signing Code integrity check 3001, 3002, 3003, 3004, 3010, 3023

Kernel Driver Signing Failed Kernel Driver Loading 219Windows Defender Detected Malware 1006Windows Defender Action on Malware failed 1008Persistence Scheduled task created 4698Powershell Use of Powershell 400New Process New process create, include

image path, hash, parent process GUID, etc.

1

New Process New Process Create with Command Line Options

4688

File Time/Data Change Change of a file create or last modified date

2

Remote Desktop Tracking use of RDP 21Network Connection Network connection src & dst,

requesting process3

4. Note that subscriptions can be disabled/enabled (See screenshot below). The ability to enable/disable subscriptions is a great way to provide flexibility in advanced threat identification. The majority of windows events that are identified in the “what data to collect” section are hyper actionable, low signal-to-noise ratio (meaning, if you are receiving events, there is likely an investigation that should take place). However, sysmon generates quite a bit of data and may not be sustainable across an entire enterprise network. In this case, sysmon could be selectively enabled based on suspected intrusion events or other triggers.

5. Configure group policy to define the log collector server, enable windows remote management, start the remote management service, allow Network Service to read security event logs, and create firewall rules to allow WEF traffic on client computers

a. Create a new group policy named “Windows Event Forwarding”

b. Define the log collector server: Expand computer configuration>policies>administrative templates>control panel>windows components>event forwarding

i. Edit the “Configure the server address, refresh interval and issuer certificate”

ii. Select “enabled”

iii. Select “show”

iv. Enter “Server=YourCollecterHostName.FQDNofYourDomain”

FIGURE 2: EXAMPLE SUBSCRIPTIONS; SCREENSHOT SHOW SYSMON SUBSCRIPTION ENABLED

c. Configure windows remote management: Expand computer configuration>policies>administrative templates>control panel>windows components>windows remote management(WinRM)

i. Select “Enabled”

ii. Under IPV4 enter “*”

iii. Under IPV6 enter “*”

FIGURE 3: GROUP POLICY CONFIGURATION FOR CLIENTS THAT SPECIFIES THE COLLECTOR SERVER

FIGURE 5: GROUP POLICY SETTING THAT CONFIGURES THE CLIENT WINRM SETTINGS (NSA, 2013)

d. Start the WEF service: Expand Computer configuration > policies> Windows Settings > Security Settings >System Services > Windows Remote Management. See image below (NSA, 2013)

i. Double click on “Windows Remote Management

ii. Check Define this setting

iii. Change the startup type to automatic

e. Allow Network Service to read security logs (without this step, WEF will not be able to pull any data from the security logs): Computer configuration > policies > windows settings >security settings > restricted

i. Right click restricted groups

ii. Select add group

iii. Type Event Log Readers and select check names, then OK

FIGURE 6: ENABLING WINDOWS REMOTE MANAGEMENT

FIGURE 7: SETTING SERVICE STARTUP TYPE

iv. Right click event log readers and select properties

v. Click add and type Network Service

vi. Select check names, then ok

f. Add a firewall rule to allow WEF traffic: Computer configuration > Policies > Windows Settings >Security Settings > Windows Firewall with Advanced… >Inbound rules

i. Allow TCP 5985

g. Link the “Windows Event Forwarding” group policy to applicable Active Directory Organizational Units (OUs)

SYSMON

SYSMON BENEFITS

Per Microsoft, Sysmon provides the following capabilities:

» Logs process creation with full command line for both current and parent processes.

» Records the hash of process image files using SHA1 (the default), MD5 or SHA256.

» Includes a process GUID in process create events to allow for correlation of events even when Windows reuses process IDs.

» Optionally logs network connections, including each connection’s source process, IP addresses, port numbers, hostnames and port names.

» Detects changes in file creation time to understand when a file was really created. Modification of file create timestamps is a technique commonly used by malware to cover its tracks.

» Generates events from early in the boot process to capture activity made by even sophisticated kernel-mode malware.”

SYSMON SETUP (WITHOUT GPO)

Sysmon.exe –i –h SHA256 –n

FIGURE 8: SYSMON SETUP

SYSMON SETUP (WITH GPO)

LOGSTASH

During the setup of logstash 1.4.2, two problems occurred. First, logstash would not pull data from the Forwarded Event log file. Although the subscriptions can be configured to save data to another file this complicated the setup and analysis. Second, logstash had a memory leak and would stop running under a heavy load. To solve these issues, the architecture was tweaked slightly. NX Log CE was installed on the Windows Server event log collector and used to forward data to a logstash instance running on a Ubuntu instance. The logstash instance would listen for connections data from NX log and forward to the ElasticSearch cluster.

INSTALL NX LOG

Download and install the binary from the link in the references. Create a configuration file and place in the configuration directory, similar to the screenshot below.

LOGSTASH CONFIGURATION FILE SETUP

» Install java

› apt-get install default-jre

› update-alternatives --config java

– Capture location (e.g. /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java)

FIGURE 9: SAMPLE COPY OF THE NX LOG CONFIGURATION

› Vi /etc/environment

– Add new line

– JAVA_HOME=”/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java”

– source /etc/environment

– echo $JAVA_HOME

» Create a configuration file similar to the screenshot below and place in the /usr/share/logstash/bin/ folder.

RUNNING LOGSTASH

C:\logstash\bin\logstash.bat agent –f logstash.conf

ELASTICSEARCH

ELASTICSEARCH BENEFITS

» Scalable horizontal compute and storage (similar to Hadoop)

» ElasticSearch can accommodate node failure and automatically re-distribute shards without data loss

» ElasticSearch leverages parallel processing to execute queries on massive data sets very quickly

» Open source (free)

» Great for long tail analysis and querying

ELASTICSEARCH SETUP

ElasticSearch will run out of the box with one command. However, it is highly encouraged that you become familiar with the different options within the elasticsearch/config/elasticsearch.yml configuration file. Note that the additional configuration items are not covered within the scope of this document.

» Prior to starting ElasticSearch, open the elasticsearch.yml file

› Remove the “#” on the line that contains “#cluster.name: elasticsearch”

› Change the “elasticsearch” to a different name that describes your cluster

› Note that new ElasticSearch instances will auto-join your cluster if they have the same cluster name

› Note that you may want to setup iptables or put the ElasticSearch cluster behind a firewall. ElasticSearch does not authenticate any connections by default. ElasticSearch listens on port 9200 for RESTful HTTP connections.

FIGURE 10: ELASTICSEARCH CONSUMING MASSIVE AMOUNTS OF DATA

FIGURE 11

» Install java

› apt-get install default-jre

› update-alternatives --config java

– Capture location (e.g. /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java)

› Vi /etc/environment

– Add new line

– JAVA_HOME=”/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java”

– source /etc/environment

– echo $JAVA_HOME

» Start ElasticSearch

› ./elasticsearch

KIBANA

BENEFITS OF KIBANA

» Kibana provides a beautiful interface that allows you to easily query ElasticSearch

» Using Kibana, you can quickly filter to perform long tail analysis and anomaly detection

KIBANA SETUP

» Edit the config.js file and point Kibana to the ElasticSearch cluster

› Cange the IP address to the host IP address on line “elastic search: http://127.0.0.1:9200”

FIGURE 12: STARTING ELASTICSEARCH ON UBUNTU

» vi /etc/init/kibana.conf

› # auto start kibana

› start on filesystem

› exec /usr/share/kibana/bin/kibana

» chmod -R 755 /usr/share/kibana/

» ln -s /etc/init/kibana.conf /etc/init.d/kibanae

» service kibana start

» Netstat -l | grep 5601

REFERENCES

Spotting the Adversary Through Windows Event Log Monitoring

» https://www.nsa.gov/ia/_files/app/spotting_the_adversary_with_windows_event_log_monitoring.pdf

FIGURE 13: KIBANA CONFIG.JS CONFIGURATION FILE

WEF

» Setting up a source initiated subscription http://msdn.microsoft.com/en-us/library/windows/desktop/bb870973(v=vs.85).aspx

» Quick and dirty large scale eventing for windows http://blogs.technet.com/b/wincat/archive/2008/08/11/quick-and-dirty-large-scale-eventing-for-windows.aspx

Logstash

» Getting started http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash

» Eventlog input http://logstash.net/docs/1.4.2/inputs/eventlog

NX Log CE

» http://nxlog-ce.sourceforge.net/

Sysmon

» Sysmon download http://technet.microsoft.com/en-us/sysinternals/dn798348

APPENDIX A – WEF SUBSCRIPTIONS

WEF Subscriptions can be created in event viewer then exported for backup or for quickly importing into other machines.

» export a subscription: wecutil gs “Powershell Invocation” /f:xml >Powershell.xml

» Note that “Powershell Invocation” is the name of the subscription in event viewer

» import a subscription: wecutil cs <InsertSubscriptionName>.xml

SAMPLE APPLICATION CRASH SUBSCRIPTION<?xml version=”1.0” encoding=”UTF-8”?><Subscription xmlns=”http://schemas.microsoft.com/2006/03/windows/events/subscription”> <SubscriptionId>Application Crashes</SubscriptionId> <SubscriptionType>SourceInitiated</SubscriptionType> <Description></Description> <Enabled>true</Enabled> <Uri>http://schemas.microsoft.com/wbem/wsman/1/windows/EventLog</Uri> <ConfigurationMode>Custom</ConfigurationMode> <Delivery Mode=”Push”> <Batching> <MaxLatencyTime>60</MaxLatencyTime> </Batching> <PushSettings> <Heartbeat Interval=”120”/>

</PushSettings> </Delivery> <Query> <![CDATA[<QueryList><Query Id=”0”><Select Path=”Application”>*[System[(EventID=1000 or EventID=1001 or EventID=1002)]]</Select><Select Path=”System”>*[System[(EventID=1000 or EventID=1001 or EventID=1002)]]</Select></Query></QueryList> ]]> </Query> <ReadExistingEvents>false</ReadExistingEvents> <TransportName>HTTP</TransportName> <ContentFormat>RenderedText</ContentFormat> <Locale Language=”en-US”/> <LogFile>ForwardedEvents</LogFile> <PublisherName>Microsoft-Windows-EventCollector</PublisherName> <AllowedSourceNonDomainComputers> <AllowedIssuerCAList> </AllowedIssuerCAList> </AllowedSourceNonDomainComputers> <AllowedSourceDomainComputers>O:NSG:BAD:P(A;;GA;;;DC)S:</AllowedSourceDomainComputers></Subscription>

Sample New Process Create Subscription<?xml version=”1.0” encoding=”UTF-8”?><Subscription xmlns=”http://schemas.microsoft.com/2006/03/windows/events/subscription”> <SubscriptionId>New Process Create With Command Line Options</SubscriptionId> <SubscriptionType>SourceInitiated</SubscriptionType> <Description>Native Eventlog Source, Moderate Traffic</Description> <Enabled>true</Enabled> <Uri>http://schemas.microsoft.com/wbem/wsman/1/windows/EventLog</Uri> <ConfigurationMode>Custom</ConfigurationMode> <Delivery Mode=”Push”> <Batching> <MaxLatencyTime>60</MaxLatencyTime> </Batching> <PushSettings> <Heartbeat Interval=”120”/> </PushSettings> </Delivery> <Query> <![CDATA[<QueryList><Query Id=”0” Path=”Security”><Select Path=”Security”>*[System[(EventID=4688)]] and *[EventData[Data[@Name=’NewProcessName’] !=’C:\Windows\System32\VSSVC.exe’]] and *[EventData[Data[@Name=’NewProcessName’] !=’C:\Windows\System32\SearchFilterHost.exe’]] and *[EventData[Data[@Name=’NewProcessName’] !=’C:\Windows\System32\SearchProtocolHost.exe’]] and *[EventData[Data[@Name=’NewProcessName’] !=’C:\Windows\System32\Defrag.exe’]] and *[EventData[Data[@Name=’CommandLine’] !=’taskhost.exe network’]] and *[EventData[Data[@Name=’CommandLine’] !=’taskhost.exe ‘]] and *[EventData[Data[@Name=’CommandLine’] !=’taskhostex.exe Regular’]]</Select></Query></QueryList> ]]> </Query>

<ReadExistingEvents>true</ReadExistingEvents> <TransportName>HTTP</TransportName> <ContentFormat>RenderedText</ContentFormat> <Locale Language=”en-US”/> <LogFile>ForwardedEvents</LogFile> <PublisherName>Microsoft-Windows-EventCollector</PublisherName> <AllowedSourceNonDomainComputers></AllowedSourceNonDomainComputers> <AllowedSourceDomainComputers>O:NSG:BAD:P(A;;GA;;;DC)S:</AllowedSourceDomainComputers></Subscription>

For blog inquiries contact blog@root9b.com

For general inquiries contact info@root9b.com