SEMplicity: The Evolution of Security Log Management 1

SIEM is Dead? The Evolution of Security Log Management

SEMplicity: The Evolution of Security Log Management 2

Is SIEM Dead? Enterprise security practitioners with a few (too) many years behind us heaved a weary sigh when the inevitable “SIEM is Dead!” cries began issuing from the tech commentariat. Much as everyone loves a better mousetrap, the unexciting truth is that widely adopted, large-scale security products often have an astounding resilience that would make any resurrected character from Game of Thrones jealous. A more likely outcome is that SIEM will take its well-earned seat among the venerable legacy security technologies that each play a role in the deeply layered defense necessary to protect enterprises today.

(Full disclosure: in the early 1990s, I wrote a RACF-based mainframe security management product which is still in use today and may well outlast me.)

This is not to say that SIEM doesn’t have any problems. It does, indeed. In many ways, these are a result of its own success. So, let’s take a quick overview of how SIEM was born and what’s wrong with it today. Then, we can discuss how to retrofit it for tomorrow.

The ever-expanding threat landscape and proliferation of log sources have combined to force SIEM (Security Information and Event Management) to re-evaluate its foundation: log management. Only the most advanced log collection and storage technologies can handle today’s massive log flows while enabling tomorrow’s detection mechanisms.

By George A. Boitano, President, SEMplicity, Inc.

Defining charateristics of SIEM

Before we start, we need to define SIEM. This exercise can be fraught, of course, in a market with so many competing commercial visions. For our purposes, I’ll define SIEM in the simplest way I can, as a technology stack with the following characteristics:

Figure 1 Common characteristics of security information and event management (SIEM)

W H I T E PA P E R

Of this list, the first two items — collecting and storing logs — necessarily form the foundation of the third and fourth, namely alerts and correlation. By the early years of the new millennium, compliance requirements such as PCI, NERC and SOX drove companies to purchase Security Information Management (SIM) systems to collect and store log records at evidentiary standards, thereby implementing #1 and #2 from our list in Figure 1. Early SIM products were designed to run reports on log data, allowing auditors to certify that a company was compliant with a mandated standard.

Nothing helps product sales like a mandated compliance requirement! Most enterprises either purchased on-prem SIM systems or hired a managed service provider (MSP) to collect, store and report upon their logs. As we’ll see, without wide adoption of this log management layer, SIEM as we know it today would be impossible.

1 Collects log records from all kinds of devices throughout

the organization,

including edge protection

devices but also critical

servers, databases and

applications;

2 Stores these records safely and allows auditors

and other personnel

to search them;

3 Occasionally raises alerts when a single record

appears to indicate

suspicious activity;

4 Groups together and raises alerts when several records,

taken together, indicate

suspicious activity —

we call this real-time

correlation.

SEMplicity: The Evolution of Security Log Management 3

Detecting breaches before intruders attack: from SIM to SEM

Having installed SIM, companies and service providers then asked the next logical question: Can we detect security breaches using these log records, preferably before the intruders accomplish their goals? From this simple request, Security Event Management (SEM) was born, and SEM products were quickly layered on top of SIM products. They scanned log records as they were gathered, looking for signatures indicating a problem, such as a US user logging on from Russia or a large amount of data being moved outside the network. The more sophisticated among the early SEM offerings persisted log records in memory or in temporary storage as they were ingested. In this way, SEM systems could be used to look for suspicious patterns, such as everyone’s favorite: multiple failed logon attempts for the same user or machine.

The happy union of SIM and SEM

Eventually, SEM and SIM got married, and the happy result was SIEM (Security Information and Event Management). SIEM combined log collection, storage, reporting, searching, single event detection and event correlation. Though real-time automated correlation was often not explicitly mandated by compliance standards, enterprises nonetheless widely adopted SIEM. While soup-to-nuts SIEM products were sometimes hard to install and configure, they largely delivered on SIEM’s promise to support security operations by detecting at least some real-time threats while fully satisfying log management compliance requirements.

As time went on, additional functionality was layered onto SIEM:

• Event enrichment incorporating outside data;

• Simple baselines for detecting anomalies;

• Richer reporting, supporting more flexible searching; and

• More sophisticated correlation rules, with GUI-based queries and reports.

Once SIEM was a proven technology, everybody wanted in! Not just firewalls, but edge routers, proxies, servers, applications, databases, endpoint protection, and so many more technologies… all of them started sending their logs to SIEM. And among all these old and new log source types, the log volumes kept increasing as enterprises expanded and merged. The regulators and auditors played their parts as well, continually expanding the scope of the logs we now needed to collect. Today, log storage and management is widely recognized as one of the original big-data use cases, and one of the most challenging.

As so often happens when an enterprise technology is successful, new features and new uses are added until the products become too unwieldy to install, run and maintain. Things bend until they ultimately break.

Alas, the increased log volumes and new use cases enabled by SIEM place a terrible strain on its supporting technology stack. Vendors have struggled to incorporate modern big-data storage technology required by ever-increasing log volumes. Simultaneously, they are asked by savvy (and sometimes war-scarred) customers to deliver innovative use cases at a rate that could keep up with the latest additions to the threat landscape — as well as the latest methods to combat them.

Fulfilling these compelling and sometimes competing requirements has proven difficult for just about every SIEM vendor. The commercial imperative to lock customers into their technology stack has not helped. Given the resulting customer dissatisfaction with SIEM products, is it any wonder that the cries of “SIEM is Dead!” are heard in certain security-focused blogs and analyst reports?

And so we come to the main proposition of this paper: we believe SIEM is not dead… but it needs a divorce.

SIEM is not dead… but it needs a divorce.

Once SIEM was a proven technology, everybody wanted in! Not just firewalls, but edge routers, proxies, servers, applications, databases, endpoint protection, and so many more technologies… all of them started sending their logs to SIEM.

SEMplicity: The Evolution of Security Log Management 4

The foundational issues of log storage and management

Before we can enable tomorrow’s exciting use cases, we need to ensure that we still have a robust, evolving solution to the foundational issues of log collection, enrichment, storage and search. Without a modern technology stack capable of ingesting ever more log data with ease and speed, advanced big-data log use cases will continue to be limited by capacity and proprietary lock-in. Only by removing such resource constraints will enterprises be able to layer the latest/greatest event detection and response methodologies on top of a solid log management foundation, regardless of vendor. Decoupling log management from event detection greatly enhances flexibility and enables nearly limitless innovation.

So what does a rock-solid log management foundation look like to a CISO, SOC manager or hunt team? To my thinking, it should be constructed following these ten commandments

Decoupling log management from event detection greatly enhances flexibility and enables nearly limitless innovation.

1. Thou shalt collect logs from every significant log source

in an enterprise, both on-premises

and in the cloud;

2. Thou shalt accommodate huge log flows, and thus scale easily and

efficiently;

3. Thou shalt manage the pipeline of logs such that none are lost, preferably using

some form of store and forward

message bus architecture that

preserves records until they reach

their destination, regardless of ebbs

and spikes in volume;

4. Thou shalt parse unstructured logs,

extracting fields into a common

schema;

5. If necessary, thou shalt re-parse previously parsed logs

into a common schema.

6. Thou shalt enrich logs just

prior to ingestion with a variety of data:

Threat intelligence, including open

source, proprietary and derived on-

prem; Vulnerability data; Identity data

on users; Network data, specifying the

location and criticality of devices; Data

on previously detected events;

7. Thou shalt ingest enriched logs quickly, efficiently and flexibly;

8. Thou shalt store such logs in a de-normalized format to assist quick searching, albeit at the

possible expense of increased disk

space;

9. Thou shalt store all logs as read-only, and store certain logs to evidentiary

standards using NIST protocols; and

10. Thou shalt enable very fast searching of stored logs using simple and complex

filters. Fast search is what enables all

advanced big data use cases.

Figure 2 The ten commandments of modern log management

SEMplicity: The Evolution of Security Log Management 5

Overhauled log management: wider range of sources, scalable log volume,

intelligent compliance reporting, flexible

parsing, fast ingestion, non-volatile storage,

blazing fast search

Enhanced human factors, dashboards and alerts

Advanced event detection, real-time correlation

flexible security analytics tools (for hunt teams)

Value-added SIEM tools and use cases

New use cases and threat mitigation techniques Long-range

correlation

Accelerated time to market

Open integration

Big data ingestion

Evidentiary storage standards

Machine learning for anomaly detection

A modern SIEM solution overhauls log management to form a robust, scalable foundation that is de-coupled from an evolving set of flexible tools and use cases provided by multiple vendors. But like any other established technology — especially those hoping to remain relevant in the dynamic and increasingly high-stakes security landscape — SIEM must evolve to survive. (See Figure 3)

Yesteryear's

SIEM product

(single-vendor)

Today’s

SIEM architecture

(multi-vendor)

Legacy SEM: alerts, compliance

reporting

Legacy SIM: log collection

storage

Figure 3 An evolved, modern SIEM solution

SEMplicity: The Evolution of Security Log Management 6

The way forward for SIEM

Once this foundation is re-established, what is left for SIEM?

Well, of course, real-time correlation will remain important. Why discard a mechanism that has a proven track record identifying suspicious activities? Also, events detected by any mechanism still need a central area for processing by the SOC, and with well-conceived alerts incorporating the latest internal and external threat intelligence, SIEM can easily serve this purpose.

With a rebuilt log management infrastructure that is flexible, expandable, fast and above all interoperable, we can layer a wide variety of mix-and-match detection methodologies that can take us well beyond the capabilities of traditional SIEM. Some of these newly enabled methodologies include:

• A wide array of security analytics tools for hunt-team members;

• Advanced visualizations for SOC managers and CISOs;

• Anomaly detection using machine learning, based on complex user, network and behavior modeling, capable of detecting deviations from usual behavior and metrics;

• Long-range correlation of events spread out in time outside the real-time correlation window;

• Retroactive application of the latest threat intelligence — both proprietary, open source and site-derived — to older events;

• Other applications that nobody has thought of yet, possibly from suppliers who do not yet even exist!

Alongside other forms of security solutions, SIEM will continue to play a crucial role in protecting enterprises from a constantly changing threat environment. But SIEM must evolve to survive. By decoupling and strengthening a robust log management layer from a wide range of SOC applications from multiple vendors, today’s enterprises can take advantage of various vendors’ best areas of expertise, picking and choosing the best set of solutions to satisfy their current and long-range IT security needs.

George A. Boitano is president and founder of SEMplicity, Inc. George is a developer, inventor and entrepreneur with more than 25 years of experience in data security for large enterprises. George holds a bachelor’s degree in physics from Harvard University.

SEMplicity is a Managed Security Service Provider (MSSP) and consulting firm that specializes in large enterprise log management, searching, correlation and analytics. We are an Elastic-licensed Managed Service Provider (MSP). Since 2010, SEMplicity has architected and implemented SIEM deployments at dozens of the world’s largest corporations, including many Fortune 500 companies. Our security engineers are recognized experts in massive deployments of secure log storage and fast log searching using best-in-class proprietary and open-source tools.

©2018 SEMplicity, Inc. All other trademarks or trade names are properties of their respective owners. All rights reserved.

semplicityinc.com