Side Tables Info

7/26/2019 Side Tables Info

1/21

2009 ArcSight, Inc. All rights reserved.

ArcSight and the ArcSight logo are trademarks of ArcSight, Inc. All other product and company names may be trademarks or registered trademarks of their respective owners.

1

Shedding Light on Side Tables

Stefan Zier

Platform TeamSeptember 2009


2/21

Agenda

Side Tables

What are they?

What data is stored?

How does the Side Table cache work?

What data causes problems; how do you fix them?

www.arcsight.com 2009 ArcSight Confidential 2


3/21



What are Side Tables?



4/21

What Are Side Tables?

ArcSight ESM uses Oracle to store all events

Event schema was expanded with ArcSight 3.0Side Tables were introduced in ArcSight 3.0

Normalization of schema to get big benefits Avoid repetition and save disk space

Make event insertion and queries much faster

Allow us to have a lot more fields in events




5/21

How Does Normalization Work?

Side Tables store parts of an event Devices

Agents (connectors) Source/target IP Geo information

Sets of event categories

Sets of additional data labels

References to annotations

All of these parts have the following properties Values repeated in many events

Limited number of different values

At most tens of thousands of each

Normalization avoids repeating values




6/21

Normalization

EVENT1 NAME SRC IP TARGET IP






AGENT ID AGENT NAME AGENT LOCATION






Unique Fields Repeated Fields

Repetitive information is saved onceand referenced to its original source!




7/21


8/21



Side Table Caches



9/21

Side Table Cache

Connectors and consoles deal with denormalizedeventsdo not keep up-to-date with Side Tables

Manager needs to map denormalized events to thenormalized schema in the DB

Read event: Find all values for columns that are in Side Tables

Insert event: Find the ID that matches entry with values

Each operation requires a database query

6-8 database operations per event

Side Table cache is an in-memory cache

Cuts number of operations down per event to 1

Side Table Cache



10/21

Side Table Cache Sizing

Side Table Cache attempts to keep 100%of entries in memory

Memory is limited, so Side Table caches need a limit

Typically, the size is larger than the number of entries

Size can be reconfigured

Defaults 50,000 Geo Descriptors

2,500 Connectors

50,000 Devices 10,000 Categories

2,000 Labels

20,000 Resource References

Side Table Cache



11/21

Side Table Cache Specifics

Two cache modes Completeall entries fit into memory

Partialmore entries than memory space

Event insert

Check whether entry is in cache (memory operation) In partial mode, try to find entry in database (database operation)

If entry not found, create entry (database operation)

Event read Check cache for entry (memory operation)

In partial mode, try to find entry in database (database operation)

Side Table Cache



12/21



Problematic Data



13/21

What Data Causes Side Table Issues?

Normalization and caching assumelimited number of entries

If number is not limited Cache will only hold a fraction of entries

For each cache missed, additional database operationsare needed when retrieving and inserting events

Additional database queries multiply the load on the database

Insertions and reads (channels, reports, trends) most likely slow down

Connectors may start to cache

Events take much more space than needed

Problematic Data



14/21

How Can This Happen?

Connectors put values into events that are not very repetitive, forexample

Unique event IDs Timestamps

Checksums

Packet sizes

Strings that contain any of the above

Various reasons Event format changed on the device

Connector changed

FlexConnector

A single misbehaving connector can affect an entire ArcSightESM deployment

Problematic Data


P bl i D


15/21

How Can This Happen?

Rules can create additional side table entries

M1 can create a new side table entry

Aggregation on subset of fields in the side table

Each different combination of values creates new entry

Rule of thumb: aggregate on ALL or NONEof the side table fields

Problematic Data


P bl i D


16/21

Countermeasures

ArcSight ESM 4.0 (and earlier versions) implementcountermeasures to prevent the issue

Connectors monitor for non-repetitive values in Side Table fields Manager monitors rate of new Side Table entries

Notifications are sent when an issue is identified

The manager pauses the misbehaving connectoruntil an operator fixes the issue

Problematic Data



17/21



Working With Side Tables


W ki ith Sid T bl


18/21

Different Notifications

Notifications logged in server.log and as internal events

Side Table cache hit rate low messageearly tell-talesign of a runaway Side Table

Agent has been paused messagelate tell-tale sign ofa runaway Side Table

Working with Side Tables


W ki ith Sid T bl


19/21

Cache Hit Rate Low Message

Carefully increase the limit in server.properties

Increasing it to a very large number may run yourmanager out of memory

Restart manager



If you know that you have an extraordinary number ofdevices or other objects (larger than the defaults)

Working ith Side Tables


20/21

Fixing a Runaway Side Table Problem

Identify the problem Contact support to get the SideTableStats.sql script

Run it, and support will tell you which column in whichtable shows too many distinct values

Run a channel filtering on the connector that was reportedin the message

Look at the values in the column

If you recognize what the data is, this will most likely be your light bulbmoment

Once you have identified it Adjust your FlexConnector

Or work with support to identify a what can be doneto fix the issue




21/21

Summary

Side Tables allow for efficient storage of large eventvolumes in Oracle

Side Table caches accelerate normalization on themanager by orders of magnitude

Misconfigured connectors can cause issues

But both connectors and manager have efficientcountermeasures in place to prevent problems

There are well-established procedures fortroubleshooting Side Table issues

Documents

Side Tables Info