ArcSight Connector Health Check

8/10/2019 ArcSight Connector Health Check

1/18


2/18

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2

Agenda

HP ArcSight Connector health check

What is a health check?

Health check steps by ArcSightcomponent

Connectors

Connector Appliances

Q & A


3/18

Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

Health Check overview


4/18


What is a health check?

Purpose

The purpose of performing a health check is to identify and remove performbottlenecks to enable top performance of the HP ArcSight implementation

issues can result in major performance degradations over time impacting s

availability and user satisfaction. Performing regular health checks will ide

issues allowing them to be remediated quickly and ensure continued top p

of the HP ArcSight implementation.

In a nutshell

A Health Check consists of common administrative tasks to verify the ArcS

solution is configured and performing optimally.


5/18


Health Check stepsby ArcSight component

Note: Its impossible to cover every scenario in this presentation,so only the common checks will be discussed.


6/18


Health check steps by ArcSight component

1Lo

CPU

Che

Sea

Cus

Che

Rec

Sto

Inde

Con

Sch

Eve

Bac

Log

Eve

Net

Onl

(On

Log

ESM Databaseand storage

DBCheck and Oracle RDA

Database Performance

Statistics Dashboard Check

Partition Check (Oracle)

Trend Jobs Check

Hardware and Operating

System Check CPU and Memory Utilization

Check

Oracle version and patch level

check

Oracle alert log check

Oracle memory parameters

check

ESM Database Storage Check

ESM Manager

Event Throughput Dashboard Check

Current Event Sources Dashboard Check

Hardware and Operating System Check

CPU and Memory Utilization Check

ESM Manager JVM (memory) Utilization

Check

Data Monitor Utilization Check

Active List/Session List Utilization Check

Rules Engine Check

Event Persistence (insertion) Performance

Check

Error Check

Scheduled Task Check

server.properties Check

Agent and Console Threads Check

Connectorappliances

Version Check

CPU and Memory Check

Network Settings

Check

Configuration Backup

Check

Connectors

Up/Down Check

(Connector or Container)

Version Check

Connector Event RateCheck (by EPS)

Cache Check

Logs Check

Configuration Check

Connectors

Tip: Check each ArcSight Component by the order of the Event Flow

Its just simple plumbing!!!


7/18


Connectors

Connector (or Container) Up/Down Check

Connector Version Check

Are there any Connectors running a version older than ~1 year?

A minimum version of 4.8.1 is required to leverage the ESM v5.2 schema.


8/18


9/18


Connectors (cont.)

Connector Logs Check ../current/logs/agent.out.wrapper.log

Java Heap Memory Utilization

Memory utilization

Frequency of Full GCs

Memory in Red Zone alerts

Unexpected Connector restarts

Connectivity errors

End Devices

ArcSight Destinations

../current/logs/agent.log

Parsing errors

DOSProtector

Chronic WARN and ERROR messages


10/18


Connectors

Connector Logs Check (cont.) Use Connector LogFu to graph the event

flow and memory utilization ../current/bin/arcsight agent logfu a


11/18


Connectors (cont.)

Connector Configuration

Check Destination Settings

Are there more than 2 Destinations on each

Connector?

Too many Destinations can negatively

impact performance of a Connector.

Common problems found:

Networks and CustomerURI are not applied

on every Connector

Fields-based Aggregation is not properlyapplied (by Connector Type)

No tuning (Filter Out) applied on high EPS

Connectors

Settings are not the same on every

Destination (ESM, Logger, etc.)


12/18


Connectors (cont.)

Connector Configuration Check (cont.)

Only check the following on problematic Connectors discovered in previous checks

../current/user/agent/agent.properties

Optimal settings are different for each Connector type

High EPS Connectors (>1200 EPS) such as Syslog, WUC, CheckPoint, and Blue Coat can be tweaked quite a bit here

../current/user/agent/agent.wrapper.conf

Only increase the Java Heap size if memory issues were found in agent.out.wrapper.log

Default Java Heap is 256MB

Maximum configurable Java Heap is 1024MB (1 GB)

Reminder: If you have 50+ Connectors in your environment, try to stay focused on problematic Connectors!


13/18


Health check steps by ArcSight component

1Lo

CPU

Che

Sea

Cus

Che

Rec

Sto

Inde

Con

Sch

Eve

Bac

Log

Eve

Net

Onl

(On

Log

ESM Databaseand storage

DBCheck and Oracle RDA

Database Performance

Statistics Dashboard Check

Partition Check (Oracle)

Trend Jobs Check

Hardware and Operating

System Check

CPU and Memory Utilization

Check

Oracle version and patch level

check

Oracle alert log check

Oracle memory parameters

check

ESM Database Storage Check

ESM Manager

Event Throughput Dashboard Check

Current Event Sources Dashboard Check

Hardware and Operating System Check

CPU and Memory Utilization Check

ESM Manager JVM (memory) Utilization

Check

Data Monitor Utilization Check

Active List/Session List Utilization Check

Rules Engine Check

Event Persistence (insertion) Performance

Check

Error Check

Scheduled Task Check

server.properties Check

Agent and Console Threads Check

Connectorappliances

Version Check

CPU and Memory Check

Network Settings

Check

Configuration Backup

Check

Connectors

Up/Down Check

(Connector or Container)

Version Check

Connector Event RateCheck (by EPS)

Cache Check

Logs Check

Configuration Check

Connector AppliancesTip: Check each ArcSight Component by the order of the Event Flow


14/18


Connector appliances

Connector appliance version check

Is the version outdated?

Are there any known issues with the current version?

Connector appliance CPU and memory

check

Review the following for excessive utilization:

CPU utilization is continuously above 70-80% in Logger Dashboard

EPS In is continuously above 5,000 EPS (a single C5400 is designed

for 5,000 max EPS)

Check the Connector Appliances Monitor Dashboards for unusual

peaks or drops

Check the System Process Status section of the Connector Appliance

If possible, SSH to the Connector Appliance and run commands such

as top, df, ifconfig, etc. to perform a deeper dive at the OS level

Connector appliance network

check

Common problems to check:

Incorrect duplex settings on the network interfa

DNS or NTP not configured properly

Connector appliance configur

check

The daily Configuration Backup job should

all Connector Appliances.


15/18


Additional resources


16/18


My favorite resources for keeping ArcSight he

1. Any HP Protect presentation on ArcSight best practices or troubleshooti

https://protect724.arcsight.com2. KB Articles on the HP Support Site

3. Solutions listed in previous Support Tickets

4. HP ArcSight University

5. HP ArcSight product documentation
https://protect724.arcsight.com/https://protect724.arcsight.com/


17/18


Thank you


18/18

Documents

ArcSight Connector Health Check