Upload
others
View
46
Download
1
Embed Size (px)
Citation preview
Created and Presented by
Balahasan V. | SIEM SME Accenture
AESA, AEIA
ARCSIGHT FLEXCONNECTOR TRAINING LEVEL 02
Introduction
Brief about Flex.
Planning your Flex.
Types of Flex Connector and related Parameters.
Sample Examples.
Detailed Configuration File Structure Topics.
Basic Flex Concepts
Declaring Regex and Configuring FlexAgentWizard /Regex Wizard.
Token Declaration
Event Mapping
Severity Mapping
Understanding the Regex Usage
Few Examples on Different Flex Types
Little Advanced Concepts of Flex
Submessages
Conditional Mapping.
Extra Processor
Multi-Line Regex
Parser Overrides.
Extra Mapping Files.
Merge operations.
Custom Categorizations.
Key Value Parsers
Creating Map Files.
Defining deviceEventClassId.
Additional Data Mapping
CounterACT Connector and REST API
Flex Active List Import and Flex Asset Import.
Few Sample Agent Property file Important Configurations
Topics Covered
Custom Defined Smart Connector.
Collects and normalizes data from unsupported devices.
Fully functioning agent, including categorization, zoning, aggregation, batching and priority calculation features.
Installed through the ArcSight smart connector installer.
Run smart connector installer
Select desired Flex connector type
What Is An ArcSight FlexConnector
Flex Connector Log-file
FlexConnecor Regex log-file
Flex Connector Regex Folder log file
Flex Connector Syslog
Flex Connector Time-based Database
Flex Connector ID-based Database
FlexConnector Multi-Database
FlexConnector SNMP
FlexConnector XML Folder Log file
FlexConnector Scanner for Text, XML, Database
Rest API
CounterACT Connectors
Types of FlexConnector
Configuration file holds the flex parser which will be used to parse the raw logs
There are 4 steps to creating a FlexConnector configuration file
Define a parsing mechanism
Identify and name tokens (Tokenization)
Map tokens to ArcSight schema (Normalization)
Map device severity to ArcSight severity
Advanced Configuration Properties
Flex Connector Configuration File
Base Directory : /current/user/agent/flexagent/
Log-file : < vendor >.sdkfilereader.properties
Regex log-file & Folder : < vendor >.sdkrfilereader.properties
Time based DB & Multi-DB : < vendor >.sdktbdatabase.properties
ID Based DB : < vendor >.sdkibdatabase.properties
Syslog : syslog/< vendor >.subagent.sdkrfilereader.properties
SNMP : < vendor >/sdksnmp.#.snmptrap.rpoperties
XML Folder log file : < vendor >.xqueryparser.properties
Scanner Text/XML/DB :
< vendor >.< scanner/vulns/openports/uris >.sdkrfilereader.properties
< vendor >.< scanner/vulns/openports/uris >.xqueryparser.properties
< vendor >.sdkdatabase.properties
Rest API : < vendor >.jsonparser.properties
CounterACT : < file_name >.counteract.properties
Flex Configuration File Location (Important)
Flex Connector Installation
Log file Flexconnector For fixed format ,delimited log file (real time log collection)
Regex Log File For variable format log file(real time log collection)
Regex folder Follower To read logs in batch mode
Regex Multiple Folder Follower Read logs from multiple folder (Real time and Batch mode)
Time-based DB Flexconnector Read event info from tables based on timestamp value
ID based DB Flexconnector Read event info from tables based on ID value
Multiple DB Flexconnector Read logs from multiple database(Time based as well as ID based)
Selecting A Flex Connector Type
SNMP Connector Collect logs from snmp traps.
SYSLOG Connector Security events from syslog messages.
XML Connector Read logs from XML-based files in a folder.
Scanner Connector To import the scan results from a scanner device.
Rest API Provides a configurable method to collect security events when you use cloud-based applications such as Box, Salesforce, or Google Apps…
CounterACT User can then execute commands on third party devices from within ArcSight and send the output of those commands back to the console.(Allowing the third party device to be controllable from ArcSight Console itself Amazing Feature isn’t it)
Selecting A Flex Connector Type
Log file Example: 08/09/2050-11:33:00,1.1.1.1,52123,2.2.2.2,80,Invalid URL
08/09/2050-12:43:00,3.3.3.3,49123,2.2.2.2,80,Buffer Overflow
Regex Log File Sep 10 15:28:49 beach sshd[24939]: Failed password for rajiv from 192.168.10.27 port 33654 ssh2
Sep 10 15:28:51 beach PAM_unix[24948]: (ssh) session opened for user rajiv by (uid=525)
Time Based Database
Examples
ID Based Database
Syslog
My application: Intruder Detected from 1.1.1.1 to 2.2.2.2 High
Examples
Parsing Mechanism
Token declaration
Event mapping with ArcSight Schema
Severity Mapping
Submessages
Additional Data and Conditional Mapping.
Extra Processor
Multi-Line Regex
Parser Overrides.
Extra Mapping Files.
Merge operations.
Custom Categorizations.
Creating Map Files.
Creating Key Value Parsers.
Defining deviceEventClassId.
Configuring FlexAgentWizard and Regex Wizard.
Detailed Configuration File Structure
Ex: Sample Log
28/09/11 08:15:00 SRC=194.168.0.12 DST=195.172.0.12 SPT=4236
DPT=80
Declaring Regex:
Requires clear idea what u need to parse and which field u need to map with
ArcSight CEF Field.
Below example gives how the Message is broken and parsed with the corresponding regular expression.
•Date & Time : (\\d+\\/\\d+\\/\\d+)\\s+(\\d+\:\\d+\:\\d+)
•Src and Dst Ip : (\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})
•Src and Dst Port : (\\d+)
Overall :
Regex=(\\d+\\/\\d+\\/\\d+)\\s+(\\d+\:\\d+\:\\d+)\\s+SRC\=(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})\\s+DST\=(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})\\s+SPT\=(\\d+)\\s+DPT\=(\\d+)
Parsing Mechanism Declaring Regex
//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//d//d//d//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)
Ways of Configuring Regex And Testing(3 Examples)
Regex Testing with
Cutom Apps(ex: Notepad++)
Regex creation using
Flex Creation Wizard
(only delimited)
Regex creation using
FlexAgent Regex Tool
token. count Number of tokens present in each line of the file
token[x].name User defined name for the tokens
token[x].type Data type of the token
token[x].format Format of the token or Modified Type
Token Declaration
Integer
Date
IPAddress
IPv6Address
Long
MacAddress
RegexToken
String
Time
TimeStamp
Reference Snap: Token Types
Token Types
Mapping the parsed token to ArcSight Event fields(400+ field event schema )
Type of token must match the ArcSight Event field Type.
In addition to the tokens that are parsed from each input record, you can also configure built-in tokens for specific Flex Connectors.
For Example
token[0].name=Msg
token[0].type=String
token[1].name=MyIP
token[1].type=IPAddress
event.sourceAddress=MyIP
event.message=Msg
event.deviceCustomDate1=_SYSLOG_TIMESTAMP (* Built-in token )
Event Mapping
Severity is an important part of the Threat Level Formula as well as for usage in reports that make use of device / event Severity.
Assume Token1 values are 23,46,69,82,95
It can be some string values too or Both string and Integer ..
Assume Token1 values as Error, Warning, Informational , Critical, Notification
Example 1
event.deviceSeverity=Token1
severity.map.veryhigh.if.deviceSeverity=95
severity.map.high.if.deviceSeverity=82
severity.map.low.if.deviceSeverity=23
severity.map.medium.if.deviceSeverity=46,69
Example 2
event.deviceSeverity=Token1
severity.map.veryhigh.if.deviceSeverity=Critical
severity.map.high.if.deviceSeverity=Error
severity.map.low.if.deviceSeverity=Informational
severity.map.medium.if.deviceSeverity=Warning, Notification
Severity Mapping
do.unparsed.events=true
Regex=(\\d+\\/\\d+\\/\\d+)\\s+(\\d+\:\\d+\:\\d+)\\s+SRC\=(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})\\s+DST\=(\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3})\\s+SPT\=(\\d+)\\s+DPT\=(\\d+)\\s+Sev\=(\\d+)\\s+URL\=(.*?)
token.count=5
token[0].name=Time_of_the_event
token[0].type=TimeStamp
token[0].format=dd/MM/yy HH:mm:ss
token[1].name=SrcIp
token[1].type=IPAddress
token[2].name=DstIp
token[2].type=IPAddress
token[3].name=Sev
token[3].type=Integer
token[4].name=URL
token[4].type=String
event.deviceReceiptTime=Time_of_the_event
event.sourceAddress=SrcIp
event.destinationAddress=DstIp
event.deviceSeverity=Sev
event.requestUrl=URL
event.deviceVendor=__getVendor(“MyVendor”)
event.deviceProduct=__stringConstant(“MyProduct”)
severity.map.veryhigh.if.deviceSeverity=404,500
severity.map.medium.if.deviceSeverity=303,302
Sample Configuration File
//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*//d+/////d+/////d+)//s+(//d+/://d+/://d+)//s+SRC/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+DST/=(//d{1,3}///d{1,3}///d{1,3}///d{1,3})//s+SPT/=(//d+)//s+DPT/=(//d+)//s+URL/=(.*
Example Type 01: Regex
Defining Regex is the main task in the Parsing mechanism from simple to complex Logs using the Regular Expressions for various Log files.
Example Type 02: Query
Defining query is the main task in the Parsing mechanism for Database Logs using the SQL Queries to retrieve the Data from Database schema.
Example Type 03: Expression
Defining node expression is one of the main task in the Parsing mechanism for XML Logs using beginning node location expression to process the Expression. A root node is at the top of the tree, hop nodes are in between, and trigger nodes are at the bottom.
Understanding The Parser Expression Usage
Regular Expression
Regex Examples
Getting the Sample Logs and Analyzing for Events of Interest from
Device/Application documents and understand its Nature.
Checking the Logging Mechanism Enabled on the End Device with Audit Level.
Choosing the Method of Logs Collection based on logging mechanism and gather the Events and define the possible Use cases.
Ex: Batch mode or Real Time.
Choosing the Suitable Flex Connector Type for Parsing.
Checking out the Log Rotation Policies of both End Device and Connector.
Defining your Flex Configuration file and Agent Properties file.
Use this in Test Environment to check your Flex Connector working without any issues. And check whether all the events are parsed properly by enabling raw events and comparing them with the Unparsed Events.
Normalization and Further Content Development.
Choosing your Categorization files, Additional Mapping, Key Value Parsers, Map files etc., and placing in the Exact Location.
Ensure the Following event severity, deviceEventClassId, categorization, deviceCustom and Flex field Labels.
Things To Remember While Defining The Flex Connectors
Example
Time based Connector
ID based Connector
Database Flex Connector
Configuration File –Time Based
version.order Specifies the order in which parser files are executed.
version.query This property enables you to perform a test query against the database to validate the database version.
version.id If the version.query succeeds, the deviceVersion token is set to the version.id.
Query Retrieves the rows(events) that were inserted between the last time query was run and the current time.
timestamp.field Specifies the field to use to determine when to run the next query.
uniqueid.fields Specifies the field to use to distinguish rows with the same timestamp field.
Configuration File –Time Based
Configuration File - Id BASED
maxid.query Specifies the query to use to retrieve the maximum ID present in the database when the query is run.
id.field Specifies the field to use to determine when to run the next query.
uniqueid.fields Specifies the field to use to distinguish rows with the same ID field.
query.limit Specifies the maximum number of rows to return when a query is run.
Configuration File –ID Based
Configuration File – XML Flex
namespace.count Specifies the number of namespaces that your XML log file Uses.
namespace.prefix Specifies the namespace prefix to use.
namespace.uri Specifies the Uniform Resource Identifier (URI) for the namespace.
hop.node.count Specifies the number of hop nodes.
hop.node.name Specifies the names for the hop nodes.
hop.node.expression Specifies the XPath/XQuery path expressions to select the nodes.
trigger.node.expression These are the nodes that trigger events.
token[x].expression Specifies the XPath/XQuery path expression that is traversed to obtain the value for the token.
token[x].node Specifies the context node—root node, hop node, or trigger node—relative to which the path expression is evaluated.
extraevent.count Specifies the number of extra events.
extraevent[x].filename Specifies the file name of the additional configuration file that this parser should use.
extraevent[x].name Specifies a name to associate with the extra events.
Configuration File – XML Flex
Example
• Nov 28 22:02:42 10.0.111.2 %PIX-6-106015: Deny TCP (no connection)from 3.3.3.3/4532 to 4.4.4.4/80 flags RST on interface outside
• Nov 28 22:06:10 10.0.111.2 %PIX-3-305005: No translation groupfound for tcp src inside:10.0.112.9/37 dst outside:4.5.6.7/3562
• Nov 29 01:46:42 10.0.111.2 %PIX-6-305005: Translation built for gaddr 1.2.3.4 to laddr 10.0.111.9
• Nov 29 01:35:15 10.0.111.2 %PIX-4-500004: Invalid transport fieldfor protocol=6, from 2.2.2.2/0 to 3.3.3.3/0
Single log source may contain more than one message format. We don’t have to define 4 different parsers for single source.
Need For Sub Messages
Message divided in two portions.(common to all messages and one that varies with each message format)
• Nov 28 22:02:42 10.0.111.2 %PIX-6-106015: Deny TCP (no connection)from 199.248.65.116/3564 to 10.0.111.22/80 flags RST on interface outside
Into: (Static Event content) • Nov 28 22:02:42 10.0.111.2 %PIX-6-106015:
And: (Variable Event content) • Deny TCP (no connection) from 199.248.65.116/3564 to 10.0.111.22/80
flags RST on interface outside
To define the sub-message we need to perform these steps:
1. Define the corresponding sub-message ID.
2. Define the regular expression(s) to use.
3. Define the mappings to event fields.
Sub Messages
regex=(\S+ \d+ \d+:\d+:\d+) (\S+) %PIX-(\d)-(\d+): (.*)
token.count=5 token[0].name=Timestamp token[0].type=TimeStamp token[0].format=MMM dd HH\:mm\:ss token[1].name=PixIP token[1].type=IPAddress token[2].name=PixSeverity token[2].type=String token[3].name=SubmessageIdToken token[3].type=String token[4].name=SubmessageToken token[4].type=String
Example
submessage.messageid.token=SubmessageIdToken
identifies the token that will hold the message identifier.
submessage.token=SubmessageToken
token that contains the actual sub-message.
submessage.count=1
count of sub-message IDs (106015).
Following will be internally equivalent to 2 Tokens
Submessage[0].messageid=106015
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=Deny (\\S+) \\(no connection\\)\\s(\\d+\.\\d+\.\\d+\.\\d+)
submessage[0].pattern[0].fields=event.transportProtocol, event.sourceAddress
submessage[0].pattern[0].types=String,IPAddress
submessage[0].pattern[0].formats=null,null
The format can also be defined using one sub-message property. Can be used for Different Time Zone Mappings.
Example Continued
//S//S//S
Event id is 532 type A with parameter 3.3.3.3
Event id is 533 type A with parameter root
Event id is 534 type A with parameter 3.3.3.3
Scenario:
Event id is 532 or 534, set the ArcSight event field event.sourceAddress to 3.3.3.3 and
if the event id is 533,
set the event.sourceUserName to root.
Conditional mappings enable you to map tokens that can contain
different types of information, based on the characteristic of the
event.
Need For Conditional Mapping
regex=Event id is (\\d+) type (\\S+) with parameter (\\S+)
token.count=3
token[0].name=EVENTID
token[1].name=TYPE
token[2].name=PARAMETER
#Standard mappings
event.deviceEventClassId=EVENTID
event.deviceEventCategory=TYPE
#Conditional mappings
conditionalmap.count=1
conditionalmap[0].field=event.deviceEventClassId
conditionalmap[0].mappings.count=2
conditionalmap[0].mappings[0].values=532,534
conditionalmap[0].mappings[0].event.sourceAddress=PARAMETER
conditionalmap[0].mappings[1].values=533
conditionalmap[0].mappings[1].event.sourceUserName=PARAMETER
Example
submessage[3].messageid=conditionalmapsample
submessage[3].pattern.count=1
submessage[3].pattern[0].regex=Event id is (\\d+) type (\\S+) with parameter (\\S+)
submessage[3].pattern[0].fields=event.deviceEventClassId
submessage[3].pattern[0].conditionalmap.count=2
submessage[3].pattern[0].conditionalmap[0].field=event.deviceEventClassId
submessage[3].pattern[0].conditionalmap[0].mappings.count=2
submessage[3].pattern[0].conditionalmap[0].mappings[0].values=532,534
submessage[3].pattern[0].conditionalmap[0].mappings[0].event.destinationAddress=$3
submessage[3].pattern[0].conditionalmap[0].mappings[1].values=533
submessage[3].pattern[0].conditionalmap[0].mappings[1].event.destinationUserName=$3
submessage[3].pattern[0].conditionalmap[1].token=$2
submessage[3].pattern[0].conditionalmap[1].mappings.count=1
submessage[3].pattern[0].conditionalmap[1].mappings[0].values=B
submessage[3].pattern[0].conditionalmap[1].mappings[0].event.destinationAddress=$3
In the above example, there are three groups:
$1 -- (\\d+)
$2 -- (\\S+)
$3 -- (\\S+)
Example 2(conditional Mapping In Submessages)
To chain two configuration files together Useful if you need to use two or more different types of FlexConnectors for the same data.
Extra processors are particularly useful when an event has more than one type of data in it and cannot be parsed by a single parser. This property is also referred to as parser linking.
Can be useful when you use Regular expression to parse data that was obtained from a time-based SQL database.
Configuration files need to be placed in the \user\agent\flexagent folder.
Example Scenario
When you use the same Log File for logging different versions of Same Application Server with varying Formats.
Need For Extra Processor
extraprocessor.count=1 (the number of extra processors)
extraprocessor[0].type=regex (extra processor type)
extraprocessor[0].filename=netiq/netiq (extra process file name)
extraprocessor[0].field=event.message
extraprocessor[0].flexagent (extra processor variable)=true (extra processor parameter or conditional value)
extraprocessor[0].clearfieldafterparsing=false
Example
Extra Processor Type
extraprocessor.count=2
extraprocessor[0].type=regex
extraprocessor[0].field=event.name
extraprocessor[0].filename=securitymanager/Name-Name
extraprocessor[0].clearfieldafterparsing=false
extraprocessor[0].flexagent=true
extraprocessor[1].type=regex
extraprocessor[1].field=event.name
extraprocessor[1].filename=scm/Name-Name
extraprocessor[1].clearfieldafterparsing=false
extraprocessor[1].flexagent=true
Example
Multiline parsing provides a mechanism for providing hints so that the parser can reconstruct messages that have been broken into multiple lines. Because some files may contain events that are split into multiple lines.
Ex Scenario:
|01/01/2005 11:00:50|1.1.1.1|7663|2.2.2.2|80|this
is
a
message
that
takes
multiple
lines|
01/01/2005 11:00:51|1.1.1.1|7663|2.2.2.2|80|this
is another large message that takes
multiple lines|
Need For Multiline Parser
multiline.starts.regex=\|\d+/\d+/\d+ \d+:\d+:\d+\|.*
multiline.ends.regex=.*\|$
Output:-
|01/01/2005 11:00:50|1.1.1.1|7663|2.2.2.2|80|this is a message that takes multiple lines|
Sample Output
multiline.starts.regex=\\|\\d+/\\d+/\\d+ \\d+\:\\d+\:\\d+\\|.*
regex=\\|(.*?)\\|(\\S+)\\|(\\d+)\\|(\\S+)\\|(\\d+)\\|(.*)|
token.count=6
token[0].name=Timestamp
token[0].type=TimeStamp
token[0].format=MM/dd/yyyy(\\s)HH\:mm\:ss
token[1].name=SourceAddress
token[1].type=IPAddress
Multiline Regex Configuration File
//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|//S+)/|(/d+)/|(/S+)/|(/d+)/|(.*)|
To support multi-line messages, we need to define the message start and end in the configuration file.
Multiline Regex
Some SmartConnector parsers map sensitive information such as source and
destination user names, host names, addresses, etc… inappropriately, i.e.
Windows Event Log SmartConnectors.
Parser Override(Parser Versioning) which enables each SmartConnector to
Parse raw events in many different ways using different Parser Versions, and thus
generate ArcSight security events with different types of mappings.
It support the current parser mappings so as to not break existing content for
users, but also to support newly corrected mappings so as to allow new and
accurate content to be developed.
Need For Parser Overrides
A SmartConnector feature that allows a SmartConnector to support multiple
versions of parsers.
Allows users to configure their SmartConnectors with any available parser version of their choice, depending on their ArcSight security event mapping requirements.
Each SmartConnector is designed to have its own internal parameter fcp.version to represent its current Parser Version.
Each SmartConnector can support a total of 8 Parser Versions
fcp.version range from 0(Base Parser Version) through 7.
To identify the Parser Version with which a raw event has been parsed, observe the last digit of the Agent Version field of the ArcSight security event. i.e.,
For Parser Version 0, the Agent Version will be 5.1.2.5823.0
For Parser Version 1, the Agent Version will be 5.1.2.5823.1
Parser Override Pg.01
Parser Override Pg.02
Extra mappings is another property of the sub-message that can be used to directly add additional mapping properties.
Example:
submessage[3].pattern[0].extramappings=event.name=
__stringConstant("Unparsed event")
|event.deviceProduct=__stringConstant("Unknown")
In this above Example, you might have multiple Submessages Patterns. If u don’t want to miss any events and want to map particular set events in one category say all unknown events in this one category but you are not using the submessage[2].pattern[0].fields directly.
Need For Extra Mapping
# Default sub-message descriptor
submessage[3].pattern.count=1
submessage[3].pattern[0].regex=(.*)
submessage[3].pattern[0].extramappings.delimiter=@
submessage[3].pattern[0].fields=event.message
submessage[3].pattern[0].extramappings=
event.name\=__stringConstant("Unparsed event")
@event.deviceProduct\=__stringConstant("Unknown")
Extra Mapping Configuration
Some devices will send information about a single event in multiple log lines. Even though in some cases it would be fine to send each line as a single event, in some other instances it is necessary to merge the information of all the events into a single one.
Ex:
[18/Jul/2005:12:30:20 -0400] conn=8 op=0 msgId=82 - BIND uid=admin
[18/Jul/2005:12:30:25 -0400] conn=7 op=-1 msgId=-1 - LDAP connection from
10.0.20.122 to 10.0.20.122
[18/Jul/2005:12:30:30 -0400] conn=8 op=0 msgId=82 - RESULT err=0
We can say I have Multiline Parser, In this instance you can’t say i can use
Multiline, coz here 1st line is input and 3rd line is output, so you can’t use the
Multiline Parser to solve this issue. Here in cases like this Merge Operation will
help you solve this mistery. So we can deploy the merge operation for the events
Which have connection in above (Conn,msgId) but with different operations.
Need For Merge Operations
Each merge operation defines:
Which events to include in the merge operation
When to start a merge operation
When to end a merge operation
The fields that identify which events belong to the same group
Note: Currently ONLY the regular expression based agents support this feature.
You need to use the predefined set of merge operation property variables.
Defining Merging Operation
merge.count Defines the number of merge operations.
merge[{mergeindex}].traceenabled When set to TRUE all operations regarding
event merging will be logged for this merge operation.
merge[{mergeindex}].pattern.count Defines how many patterns will be defined.
merge[{mergeindex}].pattern[{patternindex}].token Defines the token that will be used for
this pattern.
merge[{mergeindex}].pattern[{patternindex}].regex Defines the regular expression to use for
this pattern.
merge[{mergeindex}].starts.count Defines how many start patterns will be defined.
merge[{mergeindex}].starts[{patternindex}].token Defines the token that will be used for
this start pattern.
merge[{mergeindex}].starts[{patternindex}].regex Defines the regular expression to use for
this start pattern.
merge[{mergeindex}].starts[{patternindex}].endspreviousmerge If set to true then it means
that if the start message is found within an already merged event, then the merge processor
should end the previous merge and start a new one.
Merge Operation Property Pg.01
merge[{mergeindex}].ends.count Merge operations require end patterns to
define which events will end the merge operation.
merge[{mergeindex}].ends[{patternindex}].token Defines the token that will
be used for this end pattern.
merge[{mergeindex}].ends[{patternindex}].regex Defines the regular
expression to use for this end pattern.
merge[{mergeindex}].timeout Defines the timeout in milliseconds for the
merging operation.
merge[{mergeindex}].id.tokens Defines the list of tokens that will be used to
group the events.
merge[{mergeindex}].id.delimiter Defines an optional delimiter to use.
merge[{mergeindex}].sendpartialevents It specifies if each event in the merge
operation must be sent individually as it is merged with other events.
merge[{mergeindex}].capacity An event merging operation requires a
cache of events that hold the merged results.
Merge Operation Property Pg.02
merge.count=1
merge[0].pattern.count=2
merge[0].pattern[0].token=NAME1
merge[0].pattern[0].regex=(BIND|UNBIND|MOD|RESULT)
merge[0].pattern[1].token=NAME2
merge[0].pattern[1].regex=(BIND|UNBIND|MOD|RESULT)2
merge[0].starts.count=1
merge[0].starts[0].token=NAME3
merge[0].starts[0].regex=(BIND|UNBIND|MOD)
merge[0].ends.count=2
merge[0].ends[0].token=NAME4
merge[0].ends[0].regex=RESULT
merge[0].ends[1].token=NAME5
merge[0].ends[1].regex=RESULT2
merge[0].timeout=60000
merge[0].id.tokens=conn|msgId
merge[0].id.tokens.delimiter=|
merge[0].sendpartialevents=true
merge[0].capacity=100
Merge Properties
Sample Logs :
[18/Jul/2005:12:30:20 -0400] conn=8 op=0 msgId=82 - BIND uid=admin
[18/Jul/2005:12:30:25 -0400] conn=7 op=-1 msgId=-1 - LDAP connection from
10.0.20.122 to 10.0.20.122
[18/Jul/2005:12:30:30 -0400] conn=8 op=0 msgId=82 - RESULT err=0
Merger Property:
merge.count=1
merge[0].pattern.count=1
merge[0].pattern[0].token=OperationName OperationName set to BIND or RESULT.
merge[0].pattern[0].regex=(BIND|RESULT)
merge[0].starts.count=1
merge[0].starts[0].token=OperationName OperationName set to BIND(start the merge)
merge[0].starts[0].regex=BIND
merge[0].ends.count=1
merge[0].ends[0].token=OperationName OperationName set to RESULT(End merge)
merge[0].ends[0].regex=RESULT
merge[0].id.tokens=Connection,Operation,MessageId (Defining Fields which must be Identical)
merge[0].timeout=60000
Merge Operation Example
In Event Mapping Section:
event.deviceReceiptTime=Date
event.name=__oneOf(mergedevent.name,OperationName)
Gets to Sub message from the name Field for Mapping the Merged Operation.
event.deviceAction=ResultCode
event.destinationUserId=UserId
Merge Operation Example Pg 2
The FlexConnector developer can control categorization by creating or modifying
Existing categorization files. Categorization files are comma-separated value (CSV)
text files, placed in a folder named for the device vendor under the directory:
ARCSIGHT_HOME/user/agent/acp/categorizer/current//.csv (Note: This Overrides Existing Categorization)
Examples:
event.deviceSeverity,set.event.categoryObject,set.event.categoryBehavior,set.event.categoryTechnique,set.event.categoryDeviceGroup,set.event.categorySignificance, set.event.categoryOutcome
666,/Host/Resource,/Access/Start,,/Application,/Normal,/Success
event.deviceAction,set.event.categoryObject,set.event.categoryBehavior,set.event.categoryDeviceGroup,set.event.categorySignificance,set.event.categoryOutcome
OPEN,/Host/Application/Service,/Communicate/Query,/Firewall,/Normal,/Success
Custom Categorizations
Key-value parsers divide log lines into key-value pairs (key=value), extract the key-value pairs into tokens, and then the tokens are mapped to event fields.
Key-value parsers are used with keyvalue extra processors and syslog subagents Use key-value parsers for secondary processing.
The configuration file name for key- value parsers is
vendor.subagent.sdkkeyvaluefilereader.properties.
Ex: TIME=28/09/11 08:15:00 SRC=194.168.0.12 DST=195.172.0.12 SPT=4236 DPT=80
Key-value parsers have the following properties:
key.delimiter key.delimiter=\\s
key.value.delimiter key.value.delimiter==
key.regexp key.regexp=([^\\s]+)
text.qualifier text.qualifier=“
trim.message True trims the leading and trailing white spaces of the log line.
trim.tokens True trims the leading and trailing white spaces of each token.
trim.keys True trims the leading and trailing white spaces of each key.
Key Value Parsers
Map files are a way that we can set ArcSight event fields based on the information in another field. Essentially the Map file is a CSV file that functions much the same way as a categorization file in that it uses getters followed by setters.
The map files are always located under
/current/user/agent/map/map.X.properties
Can have multiple map files as long as they are named using a sequential number
Allow customers to perform custom field mappings
Allow override of standard parser values
A very simple example is if you do not use DNS for hostname to IP resolution. This can be handled in the map file. The structure of the map file would look something similar to below:
range.event.sourceAddress,set.event.deviceCustomString4,
set.event.deviceCustomString4Label
10.100.0.0-10.100.0.100,QATestLab Building2,Location
10.100.0.101-10.100.0.200,Building1,Location
Creating Map Files
The deviceEventClassId is a method that ArcSight uses to create a unique identifier for each event.
For example all the ArcSight internal agent messages are in the format “agent:xxx” where xxx represent a number. When tracking events using rules we are able to use these numbers as they are unique for each event.
Ex:
event.deviceEventClassId=__concatenateDeleting("Nessus=",NessusID,
"#",Name,"#",Risk,"#",INFO,"%CVE=",CVE,"%Bugtraq=",Bugtraq,"%|#=/@")
%, |, #, =, @, which are used asdelimiters in parsers.
Defining DeviceEventClassId
In some environments it is useful to map certain additional data names to normal ArcSight schema fields. The mapping can vary based on the device vendor and product and can be controlled from the
ArcSight Console, with the mappings stored on the SmartConnector machine. The Get Additional Data Names command specifies the additional data names Assigned to each
device vendor or product combination since the SmartConnector started running.
The Map Additional Data Name Field used must be a valid ArcSight event field.
Additional Data Mapping
Action connectors are built to allow integrations between ArcSight and third party devices for the purpose of allowing the third party device to be controllable from within the ArcSight console.
The user can then execute commands on third party devices from within ArcSight and send the output of those commands back to the console. The remote command can be executed as an action in the correlation rules engine, or as a right click on the action connector. The command is executed from the host that the connector resides on.
While Installing select the Flex CounterACT connector from the list of available connectors. After selecting the connector you will need to enter the name of the Configuration File (the extension will be added automatically). Complete the wizard.
Need For CounterACT Connector
Create a file named “.counteract.properties” in the
\current\user\agent\flexagent directory. This file will contain the commands that you want to be able to execute. Here is an example of such file:
command.count The number of commands that will be supported.
command[x].name The internal name that you want for the command. command[x].displayname The Command display name in the ArcSight console.
command[x].parameter.count Number of parameters that the command will receive.
command[x].parameter[x].name The internal name of the parameter.
command[x].parameter[x].displayname The parameter display name shown in the ArcSight console
command[x].action This is the command line executable that will be executed. This property should be provided as a template with variables that will be replaced by the actual values. A couple of variables are provided by default:
ARCSIGHT_HOME: The absolute path where the connector is running
PLATFORM: A platform code (win32/linux/solaris) Typically used if you have scripts for different OSs
PLATFORM_BINARY_EXT: Set to .bat for win32 and set to .sh for linux and solaris
CounterACT Config Commands
command.count=1
command[0].name=nmapit
command[0].displayname=NMap
command[0].parameter.count=1
command[0].parameter[0].name=ipaddress
command[0].parameter[0].displayname=Ip Address
command[0].action=C:\\NMAP\\NMAP.EXE ${ipaddress}
You can make use of the CounterACT Commands in 2 ways:
From Connector Will pop out the Command Parameter.
From a Rule Will pop out the Fields for your Command Parameter.
It is possible to parse this output and modify the return event to extract the Output you are looking for using a module called SecondLevelRegexParser.
To use the second level parser feature, create the file user/agent/fcp/additionalregexparsing/ngflexcounteract/regex.0.sdkrfilereader.properties
CounterACT Example
CounterACT Command Execution
The REST FlexConnector provides a configurable method to collect security events when you use cloud-based applications such as Box, Salesforce, or Google Apps.
The REST FlexConnector framework allows you to develop FlexConnectors to collect events from vendors by configuring:
OAuth2 for authentication with the vendor.
REST API endpoints exposed by the vendor for event collection.
JSON parsers for parsing and mapping data (retrieved from the REST APIs).
Refer the Rest Flex Connector Development Guide for More Information on this and how to configure one.
Rest API
Register Your Connector Application
Box OAuth2 Registration and Values.
Salesforce OAuth2 Registration and Values.
Google Apps OAuth2 Registration and Values.
Create OAuth2 Client Properties File
Determine Which Events URL (REST API Endpoint) to Use
REST API End Points General Information
Querying Based On Timestamp, Rate Limiting
Box REST API
Salesforce REST APIs
Google Apps REST API
Create a JSON Parser File
Defining the JSON Structure
Defining the JSON Parser
Viewing the Raw JSON Data
REST FlexConnector Configuration Support Tool (restutil)
REST Flexconnector Development Tasks
Enter the name of the parser file, provided the parser file is copied into the user\agent\flexagent dir
Enter the events URL. This is the REST API endpoint which is used by the connector to get the events.
Browse for the OAuth2 Client Properties File. You must create this file from values you obtain when you register your connector application, as well as providing a redirect_uri.
Rest Flex Installation
Create any regular flex connector to read the data corresponding to the
Active List.
Define Tokens only and do not map to fields.
Map tokens to additional data.
Additional Data field name can be anything.
Define the properties to invoke Model Import feature.
Define the property to invoke the custom Velocity Macro file that converts the data into the ArcSight Archive and place it in the user/agent/fcp directory.
Edit the agent.properties and add the following
agent.component[34].maxeventsbeforebuild=20000
agent.component[34].buildmodeldelay=90000
Flex Active List Import
comments.start.with=#
delimiter=,
token.count=1
token[0].name=IP
token[0].type=String
additionaldata.enabled=true
additionaldata.IP_ADDRESS=IP
additionaldata.CREATE_DATE=__concatenate(__longToString(__currentTimestampInSeconds()),"000")
event.deviceVendor=__stringConstant(ArcSight)
event.deviceProduct=__stringConstant(FlexArchiveImport)
event.deviceCustomString1Label=__stringConstant(model.sender)
event.deviceCustomString1=__stringConstant(DVLabs)
event.deviceCustomString2Label=__stringConstant(model.template)
event.deviceCustomString2=__stringConstant(ips.vm)
Flex Active List Import Example
The SmartConnector for Asset Import lets you define a comma-separated (.csv) file that imports asset modeling details in a batch.
If your asset inventory changes regularly, you can set up a process to update and export this list at regular intervals to update the assets in ArcSight ESM.
Enter the file path to the folder where the CSV files to be imported are stored for the connector to automatically import the assets into ArcSight ESM.
Assign your Asset Import connector to the ArcSight Network or Networks represented by the assets modeled in your CSV file.
The CSV File contains following headers or Fields:address, macAddress, hostname, location, category
Flex Asset Import
agents[x].usenonlockingwindowsfilereader Does not lock the log file
read by the connector on the Windows platform.
agents[x].startatend The default is true. Useful when log files to be
processed already exist and contain data at connector startup or when the log file
rotation takes place.
agents[x].wildcard Specifies a file extension. The Regex FlexConnector
processes only files with the specified file extension.
agents[x].processfoldersrecursively Specifies whether to process log files in
the subfolders of a specified folder.
agents[x].mode Specifies the action to perform on a log file after
the FlexConnector has processed it. (RenameFileInTheSameDirectory, DeleteFile,
PersistFile)
agents[x].foldertable[x].processingmode The Flex Log Processing mode either
on real time or Batch File Mode.
Few Sample Agent Property File Important Configurations
Advanced Regex Usages and Scenarios.
Agent Property file Important Configurations.
Collection of all Other Useful Files (Token Operations List, ArcSight CEF Fields….)
Basic Troubleshooting.
Lab Exercises for Practice using Regex.
Using Replay Connectors to test the Sample Logs.
Start Open Forum for Flex Connector Building and Suggestions.(Trust me if I am a coder will create a Site to Test and Generate Online Flex )
Q & A.
Modifications based on user Suggestions.
Future Concepts
Multiple References:
ArcSight Flex Dev Guide.
Protect 724 Posts.
Other Flex Documents and Discussions.
Merge Operations: Hector Aguilar – Macias, Girish Mantry.
Rate the ArcSight content of these Document in the same ArcSight Forum Thread where it is uploaded. Your appreciation and Suggestions are always helpful and motivating. Next Update will provide more useful Snaps for Flex.
Thank You
Copyright V.B 2013
REFERENCES