Writing a Log Source Extension - IBM - United States Writing a Log Source Extension Summary A Universal

Embed Size (px)

Text of Writing a Log Source Extension - IBM - United States Writing a Log Source Extension Summary A...

  • POC: Writing a Log Source Extension


    A Universal DSM, or UDSM, can be used to integrate log sources that do not have an official

    DSM. A Log Source Extension, or LSX, is then applied to the UDSM to provide the logic for

    parsing the logs. The LSX is based on Java regular expressions and can be used against any log

    protocol, e.g. Syslog, JDBC, LFPS, etc. Values can be extracted from the logs and mapped to all

    common fields within QRadar. The unsupported log source can then be fully utilized within


    Obtaining A Log Sample

    Creating The Log Source

    The first step in building a Universal DSM is to create the log source within QRadar. This

    prevents the logs from being automatically classified and allows you to export the logs for


    1. From the Admin tab, create a new source using the Log Sources icon. 2. Specify the Log Source Name. 3. Specify Log Source Type as Universal DSM. 4. Specify the protocol that will used for Protocol Configuration. This is the method that

    QRadar will obtain the logs from the unsupported log source.

    5. Enter the IP address or hostname of the unsupported log source as the Log Source Identifier.

    6. Click Save to save the new log source. Close the window. 7. Click Deploy Changes from the Admin tab.

  • Alternatively, log samples can be used if provided by a customer. This is typically done before

    the start of a POC. The log samples must be in the same format as the logs being sent to

    QRadar. Typically this is only possible if the log samples are from Syslog or FTP/SFTP. Other

    methods, e.g. screenshots of a mainframe screen or a description of an SQL schema, do not

    provide logs in a format that can be utilized by QRadar.

    Exporting The Logs

    The next step is to export the logs that were obtained from the previous step. Typically you will

    want a significant amount of logs for review. Depending on the EPS rate of the unsupported, it

    may take several hours to obtain a comprehensive log sample.

    1. Create a search for exporting the logs. 1. From the Events tab, select Search -> Edit Search. 2. Specify the Time Range as an adequate amount of time from when the log source

    was created, e.g. 6 hours.

    3. Under Search Parameters, select Log Source Equals and specify the log source that was created in the previous step.

    4. Click Search to view the results.

  • 2. Export the results of the search. 1. After the search has completed, select Actions -> Export to CSV.

    2. You will now be prompted to download a compressed file. Open the compressed file and then open the resulting CSV file using Microsoft Excel.

    3. In Microsoft Excel, copy column AM. This column should contain the raw payloads from the unsupported log source.

    4. Paste the results into a raw text file using an application such as Notepad++. 5. Save the file. The file name should be based on the unsupported log source name

    and not include spaces, e.g. Fakeware_Logs.txt.

    Reviewing The Logs

  • A critical step in creating a Universal DSM is reviewing the logs for usability. At a bare

    minimum, the logs should have a value that can be mapped to an event name. The event name

    should be a unique value that can distinguish the various log types.

    An example of usable logs: o May 20 17:16:14 dropbear[22331]: bad password attempt for 'root'

    from o May 20 17:16:26 dropbear[22331]: password auth succeeded for

    'root' from o May 20 16:42:19 kernel: DROP IN=vlan2 OUT=

    MAC=00:01:5c:31:39:c2:08:00 SRC=

    DST= PROTO=UDP SPT=67 DPT=68 An example of slightly usable logs:

    o Oct 26 08:12:08 loopback 1256559128 autotrace[215824]: W: trace: no map for prod 49420003, idf 010029a2, lal 00af0008

    o Oct 26 16:35:00 sxpgbd0081 last message repeated 7 times o Nov 24 01:30:00 sxpgbd0081 /usr/local/monitor-rrd/sxpgbd0081/.rrd

    (rc=-1, opening '/usr/local/monitor-rrd/sxpgbd0081/.rrd': No such

    file or directory)

    Creating The Log Source Extension

    Determining Which Fields Can Be Used

    The LSX template provides a large number of fields that values can be mapped to. These values

    are obtained from the unsupported logs. This includes fields such as the source IP, destination

    port, username, etc. The first step in creating the log source extension is reviewing the

    unsupported logs and determining what fields will be used. At a bare minimum, the EventName

    field must be used.

    1. Download the LSX template and save it to your desktop. Be sure to right-click and use the "Save As" option so it's not opened by Internet Explorer.

    2. Rename the file to reflect the unsupported log source. For example, for Fakweare, rename the file to Fakeware_LSX.xml.

    3. Open the file using a text editor such as Notepad++. A list of fields will be displayed.

  • 4. Determine which values in the unsupported log source can be mapped to the fields in the LSX template. It is not necessary to use all of the fields in the default LSX template.

    o Example log entry:

    May 20 17:24:59 kernel: DROP MAC=5c:31:39:c2:08:00 SRC=

    DST= LEN=351 TOS=0x00 PREC=0x00 TTL=64 ID=9582 PROTO=UDP

    SPT=67 DPT=68 LEN=331

  • o The following fields are usable for this example:

    EventName, i.e. DROP

    SourceMAC, i.e. MAC=5c:31:39:c2:08:00

    SourceIp, i.e. SRC=

    DestinationIp, i.e. DST=

    Protocol, i.e. PROTO=UDP

    SourcePort, i.e. SPT=67

    DestinationPort, i.e. DPT=68

    5. Remove any unused fields and their corresponding Pattern IDs from the LSX.

    6. Rename the Pattern IDs to a unique name. This saves confusion if multiple patterns are used and helps distinguish between field and Pattern ID names. In most instances, the log

    source type can be appended to the end of the Pattern ID as the unique name. For

    example, EventName can be renamed to EventName-Fakeware as the Pattern ID name.

    7. Change the Pattern IDs defined in the fields to match the renamed Pattern IDs.

  • 8. Any other values in the logs that do not have a corresponding field can be utilized later using Custom Event Properties.

    Determining Regular Expression Patterns

    The next step in creating a UDSM is building the regular expression patterns. Regular

    expressions, or regex, are used to match strings of text from the unsupported log source.

    Finding The Strings To Match

    Visually analyze the unsupported log source to identify unique patterns. These patterns will later

    be translated into regular expressions. When possible, you should include characters before and

    after the actual value to provide a basic level of error checking. This will prevent similar values

    from being unintentionally matched. The actual value can be isolated from the extra characters in

    the later steps.

  • Example log entry:

    May 20 17:24:59 kernel: DROP MAC=5c:31:39:c2:08:00 SRC=

    DST= LEN=351 TOS=0x00 PREC=0x00 TTL=64 ID=9582 PROTO=UDP

    SPT=67 DPT=68 LEN=331 May 20 17:24:59 kernel: PASS MAC=5c:14:ab:c4:12:59 SRC=

    DST= LEN=351 TOS=0x00 PREC=0x00 TTL=64 ID=9583 PROTO=TCP

    SPT=1057 DPT=80 LEN=331 May 20 17:24:59 kernel: REJECT MAC=5c:ad:3c:54:11:07 SRC=

    DST= LEN=351 TOS=0x00 PREC=0x00 TTL=64 ID=9584 PROTO=TCP

    SPT=25212 DPT=6881 LEN=331

    Pseudo-code for matching patterns. The quotes can be ignored. The event names are

    DROP, PASS, and REJECT. Please note that the space character is included to denote the

    beginning and end of a pattern whenever possible.

    o EventName: " kernel: VALUE "

    o SourceMAC: " MAC=VALUE "

    o SourceIp: " SRC=VALUE "

    o DestinationIp: " DST=VALUE "

    o Protocol: " PROTO=VALUE "

    o SourcePort: " SPT=VALUE "

    o DestinationPort: " DPT=VALUE "

    In the next steps we will translate the pseudo-code to regular expressions.

    Common Regular Expressions

    There is essentially an infinite numbers of regular expressions that can be used to match a

    desired string of text. The following are several common regular expressions:

    Type Expression

    IP Address \d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

    Port Number \d{1,5}

    MAC Address (?:[0-9a-fA-F]{2}\:){5}[0-9a-fA-F]{2}

    Protocol (tcp|udp|icmp|gre)

    Device Time \w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}

    White Space \s

    Tab \t

    Match Anything .*?

    Additionally, the escape character, or "\", is used to denote a literal character. For example, in

    regex the "." character means "any single character" and would match A, B, 1, X, etc. To specify

    a literal match, you would use "\." instead. This would only match the "." character itself.

  • Escaping any non-digit or non-alpha character is usually the best way to ensure you do not

    accidentally match another character.

    Translating Pseudo-Code to Regular Expressions

    The translations are based on the common expressions provided in the previous step. The quotes

    can be ignored. Always substitute a space with the \s regular expression. Please note that any

    non-digit or non-alpha character has been escaped, e.g. = becomes \= and : becomes \:.

    Field Pseudo-Code Regular Expression

    EventName " kernel: VALUE

    " \skernel\:\s.*?\s

    SourceMAC " MAC=VALU


View more >