24
Juniper Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 www.juniper.net Published: 2010-11-16 Security Threat Response Manager AQL Flow and Event Query CLI Guide Release 2010.0

AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

  • Upload
    vothuan

  • View
    231

  • Download
    5

Embed Size (px)

Citation preview

Page 1: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

Security Threat Response Manager

AQL Flow and Event Query CLI Guide

Release 2010.0

Juniper Networks, Inc.1194 North Mathilda AvenueSunnyvale, CA 94089USA408-745-2000

www.juniper.net

Published: 2010-11-16

Page 2: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

2

Copyright NoticeCopyright © 2010 Juniper Networks, Inc. All rights reserved. Juniper Networks and the Juniper Networks logo are registered trademarks of Juniper Networks Inc. in the United States and other countries. All other trademarks, service marks, registered trademarks, or registered service marks in this document are the property of Juniper Networks or their respective owners. All specifications are subject to change without notice. Juniper Networks assumes no responsibility for any inaccuracies in this document or for any obligation to update information in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.

FCC StatementThe following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. The equipment generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense. The following information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it is not installed in accordance with NetScreen’s installation instructions, it may cause interference with radio and television reception. This equipment has been tested and found to comply with the limits for a Class B digital device in accordance with the specifications in part 15 of the FCC rules. These specifications are designed to provide reasonable protection against such interference in a residential installation. However, there is no guarantee that interference will not occur in a particular installation. If this equipment does cause harmful interference to radio or television reception, which can be determined by turning the equipment off and on, the user is encouraged to try to correct the interference by one or more of the following measures: Reorient or relocate the receiving antenna. Increase the separation between the equipment and receiver. Consult the dealer or an experienced radio/TV technician for help. Connect the equipment to an outlet on a circuit different from that to which the receiver is connected.

Caution: Changes or modifications to this product could void the user's warranty and authority to operate this device.

DisclaimerTHE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR JUNIPER NETWORKS REPRESENTATIVE FOR A COPY.

AQL Flow and Event Query CLI GuideRelease 2010.0

Copyright © 2010, Juniper Networks, Inc.

All rights reserved. Printed in USA.

Revision History

November 2010—R2 AQL Flow and Event Query CLI Guide

The information in this document is current as of the date listed in the revision history.

Page 3: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

CONTENTS

ABOUT THIS GUIDEConventions 1Technical Documentation 1Contacting Customer Support 2

1 USING THE AQL QUERY CLIAbout the AQL Query CLI 3Accessing the AQL Query CLI 4Using a Select Statement 5Using Where Clauses 10Using the Group By Clause 11Using the Order By Clause 12Using the Count(*) Clause 12Using the Distinct Clause 12Using the Count (Distinct ...) Clause 13Using the Materialize View Clause 13Using the Like Clause 14Using the Boolean Clause 14Identifying AQL Fields 15Listing AQL Fields 16

Page 4: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response
Page 5: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

ABOUT THIS GUIDE

The AQL Event and Flow Query CLI Guide provides you with information for using the AQL CLI. This guide assumes you have advanced knowledge of Linux command line functionality.

Conventions Table 1 lists conventions that are used throughout this guide.

Technical Documentation

You can access technical documentation, technical notes, and release notes directly from the Juniper customer support web site at www.juniper.net/support/. Once you access the Juniper customer support web site, locate the product and software release for which you require documentation.

Your comments are important to us. Please send your e-mail comments about this guide or any of the Juniper Networks documentation to:[email protected].

Include the following information with your comments:

• Document title • Page number

Table 1 Icons

Icon Type DescriptionInformation note Information that describes important features or

instructions.

Caution Information that alerts you to potential loss of data or potential damage to an application, system, device, or network.

Warning Information that alerts you to potential personal injury.

AQL Event and Flow Query CLI Guide

Page 6: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

2 ABOUT THIS GUIDE

Contacting Customer Support

To help resolve any issues that you may encounter when installing or maintaining STRM, you can contact Customer Support as follows:

• Open a support case using the Case Management link at www.juniper.net/support/.

• Call 1-888-314-JTAC (from the United States, Canada, or Mexico) or1-408-745-9500 (from elsewhere).

AQL Event and Flow Query CLI Guide

Page 7: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

1

USING THE AQL QUERY CLI

You can use the AQL Event and Flow Query Command Line Interface (CLI) to access flows and events stored in the Ariel database. This document provides information on accessing and using the AQL query CLI including:

• About the AQL Query CLI• Accessing the AQL Query CLI• Using a Select Statement• Using Where Clauses• Using the Group By Clause• Using the Order By Clause• Using the Count(*) Clause• Using the Distinct Clause• Using the Count (Distinct ...) Clause• Using the Materialize View Clause• Using the Like Clause• Using the Boolean Clause• Identifying AQL Fields• Listing AQL Fields

About the AQL Query CLI

The AQL Event and Flow QueryCLI allows you to access raw flows and events stored in the Ariel database. The AQL query CLI includes syntax that is a subset of the SQL92 standard and provides support for two tables: events and flows.

Note: The AQL CLI does not provide support for joining tables.

The AQL Event and Flow Query CLI functions in the following modes:• Interactive mode - Using a simple shell, you can enter queries interactively

and view the results in a standard output. At the query prompt, any valid AQL statement is accepted. If time is not specified (using -start and -end options), the last minute is assumed as the time range. You can also access previous commands by using your up arrow. This is the default mode.

AQL Event and Flow Query CLI Guide

Page 8: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

4 USING THE AQL QUERY CLI

• Non-interactive mode - You can enter the non-interactive mode by adding the -execute <AQL query> parameter to the command. The -execute command must be followed by a valid AQL query surrounded by double quotes. The non-interactive mode does not include a prompt allowing you to redirect the output to a file with a regular UNIX pipe syntax. By default, the results are sent to a standard output.

Accessing the AQL Query CLI

To access the AQL query CLI:

Step 1 Log in to STRM, as root.

Step 2 Enter the following command:/opt/qradar/bin/arielClient

The Query prompt appears.

CLI Options Table 1-1 lists the supported CLI options:

Table 1-1 AQL CLI Options

Option Description-range <first record> <last record>

Limits the number of records sent to the output within the specified range. This is useful for viewing a selection of records generated by an ordered query. For example, if you want to view the first ten records, you must specify -range 1 10.

-debug Generates debugging output during execution. -start <time>, -end <time>

Specifies the start and end time of the query. Where <time> specifies the time. You must specify the time as either a UNIX timestamp or a date using the following format: yyyy/mm/dd-hh:mm:ss.For example:/opt/qradar/bin/arielClient - start 2007/08/11-01:15:00 -end 2007/08/11-01:17:00

-exectime <time limit>

Specifies the maximum period of time, in seconds, a single query may continue processing.

-execute <AQL query>

Allows you to enter non-interactive mode that enables you to process a query that is sent to standard output. If you do not include this option, the command is entered in interactive mode. You must include your query in double quotes.

-f <output format>

Allows you to specify the output format for the query results. The table format is an ASCII drawing of a multi-column table while the CSV format provides a comma separated list. Where <output format> indicates the output format. The options are table or csv.

AQL Event and Flow Query CLI Guide

Page 9: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

Using a Select Statement 5

For example:

To enter a command in interactive mode:/opt/qradar/bin/arielClient -start 2007/08/11-01:15:00 -end 2007/08/11-01:17:00 -exectime 60

/opt/qradar/bin/arielClient

/opt/qradar/bin/arielClient -start 2007/08/11-01:15:00 -end 2007/08/11-01:17:00 -f csv

To enter a command in non-interactive mode:/opt/qradar/bin/arielClient -start 2007/08/11-01:15:00 -end 2007/08/11-01:17:00 -exectime 60 -execute "select * from flows where sourceIP = '231.12.37.17' and protocol != 'TCP.tcp_ip'"

Using a Select Statement

You can use a select statement that includes one or more fields of a flow or event table. You can also use an asterisk (*) to denote all columns. All field names are case sensitive, however, the terms select and from are not case sensitive.

For example:select sourceIP, destinationIP, application from flows where protocol = ‘TCP.tcp_ip’

select category, credibility from events where severity > 8

select * from events where credibility >=9

The supported flow and event fields include:

-remote <host:port>

Specifies that you want to connect to a specific Ariel query host and port.

Table 1-1 AQL CLI Options (continued)

Option Description

Table 1-2 Supported Flow and Event Fields

Table Field Name Description Field TypeIdentified in idlist.sh?

Flows anyDestinationFlag Destination Flags Numeric NoanySourceFlag Source Flags Numeric Noapplication Application String YesapplicationId Application Numeric YesbytesIn Bytes In Numeric NobytesOut Bytes Out Numeric NodestinationAssetName Destination Asset Name String NodestinationASN Destination ASN Numeric No

AQL Event and Flow Query CLI Guide

Page 10: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

6 USING THE AQL QUERY CLI

destinationBytes Destination Bytes Numeric NodestinationByteRatio Destination Byte Ratio Numeric NodestinationDSCP Destination DSCP Numeric NodestinationDscpOnly Destination DSCP Numeric NodestinationFlags Destination Flags Numeric NodestinationIP Destination IP String NodestinationIPSearch Destination IP Search String NodestinationIfIndex Destination If Index Numeric NodestinationNetwork Destination Network String NodestinationPackets Destination Packets Numeric NodestinationPacketRatio Destination Packet Ratio Numeric NodestinationPayload Destination Payload String NodestinationPayloadHex Destination Payload As Hex Hexadecimal NodestinationPort Destination Port Numeric NodestinationPrecedence Destination Precedence Numeric NodestinationPrecedanceOnly Destination Precedence Numeric Nodestinationv6 IPv6 Destination Hexadecimal NofirstPacketTime First Packet Time Numeric NoflowBias Flow Bias String NoflowDirection Flow Direction:

• Local-to-Local (L2L)• Local-to-Remote (L2R)• Remote-to-Local (R2L)• Remote-to-Remote (R2R)

String No

flowSource Flow Source Numeric NoflowType Flow Type Numeric Yesgeographic Matches Geographic Location String NohasDestinationPayload Has Destination Payload Boolean NohasSourcePayload Has Source Payload Boolean NoicmpType ICMP Type/Code Numeric Nointerface Flow Interface String NolastPacketTime Last Packet Time Numeric NopacketsIn Packets In Numeric NopacketsOut Packets Out Numeric Noprotocol Protocol String Yes

Table 1-2 Supported Flow and Event Fields (continued)

Table Field Name Description Field TypeIdentified in idlist.sh?

AQL Event and Flow Query CLI Guide

Page 11: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

Using a Select Statement 7

remoteHost Remote Host String NoremoteNet Matches Remote Network String NoremoteServices Matches Remote Service String NosourceASN Source ASN Numeric NosourceAssetName Source Asset Name String NosourceByteRatio Source Byte Ratio Numeric NosourceBytes Source Bytes Numeric NosourceDSCP Source DSCP Numeric NosourceDscpOnly Source DSCP Numeric NosourceFlags Source Flags Numeric NosourceIP Source IP String NosourceIPSearch Source IP Search String NosourceIfIndex Source If Index Numeric NosourceNetwork Source Network String NosourceOrDestinationIP Source or Destination IP String NosourcePackets Source Packets Numeric NosourcePacketRatio Source Packet Ratio Numeric NosourcePayload Source Payload String NosourcePayloadHex Source Payload As Hex Hexadecimal NosourcePort Source Port Numeric YessourcePrecedence Source Precedence Numeric NosourcePrecedanceOnly Source Precedence Numeric Nosourcev6 IPv6 Source Hexadecimal NototalBytes Total Bytes Numeric NototalPackets Total Packets Numeric NoviewObjectPair View/Object String No

Events category Low Level Category Numeric YescreEventList Matched Custom Rule String Nocredibility Credibility Numeric NodestinationAssetName Destination Asset Name String NodestinationIP Destination IP String NodestinationMAC Destination MAC Hexadecimal NodestinationNetwork Destination Network String NodestinationPort Destination Port Numeric Nodestinationv6 IPv6 Destination Hexadecimal No

Table 1-2 Supported Flow and Event Fields (continued)

Table Field Name Description Field TypeIdentified in idlist.sh?

AQL Event and Flow Query CLI Guide

Page 12: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

8 USING THE AQL QUERY CLI

device Log Source Numeric YesdeviceGroup Log Source Group Numeric NodeviceTime Log Source Time Numeric NodeviceType Log Source Type Numeric Yesduration Duration Numeric NoendTime End Time Numeric NoeventCount Event Count Numeric NoeventDirection Event Direction:

• Local-to-Local• Local-to-Remote• Remote-to-Local• Remote-to-Remote

String No

eventProcessor Event Processor Numeric YeshasOffense Associated With Offense Boolean NohighLevelCategory High Level Category Numeric YesisCREEvent Is CRE Event Boolean NoidentityExtendedField Identity Extended Field String NoidentityGroupName Identity Group Name String NoidentityHostName Identity Host Name String NoidentityIP Identity IP String NoidentityMAC Identity MAC Hexadecimal NoidentityNetBiosName Identity NetBIOS Name String NoidentityUserName Identity User Name String Nomagnitude Magnitude Numeric Nopayload Payload String NopayloadHex Payload As Hex Hexadecimal NopostNatDestinationIP Post NAT Destination IP String NopostNatDestinationPort Post NAT Destination Port Numeric NopostNatSourceIP Post NAT Source IP String NopostNatSourcePort Post NAT Source Port Numeric NopreNatDestinationIP Pre NAT Destination IP String NopreNatDestinationPort Pre NAT Destination Port Numeric NopreNatSourceIP Pre NAT Source IP String NopreNatSourcePort Pre NAT Source Port Numeric Noprotocol Protocol String No

Table 1-2 Supported Flow and Event Fields (continued)

Table Field Name Description Field TypeIdentified in idlist.sh?

AQL Event and Flow Query CLI Guide

Page 13: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

Using a Select Statement 9

You can also use CIDR-based queries using the select statement. To query by source IP address (sourceIP) or by destination IP address (destinationIP) using a CIDR, use the following format:

select <query item> from <flows|events> where <sourceCIDR|destinationCIDR> = ‘<CIDR Range>’

For example:

select * from flows where sourceCIDR = '10.100.100/24'

This command returns all flows coming from the 10.100.100 subnet. To capture flows coming from and into the subnet, use the regular OR expression as follows:select * from flows where sourceCIDR = '10.100.100/24' OR destinationCIDR = '10.100.100/24'

For more information on flow or event fields described using idlist.sh, see Identifying AQL Fields.

You can use the following statements with any statement to assist you to filter AQL queries.

qid Event Name ID Numeric Yesrelevance Relevance Numeric Noseverity Severity Numeric NosourceAssetName Source Asset Name String NosourceIP Source IP String NosourceMAC Source MAC Hexadecimal NosourceNetwork Source Network String NosourcePort Source Port Numeric Nosourcev6 IPv6 Source Hexadecimal NostartTime Start Time Numeric Notoken Token Associated With Offense Numeric Nounparsed Event Is Unparsed Boolean NouserName Username String No

Table 1-2 Supported Flow and Event Fields (continued)

Table Field Name Description Field TypeIdentified in idlist.sh?

Table 1-3 Fields Used with Any Statement Type

Table Field Name Description Field TypeFlow anyASN Source or Destination ASN Numeric

anyHost Source or Destination IP String

AQL Event and Flow Query CLI Guide

Page 14: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

10 USING THE AQL QUERY CLI

Using Where Clauses

You can restrict your AQL queries using where clauses. The supported logical operators in the clause include and, OR, and parentheses. AQL queries also support relational operators include, =, <, >, >=, <=, and !=.

For example:select sourceIP, category, credibility from events where severity > 9 and category = 5013

select sourceIP, category, credibility from events where (severity > 9 and category = 5013) or (severity < 5 and credibility > 8)

The where clause also supports the arieltime variable, which overrides the time settings passed to the AQL CLI. The arieltime variable must be used with the between keyword to specify the start and end time bounds of the query. All time constraints must be entered as either UNIX timestamps or formatted date/time strings.

You can only use the arieltime variable once in a single query. Therefore, you can only query a continuous span of time in a single AQL command.

The logical operator for the arieltime variable and the remainder of the where clause should be the and operator. We recommend that you use the arieltime variable as the last constraint of the query and the and operator between the arieltime variable and the rest of the where clause.

anyNetwork Source or Destination Network

String

destinationTOS Destination QoS CompositeicmpType ICMP Type/Code NumericsourceOrDestinationIP Source or Destination IP StringsourceTOS Source QoS Composite

Events anyIP Any IP StringanyMac Source or Destination MAC

AddressHex

anyPort Any Port NumericsourceOrDestinationIP Source or Destination IP StringsourceOrDestinationNetwork Source or Destination

NetworkString

sourceOrDestinationPort Source or Destination Port Numeric

Table 1-3 Fields Used with Any Statement Type (continued)

Table Field Name Description Field Type

AQL Event and Flow Query CLI Guide

Page 15: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

Using the Group By Clause 11

Using the Group By Clause

You can use the group by clause to aggregate your data. Typically, data aggregation is combined with arithmetic functions on remaining columns to provide meaningful results of the aggregation. For example, to enter a query to investigate the IP addresses that sent more than 1 million bytes within all flows in a specific time frame, you must enter:

select sourceIP, SUM(sourceBytes) from flows where sourceBytes > 1000000 group by sourceIP

The output resembles:-----------------------------------| sourceIP | SUM_sourceBytes |-----------------------------------| 64.124.201.151 | 4282590.0 || 10.105.2.10 | 4902509.0 || 10.103.70.243 | 2802715.0 || 10.103.77.143 | 3313370.0 || 10.105.32.29 | 2467183.0 || 10.105.96.148 | 8325356.0 || 10.103.73.206 | 1629768.0 |-----------------------------------

However, if you compare this information to a non-aggregate query, the output displays all the IP addresses that are unique:

select sourceIP, sourceBytes from flows where sourceBytes > 1000000

------------------------------| sourceIP | sourceBytes |------------------------------| 64.124.201.151 | 1448629 || 10.105.2.10 | 2412426 || 10.103.70.243 | 1793095 || 10.103.77.143 | 1449148 || 10.105.32.29 | 1097523 || 10.105.96.148 | 4096834 || 64.124.201.151 | 2833961 || 10.105.2.10 | 2490083 || 10.103.73.206 | 1629768 || 10.103.70.243 | 1009620 || 10.105.32.29 | 1369660 || 10.103.77.143 | 1864222 || 10.105.96.148 | 4228522 |------------------------------

In addition to the sum operator, the min, max, avgnonzero and avg arithmetic aggregation functions are also supported.

AQL Event and Flow Query CLI Guide

Page 16: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

12 USING THE AQL QUERY CLI

For example:

To view the maximum number of events.

select max(eventCount) from events

To see the number of average events from a source IP.

select avg(eventCount) from events group by sourceIP

Using the Order By Clause

You can add a single order by clause to the end of your AQL CLI query. Only one field can be used in the order by clause. Also, sorting can be switched between ascending or descending by appending the asc or desc keyword to the order by clause, respectively. By default, the query returns results in descending order.

For example:select sourceBytes, sourceIP from flows where sourceBytes > 1000000 order by sourceBytes

To display results in ascending order:select sourceBytes, sourceIP from flows where sourceBytes > 1000000 order by sourceBytes asc

Combing the group by and the order by clauses in a single query is useful for creating data, such as, TopN lists to determine the most abnormal events or the most bandwidth intensive IP addresses. For example, the following query displays the top traffic intensive IP address in a descending order:

select sourceIP, sum(sourceBytes) from flows group by sourceIP order by sum(sourceBytes) desc

Using the Count(*) Clause

You can use the count(*) clause to count the number of records matching your query. For example, if you want to count all events with credibility equal to or greater than 9:

select count(*) from events where credibility >= 9

Using the Distinct Clause

You can use the distinct clause to select unique rows based on a column or a group of columns. This clause is similar to the group by clause, however, the distinct clause ensure ANSI SQL compatibility. For example:

select distinct sourceIP, sourcePort from flows where sourceBytes > 1000000

AQL Event and Flow Query CLI Guide

Page 17: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

Using the Count (Distinct ...) Clause 13

Using the Count (Distinct ...) Clause

You can use the standard SQL Count(Distinct ...) clause to obtain unique counts. Using the AQL CLI, you can only use one field. For example, if you want to view all the IP addresses that are connected to a specific IP address over time:select count(distinct sourceIP) from flows where destinationIP = '192.168.61.71'

Or, if you want to view the number of unique source IP addresses communicating with a particular destination IP address:select destinationIP, count(distinct sourceIP) from flows group by destinationIP

Using UniqueCount also returns unique counts for items in the table:

select destinationIP, UniqueCount(sourceIP) from flows

Note: Using this clause may require additional system resources. Therefore, depending on the query, the amount of time to return results may vary.

Using the Materialize View Clause

The materialize view clause allows you to produce query results as a virtual table and run subsequent queries against the view. You can also specify the period of time that the materialized view is accessible.

The syntax for the materialized view includes:

materialize view <time> NameOfView as select <statement>

Where:

• <time> specifies the time you want the materialized view to be accessible.

• <statement> specifies a valid select statement.

For example, if you want to create a materialized view containing flows with more than 1,000,000 source bytes, enter the following:

materialize view LargeSourceBytesFlows as select * from flows where sourceBytes >1000000

To select from this view, enter the select statement as you would a valid table:

select * from LargeSourceBytesFlows

You can also use an aggregation statement on a materialized view:

select sourceIP, sum(sourceBytes) from LargeSourceBytesFlows group by sourceIP

Note: You cannot create a materialized view statement based on a previously created materialized view.

AQL Event and Flow Query CLI Guide

Page 18: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

14 USING THE AQL QUERY CLI

If you want to create a materialized view to select from a record set with ambiguous column names, you can define aliases for all computed columns. For example:

materialize view MyView as select sourceIP, sum(sourceBytes) as srcBytesSum from flows group by sourceIP

Then you can refer to the alias in a subsequent query against MyView:

select * from MyView orderBy srcBytesSum

Using the Like Clause

You can search text fields using the standard like clause. You can also use the two wild card options supported by the AQL Query CLI: % and _. The percentage (%) wild card option matches zero or more characters while the _ wild card option only matches one character.

For example:

To match names such as Joe, Joanne, Joseph, or any other name beginning with Jo, enter the following clause:select * from events where userName like ‘jo%’

If you want to match names beginning with Jo that are three characters long, such as, Joe or Jon, enter the following clause:select * from events where userName like ‘jo_’

You can enter the wild card option at any point in the command. For example:select * from flows where sourcePayload like ‘%xyz’

select * from events where payload like ‘%xyz%’

select * from events where payload like ‘_yz’

Using the Boolean Clause

Boolean clauses allow you to restrict your AQL queries to return data matching values you specify using true or false with relational operators. AQL queries support the following relational operators, = and !=.

For example:To sort events that are unparsed, enter the following clause:select * from events where payload = “false”

If you want to sort flows to find a specific source IP address that has an offense, enter the following clause:select * from events where sourceIP = '231.12.37.17' and hasOffense = “true”

If you want a list of source IP addresses and protocols that have an offense you can use BooleanTrueCount, and enter the following clause:

AQL Event and Flow Query CLI Guide

Page 19: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

Identifying AQL Fields 15

select sourceIP, protocol, BooleanTrueCount(hasOffense) from events group by sourceIP

---------------------------------| sourceIP | protocol |---------------------------------| 64.124.201.151 | TCP.tcp.ip || 10.105.2.10 | UDP.udp.ip || 10.103.70.243 | UDP.udp.ip || 10.103.77.143 | UDP.udp.ip || 10.105.32.29 | TCP.tcp.ip || 10.105.96.148 | TCP.tcp.ip || 64.124.201.151 | TCP.tcp.ip || 10.105.2.10 | ICMP.icmp.ip |---------------------------------

Identifying AQL Fields

AQL queries to event or flow fields may return numeric codes making query responses or searches difficult. The shell script idlist.sh provides additional information from numeric AQL fields in the events and flows table.

To access the shell script idlist.sh:Step 1 Log in to STRM, as root.

Step 2 Enter the following command:/opt/qradar/bin/idlist.sh <-e or -f> <field name>

Step 3 Enter the appropriate parameters:

Note: You can search the idlist.sh output using standard operators such as fowardslash. Search terms are case sensitive in the idlist.sh output.

Table 1-4 AQL Field Parameters

Parameters Description-f Specify that the script is to identify a field from the flows table.

For example:/opt/qradar/bin/idlist.sh -f application

-e Specify that the script is to identify a field from the events table. For example:/opt/qradar/bin/idlist.sh -e category

<field name> Specify the field name of the event or flow you want to identify. See Table 1-2 for a complete list of fields that can be identified using idlist.sh.

AQL Event and Flow Query CLI Guide

Page 20: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

16 USING THE AQL QUERY CLI

Listing AQL Fields AQL supports a large number of field names for the events and flows table. To view a list of the most commonly used field names for a select statement, you can use the describe command.

To access the AQL describe command:

Step 1 Log in to STRM, as root.

Step 2 Enter the following command:/opt/qradar/bin/arielClient

The Query prompt appears. Step 3 Enter the following command to view a list of fields from the flows table:

describe flows

The following flows are listed.

Table 1-5 Describe Flows

Table Field Name Description Field TypeFlow application Application String

destinationASN Destination ASN NumericdestinationBytes Destination Bytes NumericdestinationDSCP Destination DSCP NumericdestinationFlags Destination Flags NumericdestinationIP Destination IP StringdestinationIfIndex Destination If Index NumericdestinationNetwork Destination Network StringdestinationPackets Destination Packets NumericdestinationPayload Destination Payload StringdestinationPort Destination Port NumericdestinationPrecedence Destination Precedence NumericdestinationTOS Destination QOS CompositefirstPacketTime First Packet Time NumericflowBias Flow Bias StringflowDirection Flow Direction:

• Local-to-Local (L2L)• Local-to-Remote (L2R)• Remote-to-Local (R2L)• Remote-to-Remote (R2R)

String

flowSource Flow Source NumericflowType Flow Type Numeric

AQL Event and Flow Query CLI Guide

Page 21: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

Listing AQL Fields 17

Step 4 Enter the following command to view a list of fields from the events table:describe events

The following events are listed.

geographic Matches Geographic Location

String

icmpType ICMP Type/Code Numericinterface Flow Interface StringintervalId Interval ID NumericlastPacketTime Last Packet Time NumericprotocolId Protocol ID NumericsourceASN Source ASN NumericsourceBytes Source Bytes NumericsourceDSCP Source DSCP NumericsourceFlags Source Flags NumericsourceIP Source IP StringsourceIfIndex Source If Index NumericsourceNetwork Source Network StringsourcePackets Source Packets NumericsourcePayload Source Payload StringsourcePort Source Port NumericsourcePrecedence Source Precedence NumericsourceTOS Source QOS Compositetoken Associated With Offense Numeric

Table 1-5 Describe Flows (continued)

Table Field Name Description Field Type

Table 1-6 Describe Events

Table Field Name Description Field TypeEvent category Low Level Category Numeric

credibility Credibility NumericdestinationIP Destination IP StringdestinationMAC Destination MAC HexadecimaldestinationNetwork Destination Network StringdestinationPort Destination Port Numericdevice Log Source NumericdeviceGroup Log Source Group NumericdeviceType Log Source Type Numericduration Duration Numeric

AQL Event and Flow Query CLI Guide

Page 22: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

18 USING THE AQL QUERY CLI

endTime End Time NumericeventCount Event Count NumericeventDirection Event Direction:

• Local-to-Local• Local-to-Remote• Remote-to-Local• Remote-to-Remote

String

hasIdentity Has Identity BooleanhasOffense Associated With Offense BooleanhighLevelCategory High Level Category NumericidentityExtendedField Identity Extended Field StringidentityGroupName Identity Group Name StringidentityHostName Identity Host Name StringidentityIP Identity IP StringidentityMAC Identity MAC HexadecimalidentityNetBiosName Identity NetBIOS Name StringidentityUserName Identity User Name Stringmagnitude Magnitude Numericpayload Payload StringpostNatDestinationIP Post NAT Destination IP StringpostNatDestinationPort Post NAT Destination Port NumericpostNatSourceIP Post NAT Source IP StringpostNatSourcePort Post NAT Source Port NumericpreNatDestinationIP Pre NAT Destination IP StringpreNatDestinationPort Pre NAT Destination Port NumericpreNatSourceIP Pre NAT Source IP StringpreNatSourcePort Pre NAT Source Port Numericprotocol Protocol Stringqid Event Name ID Numericrelevance Relevance Numericseverity Severity NumericsourceIP Source IP StringsourceMAC Source MAC HexadecimalsourceNetwork Source Network StringsourcePort Source Port NumericstartTime Start Time Numeric

Table 1-6 Describe Events (continued)

Table Field Name Description Field Type

AQL Event and Flow Query CLI Guide

Page 23: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response

Listing AQL Fields 19

token Token Associated With Offense

Numeric

unparsed Event Is Unparsed BooleanuserName Username String

Table 1-6 Describe Events (continued)

Table Field Name Description Field Type

AQL Event and Flow Query CLI Guide

Page 24: AQL Flow and Event Query CLI Guide - Juniper Networks Networks, Inc. 1194 North Mathilda Avenue Sunnyvale, CA 94089 USA 408-745-2000 Published: 2010-11-16 Security Threat Response