Web Application Firewall Test Report - FortiWeb 3000E · PDF fileNSS Labs Web Application Firewall Test Report – Fortinet FortiWeb-3000E v5.5.5 Web Application Firewall Test Report_Fortinet

WEB APPLICATION FIREWALL TEST REPORT

Fortinet FortiWeb-3000E v5.5.5

APRIL 11, 2017

Author – Matthew Chips

NSS Labs Web Application Firewall Test Report – Fortinet FortiWeb-3000E v5.5.5

Web Application Firewall Test Report_Fortinet FortiWeb-3000E_041117 2

Overview NSS Labs performed an independent test of the Fortinet FortiWeb-3000E v5.5.5. The product was subjected to

thorough testing at the NSS facility in Austin, Texas, based on the Web Application Firewall Methodology v2.1,

which is available at www.nsslabs.com. This test was conducted free of charge and NSS did not receive any

compensation in return for Fortinet’s inclusion.

This report provides detailed information about this product and its security effectiveness, performance, and TCO.

Additional comparative information is available at www.nsslabs.com.

NSS testing has found that the majority of web application firewalls (WAFs) operate in an adaptive learning mode

(i.e.,“learning mode”). In this mode, a WAF learns the behavior of applications and automatically generates policy

recommendations. These recommendations require review and approval before the WAF is deployed. Periodic

manual tuning may also be required.

As part of the initial WAF test setup, devices are configured on site by the vendor to protect the target websites,

either by “training” the device—walking through the e-commerce sites (automatically, or manually), or by

manually creating rule sets and a security policy. NSS considers it unacceptable for a product of this nature to be

sold without some standard approach and/or recommended settings, or without consultancy included to create a

policy specific to the target environment. The product version tested must be available to the general public at the

time of testing. This provides readers with the most useful information on key WAF security effectiveness and

performance capabilities based upon their expected usage. Figure 1 presents the overall results of the tests.

Product OWASP Top 10 NSS-Tested Throughput


Blocked 100% in 9 out of 10 categories 44,120 CPS

Block Rate1 Stability and Reliability

98.00%

PASS

Figure 1 – Overall Test Results

Using a tuned policy, the Fortinet FortiWeb-3000E blocked 100% of attacks in 9 out of 10 OWASP categories,

achieving an overall block rate of 98.00%. The device proved effective against all evasion techniques tested. The

device also passed all stability and reliability tests.

The Fortinet FortiWeb-3000E is rated by NSS at 44,120 connections per second (CPS). This is a minimum rating that

uses one transaction per connection. Fortinet rates this device at 5 Gbps, which would be 25,000 CPS at a 21 KB

object size. NSS-tested capacity is an average of all of the HTTP response-based capacity tests. These performance

numbers represent a baseline, which an enterprise can use to model its environment.

1 Block rate is defined as the number of attacks blocked under test.

http://www.nsslabs.com./

http://www.nsslabs.com/



Table of Contents

Overview ............................................................................................................................................................... 2

Security Effectiveness ............................................................................................................................................ 5

Attack Types .................................................................................................................................................................. 5

OWASP Top 10 ........................................................................................................................................................... 5

Weak Authentication and Session Management....................................................................................................... 6

Cross-Site Scripting .................................................................................................................................................... 7

Insecure Direct Object Reference ............................................................................................................................... 8

Security Misconfiguration .......................................................................................................................................... 8

Sensitive Data Exposure ............................................................................................................................................. 8

Missing Function-Level Access Control ...................................................................................................................... 8

Cross-Site Request Forgery ........................................................................................................................................ 8

Using Components with Known Vulnerabilities ......................................................................................................... 9

Unvalidated Redirects and Forwards ......................................................................................................................... 9

Performance ........................................................................................................................................................ 10

Maximum Capacity ...................................................................................................................................................... 10

HTTP Capacity without Caching and without Transaction Delays ............................................................................... 11

HTTP Capacity without Caching and with Transaction Delays ..................................................................................... 12

Stability and Reliability ........................................................................................................................................ 13

Total Cost of Ownership (TCO) ............................................................................................................................. 14

Installation Hours ........................................................................................................................................................ 14

Total Cost of Ownership .............................................................................................................................................. 15

Appendix A: Product Scorecard ........................................................................................................................... 16

Test Methodology ................................................................................................................................................ 20

Contact Information ............................................................................................................................................ 20



Table of Figures

Figure 1 – Overall Test Results ....................................................................................................................................... 2

Figure 2 – OWASP Category – Injection ......................................................................................................................... 5

Figure 3 – Weak Authentication and Session Management .......................................................................................... 6

Figure 4 – Cross-Site Scripting ....................................................................................................................................... 7

Figure 5 – Insecure Direct Object Reference ................................................................................................................. 8

Figure 6 – Security Misconfiguration ............................................................................................................................. 8

Figure 7 – Sensitive Data Exposure ................................................................................................................................ 8

Figure 8 – Missing Function-Level Access Control ......................................................................................................... 8

Figure 9 – Cross-Site Request Forgery ........................................................................................................................... 8

Figure 10 – Using Components with Known Vulnerabilities .......................................................................................... 9

Figure 11 – Unvalidated Redirects and Forwards .......................................................................................................... 9

Figure 12 – Concurrency and Connection Rates .......................................................................................................... 10

Figure 13 – HTTP Capacity without Caching and without Transaction Delays Tests ................................................... 11

Figure 14 – HTTP Capacity without Caching and with Transaction Delays .................................................................. 12

Figure 15 – Stability and Reliability Results ................................................................................................................. 13

Figure 16 – Sensor Installation Time (Hours) ............................................................................................................... 14

Figure 17 – 3-Year TCO (US$)....................................................................................................................................... 15

Figure 18 – Scorecard .................................................................................................................................................. 19



Security Effectiveness This section verifies that the device under test is capable of detecting, preventing, and logging attack attempts

accurately, while remaining resistant to false positives.

Attack Types

NSS testing demonstrates the effectiveness of the WAF in protecting vulnerable web application servers against

targeted exploitation. This asset/target and threat-based approach forms the basis on which the security

effectiveness of the device is measured.

The NSS Exploit Library for WAF contains publically available exploits (including multiple variants of each exploit)

and a number of complex web applications that have been constructed to include known vulnerabilities and

coding errors. It has been validated that each exploit impacts the target vulnerable host(s) by compromising either

the underlying OS, the web server, or the web application itself. A compromise may include executing a denial-of-

service (DoS); providing administrator/root access to the host server; allowing malicious users to amend system

parameters or application data before submission; giving the attacker the ability to browse and/or retrieve files

stored on the host server; escalating user privileges.

OWASP Top 10

The OWASP Top 10 represents a broad industry consensus about the most critical web application security flaws.

For details, please see the Test Methodology available at www.nsslabs.com.

Attack Type Results

SQL Injection

Injection Search box – GET PASS

Injection Malicious Character PASS

Injection in URL – GET PASS

Injection Search box – POST PASS

Injection Login Form – POST PASS

Injection User Agent PASS

Injection Stored Blog PASS

Injection Blind Boolean-Based PASS

SQLMap Attack 1 PASS

Attack 2 PASS

XML Injection PASS

SSI Injection PASS

XPATH Injection PASS

Code Injection PASS

Command Injection PASS

Figure 2 – OWASP Category – Injection




Weak Authentication and Session Management

Attack Type Results

Privilege Escalation Admin param in URL PASS

Privilege Escalation Admin param in Burp param PASS

Session Fixation back button after log out PASS

Session Timeout PASS

Figure 3 – Weak Authentication and Session Management



Cross-Site Scripting

Attack Type Results

Reflected GET Malformed img tag 1 PASS

Malformed img tag 2 PASS

IMG on ERROR and JavaScript alert encode PASS

Extraneous open brackets PASS

Escaping escapes PASS

SVG object tag PASS

Body Tag PASS

iFrame PASS

Reflected POST






SVG object tag PASS

Body Tag PASS

URL Encoding PASS

Base64 Encoding PASS

Reflected User Agent (Intercept on)






SVG object tag PASS

Body Tag PASS

Stored User Agent






SVG object tag PASS

Body Tag PASS

HTML Injection

Injected blog PASS

Injected GET PASS

Injected POST PASS

Normal iFrame PASS

Encoded iFrame URL PASS

Reflected Standard URL PASS

Reflected Encoded URL PASS

Figure 4 – Cross-Site Scripting



Insecure Direct Object Reference

Attack Type Results

Insecure Direct Object Reference Change Secret PASS

Change Ticket Price PASS

Local and Remote File Inclusion PASS

Figure 5 – Insecure Direct Object Reference

Security Misconfiguration

Attack Type Results

Fingerprint Web Server PASS

Fingerprint Web Application Framework: PASS

HTTP Methods PASS

Server-Side Request Forgery Attack 1 PASS

Attack 2 PASS

Figure 6 – Security Misconfiguration

Sensitive Data Exposure

Attack Type Results

Insufficient TLS PASS

Heartbleed PASS

Figure 7 – Sensitive Data Exposure

Missing Function-Level Access Control

Attack Type Results

Directory Traversal/File Include File Traversal PASS

Directory Traversal PASS

Figure 8 – Missing Function-Level Access Control

Cross-Site Request Forgery

Attack Type Results

CSRF Change Password PASS

CSRF Transfer Amount PASS

Figure 9 – Cross-Site Request Forgery



Using Components with Known Vulnerabilities

Attack Type Results

Denial-of-Service XML DoS PASS

Nginx DoS PASS

Shellshock PASS

PHP CGI Remote Code Execution Code Disclosure FAIL

Remote Code Execution PASS

Figure 10 – Using Components with Known Vulnerabilities

Unvalidated Redirects and Forwards

Attack Type Results

Client-Side URL Redirect Redirect and Forward 1 PASS

Redirect and Forward 2 PASS

Figure 11 – Unvalidated Redirects and Forwards



Performance There is frequently a trade-off between security effectiveness and performance. Because of this trade-off, it is

important to judge a product’s security effectiveness within the context of its performance and vice versa. This

ensures that new security protections do not adversely impact performance and that security shortcuts are not

taken to maintain or improve performance.

Maximum Capacity

The use of traffic generation equipment allows NSS engineers to create true “real-world” traffic at multi-gigabit

speeds as a background load for the tests.

The purpose of these tests is to stress the inspection engine and determine how it handles high volumes of

application layer transactions per second, and concurrent open connections. All packets contain valid payload and

address data, and these tests provide an excellent representation of a live network at various

connection/transaction rates.

Note that in all tests the following critical “breaking points”—where the final measurements are taken—are used:

Excessive concurrent HTTP connections – Latency within the WAF is causing excessive delays and increased

response time.

Unsuccessful HTTP transactions – Normally, there should be zero unsuccessful transactions. Once these

appear, it is an indication that excessive latency within the WAF is causing connections to time out.

Figure 12 depicts the results of the maximum capacity tests.

Figure 12 – Concurrency and Connection Rates



HTTP Capacity without Caching and without Transaction Delays

The aim of these tests is to stress the HTTP detection engine and determine how the device copes with network

loads of varying average packet size and varying connections per second. By creating genuine session-based traffic

with varying session lengths, the device is forced to track valid TCP sessions, thus ensuring a higher workload than

for simple packet-based background traffic. This provides a test environment that simulates real-world HTTP

transactions in the lab, while ensuring absolute accuracy and repeatability.

Each transaction consists of a single HTTP GET request and there are no transaction delays (i.e., the web server

responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and

address data, and this test provides an excellent representation of a live network (albeit one biased toward HTTP

traffic) at various network loads.

Figure 13 depicts the results for the HTTP capacity without caching and without transaction delays tests.

Figure 13 – HTTP Capacity without Caching and without Transaction Delays Tests



HTTP Capacity without Caching and with Transaction Delays

Typical user behavior introduces delays between requests and responses; for example, “think time,” as users read

web pages and decide which links to click next. This group of tests is identical to the previous group except that

these include a five-second delay in the server response for each transaction. This has the effect of maintaining a

high number of open connections throughout the test, thus forcing the device to utilize additional resources to

track those connections. Figure 14 depicts the results for the HTTP capacity without caching and with transaction

delays test.

Figure 14 – HTTP Capacity without Caching and with Transaction Delays



Stability and Reliability Long-term stability is particularly important for an inline device, where failure can produce network outages. These

tests verify the stability of the device along with its ability to maintain security effectiveness while under normal

load and while passing malicious traffic. Products that cannot sustain legitimate traffic (or that crash) while under

hostile attack will not pass.

The device is required to remain operational and stable throughout these tests, and to block 100% of previously

blocked traffic, raising an alert for each. If any non-allowed traffic passes successfully, caused either by the volume

of traffic or by the device failing open for any reason, the device will fail the test. Figure 15 depicts the results of

the tests for stability and reliability.

Stability and Reliability Results

Blocking under Extended Attack PASS

Passing Legitimate Traffic under Extended Attack PASS

Protocol Fuzzing and Mutation PASS

Power Fail PASS

Persistence of Data PASS

Figure 15 – Stability and Reliability Results

These tests also determine the behavior of the state engine under load. All WAF devices must choose whether to

risk denying legitimate traffic or allowing malicious traffic once they run low on resources. Dropping new

connections when resources (such as state table memory) are low, or when traffic loads exceed the device

capacity will theoretically block legitimate traffic but maintain state on existing connections (preventing attack

leakage).



Total Cost of Ownership (TCO) Organizations should be concerned with the ongoing amortized cost of operating security products. This section

evaluates the costs associated with the purchase, installation, and ongoing management of the device, including:

Product Purchase – The cost of acquisition

Product Maintenance – The fees paid to the vendor (including software and hardware support, maintenance,

and updates)

Installation – The time required to take the device out of the box, configure it, deploy it into the network,

apply updates and patches, perform initial tuning, and set up desired logging and reporting

Upkeep – The time required to apply periodic updates and patches from vendors, including hardware,

software, and firmware updates

For TCO analysis, refer to the TCO Comparative Report, which is available at www.nsslabs.com.

Installation Hours

This table depicts the number of hours of labor required to install each device using only local device management

options. The table accurately reflects the amount of time that NSS engineers, with the help of vendor engineers,

needed to install and configure the device to the point where it operated successfully in the test harness, passed

legitimate traffic, and blocked and detected prohibited or malicious traffic. This closely mimics a typical enterprise

deployment scenario for a single device.

The installation cost is based on the time that an experienced security engineer would require to perform the

installation tasks described above. This approach allows NSS to hold constant the talent cost and measure only the

difference in time required for installation. Readers should substitute their own costs to obtain accurate TCO

figures.

Product Installation (Hours)


8

Figure 16 – Sensor Installation Time (Hours)




Total Cost of Ownership

Calculations are based on vendor-provided pricing information. Where possible, the 24/7 maintenance and

support option with 24-hour replacement is utilized, since this is the option typically selected by enterprise

customers. Prices are for single device management and maintenance only; costs for central management

solutions (CMS) may be extra.

Product Purchase

Price Maintenance

/Year Year 1 Cost

Year 2 Cost

Year 3 Cost

3-Year TCO


$44,997 $20,998 $20,998 $20,998 $20,998 $108,591

Figure 17 – 3-Year TCO (US$)

Year 1 Cost is calculated by adding installation costs (US$75 per hour fully loaded labor x installation time) +

purchase price + first-year maintenance/support fees.

Year 2 Cost consists only of maintenance/support fees.

Year 3 Cost consists only of maintenance/support fees.

For additional TCO analysis, including for the CMS, refer to the TCO Comparative Report.



Appendix A: Product Scorecard Description Results

Security Effectiveness

OWASP Top 10 98.00%

Injection 100.00%

SQL Injection

injection Search box – GET PASS

Injection Malicious Character PASS

Injection in URL – GET PASS

Injection Search box – POST PASS

injection Login Form – POST PASS

Injection User Agent PASS

Injection Stored Blog PASS

Injection Blind Boolean-Based PASS

SQLMap

Attack 1 PASS

Attack 2 PASS

XML Injection PASS

SSI Injection PASS

XPATH Injection PASS

Code Injection PASS

Command Injection PASS

Weak Authentication and Session Management 100.00%

Privilege Escalation Admin param in URL PASS

Privilege Escalation Admin param in Burp param PASS

Session Fixation back button after log out PASS

Session Timeout PASS

Cross-Site Scripting 100.00%

Reflected GET






SVG object tag PASS

Body Tag PASS

iFrame PASS



Reflected POST



IMG on ERROR and javascript alert encode PASS



SVG object tag PASS

Body Tag PASS

URL Encoding PASS

Base64 Encoding PASS

Reflected User Agent (Intercept on)






SVG object tag PASS

Body Tag PASS

Stored User Agent






SVG object tag PASS

Body Tag PASS

HTML Injection

Injected blog PASS

Injected GET PASS

Injected POST PASS

Normal iFrame PASS

Encoded iFrame URL PASS

Reflected Standard URL PASS

Reflected Encoded URL PASS

Insecure Direct Object Reference 100.00%

Change Secret PASS

Change Ticket Price PASS

Local and Remote File Inclusion PASS

Security Misconfiguration 100.00%

Fingerprint Web Server PASS

Fingerprint Web Application Framework: PASS

HTTP Methods PASS



Server-Side Request Forgery

Attack 1 PASS

Attack 2 PASS

Sensitive Data Exposure 100.00%

Insufficient TLS PASS

Heartbleed PASS

Missing Function-Level Access Control 100.00%

Directory Traversal/File Include

File Traversal PASS

Directory Traversal PASS

Cross-Site Request Forgery 100.00%

CSRF Change Password PASS

CSRF Transfer Amount PASS

Using Components with Known Vulnerabilities 80.00%

Denial-of-Service

XML DoS PASS

Nginx DoS PASS

Shellshock PASS

PHP CGI Remote Code Execution

Code Disclosure FAIL

Remote Code Execution PASS

Unvalidated Redirects and Forwards 100.00%

Client-Side URL Redirect



Performance

Maximum Capacity CPS

Maximum HTTP Connections per Second 50,010

Maximum HTTP Transactions per Second 96,360

HTTP Capacity without Caching and without Transaction Delays CPS

44 KB HTTP Response Size – 2500 Connections per Second 25,000



4.5 KB HTTP Response Size – 20000 Connections per Second 49,640

1.7 KB HTTP Response Size – 4000 Connections per Second 59,800

HTTP Capacity without Caching and with Transaction Delays

21 KB HTTP Response Size with Delay 31,530

10 KB HTTP Response Size with Delay 36,370

Stability & Reliability

Blocking Under Extended Attack PASS

Passing Legitimate Traffic Under Extended Attack PASS

Protocol Fuzzing and Mutation – Detection Ports PASS



Power Fail PASS

Persistence of Data PASS

Total Cost of Ownership

Ease of Use

Initial Setup (Hours) 8

Time Required for Upkeep (Hours per Year) Contact NSS Labs

Expected Costs

Initial Purchase (hardware as tested) $44,997

Initial Purchase (enterprise management system) See Comparative

Annual Cost of Maintenance & Support (hardware/software) $20,998

Annual Cost of Maintenance & Support (enterprise management system) See Comparative

Installation Labor Cost (@ US$75/hr) $600

Management Labor Cost (per Year @ US$75/hr) Contact NSS Labs

Total Cost of Ownership (TCO)

Year 1 $66,595

Year 2 $20,998

Year 3 $20,998

3-Year TCO $108,591

Figure 18 – Scorecard



This and other related documents are available at: www.nsslabs.com. To receive a licensed copy or report misuse,

please contact NSS Labs.

© 2017 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval

system, e-mailed or otherwise disseminated or transmitted without the express written consent of NSS Labs, Inc. (“us” or “we”).

Please read the disclaimer in this box because it contains important information that binds you. If you do not agree to these

conditions, you should not read the rest of this report but should instead return the report immediately to us. “You” or “your”

means the person who accesses this report and any entity on whose behalf he/she has obtained this report.

1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it.

2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed. All

use of and reliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or expenses of

any nature whatsoever arising from any error or omission in this report.

3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF

MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED

BY US. IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT

DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE

POSSIBILITY THEREOF.

4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software)

tested or the hardware and/or software used in testing the products. The testing does not guarantee that there are no errors or

defects in the products or that the products will meet your expectations, requirements, needs, or specifications, or that they will

operate without interruption.

5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in

this report.

6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their

respective owners.

Test Methodology

Web Application Firewall (WAF) Test Methodology v2.1

A copy of the test methodology is available at www.nsslabs.com.

Contact Information NSS Labs, Inc.

206 Wild Basin Road

Building A, Suite 200

Austin, TX 78746 USA

[email protected]

www.nsslabs.com



Documents

Web Application Firewall Test Report - FortiWeb 3000E · PDF fileNSS Labs Web Application Firewall Test Report – Fortinet FortiWeb-3000E v5.5.5 Web Application Firewall Test Report_Fortinet