i

Intrusion Prevention Systems: How

do they prevent intrusion?

Phil Baskerville

A thesis submitted for the degree of

Master of Science

at the University of Otago, Dunedin,

New Zealand.

1 March 2006

ii

Abstract

Intrusion Prevention Systems (IPS) are the latest in a line of products created to counter network

attacks. This thesis has explored the history of products manufactured to protect network systems

from attacks. An experiment was conducted to find out what an attack on a systems looked like

and to gauge the success of current IPS software. Results indicated that current IPS were

reasonably effective and that the methodology of attacking a system was predictable, allowing

administrators scope for putting methods in place to counter attacks.

iii

Preface

There are many people to acknowledge as part of this thesis but these three I have singled out. To

my special friend Vivienne, without your help and encouragement this body of work would not be

the same and I may be in a ditch somewhere by now. To my mother who used my thesis as an

excuse to visit, your help was appreciated and I very much enjoyed working with you. A big thank

you goes to my supervisor Dr Hank Wolfe. Thank you all very much.

iv

Contents1. Introduction........................................................................................................12. Firewalls ..............................................................................................................2

2.1. Introduction......................................................................................................................................22.2. Timeline of Firewall Development.................................................................................................32.3. Packet Filter Firewalls.....................................................................................................................42.4. Circuit Level Firewalls....................................................................................................................42.5. Application Layer Firewalls............................................................................................................52.6. Stateful Inspection Firewalls...........................................................................................................72.7. Dynamic Packet Filtering Firewalls................................................................................................82.8. Kernel Proxy Firewalls....................................................................................................................92.9. Summary........................................................................................................................................10

3. Intrusion Detection Systems ...........................................................................113.1. Introduction....................................................................................................................................113.2. First IDS Papers.............................................................................................................................113.3. Host Based IDS .............................................................................................................................123.4. How host-based IDS work ............................................................................................................123.5. Network Based IDS.......................................................................................................................163.6. How Network Based IDS work ....................................................................................................163.7. Comparison....................................................................................................................................19

3.7.1. Strengths.................................................................................................................................................193.7.2. Limitations .............................................................................................................................................20

3.8. Summary........................................................................................................................................21

4. Intrusion Prevention Systems .........................................................................224.1. Introduction....................................................................................................................................224.2. Development..................................................................................................................................234.3. How IPS work ...............................................................................................................................254.4. Issues around IPS...........................................................................................................................284.5. Further Developments ...................................................................................................................294.6. Summary........................................................................................................................................30

5. IPS Vendors ......................................................................................................315.1. Introduction....................................................................................................................................315.2. Network Associates Inc - Entercept Standard Edition .................................................................315.3. McAfee - McAfee IntruShield Global Manager ..........................................................................325.4. Citadel - Hercules ..........................................................................................................................335.5. High Tower Software - TowerView .............................................................................................335.6. Argus Systems - PitBull LX..........................................................................................................345.7. Sana Security - Primary Response................................................................................................345.8. Symantec - Symantec Event Manager ..........................................................................................355.9. Sonic Wall Inc - Sonic WALL......................................................................................................355.10. Okena - StormWatch .....................................................................................................................365.11. Summary........................................................................................................................................37

6. Policy .................................................................................................................386.1. Introduction....................................................................................................................................386.2. What is policy? ..............................................................................................................................386.3. What is in a Policy?.......................................................................................................................396.4. Policy Components........................................................................................................................396.5. Where does policy fit into a Network Defence? ..........................................................................406.6. Example of Network Policy ..........................................................................................................416.7. Summary........................................................................................................................................41

v

7. Security Patches and Management ................................................................427.1. Introduction....................................................................................................................................427.2. What is a Security Patch?..............................................................................................................427.3. How are vulnerabilities discovered?.............................................................................................427.4. Who creates the patches? ..............................................................................................................437.5. Patch Management ........................................................................................................................44

7.5.1. Policy......................................................................................................................................................447.5.2. Identifying Patches ................................................................................................................................447.5.3. Implementation ......................................................................................................................................457.5.4. Maintenance ...........................................................................................................................................45

7.6. Summary........................................................................................................................................46

8. Hackers..............................................................................................................478.1. Introduction....................................................................................................................................478.2. Why do they hack? ........................................................................................................................47

8.2.1. Underlying opposition of institutions and government ........................................................................488.2.2. Monetary Gain .......................................................................................................................................488.2.3. Curiosity .................................................................................................................................................498.2.4. Aspergers Syndrome..............................................................................................................................49

8.3. The different styles of hackers ......................................................................................................508.3.1. Black hat.................................................................................................................................................508.3.2. White Hat ...............................................................................................................................................508.3.3. Grey hat ..................................................................................................................................................518.3.4. Script kiddies..........................................................................................................................................51

8.4. Geographical Location of Hackers ...............................................................................................518.5. Summary........................................................................................................................................52

9. A Common Attack Strategy ............................................................................539.1. Introduction....................................................................................................................................539.2. Vulnerability Assessment..............................................................................................................539.3. Exploitation....................................................................................................................................549.4. Specific Attacks.............................................................................................................................56

9.4.1. DoS Attack:............................................................................................................................................569.4.2. Distributed Denial of Service Attack: ...................................................................................................569.4.3. Social Engineering:................................................................................................................................569.4.4. Brute Force:............................................................................................................................................569.4.5. Buffer Overflow:....................................................................................................................................569.4.6. Packet sniffing: ......................................................................................................................................579.4.7. Cracking: ................................................................................................................................................579.4.8. Spoofing: ................................................................................................................................................579.4.9. DNS Cache Poisoning: ..........................................................................................................................579.4.10. UDP Flood Attack .................................................................................................................................58

9.5. Summary........................................................................................................................................58

10.Crafting a new Attack Strategy ......................................................................5910.1. Introduction....................................................................................................................................5910.2. What is an attack strategy?............................................................................................................5910.3. Identifying a Vulnerability ............................................................................................................5910.4. Vulnerability to Exploit turnaround..............................................................................................6010.5. Turning a vulnerability into an exploit .........................................................................................61

10.5.1. Finding the Bug......................................................................................................................................6110.5.2. Calculate String Length .........................................................................................................................6210.5.3. Prove the buffer length ..........................................................................................................................6210.5.4. Inserting the command that is to be run................................................................................................6310.5.5. Making a Beta Exploit ...........................................................................................................................6310.5.6. Using the Crafted Exploit ......................................................................................................................63

10.6. Summary........................................................................................................................................64

vi

11.Experiment........................................................................................................6511.1. Introduction....................................................................................................................................6511.2. Method ...........................................................................................................................................6511.3. Results............................................................................................................................................68

11.3.1. Foot Printing ..........................................................................................................................................6811.3.2. Scanning.................................................................................................................................................7211.3.3. Enumeration ...........................................................................................................................................7411.3.4. Gaining Access ......................................................................................................................................7611.3.5. Pilfering..................................................................................................................................................81

11.4. Summary........................................................................................................................................84

12.Conclusion.........................................................................................................8513.Bibliography .....................................................................................................8614.Appendix ...........................................................................................................93

14.1. Appendix A: Entercept DataSheet ................................................................................................9314.2. Appendix B: IntruShield DataSheet..............................................................................................9414.3. Appendix C: Hercules DataSheet .................................................................................................9514.4. Appendix D: TowerView DataSheet ............................................................................................9614.5. Appendix E: PitBull LX DataSheet ..............................................................................................9714.6. Appendix F: Primary Response DataSheet...................................................................................9814.7. Appendix G: Event Manager DataSheet.......................................................................................9914.8. Appendix H: SonicWall DataSheet ............................................................................................10014.9. Appendix I: StormWatch DataSheet...........................................................................................10114.10. Appendix J: Anti-Virus Protection Policy ..............................................................................10214.11. Appendix K: Vulnerability Advisory Squid ...........................................................................10314.12. Appendix L: Vulnerability Advisory Microsoft .....................................................................10414.13. Appendix M: Security Patching Policy...................................................................................10514.14. Appendix N: Index.html ..........................................................................................................10614.15. Appendix O: Index.html Source Code ....................................................................................10714.16. Appendix P: Phase One Network Traffic................................................................................10814.17. Appendix Q: db.triumph.com.hosts File .................................................................................10914.18. Appendix R: DNS PTR Scan Traffic ......................................................................................11014.19. Appendix S: DNS A Scan Traffic ...........................................................................................11114.20. Appendix T: Service Ports.......................................................................................................11214.21. Appendix U: Port Scan ............................................................................................................11314.22. Appendix V: Nessus Report ....................................................................................................11414.23. Appendix W: Telnet Login......................................................................................................11514.24. Appendix X: FTP Login Log...................................................................................................11614.25. Appendix Y: Index.html ..........................................................................................................11714.26. Appendix Z: Index.html Source Code.....................................................................................11814.27. Appendix AA: Named.conf File..............................................................................................11914.28. Appendix AB: Named.local File .............................................................................................12014.29. Appendix AC: db.128.64.32.hosts File ...................................................................................121

vii

List of Tables

Table 1 - Experiment Computer Configuration ...........................................................67

Table 2 - Webpage source code ..................................................................................69

Table 3 - Webpage browse times. ...............................................................................69

Table 4 - DNS scan.....................................................................................................71

Table 5 - DNS Scan Times..........................................................................................71

Table 6 - Port Scan .....................................................................................................73

Table 7 - Computer Port Scans....................................................................................74

Table 8 - Nessus Scan.................................................................................................75

Table 9 - Telnet Session..............................................................................................77

Table 10 - Telnet Commands .....................................................................................78

Table 11 - FTP session packet capture. .......................................................................79

Table 12 - FTP Log.....................................................................................................80

Table 13 - Telnet Session Commands .........................................................................82

Table 14 - Telnet Session Packet Capture ...................................................................83

Table 15 - FTP Session Commands ............................................................................83

viii

List of figures

Figure 1: Timeline of Firewall Architectures ................................................................4

Figure 2: Application Layer Firewall Proxy Demonstration ..........................................5

Figure 3 - An IPS monitoring a network .....................................................................23

Figure 4 - An IDS monitoring a network.....................................................................23

Figure 5: Exploit Code Availability. ...........................................................................61

Figure 6: Calculate string length one...........................................................................62

Figure 7: Calculate string length two...........................................................................62

Figure 8: Prove buffer length ......................................................................................63

Figure 9: Insert command. ..........................................................................................63

Figure 10: Network Topology.....................................................................................65

Figure 11: DNS forward lookup. ................................................................................70

Figure 12: DNS reverse lookup...................................................................................70

ix

List of Abbreviations

AD - Anomaly Detection

ApS - Aspergers Syndrome

ASIM - Automated Security Measurement System

ASR - Attack Signature Recognition

CIAC - Computer Incident Advisory Capability

CMDS - Computer Misuse Detection System

COM - Common Operation Models

CSTC - Cyber Solution Tools Centre

DDoS - Distributed Denial of Service

DEC - Digital Equipment Corporation

DIDS - Distributed Intrusion Detection System

DNS - Domain Name Service

DoS - Denial of Service

FDDI - Fibre Distributed Data Interface

FTP - File Transfer Protocol

HTTP - Hypertext Transfer Protocol

ICMP - Internet Control Message Protocol

IDES - Intrusion Detection Expert System

IDS- Intrusion Detection Systems

IEEE - Institute of Electrical and Electronics Engineers

IP - Internet Protocol

IPS - Intrusion Prevention Systems

LDAP - Lightweight Directory Address Protocol

MDC - Microsoft Download Centre

MRSE - Multi-Rule Search Engine

MSRC - Microsoft Security Response Centre

x

NIC - Network Interface Cards

NSM - Network Security Monitor

OS - Operating System

OSI - Open Systems Interconnection

P2P - Peer to Peer

PFA - Protocol Flow Analyser

RIA - Retrospective Intrusion Analysis

RO - Rule Optimiser

RTID - Real-time Intrusion Detection

SAIC - Science Applications International Corporation

SEAL - Secure External Access Link

SG - Statistics Gathering

SMTP - Simple Mail Transfer Protocol

SSH - Secure Shell

SSL - Secure Sockets Layer

TCP - Transport Control Protocol

UDP - User Datagram Protocol

VRM - Vulnerability Risk Model

1

1. Introduction

The first documented Internet worm “attack” was released 3rd November 1988. This attack showed

the need for a certain degree of network defense. Intrusion Prevention Systems (IPS) are the latest

advance in protecting networks from computer aided attack. The previous development of systems

such as Firewalls and Intrusion Detection Systems (IDS) laid a strong foundation for IPS.

A significant part of research in this area has been dedicated to the development and progression

of firewalls and IDS. This research shows that development was responding to the need to fix

problems in previous architectures rather than a fundamental desire to improve a proposed

faultless product.

An IPS is designed to protect a network. It is connected to the network inline so that it can monitor

all sent and received network traffic. An IPS has the ability to allow or deny network traffic. It

chooses whether to do it via attack libraries of known attacks and behavioiurs. An IPS is an

improvement on firewall and IDS technologies as it combines the two on to one device.

To complement the research about the systems themselves, the source and defence of computer

attacks is explored in this study. This examines avenues an organisation can take in order to stop

attacks through implementing strong policies and patch management. The source of an attack is

the hacker. This research evaluates views on what makes a hacker; how a hacker would normally

attack a network, and how new attack strategies are crafted.

With the continued release of insecure operating systems and software, the protection of these

vulnerable products is a persistent problem. The aim of this research is to identify the subtle and

obvious signs of a network under attack. This will be undertaken by conducting an experiment on

an isolated network with existing security holes and analysing the network traffic that is used to

probe and attack the vulnerable computers.

The research aims to answer the question: how does an IPS prevent intrusion? In order to answer

this question the thesis will proceed as follows:

• A review of developments that took place preceding and contributed towards IPS.

• An examination of countermeasures that organisations can implement to complement an IPS.

• Analysis of the types of persons endeavouring to bypass implemented IPS.

• Examination of the methods these people would use and how a new attack strategy is created.

An experiment is then conducted to view hacking methods, followed by countermeasures that if

implemented could reduce the effects of the attack.

2

2. Firewalls

2.1. Introduction

“Internetworks that connect multiple organisations create potential security

problems that cannot be solved simply by internal administrative procedures.

Organisations would like to restrict inter-organisation access to specific restricted

hosts and applications, in order to limit the potential for damage and to reduce the

number of systems that must be secured against attack. One way to restrict access

is to prevent certain packets from entering or leaving an organisation through its

gateways. This paper describes simple, flexible, and moderately efficient

mechanisms for screening the packets that flow through a Unix-based gateway”.1

The preceding paragraph was the abstract of the first recognised paper about firewalls and the

security issues surrounding them. It was presented by Jeffery C. Mogul to the USENIX summer

conference held in Baltimore in 1989.

Before the advent of computers and networking the term “Firewall” was used in a building as “a

fireproof wall used as a barrier to prevent the spread of fire”.2 As the Internet developed from an

academic resource to an open community, the need to stop unwanted attacks on computer

networks arose. The alarming speed of even the earliest attacks was seen as destructive as an

uncontained fire.

“There may be a virus loose on the internet.” These were the words of Andy Sudduth of Harvard,

34 minutes after midnight, November 3rd, 1988. They were spoken as the first documented

internet worm “attack” was beginning. This worm was later named the Morris Worm after its

creator, Robert Tappan Morris, a 23 year old student from Cornell University.

This incident elevated concerns regarding Internet security, and network access became an issue.

Before the worm outbreak there had been research into firewall technology but there had not been

any destructive event demonstrating why it was needed.

1 Mogul, Jeffery C., 1989, Simple and Flexible Datagram Access Controls for Unix-based Gateways, USENIX ConferenceProceedings, pp. 203-221.2 Dictionary.com. 2004, Dictionary.com/firewall, Lexico Publishing Group, Website. Accessed 2 May 2004.http://dictionary.reference.com/search?q=firewall

3

2.2. Timeline of Firewall Development

The following is a timeline of the evolving firewall technologies and how they evolved and

developed. It shows a definite steady trend to continually improve on previous methods and fix

faults in the previous technology. The ideas outlined in this section are discussed in more depth

later.

The first generation of firewall architectures has been used almost as long as routers, and first

appeared around 1985. These firewalls were called packet filter firewalls. However, the first paper

describing the screening process used by packet filter firewalls did not appear until 1988, when

Jeff Mogul from Digital Equipment Corporation published his studies.

In the period around 1989-1990, Dave Presotto and Howard Trickey of AT&T Bell Laboratories

pioneered the second generation of firewall architectures with their research in circuit relays,

which were known as circuit level firewalls. They also implemented the first working model of the

third generation of firewall architectures, known as application layer firewalls. However, they

neither published any papers describing this architecture nor released a product based upon their

work.

As is often the case in research and development, the third generation of firewall architectures

were independently researched and developed by several people in the United States during the

late 1980s and early 1990s. Publications by Gene Spafford of Purdue University, Bill Cheswick of

AT&T Bell Laboratories, and Marcus Ranum describing application layer firewalls first appeared

during 1990 and 1991. Marcus Ranum’s work received the most attention in 1991 and took the

form of bastion hosts running proxy services. Ranum’s work quickly evolved into the first

commercial product: Digital Equipment Corporation’s Secure External Access Link (SEAL)

product.

Around 1991, Bill Cheswick and Steve Bellovin began researching dynamic packet filtering also

known as stateful inspection, and went as far as to help develop an internal product at Bell

Laboratories based upon this architecture; however, this product was never released. In 1992, Bob

Braden and Annette DeSchon at USC’s Information Sciences Institute began independently

researching dynamic packet filter firewalls for a system that they called “visas.” Check Point

Software released the first commercial product based on this fourth-generation architecture in

1994.

During 1996, Scott Wiegel, Chief Scientist at Global Internet Software Group, Inc., began laying

out the plans for the fifth generation firewall architecture, the kernel proxy architecture. Cisco

Centri Firewall, released in 1997, was the first commercial product based on this architecture.

4

Figure 1: Timeline of Firewall Architectures3

2.3. Packet Filter Firewalls

The first firewalls used Internet Protocol (IP) router technology, the network layer, and filtering

rules to determine whether network traffic was allowed access to the network. Packet filter

firewalls could only allow or deny network communication. The filtering rules had to be manually

altered by the firewall administrator. The filtering rules examined incoming or outgoing packets,

allowing or disallowing their transmission. The basis for these rules was often the source IP

address, the destination port and the protocol used.4

One problem of the first packet filter firewall was that because it used IP router technology that

passed traffic through the connection, it allowed direct connections between networks through

address authorisation. To correct this problem, a further series of firewalls were developed

between 1989 and 1990 using circuit level firewall gateways

2.4. Circuit Level Firewalls

Circuit level firewall gateways were used for Transport Control Protocol (TCP) connections. They

examined each connection setup to ensure it follows a legitimate “handshake” for the TCP being

used. The circuit level firewall then checks its records to make sure that the sender is allowed to

send to the receiver and the receiver is allowed to receive from the sender. If the answer is “yes” to

both conditions, the connection and all associated packets are routed through with no more

security checks.5

3 Cisco Systems, 2002, Evolution of the Firewall, Cisco Systems (28 September 2002), Website accessed 13 May 2004.http://www.cisco.com/univercd/cc/td/doc/product/iaabu/centri4/user/scf4ch3.htm4 PC Help, 1999, What is a Firewall, PC Help, Website. Accessed 13 May 2004, http://www.pc-help.org/www.nwinternet.com/pchelp/security/firewalls.htm5 IT Security, 2002, Computer Security Dictionary, ITsecurity.com, Website accessed 2 May 2004,http://www.itsecurity.com/dictionary/dictionary.htm

5

The next advance in firewall technology took place in 1991 with the first commercial release of a

firewall by Digital Equipment Corporation (DEC) with the new development of an application

layer firewall called SEAL.

2.5. Application Layer Firewalls

This new generation of firewalls uses filters and application gateways or proxies to control traffic

entering or leaving their networks. The application layer firewall is an intermediary between the

internal network and the Internet.6

An application layer firewall has two primary functions, to act as a proxy server or as a proxy

client. This means that the firewall is the go-between for any communication that crosses between

the two networks (internal network and Internet).

When a Computer A wants to communicate with Computer B which is connected to the world-

wide web, the Firewall C acts as an intermediary between Computers A and B. Firewall C takes

the intended communication from Computer A and directs it to Computer B, when Computer B

replies, it replies to Firewall C thinking it is Computer A. When Computer A communicates back

to computer B, it is actually only passing data to Firewall C, as represented in Figure 2.

Figure 2: Application Layer Firewall Proxy Demonstration 6 Meyer, Helen, 1997, A History of Firewall Technology, Computers & Security, 16 (4), p. 331.

6

Inbound connections are always made with the proxy client, while outbound connections are

always made with the proxy server. There is no direct connection between the internal network

and an insecure network.7

A typical application layer firewall can provide proxy services for applications and protocols like

Telnet, File Transfer Protocol (FTP), Hypertext Transfer Protocol (HTTP), and Simple Mail

Transfer Protocol (SMTP). Note that a separate proxy must be installed for each application-level

service.

When the application layer firewall is passing the traffic from network to network the firewall has

the opportunity to analyse the data and headers within the traffic. When traffic arrives at the

external connection, the firewall evaluates IP addresses; it looks at the data within the packet to

stop an outside source hiding information, and it uses any filters or policies in place to determine if

the traffic is legitimate or not and whether it is allowed to enter the internal network.8

The advantages of this technique are that because they act as a proxy, all communication passes

though the firewall. It can also control what protocols are used, such as HTTP and FTP, but can

disallow Peer To Peer (P2P) and other unused protocols. An application layer firewall can restrict

access to certain network services and websites not related to the business such as web based

email sites or pornography. The application layer firewall can also use HTTP object caching and

user authentication. As all traffic flows through it, the application layer firewall has all the

information required to generate comprehensive audit reports.

As with many aspects of Information Science, there are drawbacks such as:

• As all traffic flows through it, the proxy server introduces delays in communication.

• A new proxy has to be written for each new protocol that has to pass though the firewall, often

causing delays.

• As the firewall is run using a third party operating system architecture, it is vulnerable to

Operating System (OS) and application layer bugs, meaning the OS’s have to be hardened

against attack.9

7 IT Security, 2002, Computer Security Dictionary, ITsecurity.com, Website accessed 2 May 2004,http://www.itsecurity.com/dictionary/dictionary.htm8 Sheldon, Tom, 2002, General firewall Paper, WindowSecurity.com, Website accessed 28 April 2004.http://www.secinf.net/firewalls_and_VPN/General_Firewall_White_Paper.html9 IT Security, 2002, Computer Security Dictionary, ITsecurity.com, Website accessed 2 May 2004,http://www.itsecurity.com/dictionary/dictionary.htm

7

2.6. Stateful Inspection Firewalls

Because of the disadvantages with cost and performance of the application layer firewalls, and the

lack of security surrounding packet filtering, a new method was developed. The new method was

called stateful inspection. Instead of examining the contents of each packet, the header information

of the packet is compared to packets that are known to be trusted.10

Because the stateful inspection used packet filtering technology, it evaluated the IP header

information such as source, source port, destination and destination port. A new feature of the

stateful inspection firewall was a state table. This table kept a list of open connections.11

When a user accessed an outside service, the stateful inspection firewall remembered details about

the original request such as port number, source and destination address. This "remembering" is

called “saving the state”. When the outside system responded to a request, the firewall compared

the received packets with the saved state to determine if they are allowed in.12

The effect of a state table meant that a packet may be generated by a third party to look like a

typical legitimate response, say a webpage. When the firewall checked the state table, there would

be no connection entry for the response thus denying access to the internal network.

A stateful inspection firewall could read all seven Open Systems Interconnection (OSI) layers,

allowing it to filter packets at the header level, as well as provide the ability to analyse

applications, overcoming the weaknesses of IP filtering devices.

Stateful inspection itself has proven to be a very effective and efficient mechanism for access

control.13

As with any new technology, stateful inspection had its disadvantages. When a service used User

Datagram Protocol (UDP), they did so insecurely. UDP’s inherent weakness was due to the lack of

a stateful connection. UDP sent one packet out and got one packet back. UDP does not have error

correction or integrity checks. If a UDP packet is sent, a single packet response can potentially

allow a hacker in. This was a problem because UDP was a widespread protocol, the Domain Name

Service (DNS) service being a prime user.

10 Sheldon, Tom, 2002, General firewall Paper, WindowSecurity.com, Website accessed 28 April 2004.http://www.secinf.net/firewalls_and_VPN/General_Firewall_White_Paper.html11 Jackson, Jeromie, 1997, Making distinctions between firewall technologies, Computer Technology Review; Winter, pp. 38-40.12 Sheldon, Tom, 2002, General firewall Paper, WindowSecurity.com, Website accessed 28 April 2004.http://www.secinf.net/firewalls_and_VPN/General_Firewall_White_Paper.html13 Jackson, Jeromie, 1997, Making distinctions between firewall technologies, Computer Technology Review; Winter, p. 38.

8

Another disadvantage was that a stateful inspection firewall was not a proxy; it let internal packets

make their way to the outside network, thus exposing internal IP addresses to potential hackers.

Some firewall vendors are using Stateful Inspection and proxies together for added security.14

However, an advantage is that the most common communication protocol used over the internet is

TCP and TCP maintains state. This means sequence numbers have to be used. In order to subvert

TCP security, not only do you need to forge the source IP address but you also need to be able to

determine what sequence number you need to use.15

An example of a stateful inspection implementation is the freeware firewall IPTables which is

standard with many Linux distributions. IPTables check and filter each packet individually. The

stateful inspection firewall also uses filters; for example, it can differentiate between packets

requesting a connection and those already connected in a session by checking whether the SYN

flag in the packet header is set (and also whether FIN and ACK flags cleared).16

As with the application layer firewalls, this technology has its strengths and weaknesses. To

choose between application layer and stateful inspection firewalls, an organisation’s policy and

requirements must be taken into account. Application gateways provide better control and logging,

while stateful inspection has both the edge in performance and much greater flexibility but at the

risk of incorrect configuration.17

2.7. Dynamic Packet Filtering Firewalls

The next development of firewall technologies was the introduction of dynamic packet filtering

firewalls. They were closely related to stateful inspection firewalls. Many definitions class them as

the same technology. The advances in the dynamic packet filtering firewalls were in similar to the

state table as it looks at each packet as opposed to the connection as a whole. Security surrounding

the communication increased so potential attackers could not adjust a packet held within normal

looking communication. The dynamic packet filtering firewall’s method did have a negative effect

on performance but with the advance in microchip technology, it was not a major concern.18

Further development with dynamic packet filtering firewalls was the commercial release of

Firewall-1 by Check Point Technologies in 1994. This firewall was the first “user friendly”

14 Sheldon, Tom, 2002, General firewall Paper, WindowSecurity.com, Website accessed 28 April 2004.http://www.secinf.net/firewalls_and_VPN/General_Firewall_White_Paper.html15 Jackson, Jeromie, 1997, Making distinctions between firewall technologies, Computer Technology Review; Winter, p. 38.16 Napier, Duncan, 2001, IPTables/NetFilter - Linux's next-generation stateful packet filter, Sys Admin, 10 (12), December, pp. 8-13.17 Meyer, Helen, 1997, A History of Firewall Technology, Computers & Security, 16 (4), 1997, p. 331.18 Whitman, Michael E and Herbert J Mattord, 2003, Principles of Information Security, Thomson Publishing.

9

product with icons and simplified installation and administration. Additionally, Firewall-1 did not

require any file editing, which had been required in other commercial products. The graphical user

interface based configuration and management interface greatly simplified installation and

administration.19

2.8. Kernel Proxy Firewalls

The current technology used for firewalls are classed as kernel proxy firewalls. This technology

evaluates packets at multiple layers of the protocol stack in the proxy server and is similar to the

application layer in its use of proxy servers.

Cisco has implemented this technology in developing a product called Centri Firewall. This

implementation uses the Windows NT Executive, which is the kernel of Windows NT, 2000, 2003

and comprised of three components:

The first component captures packets arriving at the firewall server. The packet is then analysed

by reading the header information and the signature data. Both the data about the packet and the

packet itself are passed to the second stage.

This second stage receives the data about the packet and decides whether to drop the packet, map

it to an existing session or to create a new session using the received data about the packet. If a

current session exists, the packet is passed through a custom built protocol stack created

specifically for that session, which is a customised implementation of the approach widely knowm

as a network address translation. This last stage enforces the security policy as configured into the

device in the final stage, the kernel proxy, as it inspects each packet.20

The kernel proxy comprises of proxy servers for application layer protocols such as HTTP, FTP,

Telnet and SMTP, transport layer protocols such as Internet Control Message Protocol (ICMP)

TCP and UDP and Network Layer protocols such as IP. These proxy servers are configurable so

the second stage determines what decision to make about the packet.21

19 Anti-Hack. History of Firewalls, Anti-Hack, 2001, Website. Accessed 2 May 2004.http://dmsweb.badm.sc.edu/mgsc890/firewalls/fire2.htm20 Whitman, Michael E and Herbert J Mattord, 2003, Principles of Information Security, Thomson Publishing.21 ibid

10

2.9. Summary

Firewalls as products are still a valid method for preventing basic intrusions. They can assess the

state of a connection, they can block communication between systems if the communication is not

valid or is unsolicited, and they can isolate systems that should not be connected to an unsecured

network. However, the research and development turned to a new technology called IDS.

The development of IDS and security issues surrounding these systems are discussed in the next

chapter.

11

3. Intrusion Detection Systems

3.1. Introduction

The goal of Intrusion Detection System (IDS) is to monitor network assets to detect anomalous

behaviour and misuse. Such a goal has been recognised as significant for nearly twenty years but

only recently has it seen a dramatic rise in popularity and incorporation into the overall

information security infrastructure.22

3.2. First IDS Papers

The first recognised IDS paper was published in 1980 by James Anderson, titled “Computer

Security Threat Monitoring and Surveillance”.23 It was written for a US government organisation

and introduced the notion that audit trails contained vital information that could be used to track

misuse and understand user behaviour. This insight into audit data and its importance led to

tremendous improvements in the auditing subsystems of virtually every operating system. His

work was the start of IDS.

In 1983, Dr Dorothy Denning, working as part of Stanford Research Institute International, began

work on a follow-up paper, later titled “An Intrusion Detection Model”.24 The study analysed audit

trails from government mainframe computers and created profiles of users based on activities

recorded. Later, Dr Denning helped to develop the Intrusion Detection Expert System (IDES) that

was used as a foundation for IDS technology development. As part of the study, Stanford Research

Institute also developed a means of tracking and analysing audit data from users on ARPANET

(soon to be renamed The Internet). Using her research and development work at Stanford Research

Institute, Dr. Denning published the decisive work, “An Intrusion Detection Model”, that revealed

information for commercial intrusion detection system development. Her paper was the basis for

most of the work in IDS that followed. The paper was published in 1987 by the Institute of

Electrical and Electronics Engineers(IEEE)25.

22 Innella, Paul, 2001, The Evolution of Intrusion Detection Systems, Tetrad Digital Integrity, Website accessed 7 April 2004,http://www.securityfocus.com/infocus/151423 Anderson, James P, 1980, Computer Security Threat Monitoring and Surveillance, Website accessed 23 April 2004,http://csrc.nist.gov/publications/history/ande80.pdf.24 Denning, Dorothy, E. 1987, An Intrusion Detection Model, IEEE Transactions on Software Engineering, Number 2, February, p.22225 Innella, Paul, 2001, The Evolution of Intrusion Detection Systems, Tetrad Digital Integrity, Website accessed 7 April 2004,http://www.securityfocus.com/infocus/1514.

12

3.3. Host Based IDS

At the time of the release of “An Intrusion Detection Model”, the University of California Davis

Lawrence Livermore Laboratories had also been conducting research into IDS. The Haystack

project released another version of IDS. This project produced an IDS that analysed audit data by

comparing it to defined patterns. The name “haystack” came from the analogy of “looking for a

needle in a haystack” as there is a vast amount of data to examine in the search for intrusion.26

This first implementation of the Intrusion Detection Model was later classified as a host-based

IDS. Host-based IDS directly monitor the computers on which they run, often through tight

integration with the operating system. This integration does have a cost, as the system monitors

inside users just as they monitor outside users, and the number of computers often makes it

impossible to protect every computer on the network using a host-based IDS. Another

disadvantage to the host based IDS implementation is that because network traffic is constantly

being monitored, the processing power required often has a negative impact on the operating

performance of the computer.27

3.4. How host-based IDS work

As described in “An Intrusion Detection Model”,28 an IDS should detect abnormal use of the

system; therefore security violations could be detected from abnormal patterns of system usage.

The following are examples of security violations that would be in the abnormal use patterns:

• Attempted break-in:

Someone attempting to break into a system might generate an abnormally high rate of password

failures with respect to a single account or the system as a whole.

• Masquerading or successful break-in:

Someone logging into a system through an unauthorised account and password might have a

different login time, location, or connection type to that of the account's legitimate user. In

addition, the unauthorised user’s behaviour may differ considerably from that of the legitimate

user. In particular, they might spend most of their time browsing through directories and executing

system status commands, where as the legitimate user might concentrate on editing or compiling

26 ibid27 Durst, Robert, Terrence Champion, Brian Witten, Eric Miller, and Luigi Spagnuolo, 1999, Testing and evaluating computerintrusion detection systems, Communications of the ACM, 42 (7), July, pp. 53- 61.28 Denning, Dorothy, E. 1987, An Intrusion Detection Model, IEEE Transactions on Software Engineering, Number 2, February, p.222

13

and linking programs. Many break-ins have been discovered by security officers or other users on

the system who have noticed the alleged user behaving strangely.

• Penetration by legitimate user:

A user attempting to penetrate the security mechanisms in the operating system might execute

different programs or trigger more protection violations from attempts to access unauthorised files

or programs. If his attempt succeeds, he will have access to commands and files not normally

permitted to him.

• Leakage by legitimate user:

A user trying to leak sensitive documents might log into the system at unusual times or route data

to remote printers not normally used.

• Inference by legitimate user:

A user attempting to obtain unauthorised data from a database through aggregation and inference

might retrieve more records than usual.

• Trojan horse:

The behaviour of a trojan horse planted in or substituted for a program may differ from the

legitimate program in terms of its CPU time or 1/0 activity.

• Virus:

A virus planted in a system might cause an increase in frequency of executable files rewritten,

storage used by executable files, or a particular program being executed as the virus spreads.

• Denial-of-Service:

An intruder able to monopolise a resource (e.g., network) might have abnormally high activity

with respect to the resource, while activity for all other users is abnormally low.29

An example of a widely used IDS is Snort. While Snort is marketed as a network intrusion

detection system, For the purposes of this demonstration it can be used as a host-based IDS. Snort

is capable of performing real-time traffic analysis and packet logging on IP networks and can

perform protocol analysis, content searching/matching and can be used to detect a variety of

attacks and probes, such as buffer overflows, stealth port scans, CGI attacks, SMB probes, OS

fingerprinting attempts, and much more.

29 ibid

14

Snort uses a flexible rules language to describe traffic that it should collect or pass, as well as a

detection engine that utilises modular plug-in architecture. Snort has a real-time alerting

capability, incorporating alerting mechanisms for syslog, a user specified file, a Unix socket, or

WinPopup messages to Windows clients using Samba's smbclient.

Snort has three primary uses. It can be used as a straight packet sniffer like tcpdump, a packet

logger (useful for network traffic debugging, etc), or as a full blown network IDS.30

Snort was first released on 22nd December 1998 as Snort 0.96. The latest distribution is Snort 2.4.3

which was last updated 17th October 2005. This latest release incorporates a new HTTP Protocol

Flow Analyser (PFA) and Detection Engine.

The PFA classifies network application protocols into client and server data flows. These flows are

the communication between the client and server. The client to server communication is

considered separate from the server to client communication.

The analysis of this communication is performed at a high level and inspects only the important

aspects of the protocol, such as server response codes or a client request code. This protocol

analysis is not an absolute; instead it is used in conjunction with generic analysis methods already

in use.

The main benefit of the PFA is reduced processing time and a reduction in the number of false

positives. Once the protocol flows are deduced, the detection engine uses the data flows to search

for possible policy violations.

The Detection engine has three major components, The Rule Optimiser (RO), The Multi-Rule

Search Engine (MRSE) and the Event Selector.

The RO uses a set-based methodology for managing the Snort Rules and applies them to network

traffic. Subsets of rules are then formed based on unique rule and packet criteria. The basis of the

subsets classification is unique parameters, such as source port, destination port and rule contents.

Each subset consists of a complete set of rules applicable to each packet. This guarantees that each

packet will be tested, and therefore ensures rules that can never match the packets are not tested.

When Snort begins running, the program reads and parses all the activated rules and then classifies

them into subsets. When the rules have been divided, each incoming packet is matched to a rule

subset meeting the packet’s unique parameters. For example, Snort is run with 1500 rules; they are

divided into smaller subsets based on transport and application layer protocols. So 500 of the rules

30 Caswell, Brian and Marty Roesch, 2004, What is Snort?, Snort.org. Website. Accessed 31 May 2004.http://www.snort.org/about.html

15

may go to the HTTP client rule set, and another 50 to the HTTP server rule set. This carries on

until all the rules and protocols are covered.

Once the rules have been optimised, the rule set is selected and the MRSE process begins.

The MRSE has three distinct searches based on the unique snort rule properties: the protocol field,

generic content and packet anomaly.

The protocol field search allows a rule to specify a particular field in a protocol to search. For

example, Snort uses the ‘uricontent’ keyword to search HTTP request-uri fields.

The generic content search allows a rule to specify a series of bytes to match against the packet.

For example, this functionality is used to look for buffer overflows in all packets and can also be

used by Snort users to search for any ASCII or binary byte sets that may signify an attack on their

network.

The packet anomaly search allows a rule to specify characteristics of a packet or packet header

that is cause for alarm. The packet anomaly search does not search any content in the packet, it is

focused on the packets other characteristics. This is a specific type of detection; an example of an

anomaly rule is a rule that looks for an ICMP packet over 800 bytes.

If a match was found using any of these three search types, all the processing power is used to

fully validate the specific rule. If the rule is validated, an event is generated and added to the event

queue. The search engine then carries on the search from where it was immediately after the match

was found. When the search engine has processed the packet, the event selector processes the

event queue.

The event selector allows Snort to sort every occurrence of every rule match within a packet. The

event selector prioritises events from the event queue and selects events based on the assigned

priority. This event is then sent to the Snort output system.31

With this guideline of security violations and the internal workings of Snort as an example, an IDS

should be able to monitor the actions of a suspected attacker. The system administrator can also

use the system logs and other sources of information to identify the violating user and then

proceed according to organisation policy. This could be anything from banning the user, engaging

in further investigation, or simply alerting the relevant authorities.

The latest developments in host-based IDS are aimed at using the advantage of having direct

system access coupled with a process that requires a lot less processing power. Other

31 Sourcefire, 2003, Snort 2 0: Detection revisited, Snort.org, 1 February 2003, Website accessed 31 May 2004,http://www.sourcefire.com/technology/whitepapers/sf_snort20_detection_rvstd.pdf

16

developments include the notion of a trusted automated process that manages hundreds of host-

based IDS sensors so every computer on a network can be protected.32

This idea of IDS sensors on other computers led to the introduction of network based IDS.

3.5. Network Based IDS

The next development of IDS, the Distributed Intrusion Detection System (DIDS), was developed

by Haystack. DIDS used the previous research but also tracked client machines, as well as the

servers that previous research had concentrated on.33

In 1990 Todd Heberlein introduced the idea of Network Intrusion Detection. Heberlein was the

primary author and developer of the Network Security Monitor (NSM) which was the first

example of a network intrusion detection system. This implementation was deployed at major US

government installations requiring massive network analysis. This led to the combination of NSM

with the DIDS to create the idea of Hybrid IDS. This development saw increased interest, and

therefore investment, in the IDS market, bringing it into the commercial world.

Following the commercial release of systems such as Haystack Labs Stalker, numerous products

were later released. For example, Science Applications International Corporation (SAIC)

developed a host-based intrusion detection called Computer Misuse Detection System (CMDS)

and the US Air force simultaneously developed the Automated Security Measurement System

(ASIM) to monitor network traffic on the USAF network. ASIM is still in use today.34

3.6. How Network Based IDS work

Network-based IDS monitor network traffic between hosts. Unlike host-based IDS, which detect

malicious behaviour outright, these systems deduce behaviour based on the content and format of

data packets on the network.

Among other things, they analyse overt requests for sensitive information and repeated failed

attempts to violate security policy. Many current network-based IDS are quite primitive, only

watching for the words and commands of a hacker's vocabulary. A few are more sophisticated and

analyse protocol-specific information.

If host-based IDS are analogous to a guard dog for each computer, network-based IDS are like

neighbourhood police patrols. Many of the network monitors under research can even respond to

32 Durst, Robert, Terrence Champion, Brian Witten, Eric Miller, and Luigi Spagnuolo, 1999, Testing and evaluating computerintrusion detection systems, Communications of the ACM, 42 (7), July, pp. 53- 61.33 Innella, Paul, 2001, The Evolution of Intrusion Detection Systems, Tetrad Digital Integrity, Website accessed 7 April 2004,http://www.securityfocus.com/infocus/151434 ibid

17

calls for help, either by decisively terminating an intrusion or by more graduated responses,

including filtering, isolation, changing logging or even disconnection. Other efforts are currently

investigating techniques for more reliable detection of intrusions through collaboration between

different types of detection systems.35

A working example of a network-based IDS is a product simply called NID. NID is a suite of

software tools that helps detect, analyse, and gather evidence of suspicious behaviour occurring on

an Ethernet or Fibre Distributed Data Interface (FDDI) network using IP. NID was developed for

the US Department of Energy as part of the Computer Incident Advisory Capability Cyber

Solution Tools Centre (CIAC CSTC) and at the time of writing is currently implemented as

Version 2.6.

NID operates passively on a stand-alone host (rather than residing on the hosts it is monitoring),

and is responsible for collecting data and/or statistics about network traffic.

NID operates within a specified security domain of hosts or a sub-network. A security domain

may consist of either a subnet of a network or the entire network to which NID is directly

connected. The security domain can be refined by looking at traffic from particular Internet

services.

NID has unique features:

• is passive, this means intruders don’t know that it is there.

• does not require host modification.

• can analyse data as it arrives or at a later time.

• provides real-time alerts of suspicious behaviour.

• can begin data gathering upon detection of suspicious behaviour. This can be stopped at the

absence of the suspicious behaviour.

• provides a full suite of analytical tools.

• is customisable.

NID uses three techniques for detecting suspicious behaviour; attack signature recognition (ASR),

vulnerability risk model (VRM) and anomaly detection(AD).

The ASR technique examines data packets for byte series associated with known attacks, and the

presence suggests the possibility of suspicious behaviour. Special non-printable control characters

35 Durst, Robert, Terrence Champion, Brian Witten, Eric Miller, and Luigi Spagnuolo, 1999, Testing and evaluating computerintrusion detection systems, Communications of the ACM, 42 (7), July, pp. 53- 61.

18

can also be included within the pattern definitions. If a suspicious pattern is found, NID can signal

an alarm, display the context in which the pattern is found and begin saving the session’s network

packets to an output file. These files can then be examined at a later date to determine whether

suspicious behaviour has occurred.

NID has a VRM that computes a warning value based on a host’s security level, any authentication

required for the service used, and any recent transactions of the host. This warning value is used to

rank the communication.

AD monitors and reports anomalies as they happen. The two major sets of anomalies the NID can

detect are activities associated with un-trusted or unexpected hosts and known network attacks

such as port scans and/or SYN flooding.

When suspicious behaviour has been identified, NID can react to it in three different ways;

collectively called Common Operation Models (COM); Retrospective Intrusion Analysis (RIA),

Real-time Intrusion Detection (RTID) and Statistics Gathering (SG).

RIA is used to analyse collected traffic for evidence of suspicious behaviour. Once analysis is

complete, any suspicious communication can be replayed so a human analyst can discover new

intrusion techniques. This technique is more of a surveillance technique as it looks at past records

of communication.

RTIS is used to process suspicious behaviour as it happens. Network packets associated with

suspicious behaviour are collected continuously until the session is terminated. This system is

more like an alarm, as it collects data about suspicious behaviour as it happens.

SG is used to collect information about the packets, such as header data, the sender and receiver

and protocols used. All this information is used to perform statistical analysis on the traffic.36

This example of a working network-based IDS shows the issues faced by the host-based IDS can

be overcome using a passive listening host on the network. Effectivly comparing the two different

methods is difficult because they are different technologies doing different jobs within the same

parameters.

36 Computer Advisory Incident Capability, 2004, NID Introduction, US Department of Energy, Website accessed 7 April 2004,http://ciac.llnl.gov/cstc/nid/intro.html

19

3.7. Comparison

The benefits of network-based versus host-based IDS must be viewed in context. The following

are some strengths and limitations of the respective IDS and should be considered if an IDS is to

be implemented.

3.7.1. Strengths

3.7.1.1. Network-based IDS:

• Have a real-time notification ability that allow quick and automated reactions.

• Passively monitor a network which is unobtrusive and avoids adverse effects on network

performance.

• Examine all packet headers for signs of malicious and suspicious activity.

• Look at the content of packets in search of a payload by looking for specific commands or

syntax indicative of a variety of attacks.

• Can be strategically deployed at critical access points to view network traffic destined for

numerous systems that need to be protected.

• Are “operating-system independent” and do not require software to be loaded and managed

on a variety of hosts, as is the case with the host-based approach.

• Require fewer detection points so the cost of ownership is usually lower for an enterprise.

• If required, can be placed outside a network to gather information about intrusion attempts as

they may not pass though the firewall.

3.7.1.2. Host-based IDS:

• Have user logs that contain events that have actually occurred; this has the advantage of

knowing whether an actual attack or exploit was successful or not. This information can be

more accurate and less prone to false positives.

• Are closer to the user, and so are capable of discerning attacks and misuse that would

otherwise be very hard to see from the network.

• Can monitor system-specific activity such as user logon and logoff, file access, changes to

resource permissions, attempts to install new executables and access to privileged services.

• Can monitor any policy changes made affecting what the system does log and does not log.

The system agent will immediately pick up the event.

20

• Can also help overcome some of the challenges that arise from encrypted communications

and switched networks. The problem is that encryption requires a lot of processing power to

keep track of; and switched networks produce problems as to where to locate IDS.

• Have extended logging capabilities to the application level allowing the IDS system agent to

protect applications like web or database servers.

3.7.2. Limitations

3.7.2.1. Network-based IDS:

• Have limits that originate from technical hardware limitations. Processor speed and memory

look-up time dictate the performance of the network monitoring engines.

• Room for improvement could come from three domains:

• Better algorithms

• Faster hardware

• Improved interaction between software and hardware

• Is a relatively young discipline where experience is gained and shared every day. Modern

software engineering techniques, such as hash-table lookups, are used to address

performance problem from the software side.

• Monitoring of encrypted packets is, although theoretically possible, hard to do in practice.

Not only does the monitoring system need to know the relevant keys, but the decryption also

has to happen in real time.

• Has limited capability to detect an attacker sitting at the workstation's console. Host-based

IDS that are closer to the end user side can provide additional support to detect this type of

security breach.

3.7.2.2. Host-based IDS:

• Where host-based IDS are usually closer to business critical applications, they are also

slower to respond. In particu|ar, when there are concerns about Denial of Service (DoS)

attacks, a network-based IDS should be in place.

• To provide full protection, the system agents have to be installed on a system-by-system

basis throughout the organisation.

21

• It is also important to note that the amount of management overhead tied to agent and engine

configuration and incident response is proportional to the installed base of intrusion detection

systems, and the capabilities of the security officers to respond to console notifications.

• In this respect, the technical limitations of the products may far exceed human capacity and

the efficiency of the security system in place will depend on both the configuration of the

active security policies, and how one has elected to handle real-time incident responses

within the organisation.37

3.8. Summary

In many aspects, a host-based IDS limits are alleviated by a network-based solution's strengths;

and vice versa.

This shows that network-based and host-based IDS would work best when coupled together to

create a comprehensive security strategy.

While this may be true, it is not an ideal situation. This situation started the development of

Intrusion Prevention Systems (IPS).

37 Schepers, F. 1998, Network- Versus Host-based Intrusion Detection, Information Security Technical Report, 3 (4), pp. 32-42.

22

4. Intrusion Prevention Systems

4.1. Introduction

It has been suggested that IPS is a revolutionary new security technology. To describe IPS as

revolutionary, one would have to have a limited view of the security products market. IPS

encompasses aspects of many well-known, existing security technologies including anti-virus,

software, intrusion detection and firewalls. Evolution rather than revolution is clearly the more

dominant process of change.38

In the IPS model, instead of developing reactive security policies, security policy becomes a

proactive tool to protect an organisation. This enables the organisation to become self-protecting.39

To achieve the ideal of self-protection, all attacks against any part of the protected environment

will be deflected by IPS. Because IPS are secure they can take any stream of network packets and

make the determination of intent – whether it is an attack or legitimate use – then take appropriate

action with complete perfection. The end result would be only a limited need for IDS or

monitoring solutions since everything representing a threat is blocked.40 While an admirable goal,

in reality it is incredibly difficult because new attacks strategies are constantly evolving. To

combat new attacks, IPS must constantly update their attack library in a similar way to anti-virus

scanners. This means that an IPS is only as secure as its last update.

As a result of the difficulty of implementing a truly secure IPS, most implementations today use a

combination of anomaly or behaviour-based detection and IDS to rapidly detect an attack. Most

IPS products are network-based and are deployed in the form of high-throughput appliances with

hardened operating systems and firmware.41

38 Secure Computing, 2003, Intrusion Prevention Systems (IPS), Secure Computing Corporation, Website accessed March 2004,http://www.condyn.net/download/Intru-Preven-WP1-Aug03-vF.pdf39 Henning, Ronda, and Richard Caliari, 2003, Behavior-Based Intrusion Prevention, Harris Corporation, Website accessed 7 May2004, http://www.stat.harris.com/solutions/bbip.asp40 Lindstrom, Pete, 2004, Intrusion Prevention Systems (IPS): Next Generation Firewalls, Website accessed 26 May 2004,www.forum-intrusion.com/Spire_IPS_Whitepaper.pdf41 Krull, Joseph E., 2003, What to expect from your IPS, Communications News, October, 40 (10), p. 19.

23

4.2. Development

The inadequacies inherent in current defences like IDS (both host-based and network-based) have

driven the development of IPS.42

On the surface, IDS and IPS appear to be equally effective. After all, they share a long list of

similar functions, such as packet inspection, stateful analysis, fragment reassembly, TCP segment

reassembly, deep packet inspection, protocol validation, and signature matching. But these

capabilities take a backseat to the different purposes for which they are deployed. An IPS operates

like a security guard at the gate of a private community, allowing and denying access based on

credentials and some predefined rule set, or policy as shown in Figure 3. An IDS is similar to a

video camera within the community, monitoring activities and looking for abnormal situations, but

an IDS cannot intervene to stop traffic as shown in Figure 4.

Figure 3 - An IPS monitoring a network.

Figure 4 - An IDS monitoring a network.

As described in previous sections, the purpose of IDS is to provide monitoring, auditing, and

reporting of network activity. It operates on the packets that are allowed through an access control

42 NSS Group, 2004, Intrusion Prevention Systems (IPS), The NSS Group Ltd, Website accessed 26 May 2004,http://www.nss.co.uk/WhitePapers/intrusion_prevention_systems.htm

24

device such as a firewall. IDS solutions are loaded with intelligence, using many different

techniques to identify potential attacks, intrusions, exploits, and abuses.

The primary expectation of IPS is that they will reduce the threat of attack by eliminating the

harmful and/or malicious network traffic while continuing to allow legitimate activity to continue.

IPS solutions must be deterministic in nature. Deterministic capabilities instil the confidence

required for a absolute decision such as denying network traffic. This means that IPS are ideally

positioned to deal with:

• Undesired applications attacks against private networks and applications.

• Attack packets (e.g. WinNuke) by using high-speed packet filters.

• Protocol abuse and evasive actions – network protocol manipulations like Fragroute and TCP

overlap exploits.

• DoS attacks such as SYN and ICMP floods.

• Application abuse and protocol manipulations – known and unknown attacks against HTTP,

FTP, DNS, SMTP.

• Application overload or abuse attacks.

All of these attacks and the vulnerable state that allows them to happen are well documented. In

addition, the anomalies in communication protocols from network through application layer have

no place in any sort of legitimate traffic.

The difference between IDS and IPS is that IDS uses historical traffic to determine if there is

potential for threat, including performing statistical analysis of traffic volume, traffic patterns, and

anomalous activities.

IPS must be deterministic in all of its decisions in order to perform its function of scrubbing

traffic. An IPS is supposed to work all of the time, and make access control decisions on the

network. Firewalls provided the first deterministic approach to access control on the network,

providing basic IPS capability. IPS devices add next-generation capability to these firewalls – still

operating inline and providing the type of deterministic comfort required of an inline device that is

making access control decisions.43

As IPS have developed, they have overcome the limitations encountered by previous technologies

and found new issues that still need to be addressed. IPS have generally improved on previous

technologies if properly implemented and designed as described above. 43 Lindstrom, Pete, 2004, Intrusion Prevention Systems (IPS): Next Generation Firewalls, Website accessed 26 May 2004,www.forum-intrusion.com/Spire_IPS_Whitepaper.pdf

25

4.3. How IPS work

With the development described previously pointing to new and better security solutions, IPS need

to incorporate new ideas and methods to advance or the technology is bound to fail. The following

outlines how IPS products detect and prevent unwanted traffic from protected networks.

IPS are network devices that can accept or deny traffic based on IP addresses, protocol/service,

application level analysis and verification. IPS receive traffic from the network, reassemble traffic

streams and look at application protocols and commands to detect suspicious fields that warrant

some predefined action. These actions vary from logging suspicious events to dropping the

connection completely.44

An IPS inspects all layers of packet information that travel on the network (except for the physical

layer), rather than only the first 4 layers traditionally inspected by a firewall. A six-layer

inspection method, commonly called “deep packet” inspection, allows an IPS to run signatures

against packets up to an application level. The result is a highly accurate filtering device that,

unlike a NIDS, has minimal false positives. This is an essential improvement over the false

positives that usually dominate the content of most daily reports found in standard NIDS logs.45

As with a typical firewall, the IPS has at least two network interfaces; one designated as internal

and one as external. As packets appear at either interface they are passed to the detection engine, at

which point the IPS device determines, (as any IDS would) whether or not the packet being

examined poses a threat.

However, if it should detect a malicious packet, in addition to raising an alert, it will discard the

packet and mark that flow as “bad”. As the remaining packets that make up that particular TCP

session arrive at the IPS device, they are discarded immediately.

Legitimate packets are passed through to the second interface and on to their intended destination.

A useful side effect of some IPS products is that as a matter of course they will provide a “packet

scrubbing” functionality to remove protocol inconsistencies resulting from varying interpretations

of the TCP/IP specification.

44 NetScreen Technologies, Inc., 2003, Comparison of Firewall, Intrusion Prevention and Antivirus Technologies, NetscreenTechnologies, Website accessed 26 May 2004, http://www.ncs.cz-novinky/seminar/fw_idp.pdf45 Hagopian, Stephanie, 2004, Network-Based Intrusion Prevention System Technology, SANS Institute, 7 April 2004,www.giac.org/practical/GSEC/Stephanie_Hagopian_GSEC.pdf

26

Any fragmented packets, out-of-order packets, or packets with overlapping IP fragments will be

re-ordered and “cleaned up” before being passed to the destination host and illegal packets can be

dropped completely.46

The hardware component of an IPS is based on multiple server processor technology so the device

can sit almost invisibly within a network. These processors process millions of instructions

simultaniously each second in order to handle a much larger volume of traffic than a single

processor. In fact, most IPS attain minimal to unnoticeable latency sitting in-line on a network as

they can analyse traffic at up to gigabit speeds. All IPS also use “stateful inspection” to keep

latency low. By using stateful inspection, the devices only have to analyse the parts of a session

that match an attack signature. Most organisations demand this type of functionality, especially for

any device that actually must sit in-line to a network that must perform at high speeds for many

users.47

IPS devices are required to perform the following tasks in order to avoid the same issues faced by

IDS:

• In-line operations:

Only by operating in-line can an IPS device perform true protection, discarding all suspect packets

immediately and blocking the remainder of that flow.

• Unquestionable detection accuracy:

It is imperative that the quality of the signatures is beyond question, since false positives can lead

to a DoS condition. The user must be able to be sure that the IPS is blocking only malicious traffic.

New signatures should be made available on a regular basis, and applying them should be quick

(applied to all sensors in one operation via a central console) and seamless (no se