A test of attack graph-basedevaluation of IT-security

Fredrik Johan Sandstrom

February 5, 2014Master’s Thesis in Computing Science, 30 credits

Supervisor at CS-UmU: Helena LindgrenExaminer: Fredrik Georgsson

Umea UniversityDepartment of Computing Science

SE-901 87 UMEASWEDEN

Abstract

To assess the accuracy and correctness of attack graphs I have studied several differentattack graphs and their attributes. The purpose of this study is to find out if attack graphscan successfully predict real attacks on modern systems. Test design was built to testMulVALs performance when Nexpose is used to provide system information. Based on theROC measurement method the results shows that MulVALs accuracy is only 0.02 percentwhen determining attack paths used to compromise the system. The main reason for lowaccuracy was due to the high trade off in precision, where MulVAL suggested thousands ofpaths to the decision maker which no attacker tried.

ii

Contents

1 Introduction and problem description 1

1.1 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Thesis outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Attack-graphs and associated tools 3

2.1 Monotonic or non-monotonic . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Single path or all paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.3 Backward or forward chaining . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.4 Probabilistic or deterministic models . . . . . . . . . . . . . . . . . . . . . . . 4

2.5 Visualization variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.6 Input formats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.7 Logic-based or graph-based . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.8 Proposed Attack Graph Solutions . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.9 MulVAL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.10 NeTSPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.11 TVA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.12 Sheyners attack graph-tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.13 CySeMOL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.14 Previous testing of attack graphs . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 Method 11

3.1 Selection of attack graph-tool . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Dataset and its properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.3 Target networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3.4 Target machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.5 Attackers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.6 Test design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.7 System and vulnerability collection . . . . . . . . . . . . . . . . . . . . . . . . 17

3.8 Configuration of MulVAL and the analysis . . . . . . . . . . . . . . . . . . . . 17

3.9 Definition of ground truth and analysis method . . . . . . . . . . . . . . . . . 18

3.10 Information retrival metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

iii

iv CONTENTS

3.11 Number of attack paths and attacker actions . . . . . . . . . . . . . . . . . . 19

4 Results 21

4.1 Information retrieval metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4.2 Number of attack paths and attacker actions . . . . . . . . . . . . . . . . . . 22

5 Discussion 23

5.1 The accuracy of the attackers log? . . . . . . . . . . . . . . . . . . . . . . . . 23

5.2 The competence and resources of the attackers? . . . . . . . . . . . . . . . . . 23

5.3 The large number of vulnerabilities in the tested organizations? . . . . . . . . 24

5.4 Tested attacks are arbitrary deepening on the attacker? . . . . . . . . . . . . 24

5.5 This is more a test of the accuracy of the scanners feeding MulVAL information? 24

5.6 MulVAL is not user friendly . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

5.7 Number of attackers on network . . . . . . . . . . . . . . . . . . . . . . . . . 25

5.8 The professional attacks was not chosen by random . . . . . . . . . . . . . . . 25

6 Conclusions and future work 27

7 Acknowledgements 29

8 References 31

8.1 Appendix A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Chapter 1

Introduction and problemdescription

To keep a network secure from attacks it is important to keep tracking if the runningsoftware has any reported vulnerabilities. This is tedious work for administrators and whenthe network grows it becomes harder to know the impact of a reported vulnerability have.A number of tools have been developed to assess network security and to support decision-making concerning network security. One popular class of such tools is based on attackgraphs. Examples include MulVAL(Ou, Govindavajhala, & Appel, 2005) NetSPA(Chu,Ingols, Lippmann, Webster, & Boyer, 2010) and CySeMoL (Sommestad, Ekstedt, & Holm,2012). A bit simplified, these tools take vulnerability databases and network descriptionsas input and produce security assessments as output. The most common type of securityassessment they produce is a reachability analysis, in other words, if a network host can bereached from another network host.

A number of algorithms and variants of attack graphs have been presented. The mainproblem the algorithms address is the large number of paths that can be taken betweentwo hosts in one computer network. Algorithms and procedures used in the most widelycited tools are relatively simple and documented in scientific articles. In some cases they arereleased as open source code. Review articles like (Alhomidi & Reed, 2012; Heberlein et al.,n.d.; Lippmann & Ingols, 2005; Roschke, Cheng, Schuppenies, & Meinel, 2009)) summarizethe differences and similarities between different approaches. But there is no paper foundon the subject comparing the results produced by this popular method to real reachabilityin IT-system. There are many different assumptions and abstractions made when producingattack graphs-tools which are lead to ask the question if the tools are meaningful to use fordecision making. For example, what if professional attacks don’t take the paths suggestedby the tools, what if the tools miss to calculate the attacks and let the system administratorhave a false sense of security?

The Swedish Defence Research Agency (FOI) has collected information from a two-dayexercise where security professionals with the Swedish Armed Forces IT unit and secu-rity researchers with FOI attacked a number of computer networks. The exercise and therecordings was conducted on a virtual environment realized within FOIs Cyber Range AndTraining Environment (CRATE)1. This thesis uses the collected data to test attack graphapproaches against this data. Such a test has not been done before.

1www.foi.se/CRATE

1

2 Chapter 1. Introduction and problem description

1.1 Goals

The goal of this thesis is to test the attack graph approach them under realistic circum-stances. The aim is to assess if attack-graph based tools are useful in practice. Moreconcretely, the question addressed is: Can attack-graph based tools accurately predict at-tacks success? A question which, in spite of a considerable number of articles on attackgraphs, has not been answered by the research community.

1.2 Thesis outline

This list will give a short introduction of what each chapter covers.

– Chapter 2 is an introduction to attack-graphs tools and their difference in attributes.Last is previous work in the field presented.

– Chapter 3 are split into three sections: First are the selections of attack-graph toolspresented. Second are the dataset and its properties described. And last is the testdesign and criteria presented.

– Chapter 4 presents results from the test.

– Chapter 5 discuss the result presented in chapter 4 in terms of validity reliability andanalysis on problems during the test design.

– Chapter 6 shows conclusions and future work.

– Chapter 7 gives acknowledgements and thanks to the supervisors

Chapter 2

Attack-graphs and associatedtools

Today there are many vulnerability scanners like Nexpose1 and Nessus2 to scan a hostfor vulnerabilities(Holm, Sommestad, Almroth, & Persson, 2011). However, even afterscanning all hosts in an organization with scanners like Nexpose and Nessus predictionsare still limited to the single host vulnerabilities. No predictions can be made on networksecurity as a whole and no predictions can be made about what vulnerabilities leads to theoutcome of the company server being compromised when they are not directly reachable bypresumed threat agents. Attack graphs, on the other hand, show multi-hop attacks in thenetwork and predict how vulnerabilities relate to each other.

To understand the concept of attack graphs, let us look at the different steps that isrequired for being able to construct a graph showing multiple-step attacks(Ramakrishnan,Sekar, & Brook, n.d.)(Sheyner, 2004)( S Roschke, F Cheng, R Schuppenies, C Meinel -Information Security, 2009). First, information about the running systems needs to begathered: software, network structures, host access list, and running services. Second, thecollected system information needs to be matched against a vulnerability database. Thisis not always simple since information about software vulnerabilities in public databases isstored in a non-uniform way which may lead to data loss when it is translated and usedby the tools of (Roschke, F Cheng, R Schuppenies, C Meinel - Information Security, 2009)finally, the system information and vulnerability information is analyzed and visualized in agraph. Graphs presented by attack tools are often very complex and more or less impossibleto fully comprehend, even for networks with only a few hosts and vulnerabilities.

When attack-graphs are evaluated against real attacks there is need of deciding whichattributes that are important to evaluate. All tools have the same purpose to produce allpossible attack paths on a system with information about the system and vulnerabilities,however there are some differences between them. The sections below describe differentvariants that are discussed in the literature (Ou et al., 2005)(Ammann, Wijesekera, &Kaushik, 2002)(Chu et al., 2010)(Jha, Sheyner, & Wing, 2002).

1www.nexpose.com2www.nessus.com

3

4 Chapter 2. Attack-graphs and associated tools

2.1 Monotonic or non-monotonic

When evaluating vulnerabilities scanners the concept of exploits is often mentioned, thisoften referenced to: Taking advantaged of something. Exploits in software are similar giventhe right chunk of data or commands the user can make unintended or unanticipated be-havior occur on computer software. The assumption that exploits can only give capabilitiesbut never take them away is called monotonic. So if a system uses exploits E1 and E2 thatgives capability C1 andC2 no other exploit or capability can remove or change the status ofthe already given capabilities. The effect is that execution of E1 and E2 in the same timethe result wont matter depending of which finish first.

When comparing attack graphs with real professional attacks the result from the graphsmay be less meaningful using monotonic feature because one side effect of a given exploitmay be to shut down or crash a given service, which renders the other exploit useless if itrelies on that service to run.

While monotonic attacks only gives privileges and there is no need to execute a given ex-ploit in a certain order to reach a goal, the non-monotonic is used when the order of exploitsmatter. When the preconditions to a goal G can only be met by executing the exploits E1and E2 in a given order and if E2 is executed before E1 the goal cannot be reached thisis also called non-monotonic attacks. One side effect of the non-monotonic attacks is thestate explosions given from each new state which results in exponential bigger digraph. Theauthor of [(Ammann et al., 2002)] states that most of the attacks in vulnerabilities databaseare stated as monotonic, and many tools do not state how the exploit works.

2.2 Single path or all paths

A single path is when a tool stops when finding a path of exploits that makes an adversaryreaching its goal. Singe path analysis is common when symbolic checking-tools, that onlyproduce counter statements is used. A counter statement is in this case an attack path.All paths are used when the tool produces all possible paths in a system, for example bycontinuing to search for more exploits after finding the first matches that reach the goal.

2.3 Backward or forward chaining

Backward chaining means starting at a goal (e.g., super user privilege of a host) and analyz-ing how to find conditions that can give this current goal. When all preconditions that leadto the goal are matched, the attack path is found. When using forward chaining the anal-ysis starts from the same capabilities the attacker is believed to start with. From this it isanalyzed how exploits can be used to gain other capabilities and, as long as new capabilitiesare found, where exploits can lead the attacker.

2.4 Probabilistic or deterministic models

A deterministic analysis produces yes and no statements concerning links in the attackgraph. Such definitive knowledge is certainly desirable. However, todays big and complexnetworks make this deterministic view of the network unsuitable (Singhal & Ou, n.d.).The deterministic view is assuming an attack will always execute successfully when thepreconditions are met, and never otherwise.

2.5. Visualization variants 5

The study of probabilistic modeling on attack graphs is a way to estimate the probabilitywhen executing a series of exploits successfully. The downside to probabilistic modelingis the need of empirical data to get meaningful probabilities. When using attack graphsthis becomes an extra layer of information when deciding how crucial the vulnerability is.When evaluating attack-graph there are many papers on making probabilistic models fromThe Common Vulnerability Scoring System(CVSS)( S Roschke, F Cheng, R Schuppenies, CMeinel - Information Security, 2009), where the vulnerability is evaluated on based metrics,temporal metrics and environmental metrics. However, the reliability of the method CVSSassigned to vulnerabilities is questionable; it appears to be very hard to obtain successfulprobabilities in reality( S Roschke, F Cheng, R Schuppenies, C Meinel - Information Security,2009).

2.5 Visualization variants

One known limitation of the attack graph usefulness is the visualization when dealing withbig networks with hundreds of machines or more. When evaluating real attacks on systemsit needs to have presentations that have good human readability. In figure 2.1 below,an attack graph from MulVAL are showing a small network with around 14 computers.Thereby it is necessary to evaluate attack graph results in form of human readability, so thedecision maker can make use of the information. If the presentation of attack paths has badcompression rate or many irrelevant paths the tool will render to be useless for the user.When the network grows in size together with the exploits in the system the permutationwill render the graph complexity into a complex task to find attack paths. A number ofvariants are available: GATOR,NAVIGATOR are tools that try to solve this.(Chu et al.,2010).

2.6 Input formats

To get reliable results there is a need of analyzing how formal input from the security sitesare loaded into the tool in a good way. Tools written by [(Jha et al., 2002)(Sheyner, 2004)]have no interaction or limited support for vulnerability scanners. The limited support forvulnerability scanners creates the need to put all known exploits for hand, which can beboth a tedious and error prone task for the user. Also, it can be a very tedious job to inputall running software on the machines and to know what network policies they have. Sometools support only manual input and some support import from the format of scanners likeNexpose and Nessus.

2.7 Logic-based or graph-based

The advantage of using a logic-based approach is the well-understood semantics of logic.This area is well understood, and the reasoning of logic-deduction systems is well-developedin computer science. Where are exploit are described in two ways: preconditions are true ifexploit is seen as successful giving capabilities to the adversary in which state it transformsto a post condition after an attack. Information about attacks and network configurationsmost be formulated in logic. An example of a proofgraph is presented in (Homer & Ou,2009) and shown in figure 2.2, where the MulVAL logical engine is described.

Figure 2.2 shows a proof graph from MulVAL, where p is gained priviliages, e is exploit,and c is configuration setting. The proof graph causality and relationship can be expressed

6 Chapter 2. Attack-graphs and associated tools

Figure 2.1: Attack graph visualization by MulVAL on a 14 computer network

as a Boolean formula where e1 Is inside a circle and seen as an AND expression needning itschildren-nodes to be true for the exploit to work. The p1 privileges here are inside a rombmeaning the logical OR expression needing either e1 or e2 to be true for gaining privilages.This can be expressed in logic(Homer & Ou, 2009), thereby its name logic-based approach.

The Graph-based approach can be seen as exploit-decency graphs to represent attacks onthe systems (Noel, Jajodia, OBerry, & Jacobs, n.d.)(Jajodia, Noel, & Berry, n.d.) insteadof pre and post conditions where exploits are shown as nodes and edges. This can be seenas changing single exploit paths together to build a multiple attack graph. This can forexample be done by the model-checking engine NuSMV3 as inference engine, where thefinal exploit is showing as a chain of prior exploits.

Logic-based advantage is that it has clear specifications of the casual relationship betweensystem configurations and the possible attackers privileges. When in a graph-based therewould be need of examine the Boolean variables that iterate over the graph to find whatcauses the adverse situation that enable the attack possible at a certain stage. The samesearch for the causes of the adverse would be specific at graph edges; it is also possible toenumerate all possible attack scenarios by a depth-first search. Logic attack graph should

3www.nusmv.fbk.eu

2.8. Proposed Attack Graph Solutions 7

Figure 2.2: Logic proof graph from MulVAL (Homer & Ou)

scale polynomial to the size of the network where the worst case complexity would beexponential (Ou, Boyer, & Mcqueen, 2006)

To avoid the exponential explosion model that comes from logic-based tools(Sheyner,2004)(Ou et al., 2005) where the attacks are encoded as in Computation Tree Logic(CTL)and the counter examples are modeled by NuSMV. The paper by P.Amman (Ammann et al.,2002) states that the logic-based graph likely to be prohibitively large since the state space isexponential in number of system variables. Even with advantage of the monotonic approachto keep state space linear in the number of system variables the problem to eliminate thefundamental exponential worst case do not disappear. Making the approach of graph-basedsearch a more realistic candidate on networks with thousands of machines.(Ammann et al.,2002)

2.8 Proposed Attack Graph Solutions

In figure 2.3 shows a summerized version of the main attributes in attack graphs are de-scribed. The subesctions after describes each indivudial tool that where studied in thisthesis.

8 Chapter 2. Attack-graphs and associated tools

Figure 2.3: Attack graph attributes summarized

2.9 MulVAL

The Logic-based Network Security Analyzer tool was started by a group from PrincetonUniversity and is an open source project. The main idea behind the project was thatnetwork and configuration rules could be described using Datalog. The Datalog inferenceengine uses the transition of attack graph into a Boolean formula to later solve it with SATsolvers.

Multi-host, Multi-stage Vulnerability Analysis Language (MulVAL) uses Datalog to pro-duce a reasoning system where each rule is declared as clauses. Datalog is based on firstorder logic and therefore it needs to be sound and completei, one issue however of Datalog isthat it is not Turing complete. Datalogs inference engine uses XSB(Ou et al., 2006) tablingmechanism to guarantee that the facts are only computed once. XSB is developed by StoneyBrook and is one of the key elements in why MulVAL attack simulations complexity level ispolynomial in relation to the network size.(Ou et al., 2005) The reasoning needs all relevantinformation encoded into Datalog facts where it uses a set of rules to understand the systemsbehavior. The basic structure of a literali, p(t1, . . . , tn) is a predicate with a set of argumentsencoded as Datalog facts. Datalog facts are either variables or constants where variables aredefined as upper-case and constants in lower-case letters. MulVAL represents each reasoningrules as a Horn clause filled with literals. (Ou et al., 2005) The example of Datalog tuplein MulVAL execCode(Attacker, Host, Owner) represents that an Attacker needs access toa Host and when exploited will gain the privileges of Owner. A limitation with MulVAL isthe weak modeling of expression, that limits the security polices for vulnerabilities, exploitsand attacks.

2.10. NeTSPA 9

2.10 NeTSPA

The Network Security Planning Architecture tool was built by MIT Lincoln Laboratoryand was developed in C++. NetSPA has support for input of software types and versions,firewalls rules exploits gateways between subnets and more. Future work was client sideattack support for trust relations between hosts. NetSPA Is planned to be commercializedby the CyberAnalytix Company.

2.11 TVA

The Topological Vulnerability Analysis tool was built by the George Mason University in2003 one limitations of the project was missing support for firewall and router rules. Andall exploits had to be imported by hand. There graph-search algorithm search for individualexploits between hosts, and chain them together to build multiple exploits paths.

2.12 Sheyners attack graph-tool

The Attack Graph tool was built by Carnegie Mellon University and is an open sourceproject. To build attack graphs it uses model-checking to evaluate attack paths in thenetwork(Lufeng, Hong, Yiming, & Jianbo, 2009).

The principle of validating complex systems can be done in many ways like simulationand formal testing and also model checking. Both testing and simulation has many prosand cons but the reason to use model checking is the need off finding all possible pathsgiven a set of rules and facts. The Attack Graph-tools modified model checking engine usesa finite state machine encoded with facts about the network and the hosts. The modifiedmodel-checker is applied to a finite-state machine crafted from network information so itcan provide all counterexamples from the given rules and facts. One more advantage of themodel-checking except automatic verification of the model is the exhaustive search giventrue if the model is correct or provides the counterexample to show otherwise.

When modeling the network and configurations parameters and attackers privilegesSheyner uses Booleans to describe each state. When the attacker gain or change the privi-leges it is modeled as state-transition relations.(Ou et al., 2006)

2.13 CySeMOL

The Cyber Security Modeling Language was built to model the architectures of SCADA(supervisorycontrol and data acquisition) systems. The Language contains general and security enti-ties so the user can model their own network as input. The modeled input is convertedto security calculating mechanism which gives an attack graph with a rough security in-dex on probability of a successful execution of attacks. The estimation of successful attackprobability is done by using Bayesian networks with conditional probabilities formula.

2.14 Previous testing of attack graphs

Several topics related to attack graphs has been thoroughly studied: how to predict theprobability of successful attacks (Wang, Islam, Long, Singhal, & Jajodia, n.d.), how toextract meaningful data from vulnerability databases (Ammann, Wijesekera, & Kaushik,

10 Chapter 2. Attack-graphs and associated tools

2002)(Chu et al., 2010)(Jha, Sheyner, & Wing, 2002) and the study of logic-based reasonengines together with their complexity(Ou et al., 2005)(Jha et al., 2002). Interestingly,however, no studies were found if attack graphs output was useful for decision making byevaluating the end results and comparing it to observed outcomes. However, some work hasbeen spent on making attack graphs comprehendible for humans, which is a non-trivial taskas figure 2.1 shows.

The need of compression on attack graphs is a known problem (Long, n.d.) when thereachable states produced are all hosts multiplied by the number of vulnerabilities. Thecomplexities in readability reaches quickly quadratic and output yields a small network of14 hosts and 10 vulnerabilities unreadable by human eye as figure 2.1 shows.. This is aproblem when enterprise network and small business networks rapidly can reach thousandsof computers and make the attack graph output infeasible.(Long, n.d.)

The state explosions that occur when adding more hosts and vulnerabilities. This canbe handled by removing redundant information. The reduction assumption is when we havetwo hosts with the same vulnerabilities and network configuration, the attack path is seenas one when the going through Host 1 or Host 2 shows the same ending result(Long, n.d.).If no reduction is made on bigger attack graphs it is hard to trace the different paths whichcan make the graph meaningless.

There have been some progresses to tackle the known need for scalability of humanreadability in attack graphs, where even medium sized networks attack graph can becomeincomprehensible. The assumption of monotonicity was not only a discovery to addressthe scalability of model-based checking but also a way to reduce the complexity of attackgraph representation. One solution is to only mask the problem by showing the graphs inhierarchical view. The suggested way by (Long, n.d.)is to compress the hosts with identicalnetwork and vulnerability information to a one-host model.

Chapter 3

Method

To answer the research questions from introduction an test was performed with the purposeof evaluating whether the attack graphs can predict real attacks on modern IT-systems, andif so, how well they do this. In Methods an attack graph-tool is choosen in 3.1 and thetarget networks and machines are described in 3.3,3.4 and the attackers in 3.5. Last are thetest design described with its vulnerability collection and configurations together with thetest criteria in 3.6-10.

3.1 Selection of attack graph-tool

When implementing attack-graph tools for the CRATE environment the approach was tofind the most suitable tool in collaboration with CRATE to get as realistic analyzes aspossible for it. The given time and funding to buy software limited choices. Due to nofunding money the attack-tool needed to be an open-source solution, all requests for trailsoftware were declined. This leaves the options MulVAL and Sheyners. Since the CRATEenvironment was incompatible with Sheyners choices of software there was old packagesfrom 2006 that was not compatible with the modern CRATE system, one example was theold software Plan91 that was hard to get running. There was however no time for buildingbig workarounds in the given timeframe, which leaded to MulVAL (described in Chapter2.2.1) being the best candidate for this test.

MulVAL is commonly used in research on attack graphs (Ou et al., 2005)(Azgomi,2012)(Saha, 2008)(Kordy, Pietre-cambacedes, & Schweitzer, 2013) and has several goodfeatures. It is open-source making it easier to understanding the program and easy tomodify after CRATEs specifications. It is still people working on the project making itup-to-date in terms of software and possible to ask for support. Sheyners last release was2007 compared with MulVALs 2012.

There was only time to test one vulnerability scanner, to be used for feeding MulValwith vulnerabilities. The study presented by (Holm et al., 2011) shows little differencebetween the scanners NeXpose, Nessus, SAINT, and McAfee when using authenticatedscans. MulVAL recommends Nessus, and have built a simple python script for convertingscan results to MulVAL input. Nessus is however not free and has limited trail period.NeXpose was my choice of scanner showing both good results in the comparison studypresented in (Holm et al., 2011) and it is free. Only small modifications were needed to use

1www.plan9.bell− labs.com/wiki/plan9/software for plan9/index.html

11

12 Chapter 3. Method

the pre-built python script for input conversion from NeXpose to MulVAL.

3.2 Dataset and its properties

The background of this dataset is from an exercise conducted in 2012 where two teamscompeted against each other to find secret keys hidden inside a few selected networks.Targeted networks, attackers and performed attacks are described below.

3.3 Target networks

The game network contained a mix of organizations illustrating IT-structures of Schools,IT Development firms, Industry, and much more. The goal of the competition was to findseven cryptological keys, hidden inside computers of nine organizations computer networks.After putting the seven keys together, a master key could be forged to decrypt the secretsof an USB-drive that was found earlier.

Attackers where split up to form two teams. One team was represented by securityresearchers at Swedish Defenses Research Institute (FOI). The other team was also fromthe Swedish Military specializing in IT-security, and has as daily job to protect valuabletargets from attacks.

The only way to find the hidden keys is by searching each organization for computerskeeping them. To be able to navigate the networks the skills of passing firewalls, guessingpasswords, scanning for weaknesses and using exploits is tested.

The monitored networks for results were of different types in terms of infrastructure,firewalls and patch levels on the operating system. The monitored organizations had thefollowing domains: Gamlenytt.ex, Nybanken.ex, Hollandspannkakor.ex , Glasspartiet.exe,PerleboWhiskey.ex, Bgskrot.ex, BCN.ex . . . An overview is given below of the networks.

The Gamlenytt network is designed after the IT-Architecture of a typical newspapercompany with many storage servers with information, and hosting of homepage with forumsas seen in figure 3.1. The security should be relatively good on both server side and oninternal network side. The firewall should be of professional class but with holes in it. Thereare three types of nets behind the firewall; DMZ,SERVER and CLIENT. DMZ contained thewebserver of the newspaper, Mail server, DNZ for domains, STREAM for streaming files,DB databases for web applications. SERVER keeps common files, user accounts, internalmail, database applications, and ERP businesses/economy systems. CLIENT side is a mixof OS and patches representing the workers of the newspaper.

Nybanken and Hollandspannkakor have the IT-Architecture of a middle size company ofgood IT-security standards as seen in figure 3.2. Like a consultancy firm that normally storesand keeps information about other companies, there might even be complete documents stor-ing passwords and usernames over other companies and networks. Firewall is of professionallevel but with holes. Behind the firewall the protected zones DMZ,SERVER,CLIENT ex-sists. DMZ have for example the webserver of the newspaper, Mail server, DNZ for domains,STREAM for streaming files, DB databases for web applications. SERVER keeps commonfiles, user accounts, internal mail, database applications, and ERP businesses/economy sys-tems. CLIENT side is a mix of OS and patches representing the workers of the newspaper.

3.3. Target networks 13

Figure 3.1: example of newspaper company

Figure 3.2: Example of middle size company

Glasspartiet and PerleboWhiskey are designed as nonprofit organizations, they are rel-ative insecure networks and have a lot of security holes and are running a diversity of

14 Chapter 3. Method

Figure 3.3: Example of nonprofit organization

operating systems as seen in figure 3.3. The firewall is simple and has a lot of holes directlyinto the network. All users are in the demilitarized zone (DMZ) directly behind the localfirewall. The patch levels and operating system of computers running are different.

The last category of organizations has the IT-Architecture of a common TV-station asseen in figure 3.4; it has relatively good security on the internal it-structure and on theclient side. Firewall should be of professional level, but whit a few holes in it. The patchlevels on the operating system and running software versions on the running computers aremixed.

3.4 Target machines

Machines inside the target networks are a mix of different operating systems OS and softwarerunning. The running operating systems in this test environment are Linux distributions(debian, Gentoo) Windows (2000, XP) and more with different packet updates and ser-vice pack versions installed. Software running was for example Adobe Reader, OpenOffice,Adobe Flash, Java, SolarWindTFTP, Golden FTP server, Apache, Wordpress, Internet Ex-plorer in different versions an complete list of software and os can be found in appendix(Machines software list summary) (Machines OS list summary). Important to know is thatthere was no patching or updating done on the software, so when the virtual machines hadbeen rolled out in the lab environment no more updates could be made. Because of thehigh number of missing patches on the machines it is safe to say that the target machinesin this exercise are more vulnerable than machines found in typical organizations. To makethe isolated environment in CRATE as ”real” as possible, the system had bots on each

3.5. Attackers 15

Figure 3.4: Example of media network

machine running different scripts to open incoming mails and run search queries. The traf-fic generated was built from previous studies on real users and how they used computersin their normal work. The bots simply replayed actions of the same type as real users atthe same time of day. However, the bots were limited to a predefined set of actions (surf,read email, send email, open file) and they were ignorant to the content of emails and files,probably making them more likely to open emails and attachments than users in the typicalorganization.

3.5 Attackers

The participants are working as researchers or analyst in the IT security field and will bethis paper references as security professionals. In this paper the evaluation will be based onthe two attacking teams and their ways of compromising the systems. The attackers wereonly allowed to use public known exploits in their efforts to compromise systems.

The procedure to break into a system is summarized by the offensive security tool Metas-ploits2 startup message showed in fig 3.5, where an attack is divided into four steps RECON,EXPLOIT, PAYLOAD, LOOT. Recon is for scanning systems for vulnerabilities. Exploit isthe ability to find vulnerabilities that can give attacker wanted privileges in this case superuser privileges. Payloads can be remote accessible trojans, specialized command shells thatare installed to gain and keep the escalate privileges for the user. Loot is treasures foundinside a compromised system, in this case cytological keys.

2www.metasploit.se

16 Chapter 3. Method

Figure 3.5: Metasploit console startup message

When the attacker recon networks they mainly used the open source tool Nmap (Net-work Mapper) to find live hosts and their open ports. To find organizations name servers(DNS) the program dig (domain information groper) was used. A complete list of scans andtheir tool of choice can be found in appendix under recons. When analyzing exploits usedduring the exercise, the frequency of some was more popular than others. The following vul-nerabilities were popular to exploit CVE-2008-4250, CVE-2003-0352, CVE-2007-3039. TheCommon Vulnerabilities and Exposures (CVE) is a reference-method for publicly knowninformation-security vulnerabilities and exposures. Every CVE is unique and listed onMITREs system as well as the US National Vulnerability database3, where more informa-tion can be found about the vulnerability.

The CVE-2008-4250 is a weakness in the Server service from Microsoft Windows 2000to Windows 7 Pre Beta that allows remote attackers to execute arbitrary code via a craftedremote procedure call RPC. This exploit is also known as the ”Server Service Vulnerabil-ity”. The CVE-2003-0352 is also a buffer overflow attack that work in certain DistributedComponent Object Model (DCOM) interfaces for RPC.

The CVE-2007-3039 vulnerability is stack-based buffer overflow in the Microsoft MessageQueuing (MSMQ) service in Microsoft Windows 2000 Microsoft XP SP2, but is remotelyexploitable on Windows 2000 Server which makes it popular on these machines. Whenthe vulnerability is identified the attacker needs a payload appropriate to the privilegeescalation needed for the task. The attackers wanted super user access to search the hostfor the cryptological key, so the common documented payload was ”reverse tcp connection”which is normally a Windows bind shell connection to the host. Loot in this exercise wasfound if the compromised host contained a cryptological key.

Observations from the exercise showed that most attacks were made with the operatingsystem Backtrack 5 R3 preprepared with all mentioned tools for recon. Backtrack alsocontains the penetration testing software Metasploit that was used by many attackers tolunch payloads on targeted machines. A complete list of attacks documented is found inapendix under attacks.

3www.cve.mitre.org

3.6. Test design 17

3.6 Test design

The collection of system and vulnerability information is vital for the predictions of attackgraphs. Due to the nature of dependency in attack-graphs it is important to get each hostscanned as good as possible, if one key host has invalid information the attack graph becomesmeaningless for decision makers. Another important aspect is runtime configurations ofMulVAL as it needs key input like host accesses control list (HACL), where attacker islocated, accounts of different users, network services running and much more before it canrun calculations. In the test key attributes has been filtered out based on how importantthey are for the prediction of real attacks. Another key aspect is interpretation of theattacker log that attackers produced during the exercise. Every attack and recon attemptwas documented by the attacker itself, and when decoding their notes it seemed that everyattacker used different tactics. Finally, in this test design I have sorted out the key aspectsand criteria of comparing real attacks with predicted ones. More details on the test designare given in the following sections.

3.7 System and vulnerability collection

There are pieces of information MulVAL wants to know besides vulnerabilities to make pre-dictions. The following are the ones that was used in this test and which i believe is mostimportant to make predictions as good as possible. I want a high level abstraction of the run-ning system, what network services are running and which port its using(”networkService”),what users have root accounts and on which hosts (”hasAccount”) and the network topologydescribing the connectivity of the hosts (”hacl”). To find the best possible configurationsand to make sure I used the correct settings I have frequently exchanged mails with thedevelopers behind MulVAL.

To collect running network services and their respective vulnerabilities I used NeXposesauthenticated scans on each host several times. Data about network topology and useraccounts was manually entered based on the deployment specification used in CRATE. Noother efficient way to provide the network topology to Mulval was found. To make sure Ifound as many vulnerabilities as possible, the scanners hade to run several times, only thencould I really trust that the scannings made by NeXpose was as good as possible.

3.8 Configuration of MulVAL and the analysis

Modifications to MulVAL project was need in my test design. MulVAL default mode usesthe parameter ”hacl(h,h)” to describe network topology, this means that all hosts are in thesame network zone with full connectivity.

MulVALs output can be configured to generate different files: txt, xml or a pdf file canbe selected. The visual pdf was unreadable after a few hosts were added, but also the time togenerate the pdf with DOT (Graph Description language) took longer time then generatingthe attack graph with XSB. In this design I used xml files.

Results given from MulVAL are very complex for the human eyes. To determinate everyattack path, a program was made using the XML as input to construct a digraph. Byrecursively step thru the digraph starting from the attacker located position and traverseto the search depth of 60 steps. This constant is taken to speed up searches and is set afterthe longest multi-step attack that was four steps.

18 Chapter 3. Method

Due to vast amount of paths in the digraph a time limit was set on the recursive breadth-first search function for attack paths in MulVALs generated XML file. This made it possibleto produce a result even if a complete enumeration of attack paths could be determinedwithin a reasonable time. The time limit used was 15 minutes and by this time 30 000attack paths had been created between all the network hosts. So, even if the complete set ofpossible attacks were not computed the most straightforward (in terms of number of steps)attacks paths were found and more paths than any human could handle was enumerated.

3.9 Definition of ground truth and analysis method

Security professionals documented every found path into the system with the specific typeof vulnerability. Next was checking if their attack is covered by the attack-graph tool; thisis done by checking if MulVAL have any records of the given vulnerability and if there wasone path between attacker and victim containing this particular vulnerability. The sevenprofessional hackers results were documented in a spreadsheet showing actions and attacks.All organizations traffic and attack was recorded during the event, making the variations ofattacks and hackers different on each organization of interest. After analyzing MulVALs out-put, victims were sorted by the definition: A host is a victim when the attacker can executearbitrary code on the host. The test was done by comparing victims in MulVALs outputand the victims of the attackers and assessing if MulVal predicts if hosts are vulnerable fromoutside the company firewall.

3.10 Information retrival metrics

Receiver operating characteristics (ROC) graphs are common in medical decision making,but have in recent years also used in machine learning and data mining. ROC graphs aretechnics for visualizing, organizing and selecting classifiers based on their performance(Fawcett,2006). ROC classification models are based on mapping elements into two different set ofclasses. The first set is divided into negative n and positive p instances where each elementgets mapped to the set (n, p). The second one is used to predicted class of the instanceswhere the positive y and negative n predictions are labeled (y, n). The classifiers of aninstance have in ROC models four different possible outcomes. Given an instance is posi-tive and also classified as positive it gets counted as true positive; if the instance is insteadnegative it is classified as false negative. The same model is used if the given predictionis negative and also classified as negative it gets counted as true negative; if it is insteadclassified as positive it its classified as false negative. The ROC model describes this conceptof two-by-two row column matrix as a confusion matrix (see figure 6). In this test the con-fusion matrix represents a strict evaluation system that can classify instances from attackgraphs into the right class and set.

In this test four instances were used as classifiers for evaluating predictions made byMulVAL and determine what class and set an instance corresponds to in terms of realattacks. In appendix under Test criteria a flow chart for the decision making can be found.The most common methods to measure information retrieval systems (IR) are the use ofrecall, precision and accuracy:

Precision is how many of the predicted attacks are relevant to the decision maker. Pre-cision are also a measure of quality in predictions and also in this context known as positive

3.11. Number of attack paths and attacker actions 19

Figure 3.6: Confusion matrix by the ROC model

predicted value (PPV). Recall is the measure of completeness and relevant data it covers allattacks made on the system and also in this context true positives. Accuracy is the overallcorrectness of the predictions made.

The terms precision and recall are together good common measures for both quality andprecision in this context. A normal effect is that increasing either recall or precision willgive a negative effect on each other. If a high rate of true positives is wanted to cover asmany attack paths as possible you need a high recall. The side effect it often means loweringthe quality in means of preciseness when calculating more attack paths. In ROC systemsrecall is referred as the true positive rate of a given classifier. ROC models also refer recallas sensitivity and positive predicated values as precision when classifying instances.

3.11 Number of attack paths and attacker actions

Decision makers do not only want to know which hosts are compromised they also want tofind out how the attackers breached the network so they can fix it. If focus is only on theattack-paths predictions made by MulVAL we can find the answer to the interesting question

20 Chapter 3. Method

Figure 3.7: Equations for calculating recall, precision and accuracy

”Can MulVAL predict real attacks successfully?” The usability of MulVAL is limited to thecomplexity of the output produced. For every extra path that is not relevant would beseen as a lowered score for Attack-graphs. There are many downsides of producing toomany invalid paths. Ideally, the attack graph tool would produce just those graphs thatreal attackers would exercise.

Attack graphs typically contain a large number of paths. An intuitive way of interpretingthe output is that a large number of possible paths will mean that the host or network is morevulnerable. It is also reasonable to expect that seemingly vulnerable networks will attractattackers so that they spend more effort searching them. To provide a rough estimator ofthis this test assessed the number of predicted attack paths, the number of network scansperformed by attackers, and the number of hosts that was successfully compromised in eachnetwork.

Chapter 4

Results

In this chapter the result made from the test are presented, evaluating the prediction per-formance of MulVAL. These results are divided into results regarding ROC model in 4.1and results regarding attack paths in 4.2.

4.1 Information retrieval metrics

The confusion matrix for the 199 cases that are covered by the attackers actions is givenin Table 5. As shown in the table, for the 65 cases where MulVAL predicted a successfulattack path the attackers only managed to find one in 6. Thus, the portion of true positives,or precision, is only 9 percent. In the 134 cases where MulVAL predicts that the host isunreachable (no attack path is found) the attackers were able to find a working attack pathin 26 cases. Thus, the recall is only 19 percent. Accuracy combines all fields in the confusionmatrix into one value. MulVALs accuracy is 57 percent.

Figure 4.1: MulVAL mapped with confusion matrix

All predictions were given by comparing all attack paths made by the attackers and

21

22 Chapter 4. Results

collecting all attack predicted attack paths from all networks. A complete list of attacksand predictions are found in the Appendix under (Attack predictions).

4.2 Number of attack paths and attacker actions

In fig 4.2 shows how many attackers scanned the organization to find vulnerabilities and thenumber of attackers who recon the networked varied from one to seven. MulVAL predictedhow many attacks paths an attacker should find from the current network position givenhe scanned or attack the network. Last shows how many from the MulVAL predictionsthat the attacker ended up using, when attacking the network for privilege escalation. Forexample, four attackers scanned Bgskrot.ex organization for vulnerabilities, when locatedoutside on the Internet. If attackers where located on the Internet so did MulVAL predictover 80000 critical paths that could lead to privilege escalation giving attackers super useraccount privileges. When the professional hackers later tried to attack the organization,they only found one of the predicted paths correctly, that lead to super user privileges.

Figure 4.2: MulVALs predictions and the actions of the attackers

As fig 4.2 indicates only a fraction (0.02 percent) of the attacks that MulVAL predictedas possible was actually successfully performed by the attackers although multiple scanswere performed.

Chapter 5

Discussion

This chapter discusses some of the troubles during the test, nuisance variables and thepossible side effects that threats the validity of the test design. The threats to validity ispresented as arguments in 5.1-5. Last is discussion about general reflections and experiencesmade throughput the study in 5.6-8.

5.1 The accuracy of the attackers log?

When analyzing documents showing what the attackers did during the contest a few typo-graphical errors was found. Some was copy paste errors showing successful exploits thatshould only vulnerable on Windows not against a Linux machine. If a majority of the at-tacks in the log are errors then the dataset would be useless, making the test meaningless.So yes there are errors in the documents, but a lot of attacks are done by several hackersand thereby strengthening the results against errors.

5.2 The competence and resources of the attackers?

A team is represented by seven people with different background all within the same fieldof expertise but some users may have more experience in the field of penetration techniquesthan other people. The attackers in this paper did have a mixed background in the ex-pertise of Information-security; some have majors in cryptology or specialization in reverseengineering and some in pure offensive security like penetrations. So one can think theyhave different tactics in analyzing targets and finding weaknesses. If a given attack is notperformed correctly it will fail; MulVALs predictions will always assume that an attack isalways successful and so low competence by the attackers can make the test results mean-ingless. When doing the comparison it is difficult to keep track of which attack is made bywhom. This uncertainty in results makes it hard to know if an attack failed just because itwas not possible or if there was simple mistake of configuration to the attack tool.

23

24 Chapter 5. Discussion

5.3 The large number of vulnerabilities in the testedorganizations?

The target networks may have a lot of vulnerabilities compared to similar it-infrastructuresrunning in a similar environment. One may think the large number of exploits found oneach machine would make the result less realistic compared to an up to date system. Allvulnerabilities in the system are picked from common exploitable vulnerabilities and shouldgive meaningful information for attackers and MulVAL in ways of attacks and predictions.

5.4 Tested attacks are arbitrary deepening on the at-tacker?

This was not arbitrary attacks due too profession of the attackers. One can assume thatattacks made during the contest is the simplest possible, or fastest to perform for an attacker.This assumption is due to the circumstance of the contest, finding clues in a time-limitedenvironment would pressure the attackers to act fast leaving only a few minutes per host.It is true that only because an attack was not shown it does not by any means give lessmeaningful information. But comparing all unmade attacks with MulVAL would give toomuch information to analyze for this paper. The purpose of MulVAL is to help the decisionmaker to find multiple step attacks that are located inside or outside the network. Thiscan also be seen as help to filter vulnerability scanner findings, because checking all foundvulnerabilities is a tedious and complex task. But if no reduction of vulnerabilities can bedone after running a MulVAL scan, MulVAL is not aiding the decision maker today.

5.5 This is more a test of the accuracy of the scannersfeeding MulVAL information?

When scanning with NeXpose I used authenticated scans to get reliable information. Onetrouble was that even with authenticated scans the result can vary between scans: Exampleis hosts that do not show windows service vulnerability even though it should have it andneeded over five scans to find this specific vulnerability, making me unsure how much youcan really trust a vulnerability scanner. If the vulnerability is not found by NeXpose thenthe information given to MulVAL is wrong, the host is then useless for this test. MulValrecommends using Nessus as their vulnerability scanner but shown by (see Holm et al)the performance is similar to my choice of Nexpose, giving MulVAL a fair trial. BecauseI spent a lot of time trying to get the scanners right, one might say this is an optimistsinterpretation on how well MulVAL would predict attacks. But yes, in a purely theoreticaltest where MulVAL have perfect input the numbers the experiment would show differentresults.

5.6 MulVAL is not user friendly

MulVAL needs a lot of custom parameters edited to start and yield meaningful results. Torun multiple network test there was need to edit a lot, and a lot of time was spent to programan interface to MulVAL. Real troubles started after producing the attack graphs, if networkshad too many hosts my instance of MulVAL crashed. However, the major weakness is the

5.7. Number of attackers on network 25

result; PDF format is meaningless if you have more than four hosts in ways of readability.The option of xml and txt is good, but then you need to implement your own analyzingsoftware to make sense of them. I hope the authors of MulVAL improve the usability inboth input and output, to get a more meaningful program.

5.7 Number of attackers on network

To make the attack simulation as real as possible the event had multiple organizationsrunning simultaneously, making it impossible for the hackers to recon every network forattacks. The downside of having a low number of attackers may result in that every networkmay not be attacked by all security professionals. The combinations of attacks tried by theattacker are lower compared if there was only one organization running. The attacks on thenetwork are more than a landmark on what the organization may suspect by a professionalmalicious attacker searching vulnerabilities in the network for a shorter period of time.

5.8 The professional attacks was not chosen by random

The professional attacks can have been chosen to be the most unpredictable ones by Mul-VAL, and security scanners. The attackers can by their domain knowledge use only theattacks that are most likely to work, and skip the most strange and far-fetched ones. Thisought to increase the accuracy of MulVALs predictions.

26 Chapter 5. Discussion

Chapter 6

Conclusions and future work

This thesis presented an evaluation study to test the attack graph approach them underrealistic circumstances. The aim is to assess if attack-graph-based tools are useful in practice.More concretely, the question addressed is: Can attack graphs based tools accurately predictattacks success? A question which, in spite of a considerable number of articles on attackgraphs, has not been answered by the research community. The study focuses on theattack graph tool MulVAL and the scanner Nexpoise to compare a series of attacks madeon different systems. Tests shows interesting results that attack graph tools like MulVALdo have trouble to successfully predict real attacks. The output from MulVAL shows badperformance, giving decision-makers a tedious task to filter out useful information. Testshows that prediction given by MulVAL should be evaluated with care and be used withother security tools like vulnerability scanners to make prediction on how well the networksecurity is. MulVAL are still showing hope in predicting attacks, but needs both strictand correct input for user to get meaningfully results. To the decision maker one candirectly say that the technology is not ready for commercial use, at least when evaluatingthe performance given by MulVAL. However, if one still decides to try this, one should reallyspend a lot of time on the input parameters: vulnerabilities and hacl are key elements to areliable result when trying to get the results as good as possible. What one could expect byusing MulVAL is at least a lot of false positives, in terms of predicted attack paths.

The following areas should be studied in future work, in order to gain an even betterunderstanding on the subject. First are the tests of different reasoning engines to see ifthey all have similar results in prediction performance. Secondly, different sources of inputshould be tested, in order to make sure that all detected vulnerabilities in a network arecorrect.

27

28 Chapter 6. Conclusions and future work

Chapter 7

Acknowledgements

I would like to thank my supervisors Helena Lindgren at Department of Computing ScienceUmea University and Teodor Sommestad, and all people at The Swedish Defence ResearchAgency (FOI) who has taken time to help me and answered my questions. Special thanksgo to Dr. Teodor Sommerstad whom really has helped me and guided me in this project.

29

30 Chapter 7. Acknowledgements

Chapter 8

References

Ammann, P., Wijesekera, D., & Kaushik, S. (2002). Scalable, graph-based network vulner-ability analysis. Proceedings of the 9th ACM conference on Computer and communicationssecurity - CCS 02, 217. doi:10.1145/586139.586140

Azgomi, M. T, M. A. (2012). A New Approach for Security Risk Assessment Causedby Vulnerabilities of System by Considering the Dependencies. International Journal ofComputer Science and Information Security, 8, 338-346.

Chu, M., Ingols, K., Lippmann, R., Webster, S., & Boyer, S. (2010). Visualizing AttackGraphs, Reachability, and Trust Relationships with NAVIGATOR , 22-33.

Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters,27(8), 861-874. doi:10.1016/j.patrec.2005.10.010

Holm, H., Sommestad, T., Almroth, J., & Persson, M. (2011). A quantitative evaluationof vulnerability scanning. Information Management & Computer Security, 19(4), 231-247.doi:10.1108/09685221111173058

Homer, J., & Ou, X. (2009). SAT-solving approaches to context-aware enterprise net-work security management. IEEE Journal on Selected Areas in Communications, 27(3),315-322. doi:10.1109/JSAC.2009.090407

Jajodia, S., Noel, S., & Berry, B. O. (n.d.). Chapter 5 TOPOLOGICAL ANALYSIS OFNETWORK ATTACK VULNERABILITY, 1-20.

Jha, S., Sheyner, O., & Wing, J. (2002). Two formal analyses of attack graphs. Proceed-ings 15th IEEE Computer Security Foundations Workshop. CSFW-15, 49-63. doi:10.1109/CSFW.2002.1021806

Kordy, B., Pietre-cambacedes, L., & Schweitzer, P. (2013). DAG-Based Attack and De-fense Modeling: Dont Miss the Forest for the Attack Trees, 318003(318003), 1-57. Long, T.(n.d.). Attack graph compression.

Lufeng, Z., Hong, T., Yiming, C., & Jianbo, Z. (2009). Network Security Evaluationthrough Attack Graph Generation, 407-410.

31

32 Chapter 8. References

Noel, S., Jajodia, S., O’Berry, B., & Jacobs, M. (n.d.). Efficient minimum-cost networkhardening via exploit dependency graphs. 19th Annual Computer Security ApplicationsConference, 2003. Proceedings., 86-95. doi:10.1109/CSAC.2003.1254313

Okhravi, H., Robinson, E. I., Yannalfo, S., Michaleas, P. W., Haines, J., & Comella,A. (n.d.). TALENT: Dynamic Platform Heterogeneity for Cyber Survivability of MissionCritical Applications.

Ou, X., Boyer, W. F., & Mcqueen, M. A. (2006). A Scalable Approach to Attack GraphGeneration.

Ou, X., Govindavajhala, S., & Appel, A. (2005). MulVAL: A logic-based network secu-rity analyzer. 14th USENIX Security, August, 8. Retrieved from http://portal.acm.org/citation.cfm?id=1251398.1251406

Saha, D. (2008). Extending logical attack graphs for efficient vulnerability analysis. InCCS 08: Proceedings of the 15th ACM conference on Computer and communications secu-rity (pp. 63-74). doi:10.1145/1455770.1455780

Sheyner, O. M. (2004). Scenario graphs and attack graphs. Architecture. Retrievedfrom http : //citeseerx.ist.psu.edu/viewdoc/download?doi = 10.1.1.83.6220& rep =rep1& type = pdf

Singhal, A., & Ou, X. (n.d.). Security Risk Analysis of Enterprise Networks Using Prob-abilistic Attack Graphs.

Wang, L., Islam, T., Long, T., Singhal, A., & Jajodia, S. (n.d.). An Attack Graph-BasedProbabilistic Security Metric.

(Alhomidi & Reed, 2012; Heberlein et al., n.d.; Lippmann & Ingols, 2005; Roschke,Cheng, Schuppenies, & Meinel, 2009), Towards Unifying

8.1 Appendix A

Not necessary!