Asset Criticality in Mission Reconfigurable Cyber Systems and its Contribution toKey Cyber Terrain

Peyton Price∗, Nicholas Anthony Leyba∗, Mark Gondree†, Zachary Staples∗, Thomas Parker‡∗Naval Postgraduate School

nicholas.a.leyba.mil@mail.mil, zhstaple@nps.edu†Sonoma State University‡thomas.c.parker@navy.mil

Abstract—The concept of a common operational picturehas been utilized by the military for situational awarenessin warfare domains for many years. With the emergence ofcyberspace as a domain, there is a necessity to develop doctrineand tools to enable situational awareness for key-decisionmakers. Our study analyzes key elements that define cybersituational awareness to develop a methodology to identifyassets within key cyber terrain, thus enabling situationalawareness at the tactical level. For the purposes of this work,we treat critical assets to be key cyber terrain, given thatno formal study has determined differences between assetcriticality and key cyber terrain. Mission- and operationally-based questions are investigated to identify critical assets withthe TOPSIS methodology. Results show that the ICS systemcan be evaluated using TOPSIS to identify critical assetscontributing to key cyber terrain, enabling further researchinto other interconnected systems.

Keywords-key cyber terrain, cyber situational awareness,MADM, TOPSIS, industrial control system, cyber physicalsystems.

I. INTRODUCTION

In the cyber warfare domain, the development of a com-mon operational picture is necessary for the war fighterto appropriately deploy defensive countermeasures and of-fensive capabilities. Without situational awareness, the warfighter may make suboptimal decisions that will put peo-ple and equipment at risk. Current capabilities for cybersituational awareness are limited to network defense op-erations, such as intrusion detection, attack trend analysis,information flow analysis, damage assessment, and intrusionresponse [1]. To accurately attain cyber situational aware-ness, one must identify critical assets with respect to themission and their importance to that mission. The criticalityof individual assets and the relationship between assetsdescribe the key cyber train for a given mission. Withoutidentification of key cyber terrain, the war fighter suffersfrom reduced situational awareness and a diminished abilityto defend and operate in the cyber domain.

Part of the difficulty in identifying critical assets in thecyber domain is in the dual logical and physical abstractionsof the domain, the deep technical requirements for assessingasset behavior in a contested environment and the domain’s

complexity due to the growth of individual technologiesinto systems-of-systems [1]. Without asset or event filtering,interpreting the volume of data itself poses a deep technicalchallenge [2].

In support of this goal, our study validates an existingmethodology for analyzing cyber asset criticality to deter-mine key cyber terrain in the context of a non-trivial casestudy: a reconfigurable, shipboard industrial control systemin use by the US Navy. We employ a hierarchical variantof TOPSIS, a multi-attribute decision making (MADM)strategy selected by Kim and Kang [3] as promising fordefining cyber asset criticality. Our main contributions areas follows:

- We employ hierarchical TOPSIS in a complex setting,using the analytical hierarchy process for derivingweights for use in TOPSIS, exploring its distinguisha-bility and sensitivity with respect to SME input;

- We “fill in the gap” for operationalizing this method-ology, including defining cybersecurity criteria, decom-posing a complex system into families and identifyingmissions;

- We explore criticality and key cyber terrain for a cyber-physical system for insight at the tactical level, whereasprior work focuses primarily on IT assets and theoperational/strategic levels;

- We demonstrate that, as suspected, asset criticality ismission-dependent and contextual;

- In the context of our study, we find that weightingassets in relative importance to a mission plays amore important role in identifying cyber key terrainthan ranking the relative importance of cybersecuritycriteria.

Organization: In Sec. II, we discuss background and re-lated work on identification of key cyber terrain. In Sec. III,we outline the methodology employed for our case study, ahierarchical variant of TOPSIS. In Sec. IV, we discuss themissions, criteria and assets of our case study. In Sec. V, weoutline the factors explored in our case study and the resultsof our analysis. In Sec. VI, we conclude and discuss futurework.

6042

Proceedings of the 50th Hawaii International Conference on System Sciences | 2017

URI: http://hdl.handle.net/10125/41893ISBN: 978-0-9981331-0-2CC-BY-NC-ND

II. BACKGROUND

The US Navy has prioritized the identification of keycyber terrain and cyber situational awareness in U.S. FleetCyber Command / Tenth Fleet: Strategic Plan 2015–2020.In particular, its Strategic Initiative 1.1 focuses the effort bystating:

For each network . . . and for each mission, wewill 1) define key terrain, 2) identify or defineoperational availability (Ao) for that terrain, and3) track how well we maintain Ao. This increasedunderstanding will ensure we can successfully de-fend and fight through those key— and sometimesdecisive— terrains. [4]

In our work, we treat asset crticiality as the sole measureablefactor contributing to key cyber terrain. In the absenceof work more concretely characterizing the relationshipbetween these, we believe this is appropriate. We discussexisting efforts to describe key cyber terrain, analysis meth-ods of determining critical cyber assets and related work.

A. Key Cyber TerrainTraditionally, key terrain is defined as “any locality, or

area, the seizure or retention of which affords a markedadvantage to either combatant” [5]. In the cyber domain, keyterrain involves network links and nodes that are essentialto both friendly and adversarial forces [6]. What constituteskey cyber terrain has been disputed [7]–[11], but Franz[11] defines it as “the physical and logical elements of the[cyber] domain that enable mission essential war fightingfunctions; is temporal; . . . and is applicable across strategic,operational, and tactical levels of war”.

Raymond et al. [10] propose a framework for character-izing cyber terrain along the following planes: supervisory,cyber persona, logical, physical, and geographic. This frame-work partially aligns with DoD Joint Publication 3-12(R) [6]which depicts cyberspace into the three layers: the physicalnetwork layer, the logical network layer, and the cyberpersona layer. While there are differences in identificationof cyber domain planes, the two extra planes proposed byRaymond provide the ability to highlight command andcontrol of cyber operations (supervisory plane) and tie inkinetic operations (geographical plane).

Jakobson [12] argues that key cyber terrain is made upof cyber assets and services, and their intra- and inter-dependencies. Cyber terrain is identified by focusing moreon dependencies within a system vice across planes andis defined as three separate sub-terrains: hardware, whichconsists of a collection of connected network infrastructurecomponents, like routers, servers, switches, communicationlines, etc., and dependencies between them; software, whichconsists of different software components, such as operatingsystems and applications; and service, which represents allthe services, such as database, file transfer, email, securityservices, etc., and their intra-dependencies [12].

Regardless of the framework or model utilized to identifykey cyber terrain, a commonality in determining key cyberterrain is determining critical assets. MITRE emphasizes theimportance of critical assets to key cyber terrain stating that“assets in operational environments are typically identifiedand their criticality determined via a mission impact analysisor business impact analysis” [8]. Dressler et al. [13] broadenthe identification of key cyber terrain to include “all criticalinformation, systems, and infrastructure; whether owned bythe organization or used in transit by its information”. FromFranz’ definition of key cyber terrain, MITRE and Dressler’sdiscussions of critical assets as part of key cyber terrain tiedirectly into “mission essential war fighting functions” [11].

B. Asset Criticality

There is debate about the best method to determineasset criticality. Three proposed methodologies are attackgraphs, Bayesian networks and multi-attribute decision mak-ing (MADM). Attack graphs are mathematical depictions ofpossible vectors of attacks against a specific network [14].They can detail “all possible sequences of vulnerabilities anattacker can follow,” or “with a monotonicity assumptionstating an attacker never relinquishes an obtained capability,an attack graph can record the dependency relationshipsamong vulnerabilities” [15]. Attack graphs are limited inthat they quickly become complicated and require computermodeling, even for small networks [16]. Bayesian networksare graphical representations of probabilistic relationshipswithin a network domain that can be constructed from attackgraphs. They are limited by only accounting for a singleactor and the assumed choices that the particular actor willmake. If the assumed choices are incorrect, then a newBayesian network is needed [17]. While these methods havevalue in looking at criticality from the attacker perspective orfrom the dependency aspect of asset criticality, MADM pro-vides mature and well-understood multi-discipline methodsfor selecting the best decision among all feasible alterna-tives [3].

C. Related Work

Endsley originally proposed three key aspects for situa-tional awareness [18], recently re-interpreted for the cyberdomain by MITRE [19] as a framework comprised ofnetwork awareness (asset and configuration management),threat awareness (identifying incidents and suspicious behav-ior) and mission awareness (critical dependencies, real-timeresponse, risk assessments and informed defense planning).MITRE’s analysis focuses on the first two of these aspects,leaving the third area to human analysis. Our study isin the mission-centric analysis of cyber asset criticalityand asset interdependencies, aiding the human analyst toprioritize assets for situational awareness. Jajodia et al. [20]reinforce both MITRE’s and Endsley’s work, observing thatto “protect critical network infrastructures and missions,

6043

we must understand not only the vulnerabilities of eachindividual system, but also their interdependencies and howthey support missions” [20].

Several projects within the US Department of Defenseseek to map cyber networks in a mission-centric manner—interpreting alarms and logs, monitoring processes, curat-ing known vulnerabilities and SME input—with the goalof improving situational awareness. Schultz et al. surveythese, calling them cyber network mission dependencies,or mission mapping [21]. Our apporach is complimentaryto all of these, as every individual analysis methodologyis insufficient in some aspect and most agree, ultimately, ahybrid approach is required for a comprehensive tool. In theterminology of Schultz et al., our methodology is a process-driven analysis as opposed to an artifact-driven analysis.Our approach, however, differs from existing process-drivenanalyses in terms of methodology and, thus, may provideuseful cross-validation of their criticality determination or,otherwise, enhancing its coverage.

MIT Lincoln Lab’s AMMO project uses SMEs to iden-tify an initial set of assets comprising key cyber terrainand leverages network scanning increase terrain coverage.MITRE’s CMIA project uses business modeling tools anduses SMEs to map assets to that model, to assess the impactof attacks to mission workflow during the system’s designphase. MITRE’s RiskMAP project employs a dependency-based analysis using SME input to model missions as atasks and subtasks, for assessing the impact of attacks indisrupting mission goals in terms of availability, integrityand (unlike most network mapping tools) confidentiality.Johns Hopkins Applied Physics Lab’s Dagger project isanother dependency-based modeling tool using manual SMEinput to interpret impacts of cyber effects to missions. Unlikethese tools, we employ a MADM methodology rather than adependency-based analysis, the latter being a type of graph-based analysis. Criteria and asset weighting in MADM alsopermits a more flexible mechanism for incorporating SMEjudgement and asset inter-relationships, compared to explicitflow-based or dependency-based analysis.

III. METHODOLOGY

Technique for Order of Preference by Similarity to IdealSolution (TOPSIS) is a MADM method that selects the bestfeasible alternative as the one closest to an ideal solutionand farthest from the negative ideal solution. TOPSIS re-quires alternatives to have attributes which are monotoni-cally increasing or decreasing, where all best attribute valuescomprise the ideal (zenith) solution and all worst valuescomprise the negative-ideal (nadir). We define a hierarchalvariant of TOPSIS for a re-configurable system, S, relativeto mission, M , given the following:

• attributes (criteria) x1, . . . , xn• alternatives (assets) A1, . . . , Am

• score of Ai for attribute xj is rij

• weight wj for attribute xj .The goal is to find alternative Ai closest to zenith A∗ andfarthest from nadir A−. Alternatives are grouped within a setof families for a particular mission (see Fig. 1). The scoringcommunicates the importance of that alternative within themission as it pertains to each family. The criticality scorefor a given alternative is based on the highest score for thealternative over all the families.

M

F1 F2 Fk. . .

A1 A2 Am. . .

score(Am, F2) = ⇣1

score(. . .) = . . .

score(A2, F2) = �1

score(A2, Fk) = �2

score(. . .) = . . .

score(A1, F1) = ⌘1

score(A1, F2) = ⌘2

score(. . .) = . . .

Figure 1. Scoring system assets based on attribute weights and allocationto families, relative to mission M .

A. Criteria Weighting

Feedback to weight attributes is solicited from experts,normalized and combined to get an overall weight followingthe Analytic Hierarchy Process (AHP). AHP allows “allimportant tangible and intangible, quantitatively measurable,and qualitative factors” to be included and measured [22].We (i) solicit data to weight families (i.e., subsystems) withrespect to each mission, (ii) solicit data to weight criteriairrespective of mission, (iii) solicit data to score each assetirrespective of mission and (iv) derive weights for criteriawith respect to assets in a mission. The data is solicited fromsubject matter experts (SMEs) who have expert familiaritywith the system and experience operating its subsystems.The system S is broken up into two or more hierarchicallayers. For complex systems, division of S into parts reducesthe need to gather feedback on weighting criteria per assetin each mission.

The system S is broken into subsystem component fami-lies F1, . . . , Fk. For each subsystem F` there is an associatedset of assets, {A`,1, . . . , A`,m} ⊆ {A1, . . . , Am}. SMEsprovide pair-wise comparisons among families expressingthe relative importance of each to mission M . These com-parisons are normalized via the eigenvector method (seeSec. III-B). This yields a set of normalized subsystemweights, wa1

, . . . , wak, where wa`

is the weight for sub-system F`.

Independently, attributes, x1, . . . , xn, are pair-wise com-pared by criteria subject matter experts with respect to theirimportance to system S irrespective of mission M . The

6044

outcome of this process is a set of normalized attributeweights, wb1 , . . . , wbn , where wbj is the weight for criteriaxj .

Finally, weights for scoring the attributes A`,1, . . . , A`,m

under subsystem F` are derived by multiplying subsys-tem weight wa`

by criteria weights wbj , yielding weightsw`,1, . . . , w`,n for assets in subsystem F` during missionM . Concretely, for F` with ` ∈ [1, k], this is given by:

wa`(wb1 , . . . , wbn) = (w`,1, . . . , w`,n)

This relationship between weights within AHP is depictedin Fig. 2.

M

F1 F2 Fk. . .

. . .x1 x2 xn

wa1 wa2wak

wb1 wb2 wbn

Figure 2. The role of weightings in the Analytical Hierarchy Process withsubsystems and criteria.

B. Weighting Normalization

The eigenvector method allows for inconsistencies withinpair-wise comparisons to be accounted for [22], [23]. Theaccomodated of matrix X is a non-null vector w suchthat Xw = λw and leaves w fixed. Accordingly, w is aneigenvector if it is a non-zero solution of (X − λI)w = 0for some λ. For an ideal solution with known weights, λwould be equal to the number of components n. This effectmeans that small variations within X are accounted for bykeeping λmax, the largest eigenvalue, close to n and theremaining eigenvalues close to zero [22, Ch. II, §5.1].

Next, we describe normalization in reference to attributeweights wb1 , . . . , wbn , but the process is the same forsubsystem weights wa1

, . . . , wak. First, the input weights

wb1 , . . . , wbn are normalized to have the propertyn∑

j=1

wbj = 1. (1)

Next, we construct the reciprocal matrix X using therespondent’s pair-wise comparison ratings, elicited via theAHP:

X =

x1/x1 x1/x2 · · · x1/xnx2/x1 x2/x2 · · · x2/xn

......

. . ....

xn/x1 xn/x2 · · · xn/xn

. (2)

When multiplied by w = (wb1 , . . . , wbn), the reciprocalmatrix obeys the equation

(X − nI)w = 0

since

X w =

1 x1/x2 · · · x1/xn

x2/x1 1 · · · x2/xn...

.... . .

...xn/x1 xn/x2 · · · 1

wb1

wb2...

wbn

= nI w.

Note, however, that X is not an exact measurement. Thus,we find the max eigenvalue, λmax, for the characteristicequation using:

det(X − λI) =

∣∣∣∣∣∣∣∣∣1− λ x1/x2 . . . x1/xnx2/x1 1− λ . . . x2/xn

......

. . ....

xn/x1 xn/x2 . . . 1− λ

∣∣∣∣∣∣∣∣∣ = 0.

Using λmax, and given that the sum of the weights is one(see Eq. 1), the final weights w can found by solving thefollowing homogenous system [23, Ch. 3]:

1− λmax x1/x2 . . . x1/xnx2/x1 1− λmax . . . x2/xn

......

. . ....

xn/x1 xn/x2 . . . 1− λmax

wb1

wb2...

wbn

= 0.

C. Alternatives Selection

TOPSIS selects the most preferred alternative based onits closeness to the most preferred outcome (zenith) anddistance from the least preferred outcome (nadir), usinga Euclidean distance metric in the n-dimensional scoringspace for criteria [3]. The proximity-to-zenith relationshipcan be modeled by:

Ci∗ =Ai−

(Ai∗ +Ai−). (3)

First, the normalized decision matrix R must is created totransform each attributes score into a dimensionless metricfor comparison across attributes [23, Ch. III, §2.3.5]. Whenyij is the scoring of asset Ai under criteria xj , the decisionmatrix R is defined as:

R =

r11 r12 . . . r1nr21 r22 . . . r2n

......

. . ....

rm1 rm2 . . . rmn

where

rij =yij√∑mi=1 y

2ij

.

6045

Using decision matrix R and weights w, we derive theweighted normalized decision matrix V , as follows:

V =

v11 v12 . . . v1nv21 v22 . . . v2n

......

. . ....

vm1 vm2 . . . vmn

=

r11w1 r12w2 . . . r1nwn

r21w1 r22w2 . . . r2nwn

......

. . ....

rm1w1 rm2w2 . . . rmnwn

= R w .

The zenith, A∗, is the most preferable alternative availablewithin the system and is comprised of the most desirablevalue of each asset over all the attributes. The nadir, A−,is the least desirable and has the worst value of each assetover all attributes. The Euclidean distance from zenith andnadir is calculated for asset i ∈ {1, . . . ,m} as:

Ai∗ =

√√√√ n∑j=1

(vij − v∗j )2 and Ai− =

√√√√ n∑j=1

(vij − v−j )2

From Ai∗ and Ai− , the best feasible alternative Ci∗ can becalculated from Eq. 3, and all the assets ranked by theirrelative closeness to the zenith [3].

IV. CASE STUDY

Three key decision points for our study are defining themissions relative to which families are pair-wise compared,defining the criteria relative to which assets are scored,and defining the assets comprising the system S that arescored. We draw each mission M from existing doctrinaldefinitions used by the Navy to prepare, provide, and employforces [24]. Criteria are derived from the Risk ManagementFramework (RMF) objectives outlined in Federal and De-partment of Defense security standards. Assets comprisingsystem S are routers, and their associated connected sub-systems, for a representative US Navy ship-board industrialcontrol system.

A. Missions

The Maritime Operations Center Standardization Manualprovides fleet commanders with an organization and processto bridge the gap between strategic guidance and tacticalexecution [24]. It outlines six essential mission areas andrelated supporting tasks. Our study selects three of thesemission areas, relative to which scoring is performed andasset criticality is analyzed. These three missions, as definedby Joint Publication 1-02, are:

Deterrence: The prevention of action by the exis-tence of a credible threat of unacceptable coun-teraction and/or belief that the cost of actionoutweighs the perceived benefits.

Sea Control: Employment of forces to destroy en-emy naval forces, suppress enemy sea commerce,protect vital sea lanes, and establish local militarysuperiority in vital sea areas.Power Projection: Conducting projection in themaritime environment to include a broad spectrumof military operations to destroy enemy forcesor logistic support or to prevent enemy forcesfrom approaching within enemy weapons range offriendly forces. [5]

Each mission may require different configurations ofsubsystems within S in order to successfully execute andaccomplish that mission. These missions were chosen toprovide three distinctly different system configurations.

B. Criteria

The criteria employed by our case study are given inTable I. Our criteria are based on the definitions of confiden-tiality, availability and integrity described in FIPS 199 [25]and draws impact terminology from NIST’s Risk Manage-ment Framework [26]. Criticality needs to reflect the impactan attack may have on a compromised system with respectto those subsystems directly connected to the target and tosubsystems that are indirectly connected (i.e., connected tothe same asset within a family versus connected to anotherrouter within the family). Our study includes specific criteriafor resource redundancy, reflecting the military’s need tosurvive through asset nonavailability to continue operations.

C. Assets

The target system S is an ethernet-connected multiplexsystem comprised of multiple switches in a ring + 1 topologywith multiple independent backbones (see Fig. 3). Routersare placed at the edge of the network with the backbonesbeing made of switches. We limit our case study to the sys-tem’s routers because they are the layer 3 devices connectingall major subsystem components in larger network.

subsystem1

subsystem2

subsystem3

S

R R R R R

S S S

SS S S

Figure 3. A notional topology resembling our target system, highlightingswitches (S), routers (R) and subsystems.

6046

Table ISECURITY CRITERIA.

Criteria Definition

Confidentiality to di-rectly connected sys-tems

If the target systems connected to the routerwere compromised, then the resulting unau-thorized disclosure of information held bythe target systems connected to that routercould be expected to have limited / serious /severe or catastrophic adverse effect on or-ganizational operations, organizational assetsor individuals.

Integrity to directlyconnected systems

If the target systems connected to the routerwere compromised, then the resulting modi-fication or destruction of information held bythe target systems connected to that routercould be expected to have limited / serious /severe or catastrophic adverse effect on or-ganizational operations, organizational assetsor individuals.

Availability to directlyconnected systems

If the target systems connected to the routerwere compromised, then the resulting dis-ruption of access to or use of informationheld by the target systems connected to thatrouter could be expected to have limited /serious / severe or catastrophic adverse effecton organizational operations, organizationalassets or individuals.

Confidentiality to indi-rectly connected sys-tems

If the target systems connected to the routerwere compromised, then the resulting unau-thorized disclosure of information held bysystems connected to other routers could beexpected to have limited / serious / severe orcatastrophic adverse effect on organizationaloperations, organizational assets or individu-als.

Integrity to indirectlyconnected systems

If the target systems connected to the routerwere compromised, then the resulting unau-thorized modification or destruction of infor-mation held by systems connected to otherrouters could be expected to have limited /serious / severe or catastrophic adverse effecton organizational operations, organizationalassets or individuals.

Availability to indi-rectly connected sys-tems

If the target systems connected to the routerwere compromised, then the resulting disrup-tion of access to or use of information held bysystems connected to other routers could beexpected to have limited / serious / severe orcatastrophic adverse effect on organizationaloperations, organizational assets or individu-als.

Resource redundancy

There are no / limited / multiple amount ofother systems to continue the same operationif the system connected to the router is com-promised.

Each router has multiple subsystems connected to it. Asubsystem deemed critical for a specific mission may carryinformation with a low priority but a high criticality. Thecategorization of systems in this manner allows for theefficient use of limited data, toward the task of identifyingcritical systems and assets within the ICS system withrespect to mission and operational parameters.

Given a network map for S, each router for a subsystemcan be scored against the criteria in Table I. The criteria arescored on a three point Likert scale, with one being limited

and three being severe or catastrophic.

V. ANALYSIS AND RESULTS

In this section, we explore the results of applying our vari-ant of TOPSIS methodology to our case study. As discussedin Sec. III, we solicit SME feedback to weight criteria,weight families and score assets. Typically, SME feedbackcollected via the AHP yields a single set of weights, derivedvia consensus. For our hierarchical variant, we solicit datafrom three different groups of SMEs (four criteria SMEs,four family SMEs, two asset SMEs) to derive a set ofSME scores. By exploring this “scoring space,” we intend toassess the relative impact of this data on the final results ofthe TOPSIS analysis and characterize the sensitivity of thismethodology in determining key cyber terrain. In particular,it may be the case that one set of scores plays a much moreimportant role than the others, and effectively decides keyterrain in our case study.

For both family and criteria weighting, we expore threestrategies: random weights, a randomly-generated, artificialweighting strategy; average weights, an average of all SME-derived weightings; and transitive weights, the single SME-derived weight demonstrating the best consistency ratio. Asdefined by Saaty [22], the consistency ratio is an expres-sion of how strongly transitive relationships hold amongweightings (i.e., a < b and b < c imples a < c), wherehighly transitive behavior yields ratios less than 0.1. Weacknowledge, however, that inconsistency is not always badfor pair-wise comparisons as it reflects the complexity ofreal-world systems. For asset scoring, we explore two strate-gies: informed scoring, derived through SME consensus; anduniform scoring, an artificial score in which all assets arescored the same (i.e., two on the 1–3 Likert scale).

In total, fifty-four possible analyses (i.e., 3 missions, 3family weighting strategies, 3 criteria weighting strategiesand 2 asset scoring strategies) were compared to observethe sensitivity of these factors and how they impact theidentification of key cyber terrain. For our analyses, weidentify “good” results to be those that appear to be highlydistinguishable, meaning they help establish distinguishingcriteria highlighting assets of high relative criticality. Allfigures show more critical assets as having a higher “relativecloseness” (per Eq. 3). Assets with high criticality areconsidered key cyber terrain. We leave exploring the possibledifference between critical assets and key cyber terrain tofuture work.

Our observations are divided into three categories: theeffects of family weighting, the effects of asset scoring andthe effects of criteria weighting. Our preliminary analysisfocuses on three factors: impact, or the effect of weightingsand scoring; transitivity, or the effect of highly transitiveweights versus average weights; and perturbation, or theeffect of average weights versus random weights or, in

6047

the case of asset scoring, informed scoring versus uniformscoring.

A. Impact of Family Weighting

For family weightings, we analyze the sensitivity of theresults under small of changes in weights. We look at overallimpact of family weights by observing the changes fromdifferences in family weightings. For 48 of 54 analyses, thefamily weightings yield highly distinguishing results. The6 analyses that we find not highly distinguishing are allwithin one mission (power projection) using non-randomfamily weights; we suspect the relative closeness of familyweights under that mission depressed differences in criti-cality. When analyzing perturbation, we compare pairs ofanalyses employing average or random family weightings,leading to 18 pairs of analyses. We observe, on average,15 assets change in criticality1, or approximately 50% ofthe total assets. This aligns with our expected behavior ifassets changed their criticality following a coin toss, andshows that family weights indeed have an important role inthe outcome of the analysis. In particular, if asset scoringdominates the outcome, we may not see this perturbationwhen asset scoring is held constant. In looking at transi-tivity, comparing highly transitive weightings (each time,using SME weightings with the lowest consistency ratio)to average weightings, leads to 18 pairs of analyses. Again,we compare changes in criticality. For example, comparingFigs. 4 and 5, we see three assets change in critical status andthree change in non-critical status. On average, across eachmission, we see between 4 and 9 assets change in criticality.Given that completely random changes in criticality yield 15assets changing position, we consider a change in 4–9 assetsto be quite sensitive to changes in family weightings.

In summary, we observe nearly all family weightingstrategies were distinguishing, i.e., even random weightsdistinguish. This indicates that family weights may be in-consequential and that other weighting strategies dominateoutcome; however, transitivity and perturbation analysessuggest that family weights are consequential and should beselected carefully. We observe that, given any fixed strategyfor asset scoring and criteria weighting, changes in familyweighting will perturb the key cyber terrain identified: foreach of our missions, between 17.2–31.0% assets changeposition when comparing transitive vs. average strategies.This sensitivity suggests that the family weightings for aparticular mission are highly distinguishable and highlyinfluential for the methodology in discriminating asset crit-icality and key cyber terrain.

1In each analysis, assets are deemed either critical, non-critical orneither (exceeding a non-criticality threshold, but not exceeding a criticalitythreshold). We count assets that transition in their outcome: moving acrossthese thresholds. Thus, an asset that moves from non-critical to criticalwould be counted as changing its criticality twice.

Figure 4. Asset criticality under the Deterrence Mission: average familyweighting, transitive criteria, and informed asset scoring.

Figure 5. Asset criticality under the Deterrence Mission: transitive familyweighting, transitive criteria, and informed asset scoring.

B. Impact of Asset Scoring

For assets, we analyze the impact and perturbation ef-fects of scoring. When considering the impact of scoring,informed scoring results are highly distinguishable for 23of 27 analyses. For perturbation analysis, we comparedinformed scores versus uniform scores: in 27 of 27 uniformscoring analyses, the effect of criteria weighting was largelyminimized giving results that were predominantly basedon the family weightings. For example, Fig. 6 and Fig. 7demonstrate different per-mission family weights, whichidentify significantly different assets as critical. Varyingscoring strategy and holding other factors constant leads to27 pairs of analyses, comparing uniform to informed assetscoring. For 14 of these 27 pairs, uniform scoring resultsin more assets to be deemed critical; uniform scoring neverresulted in fewer assets deemed critical, and never resulted in

6048

more assets deemed non-critical. For example, consideringa particular case within the power projection mission, sixmore critical assets were identified (compare Fig. 7 withFig. 8). We conclude that uniform scoring produces moreconservative results, which makes sense since it employslittle judgement differentiating asset criticality. The value ofhaving conservative estimates, potentially yielding a largerset of assets deemed critical, requires further study. Ourstudy does not attempt to conclude if a more conservativeapproach to determining key cyber terrain is better or worse,from a practical application standpoint.

Figure 6. Asset criticality under the Sea Control Mission: average familyweighting, transitive criteria, and uniform asset scoring.

Figure 7. Asset criticality under the Power Projection Mission: averagefamily weighting, transitive criteria, and uniform asset scoring.

C. Impact of Criteria Weighting

The sensitivity of results to criteria weights was analyzedto determine effect on key cyber terrain. We explore the

Figure 8. Asset criticality under the Power Projection Mission: averagefamily weighting, transitive criteria, and informed asset scoring.

effect of perturbation for criteria weighting, comparing ran-dom with average criteria weighting—while holding otherfactors constant, using only non-random family weightingand informed asset scores2—yields 6 pairs of analyses (i.e.,three missions, two family weightings, informed scoring).Across these 6, we observe an average of 2 assets changecriticality (6.9% of all assets); excluding one case (powerprojection mission with average family weightings underinformed scoring), this average is significantly lowered: 0.83assets (2.9% of all assets). Thus, either outcomes exhibitlow-sensitivity to criteria weights, or average criteria weightsare effectively random. In fact, we do observe high vari-ance among SME-derived weighting: the largest differencebetween derived single criteria weightings is 0.3027. Weexpected, given our use of AHP, to be unable to achieve totalconsensus on what security criteria are most important forkey cyber terrain; but there were few similarities in criteriaweighting to suggest even the general importance of eachcriteria.

We find similar trends when analyzing the effect oftransitivity for criteria weighting—i.e., employing the mosttransitive weights while holding other factors constant, asbefore—again, yielding 6 pairs of analyses. Across these 6comparisons, we observe an average of 0.5 assets changecriticality (1.7% of all assets), demonstrating that outcomesshow low-sensitivity to transitivity in criteria weighting.This extremely low effect with highly transitive weightssuggests that outcomes are more significantly decided byfamily weighting and asset scoring strategies, and see littledifference among highly transitive, average or random crite-ria weighting strategies. Qualitatively, however, it appears

2We exclude random family weighting and uniform asset scoring, aswe have previously remarked the effects of these, i.e., random changesto asset criticality or more conservative determination of asset criticality,respectively.

6049

that highly transitive weights yield simpler to interpretdistinguishing criteria among critical and non-critical assets;we leave fuller characterization of this observation to futurework.

Given the apparent low sensitivity of criteria weights, weexplore the magnitude of effect of criteria weights comparedto family weights. We conduct our analyses by observingthe change in the number of critical assets when the familyweighting strategy is changed and criteria weighting strategyis held constant (i.e., average, transitive, and random criteriaweighting strategy under informed scoring when familyweighting is most transitive or average), and the numberof critical assets when the criteria weighting strategy ischanged and family weighting strategy is held constant (i.e.,average, transitive, and random family weighting strategyunder informed scoring when criteria weighting is mosttransitive or average). This leads to two sets of 18 analysispairs: in one set, family weighting strategy is varied and inthe other, criteria weighting strategy is varied. When lookingat the family weighting strategies, we observe an average ofabout 7 assets change criticality (23.7% of all assets); whencriteria weighting strategies change, we observe an averageof less than one asset changes criticality (1.1% of all assets).As illustrated in Figs. 4 and 5, six assets changed criticalitywhen family weighting strategy changed. In comparison, asshown in Figs. 4 and 9, only two assets changed criticalitywhen criteria weighting changed3. This large difference innumber of assets changing criticality suggests that familyweightings have about 20 times larger effect on the outcomesin identifying key cyber terrain than do criteria weighting inthe context of our case study.

Figure 9. Asset criticality under the Deterrence Mission: average familyweighting, average criteria, and informed asset scoring.

3Note: This particular comparison had the largest number of criticalasset changes when the criteria weighting strategy changed. Only oneother comparison had any assets change criticality with the remainingcomparisons having zero changes in critical assets.

VI. CONCLUSION

Our application of hierarchical TOPSIS yields insight intothe cyber-physical key terrain aboard a US Navy vesselrequired for the successful conduct of various missions. Themethodology used here suggests that critical assets’ partici-pation in key cyber terrain can be identified and does changeaccording to mission. The impact of our findings means thatcyber situational awareness should account for mission. Asa result, we conclude that any cyber situational awarenesstool must account for the changes in the operational use ofa system while operating under different missions.

Within the context of our case study, our observationssuggest the most important factor in determining key cyberterrain is in defining family weights for a particular mission.Once per-mission family weights are determined, then thekey cyber terrain derived from the methodology used herewill closely mirror those weights. Our observations suggestthat criteria weights are dramatically less important thanfamily weights. When assets are scored uniformly, key cyberterrain appears to be identified in a more conservative andless informed manner.

An important corollary of our research in practical ap-plication is that identifying key cyber terrain will allowscarce resources to be prioritized to critical systems duringoperation. Further work is needed to demonstrate that terrainprioritization benefits situational awareness because it allowsfocus on the health and performance of key assets withoutdistraction from system components that do not impactmission performance. Similarly, further work may show thatterrain prioritization allows the use of tailored analyticsand limited defensive resources to focus on monitoring andprotecting infrastructure essential to the success of the ship.

A. Future Work

Further work is required to validate the soundness ofthe results obtained using this methodology, i.e., that assetsidentified as contributing to key cyber terrain are the same asthose identified as integral in attack and vulnerability studiesfor the target system. This could be done by overlayingknown vulnerabilities with critical assets to better identifythe possible impact to missions and effect on key cyberterrain, or compare our study with other methodologiesto determine the differences in identification of key cyberterrain. It would be interesting to explore the high SME-response variance we observed. In fact, no formal surveywork has been conducted to elicit family and criteria weight-ings, so no reserch exists to guide selecting a sample sizeto achieve statistical significance. Exploring how to elicitpairwise rankings with confidence in the statistical propertiesof the derived weights would be a valuable contribution,allowing future research to avoid the costly process ofachieving consensus-derived weights.

6050

REFERENCES

[1] P. Barford, M. Dacier, T. G. Dietterich, M. Fredrikson,J. Giffin, S. Jajodia, S. Jha, J. Li, P. Liu, X. Ou, D. Song,L. Strater, V. Swarup, G. Tadda, C. Wang, and J. Yen, CyberSituational Awareness: Issues and Research. Boston, MA:Springer, 2010, ch. Cyber SA: Situational awareness for cyberdefense, pp. 3–13.

[2] W. W. Streilein, J. Truelove, C. R. Meiners, and G. Eakman,“Cyber situational awareness through operational stream-ing analysis,” in IEEE Military Communications Conference(MILCOM 2011), Nov. 2011, pp. 1152–1157.

[3] A. Kim and M. H. Kang, “Determining asset criticalityfor cyber defense,” Office of Naval Research, Tech. Rep.NRL/MR/5540–11-9350, 2011.

[4] U.S. Fleet Cyber Command/TENTH Fleet: Strategic Plan2015–2020, U.S. Fleet Cyber Command, Washington,DC, 2015. [Online]. Available: http://www.navy.mil/strategic/FCC-C10F\%20Strategic\%20Plan\%202015-2020.pdf

[5] Joint Publication 1-02: Department of Defense Dictionaryof Military and Associated Terms, JP 1-02, Departmentof Defense, Washington, DC, 2010. [Online]. Available:http://www.dtic.mil/doctrine/new pubs/jp1 02.pdf

[6] Joint Publication 3-12 (R): Cyberspace Operations, JP3-12, Department of Defense, Washington, DC, 2013.[Online]. Available: http://www.dtic.mil/doctrine/new pubs/jp3 12R.pdf

[7] P. W. Phister, “Cyberspace: The ultimate complex adaptivesystem,” The International C2 Journal, vol. 4, no. 2, 2010.[Online]. Available: http://www.dodccrp.org/files/IC2J v4n203 Phister.pdf

[8] D. Bodeau, R. Graubart, and W. Heinbockel, “Mappingthe cyber terrain: Enabling cyber defensibility claims andhypotheses to be stated and evaluated with greater rigor andutility,” MITRE, McLean, VA, Tech. Rep. MTR130433, 2013.

[9] J. R. Mills, “The key terrain of cyber,” GeorgetownJournal of International Affairs, pp. 99–107, 2012. [Online].Available: http://www.jstor.org/stable/43134343

[10] G. Conti, T. Cross, M. Nowatkowski, and D. Raymond, “Keyterrain in cyberspace: Seeking the high ground,” in 2014 6thInternational Conference on Cyber Conflict. Tallinn: NATOCCD COE, 2014. [Online]. Available: http://www.westpoint.edu/acc/SiteAssets/SitePages/Publications/06916409.pdf

[11] G. J. Franz III, “Effective synchronization andintegration of effect through cyberspace for the jointwarfighter,” presented at Armed Forces Communicationsand Electronics Association TechNetLand Forces-East, Baltimore, MD, August 2012. [Online].Available: http://www.afcea.org/events/tnlf/east12/documents/4V3EffSynchIntEffthruCybrspcforJtWarfighterforpublicrelease.pdf

[12] G. Jakobson, “Mission-centricity in cyber security: Architect-ing cyber attack resilient missions,” in Proceedings of the 5thInternational Conference on Cyber Conflict (CyCon). IEEE,June 2013, pp. 1–18.

[13] J. Dressler, W. Moody, C. L. Bowen III, and J. Koepke,“Operational data classes for establishing situational aware-ness in cyberspace,” in Proceedings of the 6th InternationalConference On Cyber Conflict (CyCon 2014), June 2014, pp.175–186.

[14] P. Ammann, D. Wijesekera, and S. Kaushik, “Scalable, graph-based network vulnerability analysis,” in Proceedings of the9th ACM Conference on Computer and CommunicationsSecurity. ACM, 2002, pp. 217–224.

[15] L. Wang, S. Noel, and S. Jajodia, “Minimum-cost networkhardening using attack graphs,” Computer Communications,vol. 29, no. 18, pp. 3812–3824, 2006.

[16] R. E. Sawilla and X. Ou, “Identifying critical attack assets independency attack graphs,” ESORICS, pp. 18–34, 2008.

[17] P. Xie, J. Li, X. Ou, P. Liu, and R. Levy, “Using Bayesiannetworks for cyber security analysis,” in IEEE/IFIP Inter-national Conference on Dependable Systems and Networks,2010, pp. 211–220.

[18] M. R. Endsley, “Toward a theory of situation awareness indynamic systems,” Human Factors and Ergonomics Society,vol. 37, no. 1, pp. 32–64, 1995.

[19] MITRE. (2016, January) Situation awareness: Today’sleaders need meaningful cyber situation awareness tosafeguard sensitive data, sustain fundamental operations, andprotect national infrastructure. [Online]. Available: http://www.mitre.org/capabilities/cybersecurity/situation-awareness

[20] S. Jajodia, S. Noel, P. Kalapa, M. Albanese, and J. Williams,“Cauldron: Mission-centric cyber situational awareness withdefense in depth,” in IEEE Military Communications Confer-ence (MILCOM 2011), 2011, pp. 1339–1344.

[21] A. E. Schulz, M. C. Kotson, and J. R. Zipkin, “Cyber networkmission dependencies,” MIT Lincoln Laboratory, Tech. Rep.1189, May 2015.

[22] T. L. Saaty, The Analytic Hierarchy Process: Planning, Pri-ority Setting, Resource Allocation. New York: McGraw-HillBook Co., 1980.

[23] C.-L. Hwang and K. Yoon, Multiple Attribute Decision Mak-ing : Methods and Applications a State-of-the-Art Survey,M. Beckmann and H. P. Kunzi, Eds. New York: Springer-Verlag, 1981.

[24] Maritime Operations Center Standardization Manual, OP-NAV M-3500.42, Department of the Navy, Washington, DC,2014.

[25] Federal Information Processing Standards Publication 199:Standards for Security Categorization of Federal Informationand Information Systems, FIPS 199, National Institue ofStandards and Technology, 2004.

[26] National Institute of Standards and Technology Special Pub-lication 800-37: Guide for Applying the Risk ManagementFramework to Federal Information Systems: A Security LifeCycle Approach, NIST SP 800-37, National Institute of Stan-dards and Technology, February 2010.

6051