Data Driven Cybersecurity Governance

© 2015 Carnegie Mellon University

Software Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

Distribution Statement A: Approved for Public Release; Distribution is Unlimited

Data-Driven Cybersecurity GovernanceDouglas Gray

2Data Driven Cybersecurity GovernanceDec. 18 2015© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Copyright 2015 Carnegie Mellon University

This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the United States Department of Defense.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].

CERT® and OCTAVE® are registered marks of Carnegie Mellon University.

DM-0003094


© 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release; Distribution is Unlimited

Data-Driven Cybersecurity Governance

Introduction


• The Software Engineering Institute (SEI) is a U.S.-owned not-for-profit federally funded research and development center (FFRDC) operated by Carnegie Mellon University to focus on software and cybersecurity.

• The CERT Division of the SEI is a trusted provider of operationally relevant cybersecurity research and innovative and timely solutions to our nation's cybersecurity challenges.

• The CERT Division developed and maintains the CERT Resilience Management Model (CERT-RMM) and OCTAVE Allegro Methodology.

Who We AreIntroduction

We work with the following organizations:• Carnegie Mellon University• Discover Financial• Highlands Union Bank• Lockheed Martin Corporation• Marshall & Ilsley Corporation• PNC Corporation• Pacific Gas and Electric• University of Pittsburgh

Medical Center• U.S. Dept. of Defense• U.S. Dept. of Energy• U.S. Dept. of Homeland

Security• U.S. Dept. of Health & Human

Services• U.S. Environmental Protection

Agency• U.S. National Security Agency• U.S. Postal Inspection Service• USBank


To discuss a process to integrate data analytics into operational cybersecurity governance decision making and execution in a way that• frames the problem quickly and accurately and that enables a

fast, effective Observe, Orient, Decide, Act Loop• facilitates better data collection and synthesis, quantitative and

qualitative analysis, and visualization• enables practical and repeatable analytical battle drills and

TTPs for leaders and enablers at all echelons

PurposeIntroduction


W. Edwards Deming’s thoughts What it means to us

“If you do not know how to ask the right question, you discover nothing.” We must have a reason to analyze data

“If you don't understand how to run an efficient operation, new machinery will just give you new problems of operation

and maintenance. The sure way to increase productivity is to better administrate man

and machine.”

We can’t “tool” our way out of cybersecurity challenges

“People with targets and jobs dependent upon meeting them will probably meet the targets - even if they have to destroy the

enterprise to do it.”Compliance is the beginning, not the end

“Whenever there is fear, you will get wrong figures.”

The use of data analytics must be productive in the aggregate, punitive as

the exception

Improving People and ProcessIntroduction

Technology is useless without effective processes and trained people


Governance vs. OperationsIntroduction

Operations GovernanceScope Individual

networks, systems, users, organizations

Multiple networks, systems, user bases, organizations

Timescale Immediate to 6 months

3 to 36 months*

Level of Abstraction

Transactional Trends, aggregations

Management Impact

Direct interaction Context setting

*Although maximum technology-related decision making is limited to approximately three years due to rate of technological change, military organizations must program

their expected budget needs five years in advance.


Governance vs. OperationsIntroduction

8

OperationsWeather – “It will snow.”

Tactical Cyber – “CVE 2015-xx-xxxx is prevalent and is being

compromised.”

GovernanceClimate – “Drought in the

southwest limits irrigation.”Strategic/Operational Cyber – “FedRAMP usage improves

asset management.”


Why Focus on GovernanceIntroduction

Know Prevent Detect

Respond

Recover

Reconnaissance Weaponization Delivery Exploitation Installation Command and Control

Actions on the Objective

Threat Actor Actions1

Friendly Actions2

Harden People, Information, Information, Technology,

FacilitiesCreate Faster, More Accurate

TTPs, Battle Drills

Source:1Lockheed Martin Kill Chain2NIST Cybersecurity Framework

Effective preparation creates the context for effective response


Leveraging Situational Awareness to Enable Cyber Mission Command

Introduction

Observe

Orient

Decide

Act

mutual

trust

shared

understanding

clear

leadership

intent

disciplined

initiative

mission-

oriented

directives

prudent risk

management




Observe


Facets of Cybersecurity GovernanceObserve


Data FusionObserve

Data Fusion Activities

Automated vulnerability sensor information• H

ardware & Software

• Behavioral Observables (Insider Threat)

Threat Information• T

hreat Actor Analysis

• Prevailing Attack Patterns

Management Information• B

udget Information

• Demographic Information

• Legal & Administrative Investigation Statuses

• Mission Impact Analysis

Qualitative Assessment• I

nspections/Assessments

• Professional Sentiments Analysis

Orient

Unstructured DataMachine Learning

Text AnalysisTrend Analysis

Correlation




Orient


Decision Science vs. DashboardOrient

15

Dashboard“It’s going to snow.”

Decision Science“It’s going to snow.

Wear galoshes, gloves,scarf, winter coat.”


Level 1 Perception of the elements

in the environment

Level 2 Comprehensi

on of the current

situation

Level 3 Projection of future status

Developing Situational AwarenessOrient

Source: Endsley, M. & Jones, D. Designing for Situation Awareness: An Approach to User-Centered Design (2nd ed.). CRC Press. 2012.


Decomposing the Situation to Develop Situational Awareness

Orient

Situational Awareness

Voice of the Environment

Socio-Political

Legal and Policy

Technological

Business

Physical

Voice of the Organization

Voice of the Mission

Voice of the Service

Strategic Objectives and

Supporting Services

Organizational Culture

Organizational Assets

External Dependencies

Voice of the Threat Actor

Describe Threat Actor

Develop Threat Actor Use Cases

IndicesProbabilistic Models

Game TheoryExpert Opinion


Build and Update Targeted MetricsOrient

RequirementsIdentify requirements from mandates, doctrine, strategy

Group requirements into categories

GoalsDevelop one or more goals for each category

QuestionDevelop one or more questions that, if answered, help determine if the goal is met

IndicatorsIdentify the information requirements to answer the question

MetricsIdentify the metrics that will measure the indicator to answer the question

Use new metrics to mature current metrics

What do we want to know? Why do we want to know it? What will we do once we know it? Build and add to a metrics library.


Authoritative vs. Non-Authoritative DataOrient

Authoritative Data•Based on their ability to stand alone as a source for one or more facets of cybersecurity governance•Population•Comprehensiveness•Poor data quality does not make a source not authoritative; it means the quality problems should be fixed

Non-Authoritative Data•Source does not cover enough of the population or not comprehensive enough to be authoritative•Can speak to confidence level of an authoritative data source•Examples: reviews, assessments, inspections, surveys.


Using Behavioral Models to Target Stakeholder Need

Orient

Executives:• Elected leaders, appointees,

GOs, FOs, SESs• Target data with eye toward

organizational mission and constituents

Middle Management:• Staff officers, analysts• Target data with eye toward

routines, procedures

information

Source: Allison, G. T., & Zelikow, P. (1999). Essence of Decision: Explaining the Cuban Missile Crisis (2nd ed.) (Kindle Edition). New York: Longman.

Results of data analysis must be impactful to the recipient.Frame products according to organizational behavioral models.




Decide


• Determine confidence level in assessed data• Low – analyze through subsequent OODA loop• Medium to High – develop action plan to effect change

• Identify and prioritize governance-level risks; identify metric-supported thresholds of acceptability and unacceptability

• Support solutions. Go beyond “name and shame.” Use metrics to identify key trends and corrective governance-level actions

• Tie metrics to a resulting set of possible risk management outcomes• Identify enablers such as SMEs, funding, contract vehicles • Identify organizations that exceed expectations in certain areas and

their lessons learned• Identify what expected changes in metric values should be and how to

avoid bias/gaming• Prioritize and identify metric thresholds where costs will exceed benefits

Key Planning and Decision-Making Factors Decide




Act


Leveraging Enablers to Achieve Desired Effects

Act


• Leverage enablers at the proper organizational level; avoid the “3,000-mile screwdriver”

• Governance sets the direction through governance facets. Operations executes through disciplined project management

• Avoid numerous, rapid changes that cause enterprise turbulence

• Tie actions to expected outcomes and expected timeframes; socialize and communicate expectations

• Set decision points to check progress against expectations• Build knowledge base to make for faster and more effective

OODA loop

Success at the Point of ExecutionAct




Implementation


Identify success stories• Lessons Learned• Tie to data analysis

Identify cautionary tales• Lessons Learned• Tie to data analysis

Track event-driven events• Identify trends that respond to events• Resourcing, technology, incidents

Building a Cybersecurity Knowledge BaseImplementation

27


• Inventory on-hand data• Inventory metrics• Develop data fusion capabilitiesObserve• Refine metrics based on constraints and mandates• Define stakeholders based on behavioral models• Develop quantitative and qualitative analysis engines• Develop visualization capabilities

Orient• Inventory enablers and their capabilities• Identify desired outcomes for metrics (i.e., thresholds)• Develop decision support TTPs• Develop decision-support systems

Decide• Develop knowledge base• Simulate and practice new decision-making TTPs• Develop and refine process control mechanisms• Develop, refine and leverage communications channels

Act

How to ImplementImplementation


Outcomes of Data Driven GovernanceImplementation

• Faster, more accurate decision making

• Better use of resources• Better enterprise cohesion

and synchronization• Data-driven outcomes• Improved information

sharing• Adaptable to change

Observe

Orient

Decide

Act




Questions

Data & Analytics

Data Driven Cybersecurity Governance