Using Information Security Metrics To Demonstrate Value and Drive

E D U C AU S E S E C U R I T Y P R O F E S S I O N A L S C O N F E R E N C E

M AY 6 - 8 , 2 0 1 4

Using Information Security

Metrics To Demonstrate Value

and Drive Improvements

SHIRLEY C. PAYNE

AVP FOR INFORMATION SECURITY, POLICY, & RECORDS

UNIVERSITY OF VIRGINIA

Copyright Shirley C. Payne 2014.

This presentation leaves copyright of the content to the presenter.

Unless otherwise noted in the materials, uploaded content carries the

Creative Commons Attribution-NonCommercial-ShareAlike license,

which grants usage to the general public with the stipulated criteria.

Seminar Will Answer…

Demonstrate Value & Drive Improvements

What makes a metric effective?

What are the challenges?

Where do I start?

How should I communicate

metrics?

Where can I learn more?

and provide lots of examples…



Provide single-point-in-

time views of specific,

discrete factors

Generated by counting

Objective raw data

Derived by comparing

two+ measurements

taken over time to a

predetermined baseline

Generated by analysis

Objective or subjective

human interpretations of

those data

Measurements and Metrics – same thing?

The Mark of Good Metrics

Metrics should be SMART

Specific Well-defined, using unambiguous wording

Measurable Quantitative when feasible

Attainable Within budgetary and technical limitations

Repeatable Measurements from which metric is derived do

not vary depending on the person taking them.

Time-dependent Takes into consideration measurements from

multiple time slices

George Jelen, “SSE-CMM Security Metrics”

Albert Einstein

Everything that

can be counted

does not

necessarily count;

everything that

counts cannot

necessarily be

counted.

Truly Effective Metrics…

Indicate the degree to which security goals are being met

Show linkage between security and institutional goals

Drive actions taken to improve the overall security program

Rate This Metric

% of servers that are secure has increased fourfold since 2010

S

M

A

R

T

Rate This Metric

% of servers that are secure has increased fourfold since 2010

% of servers with patched operating systems

increased fourfold since 2010

Rate This Metric

% of employees who are aware of security threats doubled last

year

S

M

A

R

T

Rate This Metric

% of employees who are aware of security threats doubled last

year

% of employees completing annual security

awareness training doubled last year

Rate This Metric

Level of faculty frustration w/2-factor authentication compared

to reduced risk of unauthorized data access

Rate This Metric

Level of faculty frustration w/2-factor authentication compared

to reduced risk of unauthorized data access

% of faculty issued UVa’s hardware identity tokens

compared to faculty use of tokens for email login

Rate This Metric

Web application vulnerabilities found during January 2014

penetration test

S

M

A

R

T

Rate This Metric

Web application vulnerabilities found during January 2014

penetration test

Web application vulnerabilities found during

January 2014 penetration test compared to January

2013 and 2012 results

Rate This Metric

% of total IT budget spent on security increased by 2% each of

the past two years

S

M

A

R

T

Rate This Metric

% of total IT budget spent on security increased by 2% each of

the past two years

Since the implementation of xxx product costing

$50K, the occurrence of records with highly

sensitive content stored in poorly secured data

stores has been reduced by 8 million

Rate This Metric

In 2013 there were 98 reported Higher Education breaches

nationwide compared to 5 at this institution.

S

M

A

R

T

Rate This Metric

In 2013 there were 98 reported Higher Education breaches

nationwide compared to 5 at this institution.

In 2013 there were 5 reported breaches, 4 of which

were discovered by internal controls (versus

reported by outsiders)

The Value of “Truly Effective” Security Metrics(internal focus)

Discern effectiveness of particular security program

component

Indicate security of specific system, product, or process

Identify risk in not taking a given action and, thereby, help

prioritize corrective actions

Provide evidence of regulatory compliance

Demonstrate ability of security staff and departments to

address security issues for which they are responsible

Value of “Truly Effective” Security Metrics(external focus)

Provide basis for answering tough questions, such as

Are we more secure today than we were before?

How do we compare to others in this regard?

Are we secure enough?

Raise security awareness among executives and other

stakeholders

Clearly convey value of overall security program relative to

business objectives


Characteristics of

effective metrics:

SMART

Indicate % to which

security goals are met

Link security to

institutional goals

Drive improvements

Effective Metric

Specific

Measurable

AttainableRepeatable

Time-dependent


A metric is effective if it can:

Provide insight into IS program effectiveness,

regulatory compliance, and ability to address security

concerns

Help identify risks of not taking certain actions,

providing guidance for future investments.

Provide concrete facts for raising security awareness

Provide credible answers to hard questions about

status and value of IS program




The State of Security Metrics

Other disciplines, such as the field of finance, have proven

quantitative methods for determining risk, along with

decision-making frameworks based on established

measures and metrics.

[These] are just emerging for information security, however,

and as in any discipline, require realistic assumptions and

inputs to attain reliable results.

Wayne Jansen, “Directions in Security Metrics Research,” NISTIR 7564; April 2009

75%CISOs say…

53%CISOs say their…

51%CISOs say their…

Why

Not?

18

35

40

48

59

0 10 20 30 40 50 60 70

EXECS NOT INTERESTED

TIME/RESOURCES TO PREP REPORTS FOR EXECS

ONLY COMMUNICATE W/ EXECS ON INCIDENTS

HIGHER PRIORITIES

INFO TOO TECHNICAL FOR EXECS

%

%

Conclusion

“CISO’s talk about the importance of leveraging

metrics to influence business leadership…

Unfortunately, they struggle with the bigger

challenge of producing meaningful metrics

while those they use are rarely aligned with

business goals.”

Rekha Shenoy, Tripwire VP for Marketing & Corporate Development

Let’s Look At The Challenges

Measuring Risk

Determining ROSI

Limited Guidance and Practical Examples

How To Measure Risk?

Risk = Asset Value x Threat x Vulnerability

Asset Value – easiest to measure in some cases, but

how to quantify assets like institutional reputation?

Threat – very hard to measure the potential for harm,

although information from external sources may be

useful.

Vulnerability – sources of good information available,

but not all vulnerabilities can be quantified.

Determining ROSI?

“It’s a good idea in theory, but it’s

mostly bunk in practice… Security

is not an investment that provides

a return… It is an expense that,

hopefully, pays for itself in cost

savings…Security is about loss

prevention, not about earnings. “

Bruce Schneier – September 2, 2008

https://www.schneier.com/blog/srchives/2008/09/

security_roi_1.html

Guatemala Sinkholehttp://news.nationalgeographic.com

“We’re here to suggest not only

that you can use ROSI to sell

security internally, but you must.”

Scott Berinato, “Calculated Risk: Return on Security Investment,

www.csoonline.com

An Alternate View!

Challenge industry assumptions and cultural biases

Rethink Your Assumptions

Find and use data that’s out there

Do the Legwork

Subtract cost from benefits

Do the Math

Scott Berinato, “Calculated Risk: Return on Security Investment, www.csoonline.com

Challenge industry assumptions and cultural biases

Rethink Your Assumptions

• Precision is not the goal

• Think in stochastic, not binary, terms

Fire extinguisher ROI: $3 return for every $1 invested

NOT

Fire extinguisher ROI: $3.14 return for every $2.97 invested

Find and use data that’s out there

Do the Legwork

• Actuarial information, e.g., CERT, Poneman

• Annual data breach reports, e.g., Verizon, privacyrights.org

• Threat trends, e.g., IBM X-Force, Mandiant

• Talk to business managers, e.g., Risk Management Officers, Financial Managers


Do the Math

• Annual Loss Expectancy

• Modified ALE

• Other methods

Cost Examples

Lost staff productivity

Loss/compromise of data

Recovery costs

Reputational loss

Fines and lawsuits

Loss of future research grants/contracts

Etc.

Informed by Julia Allen – March 10, 2003 “Making the Business

Case for Information Security: Selling to Senior Managements”

Scenario: Need to determine ROSI on acquisition of

web app vulnerability scanning service


Do the Math

• Annual Loss Expectancy: how much $$ lost per year due to security incident

ALE =

average cost of data breach X probability of web app breach next year

ALE = $3.2M X .22 = $704,000

Poneman Study Verizon Study


Do the Math

• Modified ALE: ALE w/effect of mitigation measure incorporated

Assumption: Effect of scanning all web apps for vulnerabilities is

that probability of web app breach reduced by half

mALE = $3.2M X .11 = $352,000


Do the Math



COST SAVINGS = ALE – mALE

COST SAVINGS = $704,000 - $352,000 = $352,000


Do the Math



ROSI = BENEFITS - COST

Cost per year of xyz web app service = $80,000

ROSI = $352,000 - $80,000 = $281,600

Let’s Look At The Challenges

Limited Guidance and Practical Examples

Limited Guidance and Practical Examples?

Good News!

ISO 27004

NIST SP 800-55 Rev. 1

CIS Consensus Information Security Metrics

Top 20 Critical Security Controls

ISO/IEC 27004

Published December 2009 (new version planned)

Guidance for developing metrics for evaluating

information security programs

Key sections: Information security measurement overview;

Management responsibilities;

Measures and measurement development;

Measurement operation;

Data analysis and measurement results reporting;

Program evaluation and improvement.

http://www.iso.org/iso/catalogue_detail?csnumber=42106

http://www.iso.org/iso/catalogue_detail?csnumber=42106

NIST SP 800-55 Rev. 1

Published July 2008

Specific advice for developing, selecting, and implementing

performance measures

Security controls tied to overall mission

Practical examples

http://csrc.nist.gov/publications/PubsSPs.html


CIS Consensus Information Security Metrics

V1.1.0 published November 2010

Metrics on security outcomes and process performance.

Common definitions for data collection and analysis

Metrics grouped by purpose and audience: management,

operational, technical

Twenty metrics defined in six functions: incident management,

vulnerability management, patch management, application security, configuration

management, financial metrics

https://benchmarks.cisecurity.org/downloads/browse/?category=metrics

https://benchmarks.cisecurity.org/downloads/browse/?category=metrics

Top 20 Critical Security Controls

V5.0 published February 2014

Identifies controls having greatest positive impact on risk

posture

Includes suggested metrics for most controls

http://www.counciloncybersecurity.org

http://www.counciloncybersecurity.org/

Great News: There’s Now Helpful Guidance

…for/by HIGHER EDUCATION w/EXAMPLES!

EDUCAUSE: 7 Things You Should Know About Security

Metrics article

EDUCAUSE: Guide To Effective Security Metrics

EDUCAUSE: Security Metrics Resource Library

EDUCAUSE: Core Data Services

http://www.educause.edu/library/resources/7-things-you-should-know-about-information-security-metrics

https://wiki.internet2.edu/confluence/display/2014infosecurityguide/Effective+Security+Metrics

http://www.educause.edu/library/security-metrics

http://www.educause.edu/research-and-publications/research/core-data-service

Also, check out these conference sessions…


Lack of common vocabulary and definitions

We don’t speak the language of executives:

Institutional goals

Risks

ROI

Finally, practical guidance and examples!


Where do I start?


Seven-Step Methodology

Review and refine

Create action plan

Determine how to report

Establish benchmarks and targets

Develop strategies for generation

Decide what metrics to generate

Define goal(s) and objectives

Step 1

Define the metrics

program goal(s) and

objectives

Clearly state the end toward which

all metrics and measurements should

be directed

Indicate high level actions that must

be collectively accomplished to meet

the goal(s)

Step 2

Decide what metrics to

generate

Use existing process improvement

framework to determine metrics

Framework Examples

Six Sigma Breakthrough Strategy

Balanced Scorecard

Enterprise Risk Management

Enterprise-level Compliance Tracking

Strong Focus Within Institution On: ROI

On time/on schedule project completion

National rankings

Bond ratings

Etc.

Step 2

Decide what metrics

to generate

Use existing process improvement

framework to determine metrics

In the absence of pre-existing

framework, use top-down or

bottom-up approach for determining

what metrics might be desirable

Top-down Approach

STEPS EXAMPLES

a. Define/list objectives of the overall

security program

To reduce the number of virus infections

within the institution by 30% by 2015

b. Identify metrics that would indicate

progress toward each objective

Current ratio of virus alerts to actual

infections as compared to the baseline

2012 figure

c. Determine measurements needed for

each metric

Number of virus alerts issued to the

organization by month

Number of virus infections reported

Bottom-up Approach

STEPS EXAMPLES

a. Identify measurements that are/could

be collected for this process

Average number of critical

vulnerabilities detected monthly in

servers using xyz scanning tool

b. Determine metrics that could be

generated from the measurements

Change in number of critical

vulnerabilities detected in servers since

xyz scanning tool implemented

c. Determine the association between the

derived metrics and established

objectives of the overall security program

To reduce the number of detectable

vulnerabilities on servers by 95% by

2015.

Step 3

Develop strategies for

generating the metrics

Identify trustworthy sources of data Internal, e.g., IT operations, Audit, Risk Management,

Finance, Compliance, etc.

External, e.g., actuarial data, annual breach stats, etc.

Decide on frequency of data collection

Assign responsibility for assuring

accuracy of raw data

Develop methods for compiling data into

measurements and generating metrics

Step 4

Establish benchmarks,

baselines, and targets

Research observed trends and

recommendations from professional

associations, published research, etc.

Set reachable targets

Step 5

Determine how the

metrics will be

reported

Effective communication of metrics is

obviously key. Don’t over-simplify, but

present clearly.

Vary what is reported and how

depending upon audience

Determine context, format,

frequency, distribution method, and

reporting responsibility

Step 6

Create an action plan

and act on it

Plan and conduct actions needed to

generate metrics; test, verify,

investigate anomalies; implement

Document!

Field Data

Measure ID ISPA-1

Goal Identify levels of serious network-based threats of the kind monitored by the FireEye scanner by network

Measure Since implementation of FireEye monitoring, number of high severity level infections detected on each network

Type Implementation

Formula Number of critical issues identified by FireEye on each network

Target Baseline; comparison

Definition of Measures Contributing to Metrics

•“Critical” is defined by FireEye based on its risk analysis; it represents items receiving a score of 5-7 on a 7 point scale; UVa does not have input on this definition•The networks are defined by the UVa network architecture; the number of devices on each network will vary over time as devices are added and migrated between networks; additional networks may be added to the list of those scanned over time

Frequency Data are collected daily; they will be reported, as appropriate, on a daily, weekly, monthly, quarterly and/or fiscal year basis

Responsible Parties •Information Owner: AVP ISPRO•Information Collector: ISPA team•Information Customer: VP/CIO, AVP ISPRO, Director ISPA

Data Source FireEye console

Reporting Format Bar graph; spreadsheet

Step 7

Establish a formal

program review and

refinement cycle

Doubt about metric accuracy?

Value worth effort to generate?

New metric best practices/guidance

to consider?

Most important: did metrics guide

improvements to overall security

program?

Adjust for Maturity of Security Program

Usefulness of a Given Metric Varies Depending Upon

Maturity of the Security Program

Policies Developed

Procedures Developed

Procedures & Controls

Implemented

Procedures & Controls Tested

Procedures & Controls Integrated

“Performance Measurement Guide for Information Security,” NIST SP 800-55 Revision 1




Policies Developed



Implemented



Useful metrics difficult to produce at this early stage; limited availability of data and collection may be difficult



Policies Developed



Implemented



Primary focus on implementation metrics

• Ex: Increase in # of departments that have mission continuity plans



Policies Developed



Implemented



Primary focus on efficiency and effectiveness metrics

• Ex: % of total departments with updated, tested mission continuity plans



Policies Developed



Implemented



Primary focus on impact metrics

• Ex: Outcome of 48-hour power outage in administration bldg.



Policies Developed



Implemented



Primary focus on implementation metrics

• Ex: Sensitive data scanning tool deployed on all individual desktops/laptops



Policies Developed



Implemented



Primary focus on efficiency and effectiveness metrics

• Ex: # of unapproved storage of sensitive data found on desktops/laptops



Policies Developed



Implemented



Primary focus on impact metrics

• Ex: Reduction in sensitive data exposures due to stolen or vulnerable desktops/laptops

Where do I start?

Leverage existing frameworks for expressing: progress

toward goals, value propositions, process improvements,

etc.

Use systematic approach for defining effective metrics

Adjust metric types as security program matures

Implementation Metrics

Efficiency and Effectiveness Metrics

Impact Metrics


How should I communicate

metrics?


Good News: It’s Now A Hot Topic

“Security is THE issue of

the day and security is

everyone’s responsibility ...”

CIO, Commonwealth of Virginia Information Technology Agency

at 2014 Commonwealth of Virginia Information Security Conference

84

But, how to make your message heard?

Albert Einstein

Things should

be made as

simple as

possible, but not

any simpler

Tip…

Customize your

metrics-based

information for

the audience

Customize for Security Engineers

• Change in #

malware

infections over

time

• # web app

vulnerabilities

detected since

scan tool

implemented

• Mean time

between phish

report and

blocked malicious

sites

Customize for CISO/CIO

• Change in # of

proactive security

consultations

compared to FY13

baseline.

• Since

implementing web

application

security scanning

service, # high

severity level

vulnerabilities

detected declined

90%.

Customize for Executives

• % of IT budget

spent on security

compared to peer

institutions

• Since institution-

wide SSN

remediation

project initiated,

change in ratio of

data security

breaches to total

security incidents

investigated

Tip…

Use effective

visuals

Additional Tips for Communication

Provide right metrics for issue at hand

Provide brief interpretation and analysis

Use specific audience’s language

Link to business goals and objectives

How should I communicate metrics?

Take heart. You now have a receptive audience.

Tailor for the audience

Delivery method is as important as what you have to say

Right metric clearly conveyed

=

Right conclusion & decision


Where can I learn more?


References

Allen, Julia. “Making the Business Case for Information Security: Selling to Senior Managements.” Carnegie Mellon University at InfoSec World, 2003, March 10, 2003

Allen, Julia and Stephani Losi. “The ROI of Security.” Software Engineering Institute, Carnegie Mellon University, October 1, 2006. http://resources.sei.cmu.edu/asset_files/podcast/2006_016_100_47182.pdf

Berinato, Scott, “Calculated Risk: Return on Security Investment,” CSOonline.com, December 9, 2002, http://www.csoonline.com/article/2113094/metrics-budgets/calculated-risk--return-on-security-investment.html

Center for Internet Security. “CIS Consensus Information Security Metrics,” November 2010. https://benchmarks.cisecurity.org/downloads/browse/?category=metrics

Council on Cybersecurity. “Top 20 Information Security Controls,” http://www.counciloncybersecurity.org

Cullinane, Dave. “Security Awareness and Communication in the C-Suite,” EDUCAUSE e-Live Webinar, October 4, 2012. http://www.educause.edu/library/resources/security-awareness-and-communication-c-suite

EDUCAUSE: 7 Things You Should Know About Security Metrics, http://www.educause.edu/library/resources/7-things-you-should-know-about-information-security-metrics

References

EDUCAUSE: Guide To Effective Security Metrics, https://wiki.internet2.edu/confluence/display/2014infosecurityguide/Effective+Security+Metrics

EDUCAUSE: Security Metrics Resource Library, http://www.educause.edu/library/security-metrics

EDUCAUSE Core Data Service, http://www.educause.edu/research-and-publications/research/core-data-service

Hinson, Dr. Gary, “Seven Myths About Security Metrics,” ISSA Journal, July 2006. http://www.noticebored.com/html/metrics.html

ISO/IEC 27004 http://www.iso.org/iso/catalogue_detail?csnumber=42106

Jansen, Wayne. “Directions in Security Metrics Research,” NISTIR 7564; April 2009. http://csrc.nist.gov/publications/nistir/ir7564/nistir-7564_metrics-research.pdf

Jelen, George. “SSE-CMM Security Metrics.” NIST and CSSPAB Workshop, Washington, D.C., 13-14 June 2000. URL: http://csrc.nist.gov/csspab/june13-15/jelen.pdf (10 July 2001).

Payne, Shirley C., “A Guide To Security Metrics,” SANS Reading Room, July 11, 2001, updated June 19, 2006. http://www.sans.org/reading-room/whitepapers/auditing/guide-security-metrics-55

References

“Performance Measurement Guide for Information Security,” NIST SP 800-55 Revision 1 –http://csrc.nist.gov/publications/PubsSPs.html

Poneman Institute. “The 2013 Cost of a Data Breach: Global Analysis,” May 28, 2013. http://www.ponemon.org/library/2013-cost-of-data-breach-global-analysis

Ponemon Institute. “The State of Risk-based Security Management 2013,” http://www.tripwire.com/ponemon/2013/

Schneier, Bruce. “Security ROI,” September 2, 2008 https://www.schneier.com/blog/archives/2008/09/security_roi_1.html

Slater, Derek, “Security Metrics: Critical Issues,” CSOonline.com, Nov 12, 2012, http://www.csoonline.com/article/2123361/metrics-budgets/security-metrics--critical-issues.html

Stafford, Eugene and Christina Torode. “A bleak picture of IT security metrics and fighting malicious attacks,” ISSA conference, Nashville, Tenn., December 11, 2013 http://searchcompliance.techtarget.com/video/A-bleak-picture-of-IT-security-metrics-and-fighting-malicious-attacks

Most Of All…

Keep your eyes on the forest, not the trees!

Documents

Using Information Security Metrics To Demonstrate Value and Drive