Risk ManagementMetrics that Matter

Ed Bellis

• Co-founder and CTO at Kenna Security, an automated risk & vulnerability intelligence platform

• Orbitz CISO for 6 years

• 20+ years Info Security experience including Bank of America, CSC, E&Y

• Contributing Author Beautiful Security

• Frequent speaker at events such as…

About Me

WarningThis presentation contains large amounts of data used

for the purpose of proving an information security theory. No marketers were harmed during the making of

this presentation.

You Are What You Measure

JET FUEL X PEANUT BUTTER = SHINY-ALEX HUTTON

Inherent Risk Residual Risk

Know & Measure the Difference

vs.Hint: This is NOT a math formula

Inherent Risk: 80

Please Don’t Do This!

Control Effectiveness: 50%X

Residual Risk: 40

JET FUEL X PEANUT BUTTER = SHINY-ALEX HUTTON

Do This Instead

1. Calculate Risk 2. Identify Potential Key Controls 3. ReCalculate Risk

The Language Barrier

*source: Cyber Balance Sheet - The Cyentia Institute

The Language Barrier

*source: Cyber Balance Sheet - The Cyentia Institute

What the CISO perceives as important versus what

the BoD believes is important often don’t

match and often neither are actually given.

The Language Barrier

*source: Cyber Balance Sheet - The Cyentia Institute

But First…

Threats, Vulnerabilities & Risks.. oh my!

But First… Some Definitions

Threat: A negative scenario you want to avoid.

Threat Actor: the agent that makes the threat happen.

Vulnerabilities: a weakness that can be exploited.

Risk: a negative scenario you want to avoid combined with its probability & impact.

FAIR Example: Risk Taxonomy

Integrate or Die

Operationalizing Security Risk Management

Measurement + Integration

Risk Management Decision Making

Selecting the Right Metrics for Risk Management

Risks > Counts

Results > Work

Quantitative Where Possible

Know Your Assets

Some Useful Metrics

1.External Asset Coverage2.Internal Asset Coverage3.Time to Discover

Know Your Business

Some useful metrics here include:

1. System Susceptibility

1. Value to Attackers

2. Vulnerabilities

2. Time to Compromise: How long would it take to compromise any of the key controls for these assets and applications?

3. Threat Accessibility

1. Access Points and Attack Surface

4. Threat Actor Capability

1. Tools

2. Resources c.

3. Techniques

Does Your Threat Model Include Alexa Ratings?

Know Your Risk

Some Useful Metrics

1.Risk by Asset2.Risk by Business Unit3.Trending Risk over Time4.Mean Time to Risk Reduction

*use targets/goals and mature to SLAs

Know Your Resources

Some Useful Metrics

1.Budget Spent on Security Remediation2.Risk Carried Above Tolerance Level3.Hours spent per Security Solution

Know Your Direction

Some Useful Metrics

1.Risk Reduction by Group Over Time2.Risk Goal/SLA by Group3.Cumulative Risk Accepted Over Time

Some Not So Useful Metrics

1. Measuring Work AKA “atta boy metrics”

Number of Vulnerabilities Closed

Number of Patches Deployed

Number of Incidents Responded to

Some Not So Useful Metrics

2. Measuring Counts “vanity metrics”

Number of Packets Dropped

Number of Malware Detections

Number of IDS Alerts

Some Not So Useful Metrics

3. Averages can be a Fool’s ErrandAverage Age of Vulnerability

Average Time to Discover

Average Time to RespondHint: Averages are skewed by outliers. Medians are your friend.

Aging Can Incent Wrong Behavior

Remember This?

Your Coworkers Have Day Jobs Too

Leverage Existing Tools• Bug Trackers• Trouble Ticketing• Configuration Management• Continuous Integration & Deployment

Bonus Points: Leverage Existing Tools for Security Purposes

Your Coworkers Have Day Jobs Too

Leverage Existing Processes• Change Management• Bug Fixing• Design Reviews• QA Testing• Continuous Integration

The Payoff

Operationalizing Security Risk Management

Security Teams

Operations Teams

Development Teams

Executive Management

Common Language

Distinct Objectives

Efficiency

Effectiveness

References

FAIR Risk Taxonomy: http://www.opengroup.org/subjectareas/security/risk

Cyber Balance Sheet: https://go.focal-point.com/cyber-balance-sheet-report

Risk Management Metrics That Matter: https://blog.kennasecurity.com/2017/03/creating-risk-management-metrics-that-matter/

Q&A