Embed Size (px)
Citation preview
1
You’re Measuring The Wrong ThingsINFORMATION SECURITY PROGRAM MEASUREMENTSHAWN SINES, CISSP
2Summary
Information security programs can be ephemeral and hard to quantify. Metrics serve many goals but if it doesn’t lead to action or empower
prioritization and decisions you are not getting the value from the activity
Metrics also MUST to be relevant to the audience you present them to – there is a big difference in Executive concerns and Subject Matter/Technical Expert concerns
3The Problem
Measuring everything takes time, energy and people Information Security often works on the probability or
uncertainty principals – we work to prevent bad things – so the argument is its hard to prove something did not happen
Knowing what to measure and how are not the same skills that Information Security people cultivate or find attractive
Too much measurement defeats progress Sharing the wrong metrics will defeat your goal
“It was my understanding that there would be no math.”
4The Goal
How do we get anyone to pay attention? Make the metric relevant to the reader Present it in a visual manner that does not dilute the results
How do we determine if something is working? We measure it!
How do numbers help us? They establish a repeatable, consistent scale to judge activity
Can we use qualitative analysis in Metrics? You must set context to your labels that can be applied consistently
5Prioritize the Discussion
There are always competing activities, new risks and everything has to be done now with the same people and money
Metrics allow us to show/demonstrate where we are likely to get the best returns for the effort/investment
Identify the difference between an Executive metric and an Operational metric – you need both but Operational metrics will have little context to drive Executive decision making without a lot of discussion.
6Driving Decisions or Support
Establishing measurement allows you to move beyond feelings and instinct
You can attack or question expertise, but well documented and measured activities speak for themselves
Use measurement to illustrate your organizational need – paint a picture in words AND numbers
If you want to accomplish something you have to validate your point of view in a way that others with less expertise can digest – even if they do not “like” numbers having them to support your story strengthens your delivery.
7Measuring the Right Things
Measurements vs. Metrics: Measurements are generated by counting; Metrics are generated from analysis.
Example 1: The number of high vulnerabilities on our systems
Example 2: The average time to resolve identified vulnerabilities
Both are important for context but which one can you control better and get a better reduction of risk/investment return on?
8Counts vs. Activity
COUNTS: It is important to count items to set scope, but often counts are merely informational
and can be outside the scope of control of the organization i.e. Understanding how “big” a problem is helps to establish the priority and risk, but it
does not always lead to something I can reduce ACTIVITY:
Measuring the effectiveness of an activity OVER TIME allows the organization to identify root factors such as resources, effectiveness of tools and process, etc. Action measurement directly impacts resource allocation and should provide a way to measure the rate of return for an investment in a process or tool.
i.e. I buy a patch tool or hire two people or design a process and my effective time to resolution of unpatched vulnerabilities can be measured and the return can be illustrated
9Cadence is Important
When allocating and establishing goals you need to identify the mean time to change in order to best show change Example:
My patch management tool scans once a month. It takes my system managers two weeks to identify a plan to address a new Critical vulnerability and on average two more weeks to coordinate/test the possible fix. Change management supports a weekly review process in non-emergency cases.
What is the reasonable rate at which a conscientious admin could reasonably make changes to address a vulnerability?
10Types of Metrics
Compliance Metrics Used to show the organization is meeting an established compliant state Typically Boolean – yes/no in nature Lend themselves to a fixed reporting/measuring cycle based on
contractual/legal timeframes i.e. Quarterly PCI Compliance, Annual SOX Control Evaluation
11Types of Metrics
Operational Metrics Used to measure the ongoing state of the
Information Security program Measurements require tuning over time to drive
continuous improvement and establish acceptable performance levels of operational processes and activities
Cadence may vary between each metric based on data collection and inputs
12Types of Metrics
Program/Executive Metrics Used to measure the ongoing state of the Information Security program with
Executive decision makers Must be digestible by non technical audience Must have business context – obscure the technology from the measure to
show the business value
13Types of Metrics
Remediation Metrics Typically used to bring a process into line with expectations Usually designed as a temporal measure or corrective activity May be emergency or as a result of audit or incident findings Cadence varies based on impact of remediation driver
14Types of Metrics
Project/Task Metrics Used to measure progress of project activities or milestones Typically temporal in nature and built to measure progress toward project
deliverables May develop into long term measurements as a result of completing the
project to support ts deliverables
15Metric Update Cycles
Weekly Metrics Activity changes rapidly – often tied to
remediation efforts or evolving activities Monthly Metrics
More “regular” activities Quarterly Metrics
Longer term, fairly static or in “steady state” Annual Metrics
Very steady, unlikely to change
16Metric Indicators
Stoplights vs. Ranges and Values Red/Yellow/Green Values
Change Indication & Trending Absolute/Target Values Maturity Values
17
Metric Examples (Good & Bad)
18Incident Response Metrics
Bad: Number of times we were “attacked” last month Number of security breeches in <year>
Good (Operational): Average mean time to incident discovery Average mean time to incident resolution % of incidents detected by internal controls
Good (Executive): Average Rate of Incidents by Business Line Most common types of incident by Business Line
19Vulnerability Management Metrics
Bad: Number of unpatched vulnerabilities on critical systems Number of systems with unpatched vulnerabilities Average number of days to patch vulnerable systems
Good (Operational): % of unpatched critical vulnerabilities on critical systems Average number of days to patch critical systems % of systems with unpatched critical vulnerabilities
Good (Executive): % of systems related to <business line> with unpatched critical vulnerabilities
20Information Security Program Metrics
Bad: Number of security controls
Good (Operational): % of systems covered by control Average maturity by control
21Wrap Up
Measurement and Metrics are time consuming You have limited resources and too much data – focus on measuring
things that enable decision making and prioritization of those limited resources
Metrics are a form of communication – know your audience and play to them appropriately
No magic catalog of metrics applies universally to every organization – the key is to build and scale the approach for yours not to do what everyone else is doing or reporting
