Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
IBM Global ServicesSecurity & Privacy Services
Information Security EffectivenessInformation Security EffectivenessMetrics: What Metrics? What Role for Metrics?Metrics: What Metrics? What Role for Metrics?
Matunda Nyanchama, PhD, CISSPNational Leader, Security & Privacy Delivery Services
IBM Global Services, CanadaE-mail: [email protected]
Website: www.ca.ibm.com
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 22Copyright IBM Global ServicesCopyright IBM Global Services
AgendaAgenda• Background
– Some Definitions– Why Metrics?
• IS Metrics - Background– Value Information Security Metrics– Metrics Development Process– Scope of Measurement – ISO 17799
• Scoping out IS Metrics– Information Security Program – Example– Scope of Considerations for Measurement– Examples of Measures
• Metrics & Reporting– Data Sources for IS Metrics– IS Metrics Process & Reporting– Metrics – Breadth, Depth & Purpose– Incident Management Example– Sample IS Dashboard
• State of IS Metrics & Caveats & Some Suggestions• Summary
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 33Copyright IBM Global ServicesCopyright IBM Global Services
Some DefinitionsSome Definitions• Metric: “relating to measurement; involving, or proceeding by, measurement”
(Webster’s Revised Unabridged Dictionary) • “Information Security” pertains to integrity, confidentiality & availability;
auditability and accountability• Security Metric: “A measurable attribute of the result of a security engineering process
that could [be] evidence its effectiveness.” (see references) • Effectiveness: Having an intended/expected effect; operative; in effect; efficacy,
force, punch, power, strength, success, validity, vigor, weight (The American Heritage Dictionary)
• Efficiency: Production of desired effect/results with minimum waste of time, effort, or skill ; a measure of effectiveness; specifically, the useful output divided by input into a system; proficiency, capability, adeptness, adequacy, suitability (The American Heritage Dictionary)
• Benchmark: Reference, a standard by which something is measured; criterion, gauge, goal, measure, standard, touchstone, yardstick
• Return on Investment (ROI):A measure of profitability; it measures how effective a company uses its capital to generate profit; income that an investment provides in a specified time (e.g. one year)
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 44Copyright IBM Global ServicesCopyright IBM Global Services
Why Metrics?Why Metrics?Metrics are intended to:• Focus on measurable attributes … that could serve as evidence
of effectiveness/efficiency of a given program or process• Facilitate decision making: what are the shortcomings? How
closely are objectives met? Gaps/shortcomings if any? Need change of direction?
• Help improve performance and accountability: where are gaps? How can things be done better? Who is responsible?
• Can be objective or subjective, and quantitative or qualitative. • To be relevant, metrics should be SMART, i.e. Specific,
Measurable, Attainable, Repeatable and Time-independent
Remember: “If you cannot measure it, you cannot manage it.” - anonQuestion? Where are we with Information Security Metrics?
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 55Copyright IBM Global ServicesCopyright IBM Global Services
Value Information Security Metrics Value Information Security Metrics -- II• IS performance against defined IS goals, e.g.
– Efficacy of information security – Accountability to stakeholders
• Assess IS plans, programs, processes, etc. for– Efficiency – how well information security resources are utilized– Effectiveness of Information security program + existing security
controls• Identify IS risks
– What assets need protection? What is their value?– What threats and vulnerabilities exist to the assets? – What chances for exploitation exist?
• IS Risk Management– Risk assessment - extent of exposure to threats + potential business
impacts should attacks happen– Controls - What countermeasures/controls to identified risks– Controls assessment - How effective are those controls
• Assess IS posture
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 66Copyright IBM Global ServicesCopyright IBM Global Services
Value Information Security Metrics Value Information Security Metrics -- IIII• Security posture trends – is the “state of security” improving,
staying the same or getting worse?• Help identify priorities for resource deployment in based on
risk levels to assets• Facilitate corrective action where controls are weak, e.g. where
incident response times are unacceptable• Demonstrate the value of information security to executives • Benchmark against industry, where possible - how do we
compare with our peers in industry?• Can be used for compliance-related assessments – e.g. SOX for
internal controls assessment
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 77Copyright IBM Global ServicesCopyright IBM Global Services
Information Security Metrics Benefits SummaryInformation Security Metrics Benefits Summary• Productivity indicators:
– Effectiveness & efficiency of a security program– Security return on investment (ROI) (where possible to measure) – Information security program maturity
• Information Security posture:– Collected data can be used as baseline for measurements & trending– Risks are identified and a business case made to address the risks
• Help define a baseline and hence deviations:– Apply risk management methodology for deviations from baseline – Quantify risk and hence plan for better risk management strategy
Used appropriately:• Metrics can engender process improvement.• Demonstrate value of Information Security investment, e.g. ROI • Facilitate risk management• Allow benchmarking with industry peers
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 88Copyright IBM Global ServicesCopyright IBM Global Services
Metrics Development ProcessMetrics Development ProcessFollow ISO17799’s “plan-do-check-act” cycle• Plan
• Establish key objectives for the metrics required• Identify the required metrics and hence required data• Design & implement strategy for data collection & metrics generation• Establish targets/benchmarks; where possible compare with industry• Determine the process for collecting and analyzing data, and reporting • Establish metrics review program, and the refinement process/cycle
• DO• Communicate with stakeholders and ensure buy-in• Implement the metrics program – people, process and technology
• CHECK/Monitor• Continuously review metrics report against objectives and benchmarks• Monitor program performance against objectives and benchmarks• Identify gaps, if any, in the program
• ACT• Address gaps in program• Refine specific metrics, where necessary• Refine metrics program, where necessary
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 99Copyright IBM Global ServicesCopyright IBM Global Services
Scope of Measurement Scope of Measurement –– ISO 17799ISO 17799Sample MeasurementsISO Area
Sample MeasurementsGaps in policies; Potential impacts of policy gaps; # security violations per period of time.Security Policy
% staff with certification; formal roles and responsibilities; staff turnover; security spending/employee; IS spending as % IT budgetSecurity Organization
% assets in inventory; % assets with classification; % assets with valuation; % assets with protection plan
Asset Classification & Control
# security training sessions; level of security awareness; # of personnel security-related incidentsPersonnel Security
Frequency of review of physical access; # access anomalies or violationsPhysical & Environmental Security
# incidents; incident impacts; frequency of assessment; % systems with exposures; incident response metrics; how quickly threats are communicated; frequency of
awareness activities; change control issues
Communications & Operations Management
Access activation/termination turnaround; % of expired accounts; % accounts with expired pwds; % of accounts with weak passwordsAccess Control
% projects that use IS; # policy exceptions/risk acceptances; % projects that perform code reviews; freq. of VAs; % systems with vulnerabilities
Systems Development & Maintenance
% systems with BCP/DRP; frequency of BCP/DRP testing; % systems that pass BCP/DRP testing; System availability
Business Continuity Management
# & trend of exemptions; Compliance
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 1010Copyright IBM Global ServicesCopyright IBM Global Services
Elements of an IS Program Elements of an IS Program –– The IS Management Life CycleThe IS Management Life Cycle
Maintain & Improve Security Management
Program
Monitor & Continuously
Review Program Performance
Establish Information Security Management
Program
Implement Information
Security Management
Program
Development, Maintenance
& Improvement of the ISM Program
ACT
CHECK
DO
PLAN
Key Security Program Elements
Strategic
Governance, Policies & Business Strategy
Strategy, Policy, Procedures, Standards, Awareness Plan
Tactical
Risk Assessment, Design Reviews, Due Care, New Technology Insertion, Risk Acceptance, Policy Exceptions
Operational
• Active Security: Intrusion Detection & Alerts, Incident Management, Vulnerability Assessments, Data Aggregation & Analysis, Trending, Root Cause Analysis; what takes place daily captures the robustness or weakness of controls, e.g. incidents, external events
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 1111Copyright IBM Global ServicesCopyright IBM Global Services
Information Security Information Security -- Another ViewAnother View
TimeNow
Bus. Req . Design Development OperationsImplementationRisk/MoneyMoney
- (6-12) months
Operational -•Active Security Posture &
Analysis•Vulnerability Assessments•Intrusion Detection & Alerts•Incident Response•Anti-virus Management•Data Analysis & Trending
Reporting•Awareness
StrategicGovernance &
Policies•Business Strategy•Policies•Standards•Procedures•Guidelines•Awareness Strategy•Research
TacticalApplications & Systems
Development•Risk Assessment•Design & Code Reviews•IS Solutions•Due Care •Risk Acceptance•Policy Exceptions•Technology Insertion•Awareness
ACT
CHECK
DO
PLAN
Development, Development, Maintenance & Maintenance & ImprovementImprovement
- (1-3) years
Reference to Industry Standards: ISF, ISO17799, ITIL, COBIT
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 1212Copyright IBM Global ServicesCopyright IBM Global Services
Scope of Considerations for MeasurementScope of Considerations for Measurement
Possible MeasuresOrganizational Level
Security OperationsIncidents, Vulnerability
Assessment, Patch Management, threat advisories
Applications & System DevelopmentProject Assessments
Risk Acceptances, Code Reviews
Strategy & Governance
Info Sec Program + Framework; Information Security Budget
Spending/employee% of IT budget in Info Sec;
Policy Gaps in existenceBench marking against industry;
Industry standards adoptedAwareness plan
% projects going assessment process# Outstanding policy exceptions & Risk
acceptances% projects performing code reviews
Frequency of vulnerability assessments# outstanding vulnerabilitiesRate of fixing vulnerabilities
Rate of response to incidents & $ImpactsTrend of incident response losses
# & frequency of awareness sessions
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 1313Copyright IBM Global ServicesCopyright IBM Global Services
Examples of MeasuresExamples of Measures
Business StrategyPoliciesStandardsProceduresGuidelinesAwareness StrategyResearch
Vulnerability AssessmentsIntrusion Detection & AlertsIncident ResponseAnti-virus ManagementData Analysis & Trending ReportingAwareness
Risk AssessmentDesign & Code ReviewsIS SolutionsDue Care Risk AcceptancePolicy ExceptionsTechnology InsertionAwareness
Span of Measurement Across the ISLCStrategic: Governance & Policies Operations – Active SecurityApplications & Systems
Development
Security Spending/employeeStrength of the security
organizationSoundness of a security framework and security
program% of IT budget given to Info
SecBench marking against
industry; Industry standards adoptedExistence or otherwise of an
Examples of Measures Examples of Measures Examples of measures
• % projects going through security assessment process•Outstanding policy exceptions & risk acceptances•% projects performing code reviews
Frequency of vulnerability assessments; # outstanding vulnerabilities; Rate of fixing
vulnerabilities; Rate of response to incidents &
$Impacts; Trend of incident response losses; # & frequency
of awareness sessions; Existing policy gaps; IS program “fit” with other processes; Feedback
integration to security life cycle
Reference to Industry Standards ISO17799awareness plan management
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 1414Copyright IBM Global ServicesCopyright IBM Global Services
Sources of Data for MetricsSources of Data for Metrics
Information SecurityVulnerability Assessments
Incident dataIntrusion detection statistics
Antivirus statisticsProject assessment reports
Policy exceptions & risk acceptancesEducation & Awareness dataRisk control self-assessmentAccess management reports
Risk Management GroupsAudit – external & internalConfiguration management
Organization UnitsLog analysis exceptions
Corporate security reportsRisk control self-assessments
IS ReportingRisk Assessment Reports
IS Posture
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 1515Copyright IBM Global ServicesCopyright IBM Global Services
IS Metrics Process & ReportingIS Metrics Process & Reporting
Audience
Management
Operations Team
Planning
Divisions
Analysis outputsSecurity Posture
IS Posture Report
Benchmarks
Value @ Risk
Other
Other
Other
Process
Risk Management Methodology/
Process
Information Sources• Assessments – projects,
systems, infrastructure• Policy reviews• Vulnerability
Assessments• Intrusion detection
statistics• Incident Response Data• Anti-virus statistics• Access Management• Systems (physical &
logical) Logs• Audit reports – ext/Int.• Security Investigations• Self Assessments• Corporate Security
ReportsAdopted from Marc Stefaniu – see references
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 1616Copyright IBM Global ServicesCopyright IBM Global Services
Metrics Metrics –– Breadth, Depth & MeaningBreadth, Depth & Meaning• "What you measure is what you get." R. S. Kaplan & D. P. Norton in "Putting
the Balanced Scorecard to Work,“ the results of measurement are as good as the data collected
• “Not everything that can be counted counts, and not everything that counts, can be counted.” -- Albert Einstein
• You can have too many or too few measures?• Selected measures can be too specific or too general• Usefulness of information depends of meaning derived from the metrics• Can be performed top-down or bottom up• Metrics useful at one level of in the organization may not mean much at
another level; ensure that generated reports make sense for the purpose for which they were meant
• Metrics selected should serve a purpose; this should lead to required data.• Measurements in any specific area of IS can be onerous – see example on
incident data
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 1717Copyright IBM Global ServicesCopyright IBM Global Services
Example Example –– Incident Data collectedIncident Data collected• Incidents that took place within a reporting period? E.g. total number of incidents;
number of incidents of high, medium & low impact; • Percentage of total incidents with high material (high, medium) impact;• Associated business impacts (monetary and otherwise);• Losses (tangible & intangible) were incurred as a result of the incidents;• Incident losses comparison with industry for similar types of businesses and size;• Failures in security controls that led to the incidents:• Whether or not the failures have been fixed; outstanding gaps;• Improvement plans/processes are underway to prevent future recurrence of
similar incidents?• The trend to date; is the situation getting better or worse?• The incident reporting dashboard would have the following:
– Current incident posture (#incidents, monetary impacts, etc.)– Trend from last reporting period (are things getting better or worse?)– Overall trend to date– Comparison with industry benchmarks – Impact of past improvement plans– Existing gaps between desirable risk levels and current posture
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 1818Copyright IBM Global ServicesCopyright IBM Global Services
Sample Incident Management DashboardSample Incident Management Dashboard
Trend to date
Existing (known) Gap Trends
Net Impact of Past Improvements
Comparison with Bench Marks
Trend from last report
$100K46212Incident Posture
Monetary Costs (S)
Low ImpactMedium Impact
High ImpactTotal #
Cumulative Incident costs to date = $500KCumulative Incident costs to date = $500K
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 1919Copyright IBM Global ServicesCopyright IBM Global Services
Sample Charts Sample Charts –– Risk & Policy ExemptionsRisk & Policy Exemptions
Risk Acceptance and Policy Exemptions
0
20
4060
80
100
120
Q4 Year-2 Q1 Year - 1 Q2 Year - 1 Q3 Year-1 Q4 Year - 1
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 2020Copyright IBM Global ServicesCopyright IBM Global Services
Visualizing results of VA scansVisualizing results of VA scans
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 2121Copyright IBM Global ServicesCopyright IBM Global Services
Global Self AssessmentGlobal Self AssessmentGlobal Self Assessment Scorecard
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
IS P
olicy
Comp
lianc
eSe
curit
y Org
aniza
tion
Data
Clas
sifica
tion
Pers
onne
l & B
usine
ss R
elati
onsh
ips S
ecu
Phys
ical &
Env
ironm
enta
l Sec
ur
Comm
unica
tions
& O
pera
tions
Man
agem
Acce
ss C
ontro
l
Syste
ms D
evelo
pmen
t & M
ainte
na
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
Q1-2003Q2-2003Q3-2003Q4-20032003 ISF Benchmark
Legend:
0 = not implemented at all
5 = fully implemented
ISF Benchmark
• Based on ISO standard
• LOBs self-assessment reported to IS on a quarterly basis
• Gap between self reported level and ISF benchmark used to prioritize ongoing work and fed into awareness program
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 2222Copyright IBM Global ServicesCopyright IBM Global Services
Sample IS DashboardSample IS Dashboard
Overall11
Compliance10
Business Continuity Management9
Systems Development and Maintenance8
Access Control 7
Communications & Operations Management6
Physical and Environmental Security5
Personnel Security4
Asset Classification and Control3
Security Organization (Roles & Responsibilities)2
Security Policy (P&Ps, Standards, Guidelines)1
Year 0Year – 1Year – 1ISO Category
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 2323Copyright IBM Global ServicesCopyright IBM Global Services
The State of Security MetricsThe State of Security Metrics• There exists intense interest in IS metrics – just search
google to see # of hits• Most literature talks about how to define Info Sec
metrics i.e. qualities/properties of good metrics; few specifics are suggested
• IS metrics remain ill-defined; industry practices may in future lead to specific IS metrics
• Most suggested measurements tend to be qualitative; quantitative measures may yet emerge;
• Quality & effectiveness of IS program is dependent on individual opinion and judgment
• Debate on Return on Security Investment (ROSI) will continue for a while to come
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 2424Copyright IBM Global ServicesCopyright IBM Global Services
Metrics Metrics –– Some CaveatsSome Caveats• Ultimately IS metrics are intended to improve understanding
or support decision making wrt IS posture. However, – They are often ill-defined and require context and process for their
generation;– There is a risk that IS measurement can become an end in itself, i.e. the
consumer of the metric may lost in the definition of the metric.• Context is key – ensure that metrics are used with intended
purpose.• Metrics should be performance indicators, assess the value of
IS and offer pointers to performance improvement.• The heterogeneous nature of infrastructures make
measurements difficult• Issues pertaining to IS change rapidly and hence measures
should evolve with the changes• The nature of threat can change with circumstance & time
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 2525Copyright IBM Global ServicesCopyright IBM Global Services
Metrics Development Metrics Development –– Some SuggestionsSome Suggestions• Identify the key areas of risk to your business and
ensure appropriate focus• Select a few IS metrics that make sense to your
organization based on above risk assessment; think of the 80:20 rule
• Start small collecting required data & refine with time• Implement a program for continuous improvement;
seek feedback on the value of the measures selected• Focus on outcomes, i.e. what the analysis points to;
metrics should not be ends in themselves.• Keep abreast with industry practices and incorporate
best practices
Security & PrivacyIBM Global Services
November 21, 2004November 21, 2004 2626Copyright IBM Global ServicesCopyright IBM Global Services
ReferencesReferences• Ron Knode. Security Value Metrics – 2002. CSC Global Information Security Services.• http://www.csc.com/aboutus/lef/mds69_off/uploads/Enterprise_Info_Risk_Management.pdf• Paul W. Lowans Implementing a Network Security Metrics Program. GIAC Administrivia.
www.giac.org/practical/Paul_Lowans_GSEC.doc. • Dr. Stuart Katzke. Security Metrics. Computer System Security & Privacy Advisory Board. June 13-14, 2000.• James P. Craft. Metrics and the USAID Model Information Systems Security Program (MISSP).• Christina Kormos, Natalie Givens, Lisa A. Gallagher and Nadya Bartol. Using Security Metrics to Assess Risk Management
Capabilities.• Proceedings - Workshop on Information Security System Scoring and Ranking• Marianne Swanson, Nadya Bartol, John Sabato, Joan Hash, and Laurie Graffo Security Metrics Guide for Information Technology
Systems. NIST Special Publication 800-55. July 2003.• Shirley C. Payne. A Guide to Security Metrics, SANS Security Essentials GSEC Practical Assignment Version 1.2e. July 11, 2001• Workshop on Information-Security-System Rating and Ranking (WISSRR) - http://www.acsac.org/measurement/• Information Security Metrics. Using Foundstone’s FoundScoreTM to Assign Metrics and Measure Enterprise Risk.
www.foundstone.com. April 2003.• Proceedings. Workshop on Information Security System Scoring and Ranking Information System Security Attribute
Quantification or Ordering. May 21-23, 2001• C. Kormos, L. A. Gallagher, N. Givans & N. Bartol. Using Security Metrics to Assess Risk Management Capabilities• George Jelen. “SSE-CMM Security Metrics.” NIST & CSSPAB Workshop, Washington, D.C., 13-14 June
2000;http://csrc.nist.gov/csspab/june13-15/jelen.pdf July 2001.• Shirley Payne. A Guide to Security Metrics. Shirley C. Payne. SANS Security Essentials GSEC Practical Assignment. July 11,
2001• Eddie Schwartz, NetForensics Inc. Measuring Security. In Computerworld July 15, 2004.
http://www.computerworld.com/securitytopics/security/story/0,10801,94524,00.html• Steve Foster and Bob Pacl. Analysis of Return on Investment for Information Security. www.getronics.com• R. S. Kaplan and D. P. Norton in "Putting the Balanced Scorecard to Work“• Marc Stefaniu – Metrics & Executive Reporting. CFI-CIRT Presentation; March 2004