Establishing Good Reporting Practices and Effective Management Dashboards

© 2016 ServiceNow All Rights ReservedConfidential © 2016 ServiceNow All Rights ReservedConfidential

Establishing Good Reporting Practices and Effective Management Dashboards

© 2016 ServiceNow All Rights Reserved 2Confidential

Seven Deadly Measurement Sins

Selection Collection Reporting Usage

An IT-Centric View of IT Performance: Lack of Senior Business involvement in measurement selection

Measures That Don’t Matter: No Link between metrics and Business / IT Strategy

No Common Ground: Lack of standard metrics definitions complicates aggregation

Overreliance on Tools: Reliance primarily of reports available in the existing tools

Lack of Drill-Down Capability and Metric Context

Too Many Metrics

No Individual Impact or Consequence

Source: CIO Working Council Research - www.executiveboard.com

1

3

4

5

6

7

2

Nature of Problem

Strategic

Tactical


Three Basic Service Measurements• Service Measurement is really about providing the results of the IT service as the

customer experiences the service. This would be an end-to-end experience.• For services, there are three basic measurements that most organizations utilize:

– Availability of the service– Reliability of the service– Performance of the service

Other Measures: Financial, Quality, Customer Satisfaction


CSR vs. KPI vs. Activity Metric• CSF – Critical Success Factor

– A key element of a service or process which must be in place to achieve the goal

• KPI – Key Performance Indicator– A specific measurable attribute related to a critical success factor

• Activity Metric– A general measure element of a process activity

Need to consider both Lead & Lag KPIs


Three Types of Metrics

These metrics are the results of the end-to-end service. Component metrics are used to compute the service metrics.

• Technology Metrics

• Process Metrics

• Service Metrics

These metrics are often associated with component and application-based metrics such as performance, availability, etc.

These metrics are captured in the form of CSFs, KPIs and activity metrics for the service management processes. These metrics can help determine the overall health of a process. Four key questions that KPIs can help answer are around quality, performance, value and compliance of following the process. CSI would use these metrics as input in identifying improvement opportunities for each process.


Example Incident Management CSF and KPICSFMaintain IT Service Quality

KPI #1Targeted Percentage Of Incidents Resolved Within Target Times By Priority

Why is it important and what it tells us about the operation?

Focused on Incidents in the proper order, improved escalation within the Incident Management process, and should have improved customer satisfaction.

How can it be measured and analyzed?

Reports against priority model and defined resolution times defined in SLAs.

The frequency for which it should be measured?

Weekly to start and then monthly. For critical customers it can be even more frequently.

How to present and distribute results?

Charts and graphs, distribute with analysis, what led to positive/negative results, and plan of action to take.

How the analysis of results can drive continuous improvement efforts

Identify potential weaknesses in priority model, escalations, and need for OLAs, or enforcement of SLAs. Could point to need to renegotiate SLAs.

How targets for performance can be set and revised

Negotiated with the customer for inclusion within the SLA. Reviewed as defined within the SLA.


IT Strategy and Cascading MeasuresCore Business Measures

Revenue

Market Share

Profit

ROE

IT Core Strategic MeasuresFinance

Innovate

InternalCustomer

Balanced Scorecard

Enterprise Operating Model

IT ProcessesITSM, Development, Project Mgmt., Security, Architecture, Operations, etc.

IT Value Chain (Service Organization)

IT Operating Model / COBIT

Business Value

IT Goals

IT CapabilityMeasures

IT ProcessMetrics


Balanced Scorecard Approach to Measure BenefitsFinancial Customer

Innovation Internal

Improve Quality of Service

Improve Management Control

Employ New Technology

Reduce Cost

Example

KPI Categories Value Quality Performance Compliance

In order to improve a Service or Service Management process, you must look at it from different perspectives


Process KPI Categories• Getting the whole picture:

– Value: Is what we are doing making a difference?– Quality: How well are we doing it?– Performance: How fast or slow are we doing it?– Compliance: Are we doing it?

A single measure may contain or cover more than one category. This, in itself is not an issue. However, understand that, when this occurs, the success criteria for this measure is more difficult to satisfy.


Choosing KPIs• Ideally, to measure a process, at least one KPI per category should be chosen

to provide a balanced perspective. However, due to the difficulty of measurement or tool limitation, a process management staff may find it necessary to limit what is measured according to what category is the most important to the objective of the process.

1. Establish the core objective of the process.2. Evaluate which category is of the highest priority to achieve the process objective.3. Define measures according to the categories which are appropriate to achieve the overall

process objective.


Building a Measurement Grid1. Define the measure.2. Determine the KPI category.3. Establish the policy and target. (Target will change with process maturity.)

4. Determine the tool or medium to realize the report.5. Define the output format (graph, data, etc.).6. Define distribution list and report frequency.

Category Measure Policy Target ToolValue, Quality # of incidents resolved

outside of SLAAll incidents to be resolved within SLA

90% of the time Incident Module


Example KPIs and CategoriesProcess Core Objective Example Core KPIs Category

Incident Restore service degradations to expected level ASAP.

# of Incidents by category, priority and resolution type by LOB.# of Incidents restored within SLA Targets.

ValuePerformance

Problem Identify systemic Infrastructure Errors and eliminate them to minimize impact and improve availability.

# of Problems identified and root cause determined with solution or workaround.# of Repeat Incidents by category trending downwards.

QualityValue

Change Handle changes efficiently while minimizing impact to service delivery.

# of Changes by type / category / group / customer. (emergency changes trending down).# of Changes that have resulting Incidents, or fail and have to be backed out% of changes submitted within lead times

QualityValuePerformance

Config. Identify / control / manage IT resources within a Configuration Management Database.

% of CMDB data population and accuracy vs. actual, according to scope.% Growth or Change by CI type over an elapsed time period.

ComplianceQuality

Release Ensure production readiness, quality and authorization of new or modified CIs and their planned deployment.

# of Releases by type that satisfy release management criteria when submitted to Change.

QualityCompliance


Example KPIs and CategoriesProcess Core Objective Example Core KPI Category

SLM Define servicesAgree on level, scope, quality, performanceMonitor and Manage

% Score of customer satisfaction survey trends up over time.% of SLA reviews that are completed within 5 days of agreement expiry

ValuePerformance

Avail. Define and plan for service availability to meet or exceed stated business requirements through process, technology and people resource planning and implementation.

% of service availability within SLA negotiated requirements. % of service design packages which contain availability management requirements

ValueCompliance

Capacity Current and future resources are greater than or equal to demand, but excess is planned.

% of components the breach tolerance thresholds in correspondence to planned capacity levels for components and complete IT systems.

Quality

ITSCM Recover IT systems to normal state in an alternate way after a disaster within an expected timeframe.

% of systems that fail recovery test.Time to execute test of plan and recover IT services in a contingency state against expected targets.

QualityPerformance

Finance Plan for and deliver IT Services within a forecasted budget against actual cost.

% deviation of forecasted vs. actual cost of IT services within defined tolerance limits.(% of Deviation $ of Deviation)

Compliance


Service Measurement ModelIT Scorecard

or Balanced Scorecard

Component 1 Measure

Rolled up Service Measurement Results

Key Performance

Indicators

Service Dashboard

Real time information

Point in time information

Component 2 Measure

Component 3 Measure

Service Scorecard


Service Management Model for Processes

Activity Measure One = number of urgent

changes

Activity Measure Two = number of failed

urgent changes

Activity Measure Three = Number of unauthorized failed

changes

Process Key Performance

Indicators

Process High Level Goal

Overall Service Management Process Scorecard

Example for Change Management = reduce the number of failed changes

Example = Improve Service Quality


Dashboard Key Performance IndicatorsProcess KPI/Description Type Progress Indicator

Incident Tickets Resolved within target time Value Meets / Exceeds Target Times

Incident % of Incidents Closed – First Call Performance Service Desk only – Target is 80%

Incident Abandon Rate Service Desk with ACD. 5% or less goal (after 24 seconds)

Incident Count of Incidents submitted bySupport Group

Compliance Consistency in number of incidents – investigation is warranted for 1) rapid increase which may indicate Infrastructure investigation, 2) rapid decrease which may indicate compliance issues.

Problem % of repeated Problems over time Quality Problems that have been removed from the infrastructure and have re-occurred. Target: Less than 1% over a 12 month rolling time frame.

Problem % Root Cause with permanent fix Quality Calculated from Problem Ticket . Start date to permanent fix found. This may not include implementation of permanent fix. Internal Target: 90% of Problems – within 40 days External Target: 80% of Problems – within 30 days Internal = BMO internal External = 3rd Party/vendor

Problem % and number of Incidents raised to Problem Management Compliance Sorted by Infrastructure (Internal and External ) and Development (Internal and External).

Change % of RFCs successfully implemented without back out or issues Quality Grouped by Infrastructure / Development

Change % of RFCs that are emergencies Performance Sort by Infrastructure or Development – and by Emergency Quick fix (service down) or Business requirement

SACM Number of CI additions or Updates Compliance Configuration Item Additions or updates broken down by group – CMDB / Change Modules

SACM Number of Records related to CI Performance Number of associations grouped by process

Release % of Releases using exceptions Value Exceptions are criteria deemed Mandatory – identify by groups

Release % of Releases by-passing process Compliance Identify groups by passing Release Process

Capacity Action Required Value Number of services that require action vs.Total number of systems

Capacity Capacity Related Problems Quality Number of Problems caused by Capacity issues sorted by group


Executive ITSM Dashboards


Incident Process Owner Dashboard


Service Desk Worker Dashboards


Executive Roll-up of Dashboards

Processes Health

Incident Target Times for Resolution Critical/High not met

Problem 50% fewer problem tickets raised – no correlation

with number of incident tickets.

Change No Concerns

Config Updates to CMDB to be broken down by group. Fewer Incidents associated.

Release Low compliance to process

KPIs were defined by and agreed to by the process owners.

In some cases, the new Service Management Tool was required to collect the metrics. The KPIs collected were updated in February. Data being collected is based on February/March Data.

SUMMARY: The chart on the right denotes status of KPI’s received and evaluated.

Follow Up / Investigate Status – Does not necessarily mean there is an issue.

No Concerns Follow Up /Investigate Target Times Exceed


CSI RegisterProcess Issue Improvement Activity Assignment Due Date

Incident

1) Updates to documentation 1) Formalise/document procedure enhancements 1) Process Owner April 30/03

Problem

1) Knowledge Database does not exist. 1) Create Knowledge Database 1) Process Owner 1) February 12No Revised date provided.

Change

1) Change Management Documentation. 2) Change Management – documentation under investigation3) MAC Process

1) Finalise documentation2) Being Updated3) Being Formalised

1) Process Owner 1) April 302) April 303) April 30

Config.

1) Lack of Associations between Logical and Physical CI’s

2) Post project CIs

1) 18 Critical systems identified – Pilot system is in database. Further design and population required.

2) Additional CI’s outside scope of project to be added

1) Process Owner 1) Jan 8

2) Not defined

Release

1) Compliance – to process2) Definitive Software Library (s)are not being updates/used. Evidence of updates to the DSL not evident.

1) Low Compliance through process – Implemented ‘RFC Requirements’2) Review/Investigate COE/Infra/Development DSL usage

1) Process Owner 1) March 302) March 30

CAPACITY

1) Area data collection procedure/method not documented

1) Each team to document and detail steps for data collection 1) Process Owner 1) April 30