Upload
james-sutter
View
695
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Joel A. Manfredo
June 14, 2012
1
Getting to a Service Catalog
Background
Timeline
Phases 1 - 5
Results
Next Steps
Conclusions and Lessons Learned
Agenda
County of Orange
$5.6 Billion Budget 7 Programs 32 Agencies / Departments
CEO IT Services
Services are delivered through an Internal Service Fund
“Running a Service Delivery Business”
Pricing a Service
What does internet access cost? Hardware: •Redundant Firewalls •Redundant Routers •Redundant Core Switches •IPS / IDS Hardware
•Monitoring (Orion) Servers •Content Filtering Appliances •Content Filtering Report Servers •DNS Servers
Software: •Firewall Security Software licensing
•Content Filtering Software licensing & Report Services
•Orion Monitoring Software licensing
•DNS Software (Unix and Wintel)
Data Center Facilities Contracts: •Redundant Internet Service Providers:
1.TW Telecom
2.VPLS (this is a fail-over service) •IPS / IDS monitoring Service
•Redundant WAN Circuit Contracts
•Redundant Firewall Maintenance Contract (HW and SW) •Redundant Routers Maintenance Contract HW and SW)
•Redundant Core Switches Maintenance Contract HW and SW) •Content Filtering Maintenance Contract •County ARIN IP and BGP AS Services
Depreciation: •Firewalls - Hardware and software Depreciation associated •Routers - Hardware and software Depreciation associated •Core Switches - Hardware and software Depreciation associated •Monitoring - Hardware and software Depreciation associated •Content Filtering - Hardware and software Depreciation associated
Other CEOIT Services: •Contracts & Procurement Management •Resource Management •Financial and Cost, Budget Analysis •Billing Processing
Labor: •Network Engineering Labor •Security Engineering Labor •Availability Management Analysis •Capacity Management Analysis •Incident Management (Service Desk) •Change Management •Management
Pricing a Service
What does internet access cost? Hardware: •Redundant Firewalls •Redundant Routers •Redundant Core Switches •IPS / IDS Hardware
•Monitoring (Orion) Servers •Content Filtering Appliances •Content Filtering Report Servers •DNS Servers
Software: •Firewall Security Software licensing
•Content Filtering Software licensing & Report Services
•Orion Monitoring Software licensing
•DNS Software (Unix and Wintel)
Data Center Facilities Contracts: •Redundant Internet Service Providers:
1.TW Telecom
2.VPLS (this is a fail-over service) •IPS / IDS monitoring Service
•Redundant WAN Circuit Contracts
•Redundant Firewall Maintenance Contract (HW and SW) •Redundant Routers Maintenance Contract HW and SW)
•Redundant Core Switches Maintenance Contract HW and SW) •Content Filtering Maintenance Contract •County ARIN IP and BGP AS Services
Depreciation: •Firewalls - Hardware and software Depreciation associated •Routers - Hardware and software Depreciation associated •Core Switches - Hardware and software Depreciation associated •Monitoring - Hardware and software Depreciation associated •Content Filtering - Hardware and software Depreciation associated
Other CEOIT Services: •Contracts & Procurement Management •Resource Management •Financial and Cost, Budget Analysis •Billing Processing
Labor: •Network Engineering Labor •Security Engineering Labor •Availability Management Analysis •Capacity Management Analysis •Incident Management (Service Desk) •Change Management •Management
The Story…
Operational Transformation in the Face of Financial Distress
Environment and Results:
Organization: 1. A culture built on Heroics 2. Bad relationship w/ vendor (expiring K) 3. Siloed organization 3. Assigning blame 4. Resistance to change 5. No priorities
Results:
1. Moved to top 10% against benchmark
2. Reduced Budget 27% 3. Increased customer satisfaction 16% 4. Reduced SLA exceptions by 77% 5. Reduced incidents by 38% 6. Improved incident resolution from 5 days to 2 days
How:
Combination: 1. Leadership 2. Soft skills introduction 3. Sustained “drumbeat” 4. The persuader 5. ITSM principles
Method: 1. Plan 2. Implement 3. Absorb 4. Measure
“Be transparent, or be gone”
Phases of the Journey
Method - Plan, Implement, Absorb, Measure
Phase 1: ITSM Training April 2009 DISC (Soft Skills) Training May 2009 Capacity Management May 2009 Availability Management June 2009 Resource Planning June 2009 Project Prioritization June 2009
Phase 2: Network & Platform Services Reorganization October 2009 Service Design Package November 2009 Monthly MAP meetings March 2010
Phase 4: Network Operations Center January 2011
ITSM Full Suite (BMC – “AMP”) March 2011
Phase 3: Marketing Brochure (Mini Service Catalog - BRAND) October 2010
Service Desk re-implementation December 2010
Phase 5: Re-benchmark AIMM July 2011
Service Catalog – Version 1 October 2011 Customer Ranking of Services January 2012 Service Catalog – Version 2 February 2012
ITSM Reorganization Future
Service Level Management Future
Further Automate Capacity & Availability Future
2009 2010
Chaos
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
NPS Reorganized: -Plan & Design -Build & Transition -Operate & Maintain
ACS Program Executive hired
Availability Reporting
Capacity Reporting
(DISC)
Resource Planning
Project Prioritization
AIMM (HP) Report 46 -Technology & Architecture -Mgt Tools & Process Maturity -Culture & IT Staff Maturity -Demand, Supply & IT Governance
“One Team”
Training
Ticket Aging Process
Service request turnaround improvements (100%)
Incidents – from over 5 days to 1
Not a single failed non-standard RFC
Not a single Service Level Breach
CTO Portal
Incident Mgt Process
Customer Survey (Jul – Dec 09)
Customer Survey (Jan – Jun 10)
Communication Team Formed
ITSM Integrated Suite Service Desk
County Connection: OCDC
Data Center Tour Tech Fest
Marketing Brochure “Agency Success is
Our Mission”
IT Service Management Path to Maturity
AIMM (HP) Report 34
SDP Process
Certifications: 95
Driving Continuous Improvements
Performance Audit Report
“A journey of a thousand miles begins with a single step”
Phase 1
Phase 2
Phase 3
2011 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Network Operations Center Launched
ITSM Integrated Suite Launch – FULL SUITE (CMDB, Change, Release…)
MSB Planning Success
Virtual Server Farm Replacement
Method
Application Monitoring
Driving Continuous Improvements
Service Level Management
ITSM based Reorganization
Customer Survey (Jul – Dec 10)
AIMM (HP) Report 10
2012 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
IT Service Management Path to Maturity
Service Catalog v 1 Service Catalog v 2
Customer Grading by
Service
Phase 4 Phase 5
Phase 1
DISC Assessment
Phase 1: Dominance, Influence, Steadiness, Compliance
0
50
100
150
200
250
300
350
400
Nov Mar April May June Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jul Aug Sep
Cumulative Monthly Accomplishments
Management Action Plan
Results through Present
2010 2011
Table of Contents Executive Summary............................................................................................................ 4
Dashboard Summary........................................................................................................... 4
Risks.................................................................................................................................... 4
Warnings ............................................................................................................................. 4
Scope................................................................................................................................... 5
Capacity Measurements ...................................................................................................... 5
Network .......................................................................................................................... 5
County Wide Area Network ....................................................................................... 5
Time Warner Internet.................................................................................................. 6
Security ........................................................................................................................... 7
Storage ............................................................................................................................ 8
Tier-1........................................................................................................................... 8
Tier-2........................................................................................................................... 8
E-Mail ........................................................................................................................... 10
Virtual Environment ..................................................................................................... 13
Analysis Additional Capacity ................................................................................... 14
Conclusion ................................................................................................................ 14
Recommendation ...................................................................................................... 14
Power5 Systems............................................................................................................ 15
Mainframe..................................................................................................................... 18
Power ............................................................................................................................ 19
Data Center Space......................................................................................................... 22
Floor Space ............................................................................................................... 22
Office Space.............................................................................................................. 24
Telephone Services ....................................................................................................... 25
APPENDIX – A – Report Data Source and Contributors ................................................ 26
APPENDIX – B – Virtual Environment Cluster Information .......................................... 27
Capacity Report Availability Report
Capacity & Availability Reporting
Phase 1
Capacity Report
Capacity & Availability Reporting
Phase 1
Values
Row Labels ETC Hours 3 Week FTE
Enterprise Infrastructure Srvcs 13.92 0.12
Networks and Platform Services 523.87 4.37
Security & Bus Continuity Oper 15.50 0.13
Grand Total 553.28 4.61
Resource Planning
Key Metrics
3 Week Summary (by organization unit):
Resource Planning
Key Metrics
3 Week Summary (by project):
Projects with late tasks:
Row Labels Sum of ETC
Disaster Recovery Implementation (FY11-12) 336.00
Enterprise Secure File Transfer Migration 33.20
ePages Replacement 22.00
Exchange 2010 Production Build out 112.00
HR & FS Access Definition Project Schedule 1.50
Secure FTP Infrastructure Upgrade 48.58
Grand Total 553.28
Sum of ETC
Row Labels Grand Total
ePages Replacement 10.00
Grand Total 10.00
Phase 2
2009 2010
Chaos
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
NPS Reorganized: -Plan & Design -Build & Transition -Operate & Maintain
ACS Program Executive hired
Availability Reporting
Capacity Reporting
(DISC)
Resource Planning
Project Prioritization
AIMM (HP) Report 46 -Technology & Architecture -Mgt Tools & Process Maturity -Culture & IT Staff Maturity -Demand, Supply & IT Governance
“One Team”
Training
Ticket Aging Process
Service request turnaround improvements (100%)
Incidents – from over 5 days to 1
Not a single failed non-standard RFC
Not a single Service Level Breach
CTO Portal
Incident Mgt Process
Customer Survey (Jul – Dec 09)
Customer Survey (Jan – Jun 10)
Communication Team Formed
ITSM Integrated Suite Service Desk
County Connection: OCDC
Data Center Tour Tech Fest
Marketing Brochure “Agency Success is
Our Mission”
IT Service Management Path to Maturity
AIMM (HP) Report 34
SDP Process
Certifications: 95
Driving Continuous Improvements
Performance Audit Report
“A journey of a thousand miles begins with a single step”
Phase 2
ASG
IRM
Process
Security
NPS
OPS
TSD
PMO
BA
Plan & Design
Build & Transition
Operate & Maintain
ITSM Organizational Design
Phase 2: Reorganization of NPS
Pla
n a
nd D
esig
n
Overall Architecture and Technologies / Standards*
Capacity Requirements
Availability Requirements
Training Requirements
POC requirements
Design Diagrams and Requirements
MOUs/SLRs
Success Criteria
TCO Calculations
Alternatives
Backup requirements
Process / Procedure requirements
Monitoring requirements
Access Requirements
Standard Requests
Engineering and Ops review and acceptance
Continuous improvement or scalability plans
Approval of the Design from Peer Groups
Approval of the design from Chief Architect
Approval of the design from Customer
Bu
ild a
nd T
ransitio
n
Design Validation / Review and Acceptance
Procurements
Configuration and Installation
Update and Audit of Asset Lists/CMDB
RFC Execution and Coordination/Communication
Rollout Plan
Backout Plan
Risk Assessment for RFC
Ops Training and Handoff
QA and Validation of Systems
Feedback to Design Team
Approval of Rollout from Ops
Approval of Rollout from customer
Approval of Rollout from Design group
Support and monitoring during cool down/warranty period
Op
era
te a
nd
Ma
inta
in
System acceptance
Registering application in ops (contact information, service levels, monitoring and backup information, new or changed procedures, Access
levels)
Update procedures
System lockdown
QA and Validation of Monitoring, Backups and Access
Assist in Negotiating SLRs/SLAs
Freeze and Warranty Period Management
ITSM Organizational Design
SDR SDP – Issues identified through the process
Phase 3
2009 2010
Chaos
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
NPS Reorganized: -Plan & Design -Build & Transition -Operate & Maintain
ACS Program Executive hired
Availability Reporting
Capacity Reporting
(DISC)
Resource Planning
Project Prioritization
AIMM (HP) Report 46 -Technology & Architecture -Mgt Tools & Process Maturity -Culture & IT Staff Maturity -Demand, Supply & IT Governance
“One Team”
Training
Ticket Aging Process
Service request turnaround improvements (100%)
Incidents – from over 5 days to 1
Not a single failed non-standard RFC
Not a single Service Level Breach
CTO Portal
Incident Mgt Process
Customer Survey (Jul – Dec 09)
Customer Survey (Jan – Jun 10)
Communication Team Formed
ITSM Integrated Suite – Service Desk
County Connection: OCDC
Data Center Tour Tech Fest
Marketing Brochure “Agency Success is
Our Mission”
IT Service Management Path to Maturity
AIMM (HP) Report 34
SDP Process
Certifications: 95
Driving Continuous Improvements
Performance Audit Report
“A journey of a thousand miles begins with a single step”
Phase 3
It’s working...
Phase 3: “Mojo”
November 2009
Not a single Service Level Breech
Incidents – from over 5 days to 1
Not a single failed non-standard RFC
Service request turnaround improvements (100%)
- Marshall Goldsmith
Marketing Brochure
Phase 3: “Mini Service Catalog”
Marketing Brochure
Phase 3: “Mini Service Catalog”
Phase 4
Phase 4
Network Operations Center
Network Monitoring
Handoff/Review Logs
Orion Network Dashboard
Daily Walkthrough AM/PM
IP Monitor Int/Ext/DR
INMON
Server Monitoring
Handoff/Review Logs
Orion Server Dashboard
VM Console
IP Monitor Probation/Int
HP SIM
Results Custom Written
Phase 4: BMC Suite through the Cloud
Integrated ITSM Suite
More Results
Results
Evidence of Improvements
Incident Tickets
Percent Resolved
0
100
200
300
400
500
600
700
800
50%
60%
70%
80%
90%
100%
Jan
-09
Feb
-09
Mar
-09
Ap
r-0
9
May
-09
Jun
-09
Jul-
09
Au
g-0
9
Sep
-09
Oct
-09
No
v-0
9
De
c-0
9
Jan
-10
Feb
-10
Mar
-10
Ap
r-1
0
May
-10
Jun
-10
Jul-
10
Au
g-1
0
Sep
-10
Oct
-10
No
v-1
0
De
c-1
0
Jan
-11
Feb
-11
Mar
-11
Ap
r-1
1
May
-11
Jun
-11
Jul-
11
Au
g-1
1
Sep
-11
Results
Evidence of Improvements
0
100
200
300
400
500
CY2009 CY2010
SLA Exceptions Reduced by 77%
Results
Evidence of Improvements
0
2
4
6
8
10
12
14
CY2009 CY2010
Average Days to Resolve Tickets Reduced 75%
Results: HP’s Converged Infrastructure Maturity Model
Evidence of Improvements
Results: HP’s Converged Infrastructure Maturity Model
Evidence of Improvements
Evidence of Improvements
1.00
1.50
2.00
2.50
3.00
3.50
4.00
Technology &Architecture
Tools & Processes Culture & IT StaffMaturity
Demand, Supply & ITGovernance
Orange County
US Public Sector
Orange County versus US Public Sector Benchmark July 2011
Results: HP’s Converged Infrastructure Maturity Model
Evidence of Improvements
Results: HP’s Converged Infrastructure Maturity Model
0
10
20
30
40
50
60
70
80
90
100
Dec-09 Jul-10 Jul-11
Better
Equal
Orange County Percent of Characteristics "At or Better" July 2011
Head
count
- 27%
Top 10%
Customer Survey Scale: 1 = worst; 4 = best
Evidence of Improvements
Phase 5
2011 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Network Operations Center Launched
ITSM Integrated Suite Launch – FULL SUITE (CMDB, Change, Release…)
MSB Planning Success
Virtual Server Farm Replacement
Method
Application Monitoring
Driving Continuous Improvements
Service Level Management
ITSM based Reorganization
Customer Survey (Jul – Dec 10)
AIMM (HP) Report 10
2012 Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
IT Service Management Path to Maturity
Service Catalog v 1 Service Catalog v 2
Customer Grading by
Service
Phase 5
Sorted Top to Bottom (and vice-versa)
Sorted by: Customer
Ave
1 Enterprise Time Synchronization 96.50
2 Midrange Platform (AIX Platform) 96.50
3 Mainframe Platform 96.00
4 Enterprise Network Connectivity (WAN) 94.71
5 Enterprise Secure File Transfer Services 94.20
6 Enterprise RightFax Services 94.00
7 SLA1 – Unmanaged Server Hosting 94.00
8 Database Mgt – Information Resource Mgt 93.67
9 Enterprise Domain Name Resolution 93.20
10 Blackberry Services 92.67
11 OnBase 92.60
12 VTL Replication Service 91.67
13 Standard Backup Service 91.50
14 Enterprise Email Services 91.33
15 Finance Services 91.33
16 Enterprise Storage Services (SAN) 91.25
17 Enterprise Internet Service Provider (ISP) 90.86
18 Technical Consultation 90.80
19 Applications Services Group 90.50
20 Desktop Support 90.00
21 Enterprise Global User Directory Services 88.67
22 Publishing Services 88.67
23 Enterprise Network Security 88.50
24 DeltaCopy Data Protection Servce (DCDP) 88.33
25 Business Recovery Center 88.33
26 SLA3 Service 87.80
27 Enterprise Remote Access Infrastructure 86.86
28 OCTNET 86.39
29 Project Management 85.00
30 Service Desk 84.78
31 eGov Services 83.33
32 Enterprise Secure Web Filtering 82.50
33 Disaster Recovery 66.50
Sorted by: Customer
Ave
1 Disaster Recovery 66.50
2 Enterprise Secure Web Filtering 82.50
3 eGov Services 83.33
4 Service Desk 84.78
5 Project Management 85.00
6 OCTNET 86.39
7 Enterprise Remote Access Infrastructure 86.86
8 SLA3 Service 87.80
9 DeltaCopy Data Protection Servce (DCDP) 88.33
10 Business Recovery Center 88.33
11 Enterprise Network Security 88.50
12 Enterprise Global User Directory Services 88.67
13 Publishing Services 88.67
14 Desktop Support 90.00
15 Applications Services Group 90.50
16 Technical Consultation 90.80
17 Enterprise Internet Service Provider (ISP) 90.86
18 Enterprise Storage Services (SAN) 91.25
19 Enterprise Email Services 91.33
20 Finance Services 91.33
21 Standard Backup Service 91.50
22 VTL Replication Service 91.67
23 OnBase 92.60
24 Blackberry Services 92.67
25 Enterprise Domain Name Resolution 93.20
26 Database Mgt – Information Resource Mgt 93.67
27 Enterprise RightFax Services 94.00
28 SLA1 – Unmanaged Server Hosting 94.00
29 Enterprise Secure File Transfer Services 94.20
30 Enterprise Network Connectivity (WAN) 94.71
31 Mainframe Platform 96.00
32 Enterprise Time Synchronization 96.50
33 Midrange Platform (AIX Platform) 96.50
A
B
Service Catalog
ISP
Service Catalog
Storage Area Network
Extend ITSM Based Organization
Move the Plan & Design Group and the Build & Transition Group “up”
Conclusions and Lessons Learned
Conclusions & Lessons Learned
1. Financial distress clouds have their silver linings in IT environments. a. Working together in the face of a single galvanizing event
b. Speak with “one voice”
2. It Takes an IT mojo-driven strategy to implement sustained continued improvements in
financial distress. a. Courage and consistency; buy-in
b. Incremental changes (process, organizational), planning and soft skills provided results
3. New leadership can ignite IT Mojo. a. Common goals
b. Common messages
4. Establish a drumbeat to pace the changes and measure visibly and consistently. a. Messaging consistency
b. Allow the organization to digest changes
5. Establish key measurements of the categories shown in the customer surveys. a. Report on them
b. Make sure everyone knows what’s being measured
6. Establish a consistent program of accountability . a. Report on them
b. Make sure everyone knows what’s being measured
7. Benchmark the organization before commencing the journey.
Questions and Answers