Joel Manfredo: Accelerating IT Mojo in the Face of Financial Distress

Joel A. ManfredoChief Technology Officer

October 12, 2011

1

Accelerating IT Mojo in the Face of Financial Distress

Background

Timeline

Phases 1 - 5

Results

Next Steps

Conclusions and Lessons Learned

Agenda

County of Orange

$5.6 Billion Budget

7 Programs

32 Agencies / Departments

CEO IT ServicesServices are delivered through an Internal Service Fund

“Running a Service Delivery Business”

Pricing a ServiceWhat does internet access cost?

Hardware:•Redundant Firewalls•Redundant Routers•Redundant Core Switches•IPS / IDS Hardware•Monitoring (Orion) Servers•Content Filtering Appliances•Content Filtering Report Servers•DNS Servers

Software:•Firewall Security Software licensing•Content Filtering Software licensing & Report Services•Orion Monitoring Software licensing•DNS Software (Unix and Wintel)

Data Center FacilitiesContracts:•Redundant Internet Service Providers:

1. TW Telecom2. VPLS (this is a fail-over service)

•IPS / IDS monitoring Service•Redundant WAN Circuit Contracts•Redundant Firewall Maintenance Contract (HW and SW)•Redundant Routers Maintenance Contract HW and SW)

•Redundant Core Switches Maintenance Contract HW and SW)•Content Filtering Maintenance Contract•County ARIN IP and BGP AS Services

Depreciation:•Firewalls - Hardware and software Depreciation associated •Routers - Hardware and software Depreciation associated •Core Switches - Hardware and software Depreciation associated •Monitoring - Hardware and software Depreciation associated •Content Filtering - Hardware and software Depreciation associated

Other CEOIT Services:•Contracts & Procurement Management•Resource Management•Financial and Cost, Budget Analysis •Billing Processing

Labor:•Network Engineering Labor•Security Engineering Labor•Availability Management Analysis•Capacity Management Analysis•Incident Management (Service Desk)•Change Management•Management

The Story…Operational Transformation in the Face of Financial Distress

Environment:

Organization: 1. A culture built on Heroics 2. Siloed organization 3. Assigning blame 4. Resistance to change 5. No priorities

Financial:

1. 27% budget reduction (2 years) 2. Expiring staff augmentation contract

Results:

1. Moved to top 10% against benchmark 2. Increased customer satisfaction 16% 3. Reduced SLA exceptions by 77%

How:

Combination: 1. Leadership 2. Soft skills introduction 3. Sustained “drumbeat” 4. The persuader 5. ITSM principles

Method: 1. Plan 2. Implement 3. Absorb 4. Measure

“Be transparent, or be gone”

Phases of the JourneyMethod - Plan, Implement, Absorb, Measure

Phase 1: ITSM Training April 2009DISC (Soft Skills) Training May 2009

Capacity Management May 2009Availability Management June 2009Resource Planning June 2009Project Prioritization June 2009

Phase 2: Network & Platform Services Reorganization October 2009 Service Design Package November 2009Monthly MAP meetings March 2010

Phase 4: Network Operations Center January 2011ITSM Full Suite (BMC – “AMP”) March 2011

Phase 3: Marketing Brochure (Mini Service Catalog) October 2010 Service Desk re-implementation December 2010

Phase 5: Re-benchmark AIMM July 2011 Service Catalog October 2011

ITSM Reorganization FutureService Level Management Future

Further Automate Capacity & Availability Future

Today

20092010

2010

Chaos

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

NPS Reorganized:-Plan & Design-Build & Transition-Operate & Maintain

ACS Program Executive hired

Availability Reporting

Capacity Reporting

(DISC)

Resource Planning

Project Prioritization

AIMM (HP) Report 46-Technology & Architecture-Mgt Tools & Process Maturity-Culture & IT Staff Maturity-Demand, Supply & IT Governance

“One Team”

Training

Ticket Aging Process

Service request turnaround improvements (100%)

Incidents – from over 5 days to 1

Not a single failed non-standard RFC

Not a single Service Level Breach

CTO Portal

Incident Mgt Process

Customer Survey (Jul – Dec 09)

Customer Survey (Jan – Jun 10)

Communication Team Formed

ITSM Integrated Suite Service Desk

County Connection: OCDC

Data Center Tour Tech Fest

Marketing Brochure “Agency Success is Our Mission”

IT Service Management Path to Maturity

AIMM (HP) Report 34

SDP Process

Certifications: 95

Driving Continuous Improvements

Performance Audit Report

“A journey of a thousand miles begins with a single step”

Phase 1

Phase 2

Phase 3

2011Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec


Network Operations Center Launched

ITSM Integrated Suite Launch – FULL SUITE (CMDB, Change, Release…)

MSB Planning Success

Virtual Server Farm Replacement

Method

Application Monitoring


Service Catalog

Service Level Management

ITSM based Reorganization


AIMM (HP) Report 10

Plan

Implement

Absorb

Measure

Phase 4 Phase 5

Phase 1

DISC AssessmentPhase 1: Dominance, Influence, Steadiness, Compliance

Nov Mar April May June Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jul Aug Sep0

50

100

150

200

250

300

350

400

Cumulative Monthly Accomplishments

Management Action PlanResults through Present

2010 2011

Capacity Report Availability Report

Capacity & Availability ReportingPhase 1

Capacity Report

Capacity & Availability ReportingPhase 1

Values

Row Labels ETC Hours 3 Week FTE

Enterprise Infrastructure Srvcs 13.92 0.12

Networks and Platform Services 523.87 4.37

Security & Bus Continuity Oper 15.50 0.13

Grand Total 553.28 4.61

Resource PlanningKey Metrics

3 Week Summary (by organization unit):

Resource PlanningKey Metrics

3 Week Summary (by project):

Projects with late tasks:

Row Labels Sum of ETC

Disaster Recovery Implementation (FY11-12) 336.00

Enterprise Secure File Transfer Migration 33.20

ePages Replacement 22.00

Exchange 2010 Production Build out 112.00

HR & FS Access Definition Project Schedule 1.50

Secure FTP Infrastructure Upgrade 48.58

Grand Total 553.28

Sum of ETC

Row Labels Grand Total

ePages Replacement 10.00

Grand Total 10.00

Phase 2

20092010

2010

Chaos





Capacity Reporting

(DISC)

Resource Planning



“One Team”

Training






CTO Portal





ITSM Integrated Suite Service Desk





AIMM (HP) Report 34

SDP Process

Certifications: 95




Phase 2

ASG IRM Process Security NPS OPS TSD PMO BA

Plan & Design

Build & Transition

Operate & Maintain

ITSM Organizational DesignPhase 2: Reorganization of NPS

Pla

n an

d D

esig

n

Overall Architecture and Technologies / Standards*

Capacity Requirements

Availability Requirements

Training Requirements

POC requirements

Design Diagrams and Requirements

MOUs/SLRs

Success Criteria

TCO Calculations

Alternatives

Backup requirements

Process / Procedure requirements

Monitoring requirements

Access Requirements

Standard Requests

Engineering and Ops review and acceptance

Continuous improvement or scalability plans

Approval of the Design from Peer Groups

Approval of the design from Chief Architect

Approval of the design from Customer

Bui

ld a

nd T

rans

ition Design Validation / Review and

Acceptance

Procurements

Configuration and Installation

Update and Audit of Asset Lists/CMDB

RFC Execution and Coordination/Communication

Rollout Plan

Backout Plan

Risk Assessment for RFC

Ops Training and Handoff

QA and Validation of Systems

Feedback to Design Team

Approval of Rollout from Ops

Approval of Rollout from customer

Approval of Rollout from Design group

Support and monitoring during cool down/warranty period O

pe

rate

an

d M

ain

tain

System acceptance

Registering application in ops (contact information, service levels, monitoring

and backup information, new or changed procedures, Access levels)

Update procedures

System lockdown

QA and Validation of Monitoring, Backups and Access

Assist in Negotiating SLRs/SLAs

Freeze and Warranty Period Management

ITSM Organizational Design

SDR SDP – Issues identified through the process

Phase 3

20092010

2010

Chaos





Capacity Reporting

(DISC)

Resource Planning



“One Team”

Training






CTO Portal





ITSM Integrated Suite – Service Desk





AIMM (HP) Report 34

SDP Process

Certifications: 95




Phase 3

It’s working...Phase 3: “Mojo”

November 2009 Not a single Service Level Breech Incidents – from over 5 days to 1 Not a single failed non-standard RFC Service request turnaround improvements

(100%)

- Marshall Goldsmith

Marketing BrochurePhase 3: “Mini Service Catalog”

Marketing BrochurePhase 3: “Mini Service Catalog”

Phase 4







Method



Service Catalog




AIMM (HP) Report 10

Plan

Implement

Absorb

Measure

Phase 4

Phase 4

Network Operations Center

Network Monitoring

Handoff/Review Logs

Orion Network Dashboard

Daily Walkthrough AM/PM

IP Monitor Int/Ext/DR

INMON

Server Monitoring

Handoff/Review Logs

Orion Server Dashboard

VM Console

IP Monitor Probation/Int

HP SIM

ResultsCustomWritten

Phase 4: BMC Suite through the Cloud

Integrated ITSM Suite

More Results

Results

Evidence of Improvements

Incident Tickets

Percent Resolved

0

100

200

300

400

500

600

700

800

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09Jul-0

9

Aug-09

Sep-09

Oct-09

Nov-09

Dec-09

Jan-10

Feb-10

Mar-10

Apr-10

May-10

Jun-10Jul-1

0

Aug-10

Sep-10

Oct-10

Nov-10

Dec-10

Jan-11

Feb-11

Mar-11

Apr-11

May-11

Jun-11Jul-1

1

Aug-11

Sep-11

50%55%60%65%70%75%80%85%90%95%

100%

Results


CY2009 CY20100

100

200

300

400

500

SLA Exceptions Reduced by 77%

Results


CY2009 CY20100

2

4

6

8

10

12

14

Average Days to Resolve Tickets Reduced 75%

Results: HP’s Converged Infrastructure Maturity Model





Technology & Architecture Tools & Processes Culture & IT Staff Maturity Demand, Supply & IT Governance

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Orange CountyUS Public Sector

Orange County versus US Public Sector BenchmarkJuly 2011


Evidence of ImprovementsResults: HP’s Converged Infrastructure Maturity Model

Dec-09 Jul-10 Jul-110

10

20

30

40

50

60

70

80

90

100

BetterEqual

Orange County Percent of Characteristics "At or Better"July 2011

Headcount

- 27%

Top 10%

Customer Survey Scale: 1 = worst; 4 = best


Phase 5







Method



Service Catalog




AIMM (HP) Report 10

Plan

Implement

Absorb

Measure

Phase 5

Service CatalogInternet Service Provider

SERVICE DESCRIPTIONCEO/IT is the ISP for the County and its agencies. This service includes reasonable and equitable Internet bandwidth and access capability to and from the County backbone to the Internet.

WHAT IS INCLUDED This includes the installation, configuration, ongoing

maintenance and security of the core routers that are outward facing to the Internet

Provides redundant Internet providers configured in failover configuration

64 Class C Networks that are available and managed by CEO/IT for existing outward facing applications

Availability of an Enterprise Class, Tier 2 Data Center Network Operations Center (NOC) (staffed hours: Mon-Fri,

8am-5pm) Monitoring of Enterprise Components and Hosts on a 24x7x365

basis Command Center staffed on a 24x7x365 basis Enterprise Software Licensing and Contracts Management

associated with this service Enterprise Services

HOW WE CHARGEThis service is included in the County Network Services Rate. CEO/IT utilizes the total email address count from each agency’s Active Directory (AD) as a divisor to calculate the cost distribution for CEO/IT Security Services, Enterprise Shared Services, and County Network Connectivity.

SERVICE TIPS Limit Internet usage to business related activities

ADDITIONAL INFORMATIONFor those who wish to filter their Internet browsing, we offer a secure web filtering solution. See Secure Web Filtering Services.

Extend ITSM Based OrganizationMove the Plan & Design Group and the Build & Transition Group “up”

Conclusions and Lessons Learned

Conclusions & Lessons Learned

1. Financial distress clouds have their silver linings in IT environments.a. Working together in the face of a single galvanizing eventb. Speak with “one voice”

2. It Takes an IT mojo-driven strategy to implement sustained continued improvements in financial distress.

a. Courage and consistency; buy-inb. Incremental changes (process, organizational), planning and soft skills provided results

3. New leadership can ignite IT Mojo. a. Common goalsb. Common messages

4. Establish a drumbeat to pace the changes and measure visibly and consistently.a. Messaging consistencyb. Allow the organization to digest changes

5. Establish key measurements of the categories shown in the customer surveys.a. Report on themb. Make sure everyone knows what’s being measured

6. Establish a consistent program of accountability .a. Report on themb. Make sure everyone knows what’s being measured

7. Benchmark the organization before commencing the journey.