STATE OF CALIFORNIA Budget C h a n g e Proposal - Cover Sheet DF-46 (REV 07/17)

Fiscal Year Business Unit Department Priority No. ^ 1 8 - 1 9 2660 Transportation

Priority No.

Budget Request Name Program Subprogram 2660-013-BCP-2018-GB 9900100-Administration

-

Budget Request Description Information Technology Infrastructure Replacement

Budget Request Summary The California Department of Transportation requests State Highway Account Funds to continue replacement of the most critical information technology infrastructure equipment that has reached its end of life.

This request is part of a California State Transportation Agency sponsored program to reform, improve and enhance performance in many areas of Caltrans Information Technology identified injecent assessments.

^ H i q u i r e s Legislation

• Yes 1^ No

Code Section(s) to be Added/Amended/Repealed

Does this BCP contain information technology (IT) components? ^ Yes D No

If yes, departmental Chief Information Officer must sign.

Department CIO GEORGE AKIYAMA

Date 9/1/2017

For IT requests, specify the project number, the most recent project approval document (FSR, SPR, S1BA, S2AA, S3SD, S4PRA), and the approval date.

Project No. Project Approval Document: Approval Date:

If proposal affects another department, does other department concur with proposal? O Yes • No Attach comments of affected department, signed and dated by the department director or designee.

Prepared By ^•^^^^•"^^•^^^^ / ^MSEORGE A K l Y A f M ^ f\

Da"te~~~ 9/1/2017

Reviewed STEVEN KECK ^

• Date 9/1/2017

Department Director l \)LyfN MALCOLM DOUGHERjTY

Date 9/1/2017

Agency Secretary ^ //If BRIAN P. KELLY / y C / b ^ ̂ 9/^/2017 (Z/d^ll

H Department of Finance Use Only

Additional Review: • Capital Outlay • ITCU • FSCU • OSAE • CALSTARS • Dept. of Technology

^ P B A Original Signed By: Steve Wells

/

Date submitted to the Legislature

Analysis of Problem

A. Budget Request Summary

The California Department of Transportation (Caltrans) requests a one-time increase of S2 million in State Highway Account (SHA) funds to develop an Information Technology Roadmap. These resources are contingent upon a contract to be managed by the California State Transportation Agency (CalSTA) and approved by the California Department of Technology (CDT) and the Department of Finance (DOF) prior to execution to ensure the scope of the work meet the requirements and deliverables are agreed upon.

Additionally, upon approval of the Information Technology Roadmap by the DOF and the CDT, Caltrans requests a one-time increase of up to $12 million in SHA funds to continue replacement of information technology (IT) infrastructure equipment that has reached its end of life, has been identified as most important to ensure continuity of business operations and is in line with the approved Technology Roadmap.

This request will enable Caltrans to address end of life equipment and stabilize the management of IT infrastructure equipment per Caltrans' Information Technology Roadmap. This request will enable Caltrans to address the infrastructural refresh priorities that present high risk of failure and support the strategies defined in the Information Technology Roadmap.

This request is part of Caltrans' ongoing effort to reform its old business models and implement improvements of IT system performance.

B. Background/History

The IT infrastructure required to support Caltrans various business missions is complex and currently supports over 700 locations across the state. Caltrans IT infrastructure was built and maintained by Caltrans staff in accordance with the approved Fiscal Year (FY) 1997-98 Finance Letter for the Network Infrastructure Project. Caltrans was given $21 million in one-time funding for implementing the IT infrastructure, and $5.8 million for ongoing maintenance and operations. The project was successfully implemented on time and on budget. Since that time, Caltrans IT infrastructure has had significant growth to support the demands of business operations.

The diagram in Attachment I illustrates funding for Caltrans technical infrastructure since the initial FY 1997-98 Spring Finance Letter.

Caltrans manages more than 50,000 miles of California's highway and freeway lanes, provides inter­city rail services, issues permits for over-sized vehicles to use the highway system, inspects and maintains bridge structures and culverts, and permits more than 400 public-use airports and special-use hospital heliports. The department also works with local agencies to address highway congestion and incidents and collaborate on new modes of transportation and highway management systems.

In California's transportation system, IT infrastructure (servers, storage appliances, network devices) supports Caltrans critical operations including: management of freeway traffic, ramps, lanes and lights, and maintenance of highways and bridges. The same infrastructure informs the public of road hazards, changes to driving conditions, and warnings about congestion on the roadway with alternative routing suggestions via changeable traffic signs. In the 12 Transportation Management Centers, operated in conjunction with the California Highway Patrol, the Caltrans network enables management and control of over 18,000 individual elements in the highway infrastructure, including traffic sensors, cameras, message signs, inter-section signal controls, freeway ramps, water control systems, weather systems and highway advisory radios that are used to manage traffic overloads and incidents that can impede efficient movement on the highways. Additionally, the IT infrastructure provides Caltrans employees with essential IT services, such as user access to applications, data, file and print services, electronic mail and other network based resources required to perform their daily jobs.

1

Analysis of Problem

In FY 2017-18, Caltrans received a one-time augmentation of S12 million to address the most critical end of life infrastructure devices and the replacement program is on schedule to be completed by the end of June, 2018. Upon approval of the Information Technology Roadmap, this FY 2018-19 request v/ill continue replacement for an additional year. This funding will replace end of life devices that have been deferred for replacement due to lack of funding. Attachment II provides Caltrans IT Infrastructure Inventory information addressed in this request.

Resource History- Information Technology (Dollars in thousands)

Program Budget FY 2012-13 FY 2013-14 FY 2014-15 FY 2015-16 FY 2016-17 Authorized Expenditures S 89,202 S 93,318 S 95,692 S 96,363 S 98,165 Actual Expenditures S 95,819 S 95,143 S 100,190 S 95,231 $ 96,910 Revenues 0 0 0 0 0 Authorized Positions 539 544 544 545 543 Filled Positions 535 539 523 503 514 Vacancies 4 5 21 42 29

Resource History - Administration Program (Dollars in thousands)

Program Budget FY 2012-13 FY 2013-14 FY 2014-15 FY 2015-16 FY 2016-17 Authorized Expenditures $425,989 $434,126 $446,946 $481,572 $330,847 Actual Expenditures $423,935 $432,277 $445,952 $479,307 $329,209 Revenues 0 0 0 0 0 Authorized Positions 1,575 1,571 1,560 1,567 1,567 Filled Positions 1,537 1,527.5 1,514 1,489 1,499 Vacancies 38 44 46 78 68

Workload History (Hours)

Workload Measure FY 2012-13 FY 2013-14 FY 2014-15 FY 2015-16 FY 2016-17 NetA'ork Device Maintenance and Support 5,265 5,265 5,265 5,265 5,265 Server Hardware Maintenance and Support 3,510 3,510 3,510 3,510 3,510 Storage Appliance Maintenance and Support 1,755 1,755 1,755 1,755 1,755

State Level Considerations

The refresh of the Caltrans IT infrastructure ensures a secure and sustainable platform for the department's mission critical services. This request supports the following Caltrans Strategic Management Plan Goals:

• Safety and Health - "Provide a safe transportation system for workers and users..." by replacing end of life IT infrastructure equipment that directly supports the transportation management system before it fails which ensures continuity of operations for managing and ensuring efficient and safe highway travel for the public.

• Stewardship and Efficiency - "Money counts. Responsibly manage California's transportation related assets." This request proposes the systematic replacement of end of life IT infrastructure equipment i per industry standard lifecycles to prevent costly extended warranty service contracts that may not fully' ensure continuity of operations due to the age of some of the IT infrastructure equipment. In summary.

Analysis of Problem

Caltrans should regularly invest in predetermined cycles to lower the risk to continuity of operations and remove the need for costly end of life equipment extended maintenance contracts.

• System Performance - "...to develop an integrated transportation system that provides reliable and accessible mobility for travelers." The regular, predetermined replacement of end of life IT infrastructure equipment supports the effective and efficient management of California's transportation system by ensuring continuity of operations through reliable and flexible network communications and secure and sustainable server/data storage systems infrastructure.

Deferred IT infrastructure hardware refresh have created a significant risk for California's transportation system. Caltrans has developed an Information Technology Asset Management database and its implementation enables timely and accurate management of IT assets from acquisition to retirement with a renewed emphasis on IT asset management to enhance its' ability to avoid IT equipment failure and disruptions to business operations. The IT infrastructure replacement program provides the opportunity to plan for integration with Intelligent Transportation Systems which are advanced applications that aim to provide innovative services relating to different modes of transport and traffic management.

The Caltrans IT infrastructure is critical to Caltrans mission and daily operations as it provides a means to manage traffic on California highways, has the capability to expand as necessary to support new modes of transportation and meet enhanced safety standards, and to provide safety for the traveling public and Caltrans workers, as well as protecting the security and operability of Caltrans transportation management systems.

Justification

Caltrans is becoming increasingly dependent every year on its information and technology assets, which are becoming more complex, more interconnected, and increasingly exposed to more sophisticated cyber security threats that could destabilize Caltrans and California's transportation system.

The demand for efficient operations and evolving technology based solutions (e.g., autonomous vehicles, and a new generation of traffic sensors) has increased from business operations and the public over the years, and funding has not been able to keep up with this demand. A recent review of Caltrans' business processes and use of technology found that: "...the increasing pace of technical hardware incorporated into Caltrans project portfolios is accelerating and does not have a projected plateau...it is expected that more technology will replace or enhance even more technology, for the near future, at Caltrans. This reality is occurring at all technology scales as well, from individual highway sensors and elements on through to the largest constructions projects. One popular leading driver of this increasing wave of technology is the developing autonomous vehicle technology...technology deployments will grow at Caltrans to solve traffic problems that jeopardize the safety, health, sustainability and livability of Califomians and other travelers through Califomia." (Business Intelligence Strategy & Roadmap - April 12, 2017, Trinity Technology Group) A modern, resilient IT infrastructure is critical to Caltrans mission and daily operations because it provides the underlying technical foundation to manage traffic on California highways and must have the capability to expand and evolve as necessary to support new modes of transportation and meet enhanced safety standards for the traveling public.

In Pif 2017-18, $12 million was approved to replace the oldest and most critical infrastructure equipment. This was an important step toward the full remediation of the backlog of aging critical IT infrastructure. This FY 2018-19 request is the next step to developing an Information Technology Roadmap and incorporating a plan for the replacement of the aging infrastructure in an inaemental one year at a time basis that enables existing staff. Caltrans IT staff continues to follow the established Asset Management and Life Cycle Management Technology Practices to identify and manage IT infrastructure replacement over the equipment's lifecycle that will continue to address the problem in manageable one year increments, without requesting additional positions.

3

Analysis of Problem

System failure due to end of life or past end of life infrastructure equipment that has no cost effective maintenance/support plan v/ill affect business operations and the safety and viability of California's transportation system. Several recent examples of end of life equipment failures include:

1. In May 2017, the Transportation Management Center in District 3 had a network outage which affected traffic operations for the Sacramento Region. The outage affected all network devices in the Transportation Management Center (TMC) and traffic cameras and traffic monitoring systems. The equipment failure caused intermittent outages of 1 to 3 hours over a 4 day period. The network equipment v/as approximately 10 years old which had critical systems plugged into it including workstations in the TMC. The network equipment was end of life and end of support. The system was replaced with spare inventory that was also end of life.

2. Also'in May 2017, the Caltrans District Office in Los Angeles lost a supervisor network line card in one of the core network server switches. The supervisor line card is the 'brains' and Central Processing Unit of the network switch. These line cards are approximately 8+ years in age and are end of life and end of support. When the line card failed, it brought down connectivity to an entire server environment which housed applications for various construction, maintenance and fleet applications. The outage lasted approximately 2 hours. The line card was replaced by pulling the redundant card from another functioning core network switch v/hile personnel sought a replacement card.

3. In early 2017 at the Caltrans Farmers Market campus a network core switch supervisor line card also failed due to age of the equipment component. The Farmers Market complex houses the Department of Transportation's Engineering Services, Accounting and Human Resources departments with approximately 1,300 personnel. This outage affected all 1,300 personnel and lasted approximately 4 hours. The line card was replaced by pulling the redundant card from another functioning core network switch while personnel sought a replacement card.

4. In March of 2017 in one of Caltrans Sacramento satellite offices which house the Permits Section, Civil Rights, and Equal Employment Opportunity departments, Caltrans lost another network s w i t c h ^ ^ chassis line card due to age. This affected approximately 50 users and lasted for approximately 2 hours. This line card was replaced v/ith a spare line card which was also end of life and end of support.

5. Recent failures of end of life and end of support wireless access points in Caltrans District 5 San Luis Obispo Midway office has left that office without wireless connectivity for approximately 200 users. This office houses mainly engineers and transportation analysts for the San Luis Obispo region. The short term fix v/as to replace the non-functioning system with an older spare system; the outage lasted approximately 3 days. These access points will be replaced during the FY 2017-18 IT Infrastructure Refresh project.

6. The Lane Closure System is used throughout the state to notify the motoring public about lane closures and provide alternative route information. This system operates on a server at the State Data Center. The main electronic component that enables the server to function failed and a replacement had to be sent and replaced by a vendor field engineer. The system was out of operation for 3 Vi days.

In addition, end of life equipment that cannot be upgraded with the latest software/security patches significantly increases the risk of interruption to business operations and/or the release of sensitive data due to the intrusion by un-authorized agents. Caltrans must keep up with advanced security threats with a modern infrastructure capable of integrating new security management solutions. Without the ability to maintain the IT security system, Caltrans is exposed to the risk of security attacks that may bring business activities to a halt.

In summary, the information processing and information assets used at Caltrans are an integral part ol^jj^ the State bridge and roadway critical infrastructure to ensure effective and efficient maintenance and operations. System failures and data information errors can, and have, resulted in health and safety

4

Analysis of Problem

issues including loss of life. IT systems and information itself is playing an increasingly vital role in the effective and safe operation of the roadways and technology-empov/ered vehicles.

Outcomes and Accountability

This request will enable Caltrans to have an Information Technology Roadmap, approved by DOF and CDT, to replace its end-of-life infrastructure. Caltrans has prioritized the replacements to mitigate the risk of failure for the most critical and aged devices included in the current backlog of end of life equipment. Caltrans will develop an Information Technology Roadmap that will lay out the approach for the replacement program which will enable the use of existing staff resources from headquarters and the twelve district offices throughout the State to replace outdated devices while maintaining operations of the network, server and storage infrastructure.

Caltrans IT has established and implemented an IT Infrastructure Lifecycle Management Program v/hich ensures the regular identification and replacement of end of life equipment based on reasonable, industry standard replacement cycles. This program is a proactive approach in managing the physical components of computers, storage, and networks from acquisition to disposal including asset inventory, purchase date, asset cost, asset life expectancy, etc., to manage equipment's useful life, to lower total cost of ownership, and avoid equipment and IT system failures that can severely disrupt business operations and public service. This IT Infrastructure Lifecycle Management Program will be incorporated into the IT Infrastructure Roadmap to be approved by CDT and DOF.

The Information Technology Roadmap will allow Caltrans to:

1. Avoid high costs of maintaining and operating end of life equipment;

2. Reduce IT infrastructure hardware failure;

3. Improve capability to protect information from cybersecurity threats and vulnerabilities;

4. Adhere to the Department's Lifecycle Management standards by replacing aging IT infrastructure equipment v/ith up-to-date technologies;

5. Maintain a consistent v/orkload and predictable budget by funding and implementing the incremental replacement of equipment on an ongoing basis.

Industry standard project management tools and techniques are being used to manage the replacement of end of life IT infrastructure funded through the FY 2017-18 BCP process. These tools and techniques will continue to be used to plan and manage the replacement program throughout its life.

These tools and techniques include:

• Develop a project plan and closely manage the project activities, scope, budget, schedule, resources, risks and issues throughout the replacement cycle;

• Use of typical project artifacts, e.g., project scope document, issue log, risk log, budget tracking log, communication plan, and procurement management plan, to proactively manage staff activities during execution of the project;

• Communication with Caltrans procurement managers and preparation of procurement documents in advance of the start of the fiscal year to ensure timely processing of replacement equipment; and

• Regular meetings with and status reporting from project team members to ensure timely and accurate reporting of project activities.

5

Analysis of Problem

Projected Workload Summary* (Hours)

Workload Measure FY 2017-18** FY 2018-19 Network Device Maintenance and

Support 15,642 7,560

Server Hardware Maintenance and Support 5,292 2,478

Storage Appliance Maintenance and Support 6,880 4,000

• The additional •norkload for replacement of IT infrastructure devices wilt be addressed through an internal redirection of resources ** Workload from FY 17-18 Infrastructure Replacement BCP

Attachment ill provides details for the Projected Workload Summary.

Caltrans mission is to provide a safe, sustainable, integrated, and efficient transportation system to enhance California's economy and livability. By leveraging existing staff and right-sizing the scope and approach, based on available resources and budget, to prioritize and replace the most critical aging devices, Caltrans is proactively avoiding system failure, decreasing the risk of disruptions to business operations and the traveling public, and modernizing the underlying IT infrastructure to enable use of new technologies by Caltrans business programs, while also ensuring maintenance worker safety and the security of Caltrans transportation management systems.

F. Analysis of All Feasible Alternatives

Alternative 1: Approve: (1) a one-time increase of up to $2 million in SHA funds, contingent upon DOF and CDT approval, to develop an Information Technology Roadmap; and (2) upon DOF and CDT approval of the Information Technology Roadmap, provide a one-time increase of up to $12 million in SHA funds to replace end of life IT infrastructure equipment in line with strategies in the approved | Technology Roadmap.

Pro: • Continues progress of the replacement of end of life IT infrastructure equipment in line with the

approved Technology Roadmap and establishes the base for equipment lifecycle management. • Incrementally lowers costs for extended maintenance when end of life equipment is removed. • Addresses end of life IT infrastructure equipment replacement in 1 year phases which enables the

use of existing IT staff to be redirected to replacement tasks and address operations at the same time and enables Caltrans IT to evaluate the effectiveness of the continued use of internal staff for equipment refresh tasks.

Con: • Requires a one-time increase of up to 514 million.

Alternative 2: Approve a permanent budget augmentation to replace end of life IT infrastructure equipment on an ongoing basis per industry standard lifecycle increments with existing staff.

Pro: • Reviews of additional BCP's would not have to be done annually. • Enables the replacement of end of life equipment in priority order and establishes the base for

equipment lifecycle management. • Incrementally lowers costs for extended maintenance when end of life equipment is removed. • Addresses end of life IT infrastructure equipment replacement in 1 year phases which enables the

use of existing IT staff to be redirected to replacement tasks and address operations at the same time. ^

Con: • Requires a permanent budget increase of SI 2 million.

6

Analysis of Problem

Alternative 3: Funding of a five year replacement plan. This alternative enables Caltrans IT to systematically replace the oldest and most critical IT infrastructure devices (network devices, servers and storage appliances) over the next five years. This will address the backlog of end of life devices that is the result of a lack of replacement funding due to higher priority funding choices during past years.

Pro: • Provides a five year funding base to address the backlog of end of life equipment that has resulted

from lack of funding in the past.

Con: • Requires a commitment of $60 million over the next 5 years.

G. Implementation Plan

July 1,2018

H. Supplemental Information

None.

I. Recommendation

Caltrans recommends approval of Alternative 1: a one-time increase of up to $2 million in SHA funds, contingent upon DOF and CDT approval, to develop an Information Technology Roadmap; and, upon' DOF and CDT approval of the Information Technology Roadmap, provide a one-time increase of up to $12 million in SHA funds to replace end of life IT infrastructure equipment in line with strategies in the approved Infrastructure Technology Roadmap.

7

F Y 2018-19 Information Technology Infrastructure Replacement B C P A T T A C H M E N T I

This diagram illustrates funding for Caltrans technical infrastructure since the initial Spring Finance Letter in FY1997-98.

1997-98 SFL $21 million

Approved FY17/18 BCP

$12 million-1 Year

FY18/19 BCP $12 million-1 Year

FY 1997/98 FY 2017/18 FY 2018/19

FY 2018-19 B O ^ ^ s t r u c t u r e Refresh

FY 2018-j^piacement List

10-20-2017

ATTACHMENT II

'-c'lsmxa'Aki i ; • lulisulv

hfC-Ui' i I 'd i i -

STORAGE SAN SWITCH EMC Connectrix (Brocade B Series) DS-23a BRCRD06014925B 5 /16 /2013 Price Included In VNX5300 D07-TMC 4 5 3 3 9 $26,400 Replace in FY 18-19 521

STORAGE SAN SWITCH EMC Connectrtx (Brocade B Series) DS-230 BRCRD0601493H 5/16/2013 Price Included In VNX5301 D07-TMC 4 5 3 3 9 $26,400 Replace in FY 18-19 522

STORAGE SAN SWITCH EMC Connectrtx (Brocade B Series) DS-300 BRCAU1912H0JH 5 /16 /2013 Price included inVNX5300 D07-TMC 4 5 3 3 9 $26,400 Replace In FY 18-19 523

STORAGE SAN SWITCH EMC

Connectrix (Brocade B Series) DS-300 BRCAU2508G0H0 5/16/2013 Price Included in VNX5300 D07-TMC 4 5 3 3 9 $26,400 Replace In FY 18-19 524

STORAGE Archive Appliance EMC EMCAtmos WS2-120 APM0013130664a 4/15/2013 Archive Storage D20 4 7 3 3 9 Replace In FY 18-19 525

STORAGE SAN SWITCH EMC Connectrix (Brocade B Series) DS-300 BRCAU0626E0S8 6/1/2014 Price Included In VNX520O D08-TMC 3 5 3 3 9 $26,400 Replace in FY 18-19 526

STORAGE SAN SWITCH EMC Connectrix (Brocade B Series) DS-300 BRCAU0630E1G0 6/1/2014 Price included In VNX520O D08-TMC 3 5 3 3 9 $26,400 Replace in FY 18-19 527

STORAGE SAN SWITCH EMC Connectrix (Brocade B Series) DS-6510 BRCBRW2505K007 6/1/2014 Price Included in VNX5200 D08-TMC 3 5 3 3 9 $26,400 Replace In FY 18-19 528

STORAGE SAN SWITCH EMC Connectrix (Brocade B Series) DS-6510 BRCBRW2547J081 6/1 /2014 Price Included in VNX520O D08-TMC 3 5 3 3 9 $26,400 Replace in FY 18-19 529

Grand Total $12,013,035

Page 1 of 1

F Y 2018-19 Information Technology Infrastructure Replacement B C P ATTACHMENT III

The following tables show the number of information technology infrastructure devices that have reached end of life (EOL), the average hours to replace each type of device, and the number of positions required to perform the work during FY 18-19.

Workload Analysis

F Y 18-19

Device Type E s t . Quantity

E s t . Average Hours to Replace

E s t . Level of Effortto Replace (Hours)

E s t . Full Time Equivalent (FY 17-18)

Network 420 18 7,560 4.3 Server 59 42 2,478 1.4 Storage 50 80 4,000 2.2 Total 529 14,038 7.9

2660-013-BCP-2018-GB

Posit ions Permanent ^mporary

Total Posit ions

Salaries and Wages Earnings - Permanent Earnings - Temporary Earnings - Statutory/Exempt Overtime, Holiday, Other

Total Salaries and Wages

Staff Benefits Dental Insurance Disability Leave Health Insurance Life Insurance Medicare Taxation OASDI Retirement

Unemployment Insurance Vision Care Workers' Compensation Staff Benefits - Other

Total Staff Benefits

Total Personal Serv ices

CY

0.0 0.0 0.0 0.0 0.0

$0

$0

BY

0.0 0.0 0.0 0.0 0.0

$0

$0

BY+1 BY+2 BY+3

0.0 0.0 0.0 0.0 0.0

$0

$0

0.0 0.0 0.0 0.0 0.0

$0

$0

0.0 0.0 0.0 0.0 0.0

$0

Operating Expenses and Equipment «^ e r a l Expense Pting ommunications

Postage Insurance Travel Training Facilities Operations Utilities Consulting and Professional Services Departmental Services Consolidated Data Centers Information Technology Central Administrative Services Office Equipment Other

Unclassified/Special, Adjustment

Total Operating Expenses and Equipment

Total Budget Request

0 0 0 0 0 0 0 0 0 0 0 0

2,000,000 0 0 0 0

$0 $2,000,000 $0

$0 $2,000,000 $0

$0

$0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

$0

Fund Source - State Operations General Fund Federal Funds Other/Special Funds Reimbursements

Total State Operations Expenditures

0 0 0 0

$0

0 0

2,000,000 0

$2,000,000 $0 ^ n d Source - Local Assistance

^ B t e r a l Fund Wderal Funds Other/Special Funds Reimbursements

Total Local Assistance Expenditures

0 0 0 0

$0

0 0 0 0

$0 $0 $0