General Rate Case · SERVICE MANAGEMENT OFFICE & OPERATIONS ... 14 three primary data centers.2 In addition, SCE-06, Volume 1, ... 24 SCE’s data center infrastructure and applications,

Application No.: A.19-08-013 Exhibit No.: SCE-06, Vol. 1 Part 1A Witnesses: L. Garris

J. Ishiguro A. Ma R. Nanda E. Roddick

(U 338-E)

2021 General Rate Case

Amended Testimony on Enterprise Technology

Before the

Public Utilities Commission of the State of California

Rosemead, California February 20, 2020

SCE-06, Vol. 1, Part 1A: Enterprise Technology

Table Of Contents Section Page Witness

i

I. INTRODUCTION .............................................................................................1 A. Ma

A. Content and Organization of Volume ....................................................1

B. Summary of O&M and Capital Request ................................................2

C. ENTERPRISE TECHNOLOGY OVERVIEW .....................................5

1. Enterprise Technology Focus & Operational Improvements ............................................................................5

a) Managed Services Model/ Fixed Price ..........................6

b) Enterprise Technology Optimization .............................7

c) Digital Transformation ...................................................7

d) Risk Factors, Safety, Reliability and Connection with RAMP .................................................8

e) Regulatory Background/Policies Driving SCE’s Request ...............................................................8

f) Compliance Requirements .............................................8

2. 2018 Decision ............................................................................9

a) Comparison of Authorized 2018 to Recorded - O&M ...........................................................9

b) Comparison of Authorized 2018 to Recorded - Capital Expenditures .................................10

II. TECHNOLOGY PLANNING, DESIGN & SUPPORT .................................11 E. Roddick

A. Work Description .................................................................................11

B. Need for Activity .................................................................................13

C. Comparison of Authorized 2018 to Recorded .....................................13

D. Scope and Forecast Analysis ...............................................................14

1. Historical Variance Analysis ...................................................14

a) Labor ............................................................................14



ii

b) Non-Labor ....................................................................15

2. Basis for Forecast .....................................................................15

a) Labor ............................................................................15

b) Non-Labor ....................................................................16

III. TECHNOLOGY DELIVERY .........................................................................17 J. Ishiguro

A. Work Description .................................................................................17

B. Need for Activity .................................................................................18

C. Comparison of Authorized 2018 to Recorded .....................................18

D. Scope and Forecast Analysis ...............................................................19

1. Historical Variance Analysis ...................................................19

a) Labor ............................................................................19

b) Non-Labor ....................................................................20

2. Basis for Forecast .....................................................................20

a) Labor ............................................................................20

b) Non-Labor ....................................................................21

(1) Base Non-Labor Expenses ...............................21

(2) ..............................................................................

(3) Miscellaneous O&M Projects Less Than $1 million ................................................22

IV. SERVICE MANAGEMENT OFFICE & OPERATIONS ..............................23

A. Fixed Price Technology & Maintenance .............................................23 L. Garris

1. Work Description .....................................................................24

2. Need for Activity .....................................................................25

3. Comparison of Authorized 2018 to Recorded .........................26



iii

4. Scope and Forecast Analysis ...................................................26

a) Historical Variance Analysis .......................................26

(1) Labor ................................................................26

(2) Non-Labor ........................................................26

b) Basis for O&M Forecast ..............................................27

(1) Labor ................................................................27

(2) Non-Labor ........................................................27

B. Software Maintenance & Replacement ...............................................28 L. Garris / .................................................................................................................. R. Nanda

1. Overview of Work Activity .....................................................30

2. Comparison of Authorized 2018 to Recorded- O&M ........................................................................................31

3. Comparison of Authorized 2018 to Recorded- Capital ......................................................................................31

4. Sub-Work Activities ................................................................32

a) Perpetual License, Software as a Service (SaaS), Cloud (Subscription Based Software) ......................................................................32

(1) Work Description .............................................32

(2) Need for Activity .............................................34

(3) Scope and Forecast Analysis ...........................35

b) Application Refresh .....................................................42



(3) Scope and Forecast Analysis ...........................44

C. Technology Infrastructure Maintenance & Replacement ....................52 R. Nanda



iv

1. Overview of Work Activity .....................................................54

2. Comparison of Authorized 2018 to Recorded- O&M ........................................................................................55

3. Comparison of Authorized 2018 to Recorded- Capital ......................................................................................55

4. Sub-Work Activities ................................................................56

a) Data Center Infrastructure ............................................56



(3) RAMP Integration ............................................59

(4) Scope & Forecast Analysis ..............................59

b) End User Computing Maintenance, Services & Replacement .............................................................71




c) Technology Adoption ..................................................78

(1) Work Activity Description ...............................78



1

I. 1

INTRODUCTION 2

A. Content and Organization of Volume 3

This Volume of amended testimony presents Southern California Edison’s (SCE) 2021 Test Year 4

forecast of Operations and Maintenance (O&M) expenses and 2019-2023 capital expenditures forecast 5

for the Enterprise Technology Business Planning Element (BPE).1 This BPE includes the activities and 6

infrastructure to support SCE’s broader Information Technology (IT) needs which are foundational to 7

the operation of SCE. SCE’s Enterprise Technology forecast of $217 million (Constant 2018 dollars) in 8

O&M expenses for Test Year 2021 and capital expenditures of $538 million for 2019-2023 will allow 9

SCE to continue necessary work to manage our increasingly complex technology environment, 10

including over 7,500 midrange servers (UNIX, Linux, and Wintel), over 2,000 terabytes of data storage, 11

700 miles of data network routing and switching infrastructure inclusive of copper and fiber-optic 12

cabling, 400 appliances to support over 500 large data repository solutions, and operations of SCE’s 13

three primary data centers.2 In addition, SCE-06, Volume 1, Pt. 2 of this Exhibit presents SCE’s request 14

of $506 million in forecast capitalized software expenditures for 2019-2023. As a result of SCE’s 15

reorganization of its 2021 GRC presentation,3 Cybersecurity & Compliance and Grid Services, which 16

were previously presented within the IT Organizational Unit (OU) testimony in SCE’s 2018 GRC, are 17

now presented in SCE-04, Volume 3 and SCE-02, Volume 3, respectively, even though organizationally 18

they still reside within IT. SCE’s presentation of the remaining IT costs (collectively, Enterprise 19

Technology) are structured consistent with our Plan, Deliver (or build), Run operating model, which 20

SCE implemented in 2017 to help streamline processes and reduce inefficiencies across technology 21

domains. The Plan function includes the identification, prioritization, and design of technology 22

investments to enable SCE to meet customers’ evolving needs, address regulatory compliance 23

objectives, and support SCE’s operational needs. The Deliver function is the construction and oversight 24

of product development, configuration, and deployment of technology solutions including project 25

1 This testimony amendment reflects the incorporation of additional IT costs to support the continued operation

of legacy systems through 2021, and the realignment of post-CSRP implementation costs with the new CSRP go-live in early 2021. For administrative convenience, this testimony also incorporates SCE’s previous errata submitted on November 22, 2019.

2 SCE’s three primary data centers include the Alhambra Data Center, Irvine Data Center, and Grid Data Center. Expenditures related to SCE’s Grid Data Center are addressed in Exhibit SCE-02, Volume 3.

3 See Exhibit SCE-07, Volume 1, Chapter X.

2

management. The Run function includes management of software products, application maintenance 1

and enhancements, the operation and replacement of IT infrastructure (hardware), and oversight over the 2

Managed Services Providers (MSPs). The recorded history and forecasts for these activities have been 3

sequenced according to this work flow and are described in detail in the following chapters: 4

Chapter I - Introduction 5

Chapter II - Technology Planning, Design & Support (Plan) 6

Chapter III - Technology Delivery (Deliver) 7

Chapter IV - Service Management Office & Operations (Run) 8

B. Summary of O&M and Capital Request 9

SCE’s Enterprise Technology forecast of $217 million in O&M expense for the Test Year 2021 10

represents a $25 million increase over 2018 recorded expenses, which is necessary to support SCE’s 11

growing IT portfolio. This includes incremental costs to accommodate: 12

- Maintenance, replacement, and modernization of grid technologies; 13

- Post-CSRP implementation costs, including decommissioning of Customer Service System 14

(CSS) mainframe and non-mainframe applications; 15

- Growth in software license and maintenance agreements; and 16

- Incremental MSP support for major programs such as Grid Modernization and Digital 17

Managed Services. 18

- Growth in cloud-based services and solutions 19

- Enablement of more digital technologies and data analytics to address needs of customers 20

and SCE operations4 21

SCE’s Enterprise Technology capital expenditures of $538 million for 2019-2023 include 22

expenditures for technology infrastructure maintenance and replacement that are necessary to support 23

SCE’s data center infrastructure and applications, including disaster recovery, server refreshes, storage 24

refreshes, and application upgrades. In addition, the Enterprise Technology capital expenditures provide 25

end-user computing equipment such as desktop computers, laptops, phones, and printing for our 26

approximately 12,500 SCE employees across our 50,000 square miles of service territory. IT capital 27

software expenditures for OU and IT projects are discussed in SCE-06, Volume 1, Pt. 2. This Volume 28

includes SCE’s 2019-2023 Enterprise Technology capital expenditure forecast, which reflects: 29

4 Refer to Exhibit SCE-06, Volume 2, Chapter V testimony entitled “Digital and Process Transformation.”

3

- Routine and deferred refresh of hardware infrastructure beyond the five-year use life cycle, to 1

replace obsolete or unsupported hardware which may be more prone to failure and outages 2

which impact business services internally within SCE and externally for our customers; 3

- Refresh of high-risk applications no longer supported by vendors or which are about to reach 4

end of life, which will hamper application availability and increase unplanned outages, 5

resulting in these applications not being able to provide needed business functionalities; and 6

- Investments for future third-party hosted infrastructure, which is expected to eventually 7

eliminate some dependencies on on-premise infrastructure and improve data storage 8

capabilities and cyber security as further detailed in Section C below and Section IV.C 9

“Technology Infrastructure Maintenance & Replacement” (specifically the Technology 10

Adoptions activities). 11

Figure I-1 Enterprise Technology O&M Expense

(Total Company Constant $Million)

4

Figure I-2 Enterprise Technology Expenditures 2019-2023

(Total Company - $Million)

5

C. Enterprise Technology Overview 1

SCE’s Enterprise Technology BPE includes the activities and infrastructure necessary to support 2

the information technology needs that are foundational to SCE’s operation. The reliance on information 3

technology has become increasingly essential to business operations across many industries, including 4

the electric utility industry. The basic, yet critical, ability to store, analyze, transmit, and present large 5

amounts of data is foundational for everyday communication and employee productivity, essentially 6

enabling all of SCE’s day-to-day activities. In addition to the safe and reliable operation of the grid, the 7

Enterprise Technology BPE also impacts our core competencies in areas such as grid resiliency and 8

customer satisfaction. With our technology landscape changing at an unprecedented pace, so does the 9

availability of new options to serve our business and customer needs. SCE continues to develop new 10

tools for customers that convey the information they need to make clean energy choices and support 11

California’s aggressive climate change goals.5 We also continue to identify and implement technology 12

solutions to help our operations run more safely, reliably, and efficiently. In recent years, SCE has 13

increasingly relied on digital technology, including doing things like switching from physical phones to 14

virtual internet phones. This ensures our core collaboration systems are on a modern platform, 15

consolidates and improves tools to better manage work, and improves the ways we communicate with 16

our customers, all of which makes a significant contribution toward improving operational efficiencies. 17

In addition, we continue to integrate more third party externally-hosted cloud technology into our 18

operating environment, which will improve agility and innovation, increase business and IT efficiency, 19

and improve availability, reliability, and disaster recovery. The following sections will provide an 20

update on the various strategic efforts that continue to be a priority for Enterprise Technology and SCE 21

during this GRC period. 22

1. Enterprise Technology Focus & Operational Improvements 23

In the 2014-2018 period, as depicted in SCE’s 2018 GRC, Enterprise Technology 24

focused on mitigating cost pressures driven by changing technology landscapes. Some of these cost 25

pressures will continue to extend into this GRC period, and others also have emerged, as summarized 26

below: 27

- Support for Grid Modernization, requiring on-going maintenance for hardware and 28

software that will make our power grid safer, more flexible to accommodate customer 29

5 Assembly Bill (AB) 32 requires California to reduce its greenhouse gas (GHG) emissions to 1990 levels by

2020.

6

adoption of distributed energy resources, more responsive to energy management 1

needs and grid emergencies, and able to address various DRP compliance 2

requirements;6 3

- Support for on-going maintenance post-CSRP implementation, which requires 4

changing knowledge and skillsets in the operation of technology solutions; 5

- Support for Wildfire Initiatives, which requires new technology that increases SCE’s 6

flexibility to respond to wildfires quickly and provides predictive capabilities to 7

ensure our customers and power grids are safe from these emerging threats; 8

- Pace of technological advancement and the availability of new options to serve our 9

business and customer needs (e.g., emergence of cloud-based services and 10

applications); and 11

- Regulatory demands (e.g., NERC Critical Infrastructure Protection (CIP)) for a more 12

secure and reliable grid. 13

We managed previously described pressures in the period of 2014-2018 through the 14

operational improvements discussed below. We will continue to maintain the results of these operational 15

improvements and implement other strategic initiatives to improve our operations and to ensure 16

regulatory and compliance mandates are met during this GRC period. 17

a) Managed Services Model/ Fixed Price 18

As previously described in SCE’s 2018 GRC (A.16-09-001), SCE began the 19

transition of our day-to-day application and infrastructure maintenance and support functions to a 20

Managed Services model in 2014 and completed this at the end of the second quarter of 2015. 21

Since then, we have observed faster reaction times, an increase in effectiveness, 22

and continual improvement on support and service levels. Moreover, transitioning to an MSP model has 23

reduced operational risk, minimized IT-related business disruption and associated costs, and lowered the 24

cost structure for IT services. 25

During Test Year 2021 and continuing through 2023, MSPs will continue to 26

perform services at a steady state, while taking on additional responsibilities after the implementation of 27

major programs such as CSRP and Grid Modernization. Further descriptions of MSP services and 28

6 See Exhibit SCE-02, Volume 4 and SCE-04, Volume 3.

7

discussions regarding benefits are described in the Fixed Price Technology & Maintenance work activity 1

(section IV.A). 2

b) Enterprise Technology Optimization 3

As also stated in SCE’s 2018 GRC, the Enterprise Technology Optimization 4

(ETO) initiative was implemented to focus on improving the way SCE plans for and applies technology 5

solutions across the enterprise. This initiative focused on a few primary areas, including: 6

- Implementing best practices in IT investment planning / demand 7

prioritization; 8

- Simplifying our current IT application portfolio and underlying infrastructure 9

by consolidating, eliminating, and standardizing applications; and 10

- Reducing redundant or non-critical IT product offerings and continuing to 11

optimize IT infrastructure. 12

In SCE’s 2018 GRC, SCE projected $14.741 million in savings during the 2016-13

2018 period from the ETO initiative. SCE realized actual savings during this period of $25.080 million 14

from the ETO initiative. These savings allowed us to manage unanticipated cost pressures that occurred, 15

such as wildfire mitigation. 16

c) Digital Transformation 17

The Digital and Process Transformation activities support SCE’s efforts to 18

improve productivity and streamline operations by leveraging digital technologies. The goal of pursuing 19

these activities is to enable disruptive innovation to generate and identify opportunities and implement 20

solutions with digital technology platforms. 21

It is imperative to know where digital tools are best deployed within a process and 22

to be able to deliver those tools faster, in an iteratively beneficial manner, in contrast to the traditional 23

waterfall method where all benefits are realized at final completion. Therefore, at the end of 2018, the 24

Digital Accelerator team was created to focus on deploying digital tools such as rapid prototyping, 25

robotic process automation, mobile solutions, and advanced analytics, using agile delivery methods. The 26

Digital Accelerator, and the solutions it delivers, aims to increase quality, efficiency, and customer 27

satisfaction. SCE’s Digital Accelerator testimony and forecast is discussed in Exhibit SCE-06, Volume 28

2. 29

8

d) Risk Factors, Safety, Reliability and Connection with RAMP 1

Cybersecurity is considered one of SCE’s top risks. As such, it was included in 2

the Risk Assessment and Mitigation Phase (RAMP). SCE-04, Volume 3 discusses the risk factors, 3

controls, and mitigations proposed in RAMP, as well as changes between SCE’s RAMP and 2021 GRC 4

Cybersecurity forecast. In Chapter IV of this volume, SCE forecasts incremental costs for IT operational 5

activities resulting from the proposed Cybersecurity mitigation activities. 6

Enterprise Technology also delivers IT projects for OUs that were included in 7

RAMP. These projects are discussed primarily in the Physical Security and Generation BPGs in SCE-8

06, Volume 1, Pt. 2 (OU Capitalized Software). 9

e) Regulatory Background/Policies Driving SCE’s Request 10

Enterprise Technology’s O&M expenses and capital expenditures are impacted by 11

various CPUC and FERC regulatory mandates both directly and indirectly as IT provides companywide 12

technology-based support that is key to SCE’s ability to meet compliance mandates. Examples of these 13

areas of compliance include, but are not limited to, customer electric rate design, California Independent 14

System Operator (CAISO) tariff and operating procedures, FERC mandates applying to SCE’s Hydro 15

Generating Plants, and Customer Data Privacy. As SCE’s regulatory environment continues to become 16

increasingly complex, so does SCE’s reliance on technology projects to maintain compliance. These 17

compliance requirements are discussed in the respective IT testimony sections, primarily in SCE-02, 18

Volume 4 (System Augmentation), SCE-04, Volume 3 (Cybersecurity & Compliance), and SCE-06, 19

Volume 1, Pt. 2 (OU Capitalized Software). 20

f) Compliance Requirements 21

In D.15-11-051, the CPUC required that SCE “include its own forecast and the 22

Commission’s adopted forecast from the previous GRC alongside historical costs, and brief explanations 23

detailing any changes in the scope of a category.”7 A summary level is provided in the Introduction 24

section and the testimony for each O&M and Capital GRC activity. 25

7 D.15-11-021, p. 224.

9

2. 2018 Decision 1

a) Comparison of Authorized 2018 to Recorded - O&M 2

Figure I-3 Enterprise Technology

O&M Expenses for 2018 - Authorized versus Recorded8 (Constant $000)

As shown in Figure I-3, SCE requested $206.5 million, was authorized $200.1 3

million, and recorded $191.8 million in 2018 O&M expenses for Enterprise Technology. SCE recorded 4

less than authorized by approximately $8.3 million primarily due to the reorganization of IT 5

departments, including some employees going to support strategic programs like CSRP and Grid 6

Modernization, and minor savings realized as a result of negotiations with SCE’s MSPs. These 7

underruns were partially offset by the cost pressures SCE faced, including increased software 8

maintenance and support costs for projects such as Teradata and HR Re-Platform, a change in SCE’s 9

capitalization rules that resulted in charging hardware maintenance as O&M expense instead of capital, 10

and the centralization into Enterprise Technology of various IT products and services that were 11

previously recorded in SCE OUs. These underruns and overages are described in more detail in the work 12

activity sections below. 13

8 Refer to WP SCE-07, Vol. 1 Authorized to Recorded.

10

b) Comparison of Authorized 2018 to Recorded - Capital Expenditures 1

Figure I-4 Enterprise Technology

Capital Expenditures for 2018 - Authorized versus Recorded9 (Nominal $000)

As shown in Figure I-4, SCE was authorized $63.8 million in 2018 capital 2

expenditures for Enterprise Technology. SCE recorded $118.1 million in 2018, approximately $54.3 3

million over authorized. This was primarily due to SCE completing a $47 million strategic restructuring 4

of SCE’s SAP portfolio support, maintenance, and licensing. While not anticipated at the time SCE filed 5

its 2018 GRC in 2016, this restructuring was advantageous in numerous ways as it replaced our SAP 6

legacy software products with their new product line functionality, which strengthened SCE’s prior 2015 7

investment in HANA technology. This restructuring also included five years of maintenance, during 8

which SCE may grow our use of the SAP products at no additional cost. The spending over authorized 9

in 2018 was also due to necessary upgrades to application versions in order to avoid technology 10

obsolescence to maintain application reliability. This spending, as well as the additional benefits from 11

the SAP purchase, are discussed in Chapter IV. 12


11

II. 1

TECHNOLOGY PLANNING, DESIGN & SUPPORT 2

Figure II-5 Summary of Technology Planning,

Design & Support Recorded10 (Constant $000)

A. Work Description 3

This activity comprises the planning and design of enterprise technology, which informs 4

investment strategies and technology standards across SCE. Planning includes the identification, 5

prioritization, and design of technology investments to enable SCE to meet customers’ evolving needs, 6

address regulatory compliance objectives, and support SCE’s operational needs. It also involves working 7

closely with SCE OUs to understand both unique and shared business drivers and define long-term 8

technology capabilities that will support these business objectives. Because business priorities are 9

constantly evolving and the pace of technology is advancing quickly, this planning component is 10

important to validate that the right projects are being pursued, prioritized, and in alignment with other 11

business needs across SCE. Ultimately, technology planning is responsible for demand and capacity 12

10 See WPSCE-06V01P01 pp. 1-7.

12

management of technology projects, serving as the single gateway to all work coming into Enterprise 1

Technology. 2

Technology design is the architecture or blueprinting of technology solutions to achieve the 3

desired business capabilities. This activity is critical as system enhancements are an increasingly 4

foundational enabler to achieving SCE’s strategic objectives. For example, SCE’s focus on cleaning the 5

power system through modernizing our electric grid consists of building a smarter grid that relies on 6

technology to maintain stability and achieve a higher level of resilience. This requires extensive 7

understanding of our grid and its systems and control theory, computer science, power systems, and 8

utility operations so that the design of proposed system enhancements does not create an unnecessary 9

risk of unforeseen technology incompatibilities resulting from the introduction of new technology 10

products, or interfaces and enhancements to existing products. Technology design also includes 11

estimating the costs of IT projects requested throughout the company. Building on the Commission’s 12

direction from SCE’s 2012 GRC,11 SCE’s 2018 GRC,12 and industry best practices, SCE utilizes a 13

standard IT cost estimation model to develop cost estimates for all prospective IT projects. This model 14

identifies several cost components and factors used in the estimation process, including: 15

• Commercial off the Shelf (COTS) and Development approaches; 16

• Labor rates; 17

• Labor expenses (including project labor and post-implementation support labor); 18

• Capitalization of on-going licensing fees; 19

• Vendor allocations (including effort driven & fixed price); and 20

• Non-labor costs (hardware and data center, software license, etc.). 21

SCE uses this model, which is discussed in more detail in SCE-06, Volume 1, Pt. 2, to estimate 22

the costs of the capital software projects it requests. In doing so, the project costs have been estimated 23

using a consistent methodology for all projects that have forecast spending in 2019 and 2020.13 24

Finally, this activity includes the provisioning of IT business support services, including support 25

for IT staffing needs, all procure-to-pay and invoice processing support, coordination of space/facility 26

allocation, and training. 27

11 D.12-11-051, pp. 435-436 and Ordering Paragraph 27.

12 D.19.05-020.

13 See Exhibit SCE-06, Volume 1, Pt. 2, for SCE’s 2021-2023 OU capitalized software portfolio that is based on SCE’s recorded expenditures.

13

B. Need for Activity 1

This activity is necessary to manage and prioritize technology investments as well as estimate the 2

costs of new technology project requests across the enterprise. Because a part of the technology planning 3

activity also includes finding ways to leverage existing solutions across BPGs, this activity helps filter 4

and eliminate the creation of applications with duplicate functionality which could lead to unnecessary 5

complexity and increased maintenance costs. Through this holistic planning approach, as well as the risk 6

analyses performed on our systems’ architecture design, SCE can strategically purchase and develop 7

hardware and software assets. This enhances our ability to manage our technology costs more 8

effectively, while also helping to mitigate risks associated with implementing new technology 9

applications and solutions. For these reasons, the need for this activity will continue to be critical as SCE 10

and our customers increasingly rely on the benefits of technology. 11

C. Comparison of Authorized 2018 to Recorded 12

SCE was authorized $11.2 million for Technology Planning, Design & Support in the 2018 13

GRC. This work activity’s recorded 2018 expenses were approximately $11.5 million, which was $0.3 14

million above authorized.14 SCE overspent authorized levels primarily because of an increase in labor as 15

SCE increased its architecture staffing to support new programs such as CSRP and Grid Modernization, 16

as well as to provide an enterprise architectural focus on our design processes and standards in order to 17

produce higher quality outcomes. This labor increase was partially offset by unusually low recorded 18

non-labor expenses as SCE temporarily cancelled its benchmarking service and survey subscription 19

license. 20


14

D. Scope and Forecast Analysis 1

Figure II-6 Technology Planning, Design & Support 2014-2018 Recorded/Forecast 2019-2021

(Constant 2018 $000)

1. Historical Variance Analysis 2

a) Labor 3

As illustrated in Figure II-6, Technology Planning, Design & Support labor 4

expenses decreased from 2014-2015 by 38 percent as SCE began IT Transformation activities, which 5

included moving to an MSP model. This resulted in a significant labor decrease of approximately $6 6

million as work previously supported by SCE labor was transitioned to the MSP.15 Labor remained 7

relatively flat from 2015 to 2016 and decreased from 2016 to 2017 due to a reorganization of the IT 8

departments into the Plan, Deliver, Run model described in Chapter I of this Volume. This involved 9

some labor shifts to and from other IT Departments. Labor increased from 2017 to 2018 as SCE 10

increased its architecture staffing to support new programs such as CSRP and Grid Modernization, as 11

15 See A.16-09-001, SCE-04, Volume 1 Pt. 1, pp. 3-9 and Section IV.A contained in this Volume for additional

information regarding SCE’s transition to an MSP model.

15

well as to provide an enterprise architectural focus on our design processes and standards in order to 1

produce higher quality outcomes. 2

b) Non-Labor 3

As shown in Figure II-6, non-labor costs decreased significantly from $8.2 4

million to $1.6 million from 2014 to 2015. This was primarily due to the temporary use of contract 5

workers to implement Microsoft Office 365, which was completed in 2014. Non-labor costs then 6

increased from $1.6 million to $4.9 million in 2016 primarily as a result of utilizing consulting services 7

to assist SCE in identifying cost savings that were included as part of SCE’s Operational Excellence 8

initiative.16 Costs decreased again from 2016 to 2018 with the completion of SCE’s Operational 9

Excellence effort and a temporary cancellation of a benchmarking service and survey subscription 10

license. 11

2. Basis for Forecast 12

a) Labor 13

As discussed above, labor costs decreased significantly from 2014 to 2015, 14

followed by minor fluctuations from 2016 to 2018 as a result of the department reorganization and 15

addition of architects to prepare for CSRP and Grid Modernization. Because the 2014-2015 recorded 16

levels were representative of a one-time MSP transition and the remaining years fluctuated, SCE 17

considered a three-year average (2016-2018) and two-year average (2017-2018) forecast methodology. 18

SCE selected a two-year average as this method yields a slightly lower forecast, which is closer to 19

SCE’s expected level of labor necessary in the 2021 Test-Year. This is also consistent with prior 20

Commission guidance that stated that for those activities that have fluctuations in recorded years, an 21

average of recorded expenses is appropriate. To the two-year average base forecast of $10.1 million, 22

SCE adjusts the 2021 Test-Year forecast downward by $1 million as SCE anticipates savings related to 23

the CSRP business case. Because CSRP will remove the need to upgrade, integrate, or increase 24

functionality of the outdated CSS mainframe, SCE projects the CS OU to have fewer and less complex 25

projects, and anticipates a shifting focus to projects with more standardized implementation on the new 26

16 In SCE’s 2018 GRC Application, SCE reflected savings in the amount of $14.7 million in IT’s 2018 Test

Year Forecast. See A.16-09-001, Exhibit SCE-04, Volume 1 Pt. 1, p.10.

16

SAP platform. This will result in decreased labor spend related to planning and designing these 1

solutions.17 2

b) Non-Labor 3

Non-labor costs fluctuated significantly from 2014 to 2018 with periodic use of 4

contract labor for a major technology implementation and consulting services for a strategic initiative. 5

Prior Commission guidance has provided that fluctuating recorded expenses warrant use of a historical 6

average as the basis to estimate the Test Year forecast. However, utilizing an average method would 7

yield a greater forecast than what SCE forecasts is necessary in the Test Year 2021. Therefore, SCE uses 8

the 2018 last recorded year of $750,000 which is far below any of the other estimating methods and 9

includes a modest increase of $71,000. SCE is resuming use of the benchmarking and survey 10

subscription licenses that were not in SCE’s 2018 recorded costs as a result of temporarily cancelling the 11

subscription in 2018. This results in a total 2021 Test year forecast of $821,000.18 12

17 Although the planned CSRP “Go-Live” date has been revised to early 2021, the $1 million post-

implementation savings is still applicable. Thus, this savings is not being added back into the 2021 Test Year forecast as part of SCE’s Amended Testimony for Enterprise Technology.

18 SCE’s forecast excludes approximately $800,000 of avoided costs that would have otherwise been necessary if CS Re-Platform was not implemented in order to fund the CS Re-Platform Technical Assessment. This periodic assessment evaluated replacement of SCE’s core legacy CS system with an SAP-based solution and continues to be an avoided cost that is not impacted by SCE’s revised CSRP implementation date.

17

III. 1

TECHNOLOGY DELIVERY 2

Figure III-7 Technology Delivery

Recorded 2014-2018/Forecast 2019-202119 (Constant $000)

A. Work Description 3

Technology Delivery is the execution of non-routine system enhancements and implementation 4

of the capital software projects for SCE’s OUs that are requested in SCE-06, Volume 1, Pt. 2. This 5

includes the overall project management required after a technology solution has successfully undergone 6

the necessary planning, design, and business case approvals. At this stage, the delivery team confirms 7

the scope, schedule, and budget and manages the project through completion. In addition to the 8

execution of the project, the Technology Delivery activity encompasses management of go-live and 9

post-implementation stabilization activities, as well as verifying business readiness of the technology 10

deployment. In instances where technology deployment impacts many users, Organizational Change 11

Management (OCM) activities are also led by the delivery team. 12

19 See WP SCE-06, Volume 1 Pt. 1, pp. 8-14.

18

In addition to the delivery services, this GRC activity also includes O&M expenses incurred to 1

support the capitalized software projects across SCE’s BPGs. The O&M expenses consists of the 2

following activities required to develop and implement capitalized software: 3

Pre-project planning activities, such as business requirements development and conceptual 4

solution engineering, data migration, and data cleansing; and 5

Delivery of end-user training associated with the technology, which includes developing 6

training material and providing training to end-users. 7

Finally, this GRC activity includes certain projects which are mostly or completely O&M. These 8

projects are typically small (<$250,000) and address targeted needs in an area of our business. These 9

O&M projects primarily involve minor enhancements to existing software or installation of new 10

products when SCE does not have the software capability within our existing portfolio. 11

In certain cases, these O&M projects can be large, similar in size to a capitalized software 12

project, but which, due to accounting guidelines that determine what types of costs are capitalized, are 13

recorded as O&M.20 14

B. Need for Activity 15

The Technology Delivery activity is essential to the execution and implementation of 16

technology, both foundational and strategic across the enterprise. It is critical in order to minimize 17

disruption to existing systems and processes as new capabilities are enabled. Further, this activity drives 18

accurate and timely visibility into our technology investments and enforces project management 19

standards and quality control principles to enable IT solutions to meet the needs of the organization. 20

These quality control activities minimize rework after a project is completed. 21

C. Comparison of Authorized 2018 to Recorded 22

SCE was authorized $36.1 million in 2018 for Technology Delivery, which is $25.1 million 23

above SCE’s recorded expenses of $10.9 million.21 This underspending compared to authorized was 24

primarily due to the reorganization of IT departments. Approximately $10.9 million of Technology 25

Delivery labor was reassigned to IT Plan and Run functions, as well as to strategic programs such as 26

Grid Modernization or CSRP that required delivery management skills and expertise. As a result of 27

resources being temporarily deployed to these programs and several positions temporarily left vacant, 28

20 See Exhibit SCE-07, Volume 2 for testimony regarding accounting treatment of O&M and Capital

expenditures.

21 Refer to WP SCE-07 Vol. 1 Authorized to Recorded.

19

the 2018 recorded labor expenses contributed to SCE underspending compared to the authorized 1

amount. In addition, SCE incurred fewer expenses for projects due to an accounting rule change that 2

lowered the threshold for projects charging capital from $1 million to $250,000. Lastly, SCE recorded a 3

lower amount of non-labor expense than expected as a result of performing less Organizational Change 4

Management (OCM) and sharing the OCM costs with SCE’s business lines in some cases. 5

D. Scope and Forecast Analysis 6

1. Historical Variance Analysis 7

Figure III-8 Technology Delivery Recorded /Forecast

(Constant $000)

a) Labor 8

Labor expenses have declined over the 2014-2018 period. In 2015, labor 9

decreased significantly by 53 percent as a result of IT’s shift to its current MSP model, which included 10

the transition of work previously performed by SCE labor, such as application development and 11

maintenance, to the MSP. Costs remained flat from 2015-2016, and then decreased again by 12

approximately $5.6 million in 2017 as a result of a reorganization of the IT departments into the Plan, 13

Deliver, Run model described in Chapter I of this Volume. This involved some labor shifts from 14

Technology Delivery to other IT Departments, such as Service Management Office & Operations 15

20

(“Run”), and to strategic initiatives such as Grid Modernization and CSRP. Further, in late 2017, SCE 1

began capitalizing more projects previously charged as O&M expense as a result of a change in SCE’s 2

accounting threshold that lowered the capitalization threshold from $1 million to $250,000. This change 3

in accounting policy was the primary driver to the further reduction of labor expenses to $3.2 million in 4

2018. 5

b) Non-Labor 6

Non-labor costs increased from approximately $14.4 million in 2014 to $15.0 7

million in 2015 as the IT OU began absorbing work associated with smaller O&M projects that had 8

previously been executed and funded by SCE’s business lines prior to the centralization of the IT 9

function in 2013. Costs steadily decreased from 2015 through 2018 as SCE reduced contract resources 10

for OCM activities and shared some OCM expenses with the business lines requesting the respective 11

technology project. In addition, the 2017 change in accounting practice related to lowering the threshold 12

of capitalized software projects from $1 million to $250,000 contributed to decreasing non-labor costs 13

each year through 2018. 14

2. Basis for Forecast 15

Table III-1 provides a breakdown of the 2021 O&M Test Year forecast. 16

Table III-1 Composition of Technology Delivery Forecast

(Constant $000)

a) Labor 17

As described in Table III-1, SCE’s labor expenses have consistently declined over 18

time as a result of the transition of work previously performed by SCE personnel to MSPs, internal re-19

organizations, and beginning to charge more project work to capital versus expense. Consistent with 20

prior Commission guidance to use the last recorded year of a downward trend as an estimating 21

21

methodology, SCE uses 2018 recorded expenses of $3.175 million as the basis to estimate the 2021 Test 1

Year labor forecast in this activity. Since the impact of the 2016 change in capitalization threshold was 2

fully accounted for in the base year, the base year is a reasonable reflection for SCE’s 2021 Test Year 3

forecast. To the base level, SCE forecasts an increase of $0.4 1.6 million to account for 14 positions that 4

were vacant in 2018 but will be filled in November 2021 as a result of SCE staff returning to 5

Technology Delivery after the implementation of CSRP. These resources will provide project 6

management support for enterprise capitalized software projects. The incremental labor will also support 7

projects with high OCM needs and costs associated with decommissioning systems which cannot be 8

capitalized. The forecast is then normalized for ratemaking purposes to reflect a slight increase over the 9

2021-2023 period, resulting in a 2021 Test Year forecast of $4.330 million.22 10

b) Non-Labor 11

Non-labor costs for this activity have fluctuated as they can vary depending on 12

either the number or complexity of IT projects being actively managed. Non-labor costs increased from 13

2014-2015 and subsequently decreased through 2018 as SCE delivered projects that required lower 14

OCM expenses and capitalized more projects as a result of the change in capitalization threshold. While 15

use of the last recorded year would typically be an appropriate forecast methodology when there is a 16

downward trend, SCE selected an itemized forecast in this case due to the varying types of costs that are 17

contained in this GRC activity. As shown in Table III-1, SCE’s 2021 Test Year non-labor forecast of 18

$6.859 million consists of the following three components: (1) Base Non-Labor Expenses (2) the capital 19

-related expense for the HR Re-Platform capital project; and (3) Miscellaneous O&M projects under $1 20

million. The forecast is then normalized for ratemaking purposes to account for the expected decrease in 21

non-labor expenses in 2022-2023.23 See below for additional discussion on each of these components. 22

(1) Base Non-Labor Expenses 23

This category of the forecast includes employee-related expenses such as 24

training and office supplies for SCE labor recorded to the Technology Delivery activity. It also includes 25

capital-related expenses incurred for OU capitalized software projects such as OCM material and 26

22 Normalization calculation derived by 2021 L forecast + 2022 L forecast + 2023 L forecast/3= ($ 3.598 4.822

million + $ 4.920 million + $ 4.471 million)/3 = $ 4.330 million.

23 Normalization calculation derived by 2021 NL forecast + 2022 NL forecast + 2023 NL forecast/3= ($7.547 million+ $6.925 million + $6.103 million)/3 = $6.859 million.

22

contract or third-party service providers who have subject matter expertise with the industry or 1

technology-specific knowledge in implementing projects. Historically, SCE incurs roughly 4% of 2

capital-related expenses relative to SCE’s capitalized software portfolio. SCE applies the 4% to the $110 3

million Test Year OU capitalized software portfolio SCE forecasts in SCE-06, Volume 1, Part 2.24 4

Adding these capital-related non-labor expenses to the employee-related expenses results in a base non-5

labor forecast of $3.420 million.25 6

(2) Miscellaneous O&M Projects Less Than $1 million 7

This component of the non-labor forecast is $2.313 million and consists of 8

smaller O&M projects, typically involving minor enhancements to existing software or installation of 9

new products when SCE does not have the software capability within our existing portfolio. Examples 10

of the eight O&M projects in this rate case period include Supply Management Contract Authoring and 11

Exact Target Replacement, which is an internal communications solution. SCE’s workpapers contain an 12

itemized list of the O&M projects that result in SCE’s 2021 forecast of $2.313 million for miscellaneous 13

O&M projects.2614

24 See Chapter IX of SCE-06, Volume 1, Pt. 2.

25 See WP SCE-06, Volume 1, Pt. 1, pp. 15-19.

26 See WP SCE-06, Volume 1, Pt. 1, pp. 20-21.

23

IV. 1

SERVICE MANAGEMENT OFFICE & OPERATIONS 2

SCE’s Service Management Office and Operations (SMOO) is the organization within IT that 3

oversees software application development and maintenance; plans, designs, builds and operates IT 4

infrastructure; manages and delivers End User Computing services; manages IT services delivery and 5

testing; and oversees the MSPs. SMOO also oversees process management of the services that are 6

delivered via the service management framework that the IT Department follows. SMOO is responsible 7

for three work activities: (1) Fixed Price Technology & Maintenance; (2) Software Maintenance & 8

Replacement; and (3) Technology Infrastructure Maintenance & Replacement. 9

A. Fixed Price Technology & Maintenance 10

Figure IV-9 provides 2014-2018 recorded and 2019-2021 forecast O&M expenses, broken down 11

by labor and non-labor, for the Fixed Price Technology & Maintenance work activity. 12

24

Figure IV-9 Fixed Price Technology & Maintenance

Recorded / Forecast27 (Constant $000)

1. Work Description 1

The Fixed Price Technology and Maintenance work activity is responsible for IT services 2

provided primarily by two incumbent MSPs. As discussed above, SCE began the transition to a 3

Managed Services model in 2014, which leverages the expertise of service providers to manage many of 4

our day-to-day IT functions. Formal transition of the day-to day IT functions to our MSP partners began 5

in the fourth quarter of 2014 and completed at the end of the second quarter of 2015. During this time, 6

the following functions were fully transitioned to our MSPs: support/development/testing for 800 7

applications, management of three enterprise data centers (including computer and communication 8

infrastructure), support and maintenance for the customer service system (CSS) mainframe (application 9

and infrastructure), all IT service management functions (ITSM), 7 X 24 Service Desk, and support and 10

maintenance for 16,000 end user laptops and desktops. SCE’s MSPs now perform these work functions, 11

which were previously staffed by a combination of third parties and internal SCE personnel and will 12

continue to do so in the Test Year 2021. This activity also includes three related SCE labor functions: 13

27 See WPSCE-06V01P01 pp. 22 – 28.

25

IT Service Management (ITSM) – This function focuses on governance of key IT 1

processes that are heavily used by the SCE end-user base and our MSPs. These processes are Incident, 2

Problem, Major Incident, Change and Release, Testing, and Audits. These key processes are integral in 3

the day-to-day operations of IT and therefore require a dedicated SCE staff to oversee and govern the 4

performance and continual improvement of these processes. 5

Sourcing – This function focuses on the planning and contracting for all project work 6

outside the fixed MSP scope of work. This activity includes development of project contractual 7

obligations and the development of Statements of Work (SOW) for projects across all SCE OUs. 8

Service provider management office (SPMO) – This function is dedicated to governing 9

the contractual performance and compliance of the MSPs. This includes Service Level Agreement 10

(SLA) adherence, root cause analysis, contract compliance, financial management, performance 11

management, and continuous improvement. This group is staffed with both dedicated and matrixed SCE 12

labor to manage each of the MSP towers: infrastructure, SAP application and maintenance, non-SAP 13

applications and maintenance, end user computing, Service Desk, and IT service management. 14

2. Need for Activity 15

The Fixed Price Technology and Maintenance activity is necessary to perform day-to-day 16

IT functions for the entire IT enterprise. SCE benefits from this model as it provides a standardized 17

service delivery and performance approach that is also flexible and scalable to respond to SCE’s 18

dynamic business needs. Because our selected MSPs have a significant level of experience in delivering 19

common and repeatable IT services and support functions, using MSPs for these activities enables SCE 20

to react faster, be more effective, and continually improve on support and service levels. This model 21

reduces operational risks, minimizes IT-related business disruption, and reduces associated business 22

disruption costs. SCE has extensive contractual terms and conditions built into our MSP partnerships to 23

provide SCE with the ability to drive performance and manage risk. SCE maintains strong governance 24

and oversight of all MSP contractual SLAs in order to maintain the committed contractual performance 25

levels. This allows SCE to shift risk away from SCE and our customers. Ultimately, this model enables 26

SCE to achieve a lower cost structure for IT services and to continue to mature IT operating capabilities. 27

Savings achieved through combining all these operational functions under two MSPs provides 28

economies of scale enough to partially absorb the growing demand for IT services and technologies. For 29

instance, we have been able to support large strategic programs such as Grid Modernization, CSRP, and 30

Wildfire Mitigation activities at a reasonable cost by using the preset pricing provisions in our contract. 31

26

Otherwise, if we had to resource and support these new activities outside our MSP structure, we would 1

have incurred a substantial O&M increase. This point is reflected in SCE’s moderate test year forecast 2

increase of $6.8 million from 2018 recorded. 3

3. Comparison of Authorized 2018 to Recorded 4

SCE was authorized $77.3 million for Fixed Price Technology & Maintenance in the 5

2018 GRC. This work activity’s recorded 2018 expenses were approximately $69.5 million, which was 6

$7.9 million below authorized.28 This underspending compared to authorized was primarily savings 7

incurred through prudent negotiations and planned work in Application Development and Maintenance 8

that did not materialize but is still anticipated to be performed in 2019 and beyond. 9

4. Scope and Forecast Analysis 10

Table IV-2 Fixed Price Technology & Maintenance

(Constant $000)

a) Historical Variance Analysis 11

(1) Labor 12

As shown in Table IV-2, historical labor for Fixed Price Technology & 13

Maintenance increased from $617,000 to $11.0 million from 2014 to 2015. This increase is attributable 14

to the SCE cost of the MSP transition, which primarily took place in 2015. Labor costs then decreased in 15

2016 to $2.8 million, which reflects the end of the MSP transition. Since then, labor has remained 16

relatively flat in years 2016-2018. 17

(2) Non-Labor 18

As shown in Table IV-2, non-labor costs significantly increased from 19

2014-2015 which was the period in which we transitioned from our existing model to the MSP model. 20

From 2015 to 2017, non-labor cost steadily increased which was anticipated and consistent with the 21


27

MSP contract schedules. In 2018, non-labor costs decreased by approximately $5.5 million, which is 1

primarily attributable to the delayed implementation of two major programs: Grid Modernization and 2

Digital Managed Services. 3

b) Basis for O&M Forecast 4

(1) Labor 5

As shown in Table IV-2, SCE’s 2016-2018 recorded expenses have been 6

relatively stable, reflecting the stabilization of SCE’s MSP transition. As in 2018, SCE will continue to 7

require SCE labor to perform oversite and governance activities related to the MSPs in 2021. Therefore, 8

SCE utilizes the last recorded year estimating methodology to form the basis of our test year forecast. 9

This is also consistent with Commission guidance to use last recorded year when historical recorded 10

costs exhibit a downward trend or are relatively stable for three or more years. From the 2018 recorded 11

labor amount of $2.8 million, SCE forecasts a minor increase of $200,000, which results in a Test Year 12

2021 forecast of $3.0 million.29 The forecast increase is necessary to accommodate additional labor for 13

the purpose of strengthening the MSP governance model for the major programs (e.g., Grid 14

Modernization and Digital Managed Services) that will begin being supported by our MSPs in years 15

2019 and beyond. 16

(2) Non-Labor 17

As shown in Table IV-2, the non-labor forecast for the 2021 Test Year is 18

$73.6 million and is based upon the agreed contractual MSP pricing that will be required in order to 19

support our operations. The forecast increase from recorded year 2018 to 2021 of $7.0 million is 20

attributable to a forecasted increase in MSP costs to provide operational support for major programs 21

such as Digital Managed Services and Grid Modernization, as well as operational support for scope 22

additions attributable to smaller projects that will also be moving into production. The amount 23

attributable to the individual MSP service areas impacted by these additions is further defined in the 24

detailed work papers.30 In alignment with the revised schedule to implement CSRP in early 2021, the 25

29 Note that SCE’s labor forecast increase reflects the absence of MSP expenses that would have been necessary

in 2022 and 2023 to support incremental mainframe maintenance and infrastructure needs if CSRP had not been implemented.

30 See WPSCE-06V01P01 pp. 29 – 32

28

amended 2021 forecast also reflects a $0.2 million increase over 2018 recorded to accommodate 1

incremental services to support the legacy CSS system. 2

B. Software Maintenance & Replacement 3

Figure IV-10 and Table IV-3 provide 2014-2018 recorded and 2019-2021 forecast O&M 4

expenses, broken down by labor and non-labor, for the Software Maintenance & Replacement work 5

activity and by sub-work activities.31 6

Figure IV-10 Software Maintenance & Replacement- O&M


31 See WPSCE-06V01P01 pp. 33 – 39.

32 Figure IV-10 provides the 2021 Test Year forecast after normalizing the 2022 and 2023 forecast fluctuations for ratemaking purposes. Table IV-3 provides the forecast prior to making this adjustment.

29

Table IV-3 Software Maintenance & Replacement- O&M by Sub Work Activity


Figure IV-11 and Table IV-4 provide 2014-2018 recorded and 2019-2023 forecast capital 1

expenditures for the Software Maintenance & Replacement work activity and by sub-work activities. 2

30

Figure IV-11 Software Maintenance & Replacement

Capital Recorded / Forecast (Nominal $000)

Table IV-4 Software Maintenance & Replacement

Capital by Sub Work Activities Recorded / Forecast (Nominal $000)

1. Overview of Work Activity 1

The Software Maintenance and Replacement work activity includes costs required to 2

maintain SCE’s operating software assets through on-premise license, cloud, subscription, and 3

maintenance contract agreements. This also includes refresh of the core Operating Software comprised 4

of operating systems, business intelligence systems, database management systems, cross-system 5

31

integration tools, IT monitoring tools, and end-user productivity and collaboration software which 1

enable business applications to take advantage of the underlying hardware features and functions. Based 2

on the United Nations Standard Products and Services Code categorization,33 our software portfolio is 3

categorized into 110 different categories comprised of over 2,950 different commercial products. Lastly, 4

this work activity includes application refresh activities, which consist of the management, upgrade, 5

maintenance, optimization, monitoring, and testing of about 800 existing IT applications and more than 6

3,000 interfaces through their lifecycles. 7

This work is divided into four sub-work activities: (1) Perpetual License; (2) Software as 8

a Service (SaaS); (3) Cloud; and (4) Application Refresh. Perpetual License, SaaS, and Cloud are 9

written together below due to their interrelationship, with Application Refresh following. 10

On-going maintenance and refresh of operating software are essential activities in order 11

to sustain business applications and systems (e.g., monitoring/alerting utility operations, outage 12

communication to customers, usage billing and reporting, and weather and safety considerations). 13

Overall, the forecast for this work activity includes $196.97 million (nominal dollars) for 2019-2023 14

capital expenditures. The calendar year 2021 O&M forecast is $89.59 million, which has been 15

normalized to a test year request of $97.25million (constant 2018 dollars). 16

2. Comparison of Authorized 2018 to Recorded- O&M 17

SCE was authorized $61.5 million in O&M expenses for Software Maintenance & 18

Replacement in the 2018 GRC. This work activity’s recorded 2018 O&M expenses were approximately 19

$73.2 million, which was $11.7 million above authorized.34 20

This overspending is due to two primary reasons: (1) increased maintenance costs in 21

support of projects primarily because of changes in software decommissioning plans and increases in 22

project scope which required additional maintenance support (Teradata, Success Factors, CA Inc.), and 23

(2) increased staffing from other IT departments as part of the 2017 re-organization. 24

3. Comparison of Authorized 2018 to Recorded- Capital 25

SCE was authorized $11.4 million in capital expenditures for Software Maintenance & 26

Replacement in the 2018 GRC. This work activity’s recorded 2018 capital expenditures were 27

33 See http://www.ahrmm.org/advocacy/issues-topics/data-standards/unspsc.shtml.


32

approximately $65.9 million, which was $54.5 million above authorized.35 This spending over 1

authorized is primarily attributable to executing a major restructuring of our SAP portfolio36 at a cost of 2

$47 million. Within this re-structure, SCE purchased a completely new set of replacement products 3

including 5 years of maintenance. This new enterprise deal includes rights to grow our use of the 4

products during the 5-year agreement at no additional cost and supports and strengthens our prior 2015 5

investment in SAP’s HANA technology. This purchase was advantageous as it included additional SAP 6

Financial suite and Tax Compliance products, additional HANA licenses, Master Data Governance 7

software, Landscape Management software, and Geo Framework software. This was a strategic spend 8

premised on a conversion from the legacy SAP software products to a new product line functionally, 9

technically, and architecturally superior to the legacy SAP products. The spending over authorized is 10

also attributable to necessary spend to upgrade applications to versions that vendors will support. As 11

applications neared end of life, the refresh of these applications became critical in order to prevent 12

technology obsolescence and to mitigate business impact to customers by ensuring application 13

availability and reliability. 14

4. Sub-Work Activities 15

This activity consists of four sub-work activities: (1) Perpetual License; (2) Software as a 16

Service (SaaS); (3) Cloud; and (4) Application Refresh. Perpetual License, SaaS, and Cloud are written 17

together below due to their interrelationship, with Application Refresh following. 18

a) Perpetual License, Software as a Service (SaaS), Cloud (Subscription Based 19

Software) 20

(1) Work Description 21

The Perpetual License, SaaS, and Cloud non-project related sub-activities 22

include maintenance of SCE’s existing IT operating software assets through on-premise license, cloud, 23

subscription, and maintenance agreements. This, in turn, supports our OUs to ensure maintenance of 24

their business applications. SCE manages over 460 contractual agreements with over 250 publishers 25

which include: 26

Access to subscription SaaS offerings; 27

Access to externally hosted (Cloud) computing platforms; 28


36 See WPSCE-06V01P01C pp. 1 – 2.

33

Access to break/fix support to ensure we are running error free 1

programs; 2

Version upgrades to ensure software currency and security; and 3

Access to vendor updates to provide security updates to protect against 4

cyber threats. 5

A significant portion of the above agreements is key to the support of the 6

migration to cloud. 7

Specific examples of operating software for which SCE has support 8

agreements ensuring the business services mentioned above are delivered include: 9

Microsoft Server and Desktop Operating Systems and Database 10

software; 11

RedHat and SUSE Linux Operating Systems; 12

IBM Mainframe Operating System, Database and Middleware; 13

Oracle Database Software; 14

VMware virtualization Software; 15

DevOps development and orchestration software; and 16

SAP S/4HANA for complex in-memory data analytics. 17

Some examples of the business applications supported by the previously 18

mentioned licensing, subscription, and maintenance agreements are: 19

The SAP suite of enterprise applications used for Customer Service 20

and Billing, Financials, Human Resources, and work management; 21

The Microsoft Suite of O365 products providing collaboration and 22

productivity tooling (Outlook, Word, Excel, PowerPoint, and 23

SharePoint), as well as InTune for Mobile Data Management and Data 24

Loss Preventions for mobile devices; 25

Success Factors cloud services for Human Resources applications; and 26

Autodesk subscription service for managing enterprise engineering 27

drawings and related services. 28

Finally, this sub-work activity also includes refresh of the core Operating 29

Software made up of operating systems, business intelligence systems, database management systems, 30

cross-system integration tools, IT monitoring tools, and end-user productivity and collaboration software 31

34

which enable business applications to take advantage of the underlying hardware features and functions. 1

Unlike projects talked about in SCE-06, Volume 1, Pt. 2, these refreshes are generally cyclical/routine, 2

small enough where they would not be delivered by a project team and do not provide new OU 3

capabilities. 4

(2) Need for Activity 5

Most IT software license management is centralized in this activity for 6

technical, legal, and financial oversight and governance. The regular renewal of vendor support for our 7

perpetually-licensed products, as well as renewal of cloud and subscription agreements for our cloud and 8

SaaS, is necessary to ensure both vendor availability to respond in a timely fashion to critical system 9

failures as well as uninterrupted access to cloud and SaaS offerings. Of equal importance is the on-going 10

maintenance support facilitated by these vendors’ agreements to provide security patches and system 11

capability updates. The maintenance support costs here are summarized in the O&M cost section of this 12

testimony. If SCE does not implement the maintenance inclusive of critical security patches, the security 13

of customer data and critical system infrastructure could be at risk. This activity also monitors system 14

capacity and asset lifecycles (obsolescence), reducing risk to system reliability and business 15

productivity. Centralizing this activity in IT helps us provide consistent service terms with the vendors 16

and helps to minimize duplication of tools or services. 17

Operating System and other COTS software products provide a stable and 18

reliable foundational platform for SCE’s critical business systems, and are required for the operation of 19

mainframe servers, midrange servers, storage devices, customer call centers, and personal computers. 20

These software products also provide automated collaboration and communications capabilities, 21

automated Service Desk and systems management, and Data Center Infrastructure Management 22

systems. As is the case with other IT assets, periodically these software products require full 23

replacement as they reach the end of their useful life and have been fully depreciated. To provide 24

optimal protection against security breaches, make use of improved features and functional 25

improvements, and to minimize our O&M expenses, we plan to replace a number of these aging 26

systems. Our plans for this GRC period include, but are not limited to, the capital expenditures needed to 27

refresh the applications below that would otherwise become incremental O&M expenses if not 28

refreshed: 29

35

Microsoft Operating, Database, and End-user Productivity and 1

Collaboration products in support of end user computing feature and 2

functionality; 3

BMC Service Desk, HR Case Management, System Monitoring, and 4

Datacenter Infrastructure Management products in order to 5

maintain/sustain business application serviceability and reliability; 6

Oracle Database products as the repository for, and management of, 7

data needed by enterprise business applications, which provide 8

electricity interval usage data collection, processing, storage, and 9

outage management; 10

An integrated application performance management product to replace 11

multiple individual products; and 12

NetApp system and data backup software in order to improve and 13

maintain application resiliency and reliability. 14

(3) Scope and Forecast Analysis 15

Table IV-5 provides 2014-2018 recorded and 2019-2023 forecast O&M 16

expenses, broken down by labor & non-labor prior to being normalized for ratemaking purposes, for 17

Perpetual License, Software as a Service (Saas), and Cloud sub-work activities. 18

Table IV-5 Perpetual License, Software as a Services (SaaS), Cloud

O&M Recorded / Forecasts (Constant $000)

Table IV-6 provides 2014-2018 recorded and 2019-2023 forecast capital 19

expenditures for the Perpetual License, Software as a Services (Saas), and Cloud sub-work activities. 20

36


Capital Recorded / Forecasts (Nominal $000)

(a) Historical Variance Analysis 1

(i) Labor (Perpetual License, Software as a Service (SaaS), 2

Cloud) 3

From 2014 to 2015, SCE IT transitioned into the Managed 4

Services model wherein most of the technical work is performed by MSPs. As a result, O&M labor costs 5

decreased. From 2015 to 2018, costs continued to decrease due to this transition. 6

(ii) Non-Labor (Perpetual License, Software as a Service 7

(SaaS), Cloud) 8

The non-labor fluctuations in this account between 2014 9

and 2018 are due to changes in various license and maintenance agreements. Consistent with SCE’s 10

prior GRCs, the number and cost of license renewals varying from year to year is driven by licenses 11

rolling off prepaid five-year maintenance from capital to O&M, the shift from perpetual licenses plus 12

maintenance (capital) to subscription models (O&M), and the need to renew new license agreements due 13

to end of life. 14

The non-labor increase in 2015 is associated with increased 15

software costs due to, among other things, the addition of: (1) $9.0 million for software maintenance 16

costs previously covered under capital purchases (IBM); (2) $1.5 million for additional software support 17

costs for data center software (BMC and RedHat); and (3) $0.76 million for SaaS subscriptions 18

(Microsoft Azure and SAP Success Factors). 19

The 2016 increase in non-labor software costs was driven 20

by, among other things: (1) $4.4 million for mainframe subscription costs previously included in a 21

multi-year agreement; and (2) other subscription software costs including $1.16 million for SaaS 22

subscriptions (Microsoft Azure). 23

37

In 2017, a one-time accounting policy change took place, 1

which changed the amortization threshold for O&M from $250,000 to $1 million. The resulting impact 2

was an $8 million increase in O&M costs due to the acceleration of payments from 2018 into 2017. This 3

acceleration accounts for the majority of the increase from 2016 to 2017 and, in turn, the majority of the 4

2018 non-labor decrease. 5

(iii) Capital Expenditures (Perpetual License, Software as a 6

Service (SaaS), and Cloud) 7

During 2014-2018, we made significant improvements to 8

our software portfolio. We invested in new technologies, refreshed major suites of software with new 9

and improved features and functions, and restructured our Enterprise SAP portfolio. These and other 10

capital investments in our software portfolio provide the foundational computing platform for our 11

business. 12

In 2015, the investment in SAP’s HANA technology – a 13

database system that uses computer main memory instead of disk storage for better performance – at a 14

cost of $24 million contributed to the $5.3 million increase from 2014 to 2015. This transition from the 15

previously-used legacy database and business warehouse systems to HANA enables SCE to better 16

manage and analyze our ever-growing pool of data leading to more informed and timely business 17

decisions. 18

In 2016 our cost increases were driven by, among other 19

things, purchases of software from: (1) IBM - $12 million in additional licenses including 5 years of 20

maintenance; (2) Redwood - $1 million for an enterprise agreement including 5 years of maintenance; 21

and (3) BMC - $1 million for additional service management software including 5 years of maintenance. 22

In 2017, we restructured multiple transactional contracts for 23

both VMware and HP suites of products. We entered into 5-year enterprise agreements including 24

products and 5 years of maintenance at a cost of $6.2 million and $7 million, respectively. We also 25

purchased an additional $7 million in Oracle database software, $6 million for IBM, $4.1 million in 26

operating software upgrades (i.e., mainframe DB, midrange operating system, and BMC licenses) to 27

convert subscription database and tooling software to perpetual rights with 5 years of maintenance, and 28

$2 million in additional SAP products including 5 years of maintenance. 29

In 2018, we executed a major re-write and restructuring of 30

our SAP portfolio at a cost of $47 million, which contributed to the overall historical cost of $58.5 31

38

million in 2018. Within this re-structure, SCE purchased a completely new set of replacement products 1

including 5 years of maintenance. This new enterprise deal includes rights to grow our use of the 2

products during the 5-year agreement at no additional cost and supports and strengthens our prior 2015 3

investment in SAP’s HANA technology. 4

(b) Basis of Forecast 5

Table IV-7 provides 2014-2018 recorded and 2019-2023 forecast 6

O&M expenses,37 broken down by labor & non-labor prior to being normalized for ratemaking 7

purposes, for the Perpetual License, Software as a Services (Saas), and Cloud sub-work activities. 8


O&M Recorded / Forecasts (Constant $000)


capital expenditures for the Perpetual License, Software as a Services (Saas), and Cloud sub-work 10

activities. 11

37 See WPSCE-06V01P01 pp. 40 – 42.

39


Capital Recorded / Forecasts (Nominal $000)

(i) Labor (Perpetual License, Software as a Service (SaaS), & 1

Cloud) 2

There are no labor forecasts for these sub-work activities 3

due to completion of the transition to Managed Services. 4

(ii) Non-Labor (Perpetual License, Software as a Service 5

(SaaS), & Cloud)) 6

Our Test Year forecast for non-labor O&M expenses, based 7

on an itemized forecast, is $72.1 million. 38 Our non-labor software license maintenance and 8

subscriptions costs are forecasted to remain relatively flat in 2019 and 2020 with an increase in 2021 for 9

support of additional application refresh activities as a result of extending licenses required to be in 10

place until CSRP is implemented in early 2021. In 2022, SCE forecasts an increase of $5.5 million from 11

2021 largely for extension of mainframe operating software maintenance that will be required through 12

the CSRP stabilization period. As the SCE Customer Service Systems are being re-platformed by the 13

CSRP program in 2021, we will be able to decrease, then fully decommission, legacy software products 14

running on our IBM mainframe systems, as well as the mainframe computers themselves over the 2022-15

2023 period. This cost reduction, as well as other planned refresh activity for entire software product 16

suites by capital projects, will help to mitigate increasing O&M cost pressures that result from an 17

increase in cloud and SaaS solutions. In 2023, the period over which our 2018 refresh of the entire SAP 18

suite of software can be capitalized will expire and these costs will switch to O&M, which is reflected 19

within this sub-activity’s 2023 forecast. 20

38 See WPSCE-06V01P01C pp. 3 – 13 and WPSCE-06V01P01C pp.14-15.

40

Similar to historical non-labor, due to the variability in 1

software license maintenance agreements and timing of renewals, an itemized forecast best represents 2

the costs needed in 2021 and has similarly been normalized over the 2021-2023 period. 3

(iii) Capital Expenditures (Perpetual License, Software as a 4

Service (SaaS), & Cloud) 5

As mentioned previously, SCE continues to make 6

significant improvements to our software portfolio with investments in new technologies, refreshing 7

major suites of software with new and improved features, and restructuring of the portfolio. These and 8

other capital investments in our software portfolio provide the foundational computing platform for our 9

business. SCE plans to replace several end-of-life suites of software products through capital 10

replacement projects. Refreshing end-of-life, fully depreciated licenses and maintenance with new 11

replacement software licenses has the additional benefit of mitigating increases to O&M expenses. In 12

other words, if these capital expenditures were not made, then there would be an increase in O&M 13

expenditures in order to continue maintenance of these end-of-life software products. SCE’s 2021 14

calendar year O&M non-labor forecast of $68.5 million for software costs described above in Table IV-15

7 (Basis of Forecast Non-Labor) is predicated on such software licenses as follows: 16

Table IV-9 Software Licenses

Forecast Capital Expenditures (Nominal $000)

2019 – There are no plans for capital replacement 17

projects for any Perpetual License, SaaS, or Cloud 18

agreements in 2019. 19

2020 – We forecast $11 million for a refresh of our 20

Oracle technology stack portfolio. This stack is 21

primarily database software upon which a large portion 22

of SCE’s applications run. 23

2019 2020 2021 2022 2023 Totals

Software Licenses ‐$ $ 11,000 $ 36,000 $ 4,500 $ 4,000 55,500$

Forecast ($)

41

2021 – SCE’s Microsoft, Veritas backup software, and 1

Redwood Process Scheduling Enterprise Agreements 2

are due for renewal in 2021. We are planning a full 3

refresh of the agreements at a forecast of $36 million. 4

2022 – Our BMC Enterprise Agreement is up for 5

renewal in 2022. This portfolio contains our 6

infrastructure monitoring, capacity planning, and 7

service management suites. We are planning a complete 8

refresh of the portfolio at an estimated cost of $4.5 9

million. 10

2023 – SCE forecasts $4 million in capital expenditures 11

in 2023 for a refresh of our HP Performance and 12

Testing suite of products. 13

The 2019-2023 capital expenditures forecast for Perpetual 14

License, Saas, and Cloud also includes forecast expenditures of $45 million for Operating Software and 15

Middleware, which are focused on the efforts for upgrading, configuring, and testing of operating 16

software tools as follows: 17

Table IV-10 Operating Software and Middleware

Forecast Capital Expenditures (Nominal $000)

2019 – Major efforts include migration of Flexpod 18

platform to HCI, ASA network firewall to Palo Alto, 19

Citrix upgrade and migration, and upgrade of Windows 20

2008 operating system. 21

2020 – SCE forecasts $8 million for work around the 22

upgrade of Redhat and SUSE OS, MDMS migration to 23

2019 2020 2021 2022 2023 Totals

Operating Software & Middleware 12,911$ $ 8,000 $ 7,200 $ 8,800 $ 8,000 44,911$

Forecast ($)

42

new Exadata platform, and continuation of the 1

Windows 2008 upgrade. 2

2021 – Upgrade efforts are forecast to continue with 3

Windows 2008, Redhat and SUSE OS, as well as the 4

initial migration of VMware software to Acropolis 5

technology. 6

2022 – SCE anticipates the migration of database 7

software to new Exadata environment, optimization of 8

dedicated backup infrastructure, and the upgrade of the 9

analytic Hadoop platform. 10

2023 – SCE forecasts $8 million in capital expenditures 11

in 2023 for a refresh of existing operating software 12

systems, and continuous migration to the Acropolis 13

technology. 14

SCE’s standard is to maintain software at a version level 15

not less than one version level below N-1 where N is the current production level. This ensures the 16

software is current on bug fixes and general code maintenance as required to mitigate cybersecurity 17

exposures and support SCE business operational needs. As software deviates from this, it is refreshed 18

accordingly. The expenditures in the forecast period will replace end-of-life operating system software, 19

database systems, middleware tools, security solutions, and environment monitoring and alerting 20

applications.39 21

b) Application Refresh 22


The Application Refresh sub-work activity consists of the management, 24

upgrade, maintenance, optimization, monitoring, and testing of about 800 existing IT applications and 25

more than 3,000 interfaces40 through their lifecycle.41 These applications collectively support a majority 26

of SCE’s business processes and capabilities, including mission critical applications that help provide 27

39 See WPSCE-06V01P01 pp. 43 – 47.

40 The specific number of applications and interfaces vary from year to year as systems are added to the production environment and others are decommissioned.

41 An application’s lifecycle spans from plan, develop, test, deploy, maintain, and eventually decommission.

43

customers with safe and reliable energy along with satisfying mandated compliance and security 1

requirements. Users of these applications include all SCE employees, multiple external vendor 2

companies, and the general public which SCE services. This activity supports both company-wide 3

enterprise technologies and outward facing interfaces. An example of applications that interface with the 4

general public include SCE.com, which is a website portal that runs on SAP ERP Central Component 5

(SAP ECC) and Hana databases. Refresh activities for SAP ECC and Hana databases are critical in 6

order to support the reliability of SCE.com, which is used by SCE customers to pay their bills, to view 7

their balance and usage, to view and report outages, and to turn on or off services. O&M and capital 8

expenditures for Application Refresh focus on work associated with maintaining existing applications 9

and interfaces. This is distinct from O&M and capital expenditures for the application refresh 10

components of IT software projects included in SCE-06, Volume 1, Pt. 2, which focus on work 11

associated with developing new applications to support the work performed in the various OUs. After 12

those new applications are transitioned to Operations from the OUs, the maintenance is covered by the 13

Application Refresh sub-work activity. Application Refresh O&M also consists of labor and non-labor 14

costs for efforts to decommission. Over the 2021-2023 forecast period, SCE forecasts an increase in 15

O&M expenses, driven primarily by an increase in C&PS costs due to expected third party costs 16

supporting the SAP platform after CSRP transition, Customer Service Application Decommissioning as 17

a result of the CSRP program, and on-going maintenance costs for OU applications that are incurred 18

after projects have been implemented. 19


Application Refresh is necessary because applications must be running on 21

vendor-supported versions in order to receive vendor support. If the applications do not run on vendor-22

supported versions, then our MSP’s technical team cannot receive vendor support when troubleshooting 23

technical issues that arise in the application. When the code is proprietary, receiving vendor support is 24

essential to troubleshoot application code as the vendor will not release source code to companies that 25

purchased and use the application. Examples of applications with proprietary code include Consolidated 26

Mobile Solution (CMS), Power Plan, Spida, Real Time Energy Metering, Utilities International Planner, 27

iVOS, VOVICI Survey, Openlink Formula Engine, and Syntellect Customer Interaction Management 28

(Syntellect CIM). 29

Application Refresh is even more critical for applications that are no 30

longer supported by vendors or are about to reach end of life. In those circumstances, issues encountered 31

44

by these applications in the production environment may take time to fix or may persist without 1

resolution. This will hamper application availability and may result in these applications not being able 2

to provide needed business functionalities. In addition, the cost to maintain these legacy and 3

unsupported systems will increase, as will the challenge of finding qualified resources to support them. 4

The Application Refresh activity provides further benefits by improving 5

system availability, stability, sustainability, and reliability. Refreshing the applications reduces the 6

impact to business operations and allows for business continuity by decreasing unplanned outages and 7

by providing the functionality needed by the business units. Moreover, not performing this work 8

increases SCE’s cybersecurity exposure and the potential loss of data or data breaches. 9

In addition, safety to the public can be impacted by not refreshing 10

applications. For example, the Medical Baseline (MBL) system allows residents with life-support 11

equipment or devices, or who have life threatening illnesses or compromised immune systems, to 12

receive allocated additional kilowatt hours at a lower rate. In MBL, the Automated Outage 13

Communication (AOC) system sends outage information to customers through SAP Process 14

Orchestration (SAP PO). This outage information is then used to generate field tickets so that SCE 15

employees can visit the MBL customers, especially in case of emergencies like outages that impact life-16

support equipment or devices. Refresh activities associated with AOC and SAP PO are critical because 17

unavailability of the applications supporting MBL can negatively impact the well-being of MBL 18

customers. 19

Consolidated Mobile Solution (CMS) – a COTS application supported by 20

the GoMocha vendor – is another example of an application whose reliability and availability are critical 21

to SCE crew safety and public safety. CMS is used by field users for the planning, restoration, 22

inspection, and yearly maintenance of the T&D electrical grid. During emergency situations like wire 23

down situations, troublemen use the distribution circuit data from CMS to quickly determine the circuit 24

configuration and then work with substation operators to quickly isolate and turn off the down wires. 25

Unavailability of CMS can negatively impact the safety of the SCE crew and public. 26

(3) Scope and Forecast Analysis 27


expenses, broken down by labor & non-labor prior to being normalized for ratemaking purposes, for the 29

Application Refresh sub-work activity. 30

45

Table IV-11 Application Refresh

O&M Labor/Non-Labor Recorded / Forecast (Constant $000)


expenditures for the Application Refresh sub-work activity. 2




(i) Labor 4

The labor costs for Application refresh represent SCE 5

employees who manage third party MSPs to upgrade, maintain, optimize, monitor, and test about 800 6

existing IT applications and more than 3,000 interfaces throughout their lifecycles. SCE employees 7

oversee the MSPs and facilitate service restoration to users of these applications and interfaces in order 8

to ensure system availability, stability, sustainability, and reliability. In addition to facilitating 9

maintenance of existing applications, SCE employees also facilitate the maintenance of new capital 10

projects after these projects are transitioned to Operations from the OUs. They also assist in the 11

maintenance of O&M refresh activities while managing the MSP and helping with application service 12

restorations. Management of MSPs and restoration of SCE application services are necessary due to the 13

criticality of the applications and services that we provide to internal SCE departments and to the public 14

for business continuity. 15

46

From 2014 to 2015, O&M labor costs decreased by 1

approximately $5 million due to SCE’s transition to the Managed Services model wherein most of the 2

technical work is performed by MSPs. The completion of the Microsoft O365 refresh work also 3

contributed to this decrease as there was a $1.7 million reduction in labor associated with that refresh. 4

From 2015 to 2018, O&M labor steadily increased due to the increased number and complexity of 5

applications that were transitioned from OU to Operations. 2017 shows an increase of $785 thousand 6

due to re-organization staffing level changes wherein FTEs were moved from Business Integration and 7

Delivery to Operations. 8

(ii) Non-Labor 9

Non-labor costs in this category are costs incurred from 10

O&M refresh activities and C&PS. The 2014 O&M non-labor expenses of $11 million were primarily 11

associated with the Microsoft O365 refresh activity, and were incurred from various vendors like 12

Microsoft, Cognizant, @Business Inc, and Smart Utility Systems LLC. The Microsoft O365 refresh 13

activity simplified and modernized the desktop office experience, integrated the desktop office, 14

telephone, and social capabilities, and was implemented in work streams. One work stream focused on 15

implementation of Exchange and O365; another work stream, implementation of Lync, Office 2013 16

client, and Internet Explorer 10; and another work stream, implementation of SharePoint. This project 17

was executed from 2014 to 2015. In 2015, non-labor expenses dropped to $1.2 million with $457,000 18

spent on Microsoft customer support for O365 refresh, on GridView annual maintenance renewal, and 19

on consulting services support for Wireless Telecom refresh. 20

Non-labor expenses then incrementally decreased to 21

$444,000 in 2016 with $148,000 due to Microsoft customer support and $139,000 due to Clicksoft 22

professional services and MSO Tech Inc. for LENS maintenance release. 23

In 2017, non-labor costs increased to $3 million due to 24

increased O&M refresh activities for Internet Explorer 11, VERINT Survey Tool, Unclaimed Property 25

Compliance Systems, and OKTA, as well as centralization of C&PS costs within this sub-work activity. 26

In 2018, there was a slight rise to $3.8 million in non-labor 27

costs due to an increase in C&PS cost associated with Power Plan support, Spinitar support services, 28

SAP Compensation Consulting services, LENS Professional Services, Substation Engineering Modeling 29

Tool (SEMT) Professional Services, and UI Planner Technical Support. 30

47

(iii) Capital Expenditures 1

Prior to 2016, a portion of the capital costs for Application 2

Refresh was tracked in the Software Asset Management (SAM) bundle, presented within prior GRCs.42 3

In 2016, the $2.8 million capital spend was due to the Nia (formerly Mana) refresh activity. In 2017, 4

capital expenditures increased to $4.8 million due to SAS Visual Analytics (VA) / Grid Upgrade, Afaria 5

and MXE Upgrade, Datastage Upgrade, SAP Hana Database Upgrade, and Nia refresh activity. 6

In 2018, capital spend rose to $7.4 million due to an 7

accumulated number of refreshes for applications whose vendor supportability ended (or will soon end) 8

and were critical to refresh in order to promote business continuity and limit operational risks. The 9

increase in capital expenditures in 2018 was also due to the refresh of certain applications to maintain 10

availability, sustainability, and reliability of the applications in the production environment. For 11

example, eWorld is a web-based application that contains SCE asset information, circuit maps, 12

substation maps, facility maps, and land bases. eWorld was highly unstable and the application was 13

unavailable almost every day due to a memory leak in the application infrastructure. As a work around, 14

the application was restarted each day to make the application available to eWorld clients. To resolve 15

the instability issue, eWorld was upgraded. After the upgrade, eWorld became stable and no longer 16

required an everyday restart. 17



O&M expenses, broken down by labor and non-labor prior to being normalized for ratemaking 20

purposes, for the Application Refresh sub-work activity. 21

42 See A.10-11-015, Exhibit SCE05, Volume 3.

48


O&M Labor/Non Labor Recorded / Forecast (Constant $000)


capital expenditures for the Application Refresh sub-work activity. 2


Capital Recorded & Forecast (Nominal $000)

(i) Labor 3

SCE’s 2021 O&M Test Year Forecast for labor is $8.7 4

million. As in 2018, SCE will continue to require SCE labor in 2021 in order to manage the MSPs, to 5

facilitate maintenance of existing applications and new applications that are transitioned from OUs to 6

Operations, and to manage O&M application refreshes. From 2016-2018, labor costs steadily increased. 7

Therefore, SCE utilizes the last year recorded estimating methodology to form the basis of our Test Year 8

Forecast. This is consistent with Commission guidance to use the last recorded year when historical 9

recorded costs exhibit a trend in one direction over three or more years. From the 2018 recorded labor 10

amount of $7.2 million, SCE forecasts an increase of $1.5 million. 11

The forecast increase is needed to accommodate additional 12

SCE FTEs to manage the projected increase in application refreshes and to manage the projected volume 13

increase in the applications transferred from OUs to Operations. In addition, there is also a forecasted 14

increase in SCE labor as SCE employees involved with the CSRP project will resume charging to 15

49

Operations in order to perform enhancements and daily operational work. There were other resources 1

that support other major programs in 2018 that will be returning to Operations, which contributes to the 2

increase in the test year. 3

When OU applications are transferred to Operations, the 4

Operational team ensure that the applications are available, stable, and reliable. The newly transferred 5

applications normally undergo a stabilization period of 6 months to 2 years depending on the size of the 6

application. 7

For further descriptions of functions, refer to the Labor 8

section Historical Variance Analysis for this sub-work activity. For detailed discussion, please see 9

SCE’s supporting work paper.43 10

(ii) Non-Labor 11

Our 2021 O&M Test Year Forecast for non-labor is $8.8 12

million, which includes C&PS, O&M refresh activities, Customer Service Application 13

Decommissioning as a result of the implementation of the CSRP program, and on-going maintenance 14

costs of OU capital software projects that transitioned from OUs to Operations. The $8.84 million Test 15

Year forecast consists of an estimated (1) $2.3 million in O&M refresh activities, (2) $0.544 million in 16

Customer Service Application decommissioning, (3) $3 million in on-going maintenance costs of 17

capitalized software projects, (4) $2.945 million in C&PS costs, (5) $55 thousand in Enterprise and 18

Transmission and Distribution Application Services, and (6) $45 thousand in Customer Service and 19

Power Supply Application Services. See Table IV-15 for a summary of the costs.46 20

43 See WPSCE-06V01P01 pp. 48 – 49.

44 At the time of filing SCE’s 2021 GRC Application in August 2019, SCE forecasted the Customer Service Application decommissioning cost of $5.0 million in 2021. SCE’s amended testimony assumes $4.5 million of this $5.0 million is deferred to 2022-2023, for a total of $0.5 million in 2021, $7.8 million in 2022, and $1.8 million in 2023.

45 As a result of the revised CSRP implementation date, SCE’s amended testimony removes $2.8 million of third party costs to support the CSRP SAP platform in 2021 , offset by the addition of minor application refresh support of $215,000 to pay extended vendor support until SCE’s CSRP freeze is lifted and the applications are upgraded appropriately. SCE now expects the $2.8 million in annual third party costs to support the CSRP SAP platform to begin in 2022.

46 See WPSCE-06V01P01 pp. 50 – 52.

50

Table IV-15 Application Refresh Activities

(Constant $000)

There is a significant forecast increase in 2022 to 19.1 1

million due to O&M refresh activities, Customer Service Application decommissioning costs, and third 2

party costs to support the SAP platform after the CSRP transition. To align with the revised plan to 3

implement CSRP in early 2021, SCE’s amended forecast assumes the majority of the Customer Service 4

Application decommissioning costs will be deferred from 2021 to 2022-2023. In addition, $2.8 million 5

of third-party costs to support the CSRP SAP platform were previously included annually beginning in 6

2021. SCE’s amended forecast removes the expense from 2021 to reflect the annual cost beginning in 7

2022. Non-labor costs are then expected to decrease to $13.0 million in 2023, largely due to completion 8

of Customer Service Application decommissioning. 9

For supporting details regarding O&M refresh activities47 10

and Customer Service Application Decommissioning,48 on-going maintenance for OU Cap Software 11

Projects,49 and C&PS increase justifications,50 see the associated work papers. 12

47 See WPSCE-06V01P01 pp. 53 – 57.

48 See WPSCE-06V01P01 pp. 58 – 59g.

49 See WPSCE-06V01P01 pp. 60 – 61 and WPSCE-06V01P01 pp. 62 – 64.

50 See WPSCE-06V01P01 pp. 65 – 67a.

51


The total forecast Application Refresh capital expenditures 2

for 2019 to 2023 are approximately $97 million. Application refresh items that are typically less than 3

$250 thousand are categorized as O&M refresh activities. Application refresh items that are $250 4

thousand or greater with the minimum expected useful life of five years are categorized as Capital 5

refresh activities.51 For each application to be refreshed, the cost is determined by complexity for each 6

stage of software refresh. Also included in the cost are the additional functionality being incorporated, 7

additional testing performed by the Testing Center of Excellence (TCoE), and additional overhead costs 8

like license costs and server procurement or upgrade. These refreshes are then categorized and 9

prioritized based on the application’s risk, urgency, and magnitude of impact. 10

Capital expenditures vary each year depending on the life 11

cycle of the application. In 2019, capital expenditures are expected to increase by $3 million. This 12

increase is primarily due to the refresh of applications to maintain vendor supportability. The 13

applications that need to be refreshed in 2019 include Power Plan and SAP Hana Database. 14

SCE forecasts that capital costs in 2020 will increase by 15

$6.3 million due to additional applications that need to be upgraded in order to maintain vendor 16

supportability. Examples of these applications driving the cost increases are CMS, SAP Governance 17

Risk and Compliance (SAP GRC), and SAP EHS (Environment, Health, and Safety) Environmental 18

Compliance. 19

The capital spend in 2021 is forecasted to slightly increase 20

by $484 thousand. Examples of applications that will need to be upgraded in 2021 include Click and 21

SAS Grid. 22

In 2022, there is a projected $5.1 million increase of capital 23

expenditures as more applications reach a period in their lifecycle wherein vendor support will be lost if 24

not upgraded to a supportable version. Examples of these applications include SAP Hana Analytics and 25

Syntellect CIM. 26

51 Refer to SCE 07 Vol. 2 for testimony regarding SCE’s accounting practices and classification of O&M versus

capital.

52

In 2023, there is a projected increase of $6.8 million. 52 1

Examples of applications being upgraded to retain vendor supportability include Power Plan and 2

Electronic Document Management and Records Management (eDMRM). 3

C. Technology Infrastructure Maintenance & Replacement 4

Figure IV-12 provides 2014-2018 recorded and 2019-2021 forecast O&M expenses, broken 5

down by labor and non-labor, for the Technology Infrastructure Maintenance & Replacement work 6

activity and its sub-work activities. Table IV-16 provides this information prior to being normalized for 7

ratemaking purposes. 8

Figure IV-12 Technology Infrastructure Maintenance & Replacement

O&M Recorded / Forecast53 (Constant $000)

52 See WPSCE-06V01P01 pp. 68 – 74.

53 See WPSCE-06V01P01 pp. 75 – 81.

53

Table IV-16 Technology Infrastructure Maintenance & Replacement

O&M Recorded / Forecast by Sub Work Activity (Constant $000)

Figure IV-13 and Table IV-17 provide 2014-2018 recorded and 2019-2023 forecast capital 1

expenditures for the Technology Infrastructure Maintenance & Replacement work activity and its sub-2

work activities. 3

Figure IV-13 Technology Infrastructure Maintenance & Replacement


54

Table IV-17 Technology Infrastructure Maintenance & Replacement


1. Overview of Work Activity 1

The Technology Infrastructure Maintenance and Replacement work activity is critical to 2

the support of the business applications and services that allow SCE to safely deliver reliable, clean, and 3

affordable energy for customers. This activity consists of three sub-work activities: (1) Data Center 4

Infrastructure; (2) End User Computing Maintenance, Services & Replacement; and (3) Technology 5

Adoption. 6

Data Center Infrastructure covers the compute, storage, and network infrastructure 7

housed in three SCE enterprise data centers: Alhambra Data Center (ADC), Irvine Operations Center 8

(IOC), and Rancho Customer Contact Center (RCCC). This sub-work activity involves the procuring, 9

installing, and maintenance of all enterprise data center hardware infrastructure. This infrastructure is 10

inclusive of a mainframe platform (primary and disaster recovery), over 7,500 midrange servers (UNIX, 11

Linux, and Wintel), over 2,000 terabytes of data storage, 700 data network routing and switching 12

infrastructure inclusive of copper and fiber-optic cabling, and 400 appliances to support over 500 large 13

data repository solutions. 14

The End User Computing Maintenance, Services & Replacement sub-work activity 15

covers the performance management of SCE’s Service Desk and maintenance of devices. SCE’s Service 16

Desk handles approximately 17,000 calls per month. SCE has a system to funnel calls into different tiers 17

based on the type and complexity of the issue. The Service Desk also resolves about 204,000 service 18

tickets per year (about 60% are restoration services, while 40% are service requests). SCE employs 19

certain SLAs that govern response times for service requests and service restoration incidents. For 20

example, the Service Desk must respond to 98% of electronic requests within 20 minutes from the time 21

of submission. In addition, End User Computing Maintenance, Services and Replacement includes 22

management of approximately 7,500 smart phone plans, 1,500 tablet cellular data and Apple care, 4,500 23

55

air cards, 1,256 printers, 225 plotters, 16,000 laptops and desktops, 108 teleconference rooms with AV 1

equipment across the company, and 19 Mauell Walls (monitors). 2

Lastly, the Technology Adoption sub-work activity consists of the retirement of 3

computer, storage, network, and operating software assets and the replacement of these assets with 4

hardware and operating software that may be more operationally efficient with improved price 5

performance to leverage new and emerging technologies such as the cloud. As SCE continues its 6

adoption of cloud technologies utilizing the Microsoft Azure Cloud computing platform and services, 7

the focus over the 2019-2023 time frame will be in the following areas: 8

Reliability and Business Resiliency 9

Backup and Recovery 10

Operational Reliability 11

Security 12

Identity Management 13

Overall, for Technology Infrastructure Maintenance and Replacement, SCE forecasts 14

2019-2023 capital expenditures of $340.84 million and test year 2021 O&M expenses of $21.78 million. 15

2. Comparison of Authorized 2018 to Recorded- O&M 16

SCE was authorized $14.0 million in O&M expenses for Technology Infrastructure 17

Maintenance & Replacement in the 2018 GRC. This work activity’s recorded 2018 O&M expenses were 18

approximately $26.7 million, which was $12.6 million above authorized.54 19

This spending above authorized was primarily due to two reasons: (1) an accounting 20

methodology change for IT products and services wherein O&M costs that previously were billed to 21

SCE OUs for these products and services are now recording directly to IT O&M, and (2) a change in 22

capitalization rules for hardware maintenance resulting in a transfer of $3.9 million from capital to 23

O&M cost accounting. 24

3. Comparison of Authorized 2018 to Recorded- Capital 25

SCE was authorized $52.4 million in capital expenditures for Technology Infrastructure 26

Maintenance & Replacement in the 2018 GRC. This work activity’s recorded 2018 capital expenditures 27

were approximately $52.2 million, which was only $0.2 million below authorized, which is within 28

normal operating expectations. 29


56

4. Sub-Work Activities 1

a) Data Center Infrastructure 2


The Data Center Infrastructure sub-work activity is comprised of activities 4

related to refreshing and maintaining the compute, storage, and network infrastructure housed in the 5

three enterprise data centers: Alhambra Data Center (ADC), Irvine Operations Center (IOC), and 6

Rancho Customer Contact Center (RCCC).55 These activities, typically deemed as capital expenditures, 7

fall into the following categories: (1) Mainframe Replacement; (2) Server Replacement; (3) Storage 8

Replacement; (4) Data Center Network Replacement; (5) Appliance Replacement; and (6) Organic 9

Growth. 10

(a) Mainframe/Server/Storage Replacement 11

For our mainframe and server environment, SCE leverages a 12

virtualization architecture to deliver critical business functions for our customers. This environment 13

enables us to run multiple applications on a shared physical server environment. For data storage, SCE 14

utilizes a storage architecture providing discrete storage devices supporting enterprise systems. Routine 15

refresh of the mainframe, midrange server, and associated data storage infrastructure is encompassed in 16

this category. 17

(b) Data Center Network Replacement 18

As SCE continues to focus on building a more resilient network 19

infrastructure both for internal business operations as well as to serve our customers, we will have to 20

adopt technologies to meet this objective. The data center network must be maintained in alignment with 21

the wide area network (WAN) infrastructure discussed in Exhibit SCE-02, Volume 4 to support internal 22

SCE business operations and communications as well as the internet to support customer services and 23

communications with the appropriate external organizations. This category includes the replacement of 24

the network infrastructure components needed to accommodate client communications needs throughout 25

the enterprise. 26

(c) Appliance Replacement 27

SCE’s IT infrastructure leverages the “appliance model” for large 28

data processing systems such as data warehouse solutions, various database systems, and specialized 29

55 The two co-primary production data centers, ADC and IOC, provide disaster recovery (DR) for one another

while IOC provides DR for RCCC.

57

solutions. An “appliance” combines individual hardware (e.g., servers, storage) and software assets into 1

a single device performing previously-discrete functions in an operationally simplified manner, resulting 2

in operational efficiencies and improved price/performance. The “Appliance Replacement” category 3

refers to the refresh activities of these appliances, including refresh of the hardware and associated 4

operating software bundled in the appliance platform maintained under the same 5-year lifecycle as with 5

other data center infrastructure components. This is a new hardware infrastructure category for the 2021 6

GRC. 7

In previous years, the appliance replacement budget was part of the 8

“Server Replacement” category. For 2020 and onward, the replacement of appliance-type solutions will 9

be a separate category. For some servers, both server and storage functional capabilities along with 10

supporting operational software will be combined in “appliances,” realizing operational efficiencies not 11

previously available through separate server and storage devices. Given the technological differences, 12

SCE deemed it best to track the costs of appliances separately. 13

(d) Organic Growth 14

The growth of SCE’s IT hardware infrastructure supporting 15

business applications is driven by the number of users, peak concurrent usage, functionality in the 16

application, and the data volume generated and stored according to SCE’s data retention policies. This 17

category addresses replacement of infrastructure supporting mission-critical, as well as other business, 18

applications. This integrated infrastructure landscape must meet the growth in usage of SCE business 19

applications required to run our day-to-day operations. Retaining data for legal and business 20

requirements is the most significant business driver of data growth. For example, for supply chain 21

management, all purchase orders must be maintained for 5 years after the purchase order expires. 22

Similarly, all SCE financial data stored in the Business Warehouse is retained for 7 years. 23

This need for the organic growth capacity is in addition to the base 24

level of capability provided by routine hardware replacement. Additional growth capacity needs may 25

include any of the following infrastructure components: 26

Servers 27

Storage hardware 28

Appliances 29

Data center network communications equipment and other 30

datacenter facility equipment 31

58

Disaster recovery infrastructure 1

Operating software for the above components 2

Our methodology for determining the growth in capacity needed is 3

driven by the natural growth of processing, data storage, and associated internal SCE network 4

communications resulting from increased transactional activity and the associated data retention 5

requirements. Our transactional activity has driven our data storage usage growth for the 2016-2019 6

timeframe year over year by 8% and 12% for our Network Attached Storage and Storage Area Network 7

(SAN) infrastructure, respectively. Major contributors to this data storage growth have been our data 8

warehousing and reporting platforms like the SAP Business Warehouse and Hadoop. Averaging this 9

data usage growth over this timeframe yields an average growth trend of 10% per year.56 10

The infrastructure hardware mentioned above represent the types 11

of capital expenditures in this sub-work activity. The corresponding O&M expenses in this sub-activity 12

consist of SCE staff to manage performance of MSPs performing acquisition, configuration, and 13

installation of infrastructure hardware/software, as well as troubleshooting activities. It also consists of 14

expenses necessary to maintain the IT infrastructure hardware within SCE’s production data centers and 15

which are provided through support agreements with the respective hardware vendors. The capitalized 16

hardware replacements benefit from purchasing prepaid maintenance agreements, typically over five 17

years. After the five-year period ends, the O&M hardware support expenses are accumulated, tracked, 18

and reported through non-labor expenses in this account. 19


In order to mitigate risk to SCE business operations, it is important for 21

SCE to sustain our standard hardware refresh cycle. SCE uses a five-year life cycle57 as an effective and 22

operationally prudent standard to maintain IT systems reliability. This applies to all IT infrastructure 23

hardware. Refresh costs for IT infrastructure hardware are captured as capital expenditures. Historical 24

experience has shown that extending hardware beyond this five-year life cycle results in hardware more 25

56 See WPSCE-06V01P01 pp. 82 – 88.

57 Industry expert Gartner Group recommends service lifecycles should be five years. See also https://www.revolutiongroup.com/blog/how-often-should-i-replace-my-servers/ (“[W]e recommend to always follow the manufacturer’s warranty and their recommended replacement timeline. Those timelines typically vary from 3-5 years and very rarely extend past 5 years. Why? Because it becomes extremely expensive to support a server after it has been running for 5 years. Statistics show it costs 200% more to support for a server that is 5 years old or older.”).

59

prone to outages due to lack of spare parts, lack of support for operating software and firmware, and an 1

inability to consistently stay current not only on a specific hardware component but also on the 2

integration with other hardware equipment/components. This can and does have a negative effect on 3

business application functionality and reliability and, more importantly, on business services internally 4

within SCE and externally for our customers. To extend hardware beyond this five-year life cycle not 5

only increases the potential for interruptions to business operations but will also result in unnecessary IT 6

operational expenditures. The benefit to maintaining a five-year refresh life cycle is a not only 7

mitigation of the aforementioned issues but also increased performance, reliability, accessibility, and 8

serviceability. 9

(3) RAMP Integration 10

In order to reduce the risks identified in the Cybersecurity Testimony 11

included in SCE’s RAMP filing,58 the 2021 O&M forecast in this GRC activity has been adjusted after 12

the RAMP filing to include $1.0 million for additional operational activities such as deploying new, and 13

maintaining existing, IT architecture on which Cybersecurity applications reside; researching and testing 14

vendor technology to determine cybersecurity capabilities and compliance with SCE cybersecurity 15

standards; and integrating identity management into more internal applications to support the mitigation 16

activities under the Cybersecurity RAMP programs: Perimeter Defense, Interior Defense, and Data 17

Protection. Some of these activities were previously provided by OUs outside of Cybersecurity at SCE. 18

The $1.0 million amount requested was derived from an internal analysis 19

that calculated estimated risk reduction by increasing the capabilities of projects under the Cybersecurity 20

program areas by adding resources. Some of those projects required personnel outside of Cybersecurity 21

so resources from Service Management Office and Operations (SMOO) were included and allocated for 22

these projects. 23

(4) Scope & Forecast Analysis 24


expenses, broken down by labor and non-labor prior to being normalized for ratemaking purposes, for 26

the Data Center Infrastructure sub-work activity. 27

58 Cybersecurity Testimony is SCE 04- Volume 3. See Cyber Delivery sections for descriptions of the projects

mentioned in this section.

60

Table IV-18 Data Center Infrastructure


Table IV-19 below provides 2014-2018 recorded and 2019-2023 forecast 1

capital expenditures for the Data Center Infrastructure sub-work activity and further breaks these 2

expenditures down into the categories discussed in the Work Description section above. 3

Table IV-19 Data Center Infrastructure Capital Recorded / Forecast

(Nominal $000)


(i) Labor 5

From 2014-2015, there was a reduction in workforce that 6

resulted in a decline in labor costs from $19.1 million to $4.6 million as SCE made the transition to 7

MSPs. The SCE workforce for the Data Center Infrastructure sub-work activity then continued to 8

decline from 2015 through 2018 due to subsequent re-organizations of SCE IT. 9

2014 2015 2016 2017 2018 2019 2020 2021 2022 2023

Mainframe Replacement 1,943,628 6,775,287 578,085

Server Replacement 18,680,699 9,141,670 8,826,355 8,771,722 12,338,495 17,217,758 9,200,000 5,021,695 3,862,711 3,221,412

Storage Replacemet 15,438,946 7,633,637 3,685,352 15,719,934 979,184 1,010,604 6,500,000 3,316,295 2,584,419 2,132,608

Data Center Network Replacement 4,851,257 1,630,018 7,707,452 9,634,957 7,223,944 7,235,990 8,000,000 7,087,500 8,662,500 7,000,000

Appliance Replacement 14,461,250 30,169,575 10,890,675 20,723,000

Organic Growth 681,623 2,417,177 1,646,801 15,417,471 9,370,370 7,955,679 10,996,067 6,372,000 7,668,000 11,047,173

Subtotals 39,652,525 20,822,502 23,809,588 49,544,084 36,687,280 33,998,116 49,157,317 51,967,065 33,668,305 44,124,193

Totals 170,515,979 212,914,996

61

(ii) Non-Labor 1

Recorded costs during 2014-2018 experienced two peaks, 2

in 2014 and 2018. In the years 2015-2017, costs were relatively flat at around $5-6 million per year. 3

Recorded costs were higher in 2014 than 2015 because, prior to 2015, hardware and software 4

maintenance costs were tracked together. An accounting process change was implemented in 2015 to 5

separate software maintenance costs into the appropriate cost centers, which are then tracked in 6

Software Maintenance & Replacement (Perpetual License sub-work activity). In 2018, there was an 7

increase in O&M expenses due to a change in capitalization rules for hardware maintenance resulting in 8

a transfer of $3.9 million from capital to O&M cost accounting. 9


From 2014 to 2018, the recorded capital expenditures for 11

the Data Center Infrastructure sub-work activity were $170.52 million. The main drivers for the year-12

over-year variances for Data Center Infrastructure expenditures during the 2014-2018 time frame were 13

the significant decrease in server and storage replacement expenditures from 2014 to 2015 and a 14

significant increase in storage replacement and organic growth expenditures in 2017 to refresh the ADC 15

infrastructure hardware initially acquired in 2012. See below for the historical variance analysis divided 16

into the categories described in the work activities section: 17

Mainframe Replacement: Recorded expenditures of 18

$8.7 million for the 2014-2018 timeframe reflect a 19

refresh of the disaster recovery mainframe in 2016 for 20

$1.9 million and the refresh of the primary production 21

mainframe in 2018 for $6.8 million. 22

Server Replacement: SCE recorded $57.8 million for 23

2014-2018 to acquire/replace midrange server 24

hardware. These recorded expenditures include costs 25

for acquiring, building, configuring, and testing 26

midrange servers to replace existing servers at the end 27

of their useful life. The higher expenditure in 2014 is 28

attributable to SCE IT’s transition to Flexpod 29

architecture for application level computing and in-30

memory database appliances for backend data 31

62

repository. In total, $18 million was allocated for 1

procurement of new platforms for those transitions. The 2

expenditures on server refresh, utilizing our standard 3

five-year life cycle, continued for 2015-2018 as per 4

previous practice with an exception in 2018, when there 5

was an additional $4 million procurement of IBM 6

DataPower appliances for the purpose of accelerating 7

web services deployment. 8

Storage Replacement: SCE recorded $43.5 million 9

for 2014-2018 to acquire/replace storage systems. 10

These recorded expenditures include costs for 11

acquiring, building, configuring, and testing storage 12

arrays to replace existing equipment at the end of their 13

useful life. Above-normal expenditures in 2014 and into 14

2015 are attributable to the start of migration from 15

traditional Storage Area Network (SAN) storage to 16

Network Attached Storage (NAS). During this time 17

frame, SCE had to maintain both types of the storage 18

systems while refreshing legacy SAN storage. In 2017, 19

expenditures increased significantly due to the need to 20

refresh both the out-of-service IBM Shared Volume 21

Controller storage subsystems along with a refresh of 22

storage infrastructure first acquired in 2012 with the 23

initial build-out of the Alhambra Data Center. 24

Data Center Network Replacement: SCE recorded 25

$31.04 million for 2014-2018 within this category. Data 26

center network equipment refresh was minimal in the 27

2014-2015 time frame. Expenditures in 2015 were 28

lower than normal because this was the first year of our 29

transition to managed services and thus the focus was 30

on operations as opposed to equipment refresh. In 31

63

contrast, during 2016-2018, we completed two major 1

network refresh cycles encompassing both of SCE’s 2

primary production data centers. Expenditures in 2017 3

were higher than normal due to an operational need to 4

accelerate the refresh of our data center network 5

equipment infrastructure in our Irvine Operations 6

Center. 7

Appliance Replacement: This is a new category 8

created for 2020-2023. Previous recorded spend was 9

captured under the “Server Replacement” category. 10

Organic Growth: SCE recorded $29.5 million for 11

2014-2018 to meet capacity growth demand. The 12

increase in 2017 was primarily due to capacity growth 13

required to meet project requirements: $2 million for 14

Exadata; $1 million for BMC licenses and $1 million 15

for Computer Associates licenses (both of which were 16

to support a fifth mainframe processor); and $4 million 17

for the sixth mainframe processor licenses. The 18

recorded expenditures for the remaining years in the 19

period 2015-2018 included costs for acquiring, 20

building, configuring, and testing midrange servers, 21

appliances, and datacenter network components. While 22

the capacity growth line items addressed the organic 23

growth needs of the business applications, this section 24

also includes a corresponding capacity expansion for 25

purposes of risk management in order to address the 26

needed disaster recovery infrastructure to ensure the 27

proper application failover as needed in the two co-28

primary production data centers—the Alhambra Data 29

Center (ADC) and Irvine Operations Center (IOC). 30

More specifically, when additional compute, storage, 31

64

and/or network equipment was added to address 1

organic growth, a proportional or, in some cases, an 2

exact matching of infrastructure was provisioned and 3

added to the appropriate disaster recovery data center 4

location. 5



O&M expenses, broken down by labor and non-labor for the Data Center Infrastructure sub work 8

activity. 9

Table IV-20 Data Center Infrastructure O&M



capital expenditures for the Data Center Infrastructure sub-work activity and further breaks these 11

expenditures down into the categories discussed in the Work Description section above. 12

65

Table IV-21 Data Center Infrastructure Capital Recorded / Forecast

(Nominal $000)

(i) Labor 1

The 2021 test year labor forecast for this sub-work activity 2

is $5.0 million. As shown in Table IV-20, SCE’s 2016-2018 recorded expenses have trended downward. 3

Therefore, SCE utilizes the last recorded year estimating methodology to form the basis of our test year 4

forecast. This is also consistent with Commission guidance to use the last recorded year when historical 5

recorded costs exhibit a downward trend or are relatively stable for three or more years. From the 2018 6

recorded labor amount of $3.5 million, SCE forecasts an increase of $1.5 million, which results in a Test 7

Year 2021 forecast of $5.0 million. 8

This forecast increase over last year recorded is primarily 9

due to the increased staff needed to provide cybersecurity operational support. As outlined in the 10

Cybersecurity Testimony, there will be an increase in Data Center Infrastructure staff required to 11

provide operational support for programs such as Perimeter Defense, Interior Defense, and Data 12

Protection. This is reflected in the incremental labor increase forecast from 2018-2021, with staffing 13

requirements normalized in 2022-2023. The incremental budget request of $1.5 million over the 2019-14

2023 timeframe will be inclusive of $1.0 million for SCE staff to support Cybersecurity program 15

operational requirements. The remaining $0.4 million is due to CSRP resources returning to SMOO after 16

CSRP stabilization and additional service manager vacancy.59 17

59 See WPSCE-06V01P01 pp. 89 – 91.

2014 2015 2016 2017 2018 2019 2020 2021 2022 2023

Mainframe Replacement 1,943,628 6,775,287 578,085

Server Replacement 18,680,699 9,141,670 8,826,355 8,771,722 12,338,495 17,217,758 9,200,000 5,021,695 3,862,711 3,221,412

Storage Replacemet 15,438,946 7,633,637 3,685,352 15,719,934 979,184 1,010,604 6,500,000 3,316,295 2,584,419 2,132,608

Data Center Network Replacement 4,851,257 1,630,018 7,707,452 9,634,957 7,223,944 7,235,990 8,000,000 7,087,500 8,662,500 7,000,000

Appliance Replacement 14,461,250 30,169,575 10,890,675 20,723,000

Organic Growth 681,623 2,417,177 1,646,801 15,417,471 9,370,370 7,955,679 10,996,067 6,372,000 7,668,000 11,047,173

Subtotals 39,652,525 20,822,502 23,809,588 49,544,084 36,687,280 33,998,116 49,157,317 51,967,065 33,668,305 44,124,193

Totals 170,515,979 212,914,996

66

(ii) Non-Labor 1

The 2021 test year non-labor forecast for this sub-work 2

activity is $ 5.260 million. The actual spend for 2014-2018 was $49.4 million whereas the forecast spend 3

for 2019-2023 is $25.3 million. In 2014-2018, SCE implemented several capitalized hardware purchases 4

with prepaid maintenance agreements that will end during 2019-2023.61 Unless this hardware is 5

refreshed on the standard five-year refresh cycle, O&M costs are incurred. The forecast period reflects 6

relatively flat expenditures year-over-year to cover the maintenance costs for IT hardware infrastructure 7

which has been extended beyond its standard five-year refresh life cycle. The forecast reflected in this 8

GRC period is derived from an itemized list of hardware assets that go beyond the five-year refresh life 9

cycle.62 10


The forecast capital expenditures for 2019-2023 are 12

$212.91 million. Forecast expenditures for 2019 are flat compared to 2018, with significant increases in 13

appliance and storage replacement expenditures forecast in 2020 and additional increases in appliance 14

replacement expenditures forecast in 2021 and 2023. See below for the basis of this forecast broken 15

down into the Data Center Infrastructure categories described in the work activities section: 16

Mainframe Replacement: The only forecast capital 17

expenditures in the 2019-2023 time frame are for a disk 18

library for the mainframe in 2019. With implementation 19

of the CSRP program, the mainframe environment will 20

be decommissioned beginning in 2021, with much of 21

the decommissioning work occurring in years 2022-22

2023. Description of the decommissioning activities are 23

summarized in the Application Refresh63 section of the 24

60 An additional $250,000 for one-year extension of application server maintenance is reflected in the forecast to

align with the CSRP implementation in early 2021.

61 Refer to SCE 07 Vol. 2 for accounting practices.

62 Note that pre-paid hardware maintenance for 2021 will be considered capital, similar to what is reflected in the 2018 GRC testimony.

63 Refer to Section IV. B. 3. B section Application Refresh Non-Labor (O&M) for references and work papers related to Customer Service Applications Decommissioning.

67

testimony. For more detail on these calculations, refer 1

to the work papers for this section.64 2

Server Replacement: Our forecast expenditures of 3

$38.5 million for 2019-2023 include costs for 4

acquiring, building, configuring, and implementing 5

midrange servers to replace existing servers at the end 6

of their useful life. There is an increase in 2019 funding 7

for the investment in a software-based hardware 8

virtualization technology, Hyper Converged 9

Infrastructure (HCI), and for the replacement of the 10

current Flexpod technology (hardware-based 11

component integration). As mentioned previously, SCE 12

expects improved operational efficiencies with HCI 13

and, in order to better track implementation of this new 14

technology, has created the Appliance Replacement 15

sub-work activity to capture forecast expenditures 16

going forward from 2020. This transition aligns with 17

our cloud migration strategy, which will in turn 18

improve reliability and availability through innovation. 19

The expenditures in the forecast period will replace 20

midrange servers and associated converged 21

infrastructure appliances for mission-critical systems 22

such as SAP Enterprise Resource Planning, T&D 23

applications, and Outage Management System. SCE 24

developed this midrange server forecast through a 25

detailed analysis of existing midrange server assets, 26

their useful lives, and the expected midrange server 27

needs in this GRC period. Once SCE quantified the 28

need (number of units) for each type of midrange server 29

64 See WPSCE-06V01P01 pp. 92 – 97.

68

equipment category, SCE applied a unit cost to derive 1

the total forecast for that equipment category. These 2

unit costs were based on actual vendor quotes or an 3

extrapolation of historical cost data for each equipment 4

category.65 5

Storage Replacement: Our forecast expenditures of 6


acquiring, building, configuring, and testing storage 8

arrays to replace existing systems at the end of their 9

useful life. Expenditures in the forecast period will 10

replace storage devices and associated infrastructure 11

appliances for mission-critical systems such as Edison 12

SmartConnect®, Outage Management System, and 13

Customer Service System (CSS). SCE developed this 14

storage forecast through a detailed analysis of existing 15

storage assets, their useful lives, and the expected data 16

growth needs in this GRC period. Once SCE quantified 17

the need (number of units) for each type of storage 18

equipment category, SCE applied a unit cost to derive 19

the total forecast for that equipment category. These 20

unit costs were based on actual vendor quotes or an 21


category.66 The forecast accounts for the future 23

migration from the traditional storage architecture to 24

the Hyper Converged Infrastructure (HCI), which will 25

drive down storage costs over time. 26

Data Center Network Replacement: The forecast 27

expenditures for 2019-2023, totaling $38.0 million, are 28

65 Id.

66 Id.

69

evenly distributed to include costs to refresh roughly 1

20% of our overall data center network infrastructure 2

on a like-for-like basis. SCE developed this forecast 3

through a detailed analysis of the useful life for existing 4

data center network assets (i.e., routers, switches, 5

monitoring tools and load balancers). SCE applied unit 6

costs for each equipment type in order to derive the 7

total forecast expenditure during 2019-2023. These unit 8

costs were based on actual vendor quotes or an 9


type.67 11

Appliance Replacement: Our forecast expenditures of 12


acquiring, building, configuring, and implementing 14

appliances to replace existing midrange servers and 15

storage devices at the end of their useful life. As 16

mentioned in Server Replacement, due to the 17

operational efficiencies gained, Hyper Converged 18

Infrastructure (HCI) has begun in 2019 and will 19

continue implementation as a replacement for our 20

conventional server technology. Beginning in 2020, 21

costs for HCI will be captured in this new category. The 22

expenditures in the forecast period will replace existing 23

appliances for mission-critical systems such as SAP 24

Enterprise Resource Planning, Design Manager, and 25

Outage Management System. In addition, this reflects 26

an increase in appliances for refresh of Enterprise 27

Analytics, SAP HANA, CSRP, Enterprise Platform 28

Core Refresh (EPCR), and Network Appliance 29

67 See WPSCE-06V01P01 pp. 98 – 102.

70

(NetApp). SCE developed this appliance replacement 1

forecast through a detailed analysis of existing assets, 2

their useful lives, and the expected needs in this GRC 3

period. Higher spend is expected for 2021 as the 4

majority of the SAP solutions are due for a refresh with 5

newer HANA backend hardware. Because SAP 6

solutions are integrated, the SAP landscape (multiple 7

applications) is required to be refreshed together at the 8

same time resulting in the increase in expenditures. In 9

2022 and 2023, there is an increased expenditure on 10

Exadata supporting our Meter Data Management 11

System (MDMS), Outage Management System (OMS), 12

and Comprehensive Geographic Information System 13

(cGIS). Once SCE quantified the need (number of 14

units) for each type of appliance category, SCE applied 15

a unit cost to derive the total forecast for that equipment 16

category. These unit costs were based on actual vendor 17

quotes or an extrapolation of historical cost data for 18

each equipment category.68 19

Organic Growth: Our forecast expenditures of $44.0 20

million for 2019-2023 include costs for acquiring, 21

building, configuring, and testing midrange servers, 22

appliances, and datacenter network components 23

required to meet the organic growth. Due to the nature 24

of these appliance models, such as HANA, Big Data 25

Appliance (BDA), and Exadata, capacities are now 26

procured in pre-configured increments and therefore 27

organic growth hardware outlays are increased. 28

Consequently, the increases for HANA and Oracle 29

68 See WPSCE-06V01P01 pp. 103 – 105.

71

appliances (BDA and Exadata) in years 2020 and 2023 1

require larger capital investment than other years within 2

this period. The organic growth projection is based on 3

historical incremental growth as a percentage of the 4

total capacity of a particular component type. As 5

mentioned in the Historical Variance Analysis section, 6

the forecast capacity growth line items address the 7

organic growth needs of the business applications as 8

well as the corresponding capacity expansion for 9

purposes of risk management (i.e., in order to address 10

the needed disaster recovery infrastructure to ensure the 11

proper application failover as needed for the two co-12

primary production data centers, ADC and IOC). Here 13

again, when additional compute, storage and/or network 14

infrastructure are forecast to address organic growth, 15

there is a need to forecast a proportional or, in some 16

cases, exact matching of infrastructure to be 17

provisioned for disaster recovery in either ADC or IOC. 18

These projected costs were based on actual vendor 19

quotes or an extrapolation of historical cost data for 20

each equipment category. 21

b) End User Computing Maintenance, Services & Replacement 22


This sub-work activity includes three types of O&M costs. First, it 24

includes service operations labor which is responsible for monitoring and managing the MSPs’ 25

performance, specifically for the End User Computing & Service functions referenced in section IV.A 26

(Fixed Price Technology & Maintenance). IT End User Computing employees consistently engage 27

clients (SCE Employees) to address issue escalations and elicit requirements for ad-hoc requests and 28

project consultations. Second, this sub-work activity includes the management of the third-party vendor 29

contractual obligations and performance for cellular and wireless, product ordering, printing, audio and 30

72

visual. Finally, it includes the management of cellular devices and monthly plans, printers, software 1

licensing renewals, computer accessories, and printers. 2

In addition to the O&M expenses, this sub-work activity encompasses 3

capital expenditures in the form of refreshing devices for both office and field workers. Office workers 4

are provided with PCs to carry out routine tasks, including email, timesheets, word processing, 5

budgeting activities, and using business-related applications. Employees whose jobs require them to 6

support multiple locations or work remotely after hours are provided a standard laptop; all other 7

employees receive a desktop PC and monitor. Field employees (Troublemen, Linemen, Apparatus 8

teams) are provided with ruggedized devices69 to respond to distribution and transmission line issues. 9

Advances in technology and improvements in business processes have enabled up-to-date electronically 10

stored information to replace potentially-outdated paper versions for work in the field. The ruggedized 11

laptop deployment was expanded to include Transmission Patrolmen, Substation Technicians, Field 12

Service Representatives, and Meter Technicians in 2017. 13


This activity is necessary for a variety of reasons. First, oversight of MSPs 15

and third-party vendors is necessary because it provides an understanding of the business impacts of the 16

issues being worked on by the MSPs and third-party vendors. Additionally, it confirms that the quality 17

of the work being performed meets the standards and requirements of end users. 18

Second, with advances in technology and with recent wildfire events, 19

office and field workers are requiring PCs that are more powerful and more reliable in order to support 20

complex applications for mapping, outage management, and engineering design and modeling. 21

Third, refresh of PCs is necessary as result of age and technology 22

obsolescence. Even though a PC’s durability and capabilities have increased year over year allowing us 23

to expand their useful life, expansive data and computing requirements of the business application 24

portfolio have required a refresh rate of every three years. Mobile and remote workforce requirements 25

have increased over the years, driving our current PC environment to 70% laptop and 30% desktop. 26

Fourth, business in the field requires up-to-date electronic versions of 27

circuit maps, safety manuals, and work management information to replace outdated paper versions. 28

Electronic files allow SCE to provide field employees with the most current versions of documentation, 29

69 See WPSCE-06V01P01 pp. 106 – 108.

73

while also providing a more efficient means to house and retrieve the vast documentation that 1

employees may need to reference in the field (for up-to-date safety procedures, troubleshooting, etc.). 2

Through the advancement of technologies and introduction of electronic forms for submissions, new 3

work processes have been established that take advantage of these mobile capabilities. Ruggedized 4

laptops are securely mounted in field trucks and accompanying mobile specialty printers enable printing 5

of customer receipts and circuit maps. The challenging field environment includes travel over rough 6

terrain and in inclement weather. Ruggedized laptops are built to operate under these extreme 7

conditions. The devices have been tested to operate in adverse working conditions. Cellular modems 8

provide wireless connectivity for work dispatch, and global positioning system receivers enable street 9

mapping software so crews can locate work sites. Just as with the office environment PCs, one of the 10

most significant drivers for the refresh of ruggedized laptops is end-of-life, which is addressed on a four-11

year plan for model releases. How computing devices will be used guides how PCs and Laptops are 12

distributed. 13

The benefits of this sub-activity include replacing End Of Life (EOL) 14

devices with more modern hardware and software. If this sub-activity is not performed, there will be 15

more operational interruptions for office and field employees due to aging devices. Additionally, a 16

reasonable refresh program that replaces devices on a scheduled basis is an important part of providing 17

safe and reliable service, including emergency readiness.70 18



expenses, broken down by labor and non-labor for the End User Computing Maintenance, Services & 21

Replacement sub-work activity. 22

70 See WPSCE-06V01P01 pp. 109 – 110.

74

Table IV-22 End User Computing

O&M Labor/Non Labor Recorded & Forecast (Constant $000)


expenditures for the End User Computing Maintenance, Services & Replacement sub-work activity. 2

Table IV-23 End User Computing



(i) Labor 4

Labor costs decreased from $9.4 million in 2014 to $2.6 5

million in 2015 due to the transition to MSPs, which outsourced most of the functions previously done 6

in house. In 2015-2018, the labor numbers then remained relatively constant, although we did see an 7

incremental increase of $400,000 beginning in 2017 due to re-organization of employees performing 8

field support services into the SMOO organization under this sub-work activity. The minimal increase 9

from 2017-2018 of $300,000 is due to costs for professional trainees and filling of two positions that 10

support end user computing critical operations and elicit requirements for various requests. 11

(ii) Non-Labor 12

From 2014 to 2015, non-labor costs for this sub-work 13

activity decreased significantly (from $6.8 million to $1.4 million) due to End User Computing repair 14

and maintenance work that was done by contractors being moved to the MSP contract. In addition, there 15

75

was a reduction in IT wireless plans and device peripherals (e.g., keyboards, mice, headsets, etc.) in 1

alignment with the workforce reduction activities and policy changes. 2

From 2015 to 2016, non-labor expenses slightly increased 3

to $3.1 million due to higher audio-visual vendor costs as well as IT employee facility movement, which 4

incurs costs related to installation and configuration of desktops. 5

Expenses then significantly increased in 2017 to $13.7 6

million due to the centralization of costs for IT products and services (e.g., Cellphone, Pager, Air-Cards, 7

WebEx, etc.). In 2017, costs of IT’s products and services were centralized in IT, whereas in years 2014-8

2016, the costs of IT products and services were paid for by SCE’s OUs. 9

2018 non-labor expenses then dropped slightly to $11.2 10

million due to an effort to reduce IT products and services (e.g., Air-Cards and Desktop Phones, etc.). 11


From 2014-2018, the refresh of devices increased capital 13

expenses. In 2014, SCE only refreshed a minimal amount of such devices due to the transition to the 14

Managed Services model. As the MSP transition stabilized, more devices were refreshed, with the 15

amount peaking in 2017. These increased refreshes were also driven by increasing business needs in the 16

field. Note that the price of these devices varies between $2,200 to $4,400, meaning that variances in the 17

number of devices that need refreshing has a significant impact on recorded costs. 18

In addition, in 2017, SMOO incurred costs for Motorola 19

radio refresh costs, which account for the significant increase in 2017. Prior to 2017, costs regarding the 20

upgrade of an entire radio system was captured as part of a radio upgrade project in the Grid Services 21

organization. 22

Also, in 2017, SMOO began separately incurring the costs 23

for Monitors, which had previously been recorded as part of the PC bundle in prior years. In 2017, we 24

also saw an increase in the recorded costs for monitors due to an increase in need for employees having 25

dual monitors. 26


(i) Labor 28

For Test Year 2021, we forecast labor expenses of $3.5 29

million for this account, which is relatively flat compared to the 2018 recorded costs of $3.4 million. 30

This forecast represents full time employees to manage MSP performance and completion of activities 31

76

as per contract commitments. It also represents activities for issue resolutions with third party vendors 1

when necessary. These Test Year forecast expenses will allow for execution of the following functions: 2

Management of Service Desk 3

Management of end user computing resources (e.g., 4

desktop support) 5

Management of Field Services 6

Management of desktop engineering function 7

Management of wireless support 8

As shown in Table IV-22, SCE’s 2015-2018 recorded 9

expenses have been relatively stable, reflecting the stabilization of SCE’s MSP transition. Therefore, 10

SCE utilizes the last year recorded estimating methodology to form the basis of our test year forecast. 11

The last year recorded is an appropriate basis for our test year forecast because in 2021 SCE expects a 12

similar level of resources to execute similar activities. SCE then made an upwards adjustment of $0.2 13

million to this basis to account for additional phone operators, which provide efficient call routing and 14

directory services to SCE employees and customers, by: understanding the employee role and business 15

needs of customers, responding appropriately to emergency calls, and assisting the company in media 16

relations. The labor costs are expected to remain flat from 2021 to 2023 due to stabilization of the 17

working model with our managed services vendors as well as aggressively managing service costs. 18

(ii) Non-Labor 19

For Test Year 2021, we forecast non-labor expenses of $8.1 20

million for maintenance of approximately 7,500 smart phone plans, 1,500 tablet cellular data and Apple 21

care, 4,500 air cards, 1,256 printers, 225 plotters, 16,000 laptops and desktops, 108 teleconference 22

rooms with AV equipment all across the company, and 19 Mauell Walls (monitors). This forecast is a 23

decrease from 2018 recorded expenses of $11.2 million. SCE used the last year recorded methodology 24

as the basis for this forecast because the costs recorded from 2014-2016 are not comparable to the costs 25

expected in the test year due to the costs in those years being managed in a de-centralized manner. SCE 26

then made a downward adjustment of $3.1 million to this basis to account for: 27

Re-negotiating contracts with printer vendors 28

Reducing color printing by implementing printer 29

configurations that default to black and white 30

77

Reducing numbers of wireless devices (e.g., smart 1

phones, air cards) 2

Non-labor costs are expected to remain flat from 2021 to 3

2023 due to the stabilization of the working model with our managed services vendors as well as SCE 4

aggressively managing service costs. 5


For capital expenditures, we forecast a total of $97.2 7

million for 2019-2023. SCE developed its 2019-2023 forecast for this sub-work activity through a 8

detailed analysis of PCs, desktops, laptops, plotters, and monitors (including Mauell Walls), their useful 9

lives, and the expected refresh requirements in this GRC period. 10

Increasing needs in the business for rugged and high-11

performance devices, as well as enterprise programs and OU needs (such as Corporate Real Estate 12

projects, CSRP, and Windows 10 Refresh projects),71 have driven the purchase of thousands of new 13

devices, monitors, and audio-visual equipment. This, in turn, increases the refresh costs due to device 14

lifecycle and device costs. Although we strive to use the lowest cost options (e.g., provisioning 15

refurbished equipment), this sub-work activity will incur increased outer year charges from 2021-2023 16

in order to refresh the purchased equipment. 17

In addition to the OU needs mentioned above, Grid 18

Operations utilizes Video Wall displays to provide a high level of Situational Awareness of the SCE 19

Grid as well as the interconnections to other utilities. These video walls are in service at the two Grid 20

Control Centers (GCC), 13 Switching Centers, and four Distribution Operations Centers (DOC). With 21

most of these displays installed in 2014, the LCD screens are at the end of their useful life. In addition, 22

the software to control the video walls needs to be upgraded to be compatible with Windows 10. This 23

refresh includes other hardware replacement (in addition to the screens) and upgrades to support the 24

currently available functions. 25

SCE forecasts the cost for these devices and the cost of 26

refreshes based on the number of units in our current environment. SCE then applied a unit cost, which 27

71 See Testimony for CRE projects in SCE-06 Vol. 05, See Testimony for CSRP in SCE-03, Vol. 03, and See

Testimony for Windows 10 Refresh project in SCE-06, Vol 1, Part 2.

78

is based on actual vendor quotes or an extrapolation of historical cost data for each equipment category 1

type, to the unit forecast by year, to arrive at the End User Computing forecast.72 2

c) Technology Adoption 3

(1) Work Activity Description 4

The Technology Adoption sub-work activity consists of the retirement of 5

computer, storage, network, and operating software assets no longer aligned with SCE’s strategic 6

roadmap. It also includes the replacement of these assets with hardware and operating software aligned 7

with SCE’s strategic direction to: (1) leverage cloud computing and cloud-hosted software-as-a-service 8

(SaaS) subscriptions; and (2) increase on-premise data center virtualized compute, network, and storage. 9

Activities within this sub-work activity include (1) improving cloud 10

foundational areas73 in order to increase reliability, cyber security, and identity and access management; 11

and (2) migrating and re-platforming the resident applications to technologies aligned with SCE’s cloud 12

strategy. 13

As previously discussed, as SCE continues its adoption of cloud 14

technologies, the focus over 2019-2023 will be on Reliability and Business Resiliency, Backup and 15

Recovery, Operational Reliability, Network Management, Security, and Identity Management. 16

In furtherance of these focus areas, SCE plans for the acquisition and 17

implementation of the following cloud capabilities, which will be capitalized over 2019-2023: 18

In order to enhance reliability and business resiliency, SCE will 19

implement tools such as Azure Disaster Recovery, which utilizes 20

geographic redundancy in order to allow workloads using compute, 21

storage, database, and managed Azure services to replicate between 22

on-premise data centers and the Azure cloud. This will significantly 23

mitigate risk because power outages, physical network outages, natural 24

disasters, and civil unrest would have to affect both the on-premise 25

72 See WPSCE-06V01P01 pp. 111 – 124.

73 Whereas cloud sub-work activity mentioned in SW Maintenance & Replacement is about maintenance of the license agreements, cloud improvement activity in this section is about adopting new cloud technologies & Continued from previous page

services around reliability & business resiliency, backup & recovery, operational reliability, security, & Identity management.

79

data center and the cloud to potentially cause an inability to use critical 1

business applications. 2

Migration of application workloads will be done utilizing Azure Site 3

Recovery (ASR), among others, which has a dual benefit as a tool to 4

be used for disaster recovery. This is a foundation capability as its use 5

avoids the risk and inefficiency associated with performing migrations 6

on a project-by-project basis. 7

Operational reliability will be enhanced utilizing network management 8

probes exemplified by Unified Infrastructure Management tool in 9

order to monitor the health and performance of the Azure Cloud 10

infrastructure and services. 11

Network traffic routing and maintenance will be simplified with the 12

Cross-Tenant Peer Connect tool and other supplemental tools while 13

also reducing operational costs. A system test environment will be set 14

up in order to help mitigate failures before production implementation 15

of business applications in the Cloud. Additionally, infrastructure 16

provisioning will be automated wherever possible in order to avoid the 17

mistakes which too often are associated with manual infrastructure 18

provisioning. 19

Cloud services security will be improved with the use of tools such as 20

Azure ExpressRoute by providing for the creation of private 21

connections between the Microsoft data centers housing Azure cloud 22

infrastructure and services and the SCE data centers. As a foundational 23

capability, these private connections will be utilized in lieu of the 24

public internet for facilitating workload migration to, and utilization 25

of, Azure cloud services. As part of SCE’s program for replacing its 26

legacy Identity & Access Management (IAM) infrastructure with a 27

modern Identity Governance & Administration (IGA) platform for 28

both the Corporate Enterprise and Grid environments, SCE has 29

developed a program, Identity Governance & Access Management 30

80

(IGAM),74 to implement a series of projects that will be focused on 1

delivering new services and capabilities to SCE in support of 2

Cybersecurity. As part of these projects, there will be additional 3

enhancements and expansion of core capabilities delivered by projects 4

to increase the adoption of the services and capabilities. SCE will be 5

driving projects to perform additional application integration and 6

remediation through new IGAM platform for SCE business 7

applications and support infrastructure. Additionally, SCE has 8

increased cloud services which require additional security capabilities 9

to adequately secure the applications, services, and environments. 10

Finally, as part of the IGAM project, SCE will bring on advanced 11

analytics, reporting, and newly developed AI technology to increase 12

the adoption and end user satisfaction with the IGAM platform. 13


Adoption of the technologies previously described are advantageous for 15

several reasons. 16

First, the implementation and migration to cloud-based services 17

infrastructure mitigates the impact of hardware obsolescence since the hardware infrastructure will be 18

the responsibility of the cloud-services provider. This will minimize the risk and impact to SCE IT 19

operations. Typically, as hardware assets move into obsolescence, they generate more failures, create 20

more compatibility issues, introduce more security vulnerabilities, and ultimately result in more outages. 21

Shifting responsibility to the cloud services provider mitigates risk to SCE IT operations. In addition, 22

there will be reduced hardware maintenance and support costs. 23

Second, SCE has identified several cloud services for which we need to 24

develop a foundation, before moving more business applications onto the cloud. Development of, and 25

support for, these SCE business applications and services will take advantage of technology 26

enhancements available in the cloud at a much faster rate than can be deployed in a traditional software 27

lifecycle. As more SCE data moves from on-premise to the cloud, improving these foundational areas 28

74 Cybersecurity will be implementing IGAM new projects and capabilities (see SCE-04, Vol. 3 Cybersecurity

in Cybersecurity Delivery sections), whereas the adoption, expansion and operational support for IGAM will be addressed in the Technology Adoption sub-work activity within SMOO.

81

also allows SCE to improve reliability and cybersecurity at a faster rate than could otherwise be 1

deployed in a traditional software lifecycle. 2

Lastly, Technology Adoption’s implementation of Identity & Access 3

Management cloud capabilities through the IGAM program and SMOO’s expansion/adoptions activities 4

will enable SCE end users to navigate through the various SCE cloud applications and services 5

seamlessly, while adding the needed level of security protection for SCE’s assets and information. As 6

part of Technology Adoption, there will also be an integration of Identity Artificial Intelligence (Identity 7

AI) to better enable SCE to manage risk and security and improve decision making relative to access 8

being granted to end users. In addition, Technology Adoption’s cloud services infrastructure 9

enhancement will address the aforementioned IGAM expansion and adoption to ensure a more secure 10

environment. 11



expenditures for the Technology Adoption sub-work activity. 14

Table IV-24 Technology Adoption



(i) Capital Expenditures 16

For 2018, there was a $900,000 expenditure for the 17

implementation of the first level of internet access for our hybrid cloud environment. 18


(i) Capital Expenditures 20

Our forecast capital expenditures are $30.7 million for 21

2019-2023. This forecast can be broken down into the following functional capabilities: 22

Cloud - $1.8 million 23

Reliability - $3.9 million 24

TOTAL CONSTANT AMOUNT

2018 2019 2020 2021 2022 2023Technology Adoption $914 $3,350 $5,455 $4,868 $5,248 $11,780Totals $914 $3,350 $5,455 $4,868 $5,248 $11,780

Recorded Forecast

82

Business/disaster Recovery - $4.6 million 1

Storage and Backup - $10 million 2

Networking - $104,000 3

Security - $64,000 4

Identity Management - $10.3 million 5

The forecast spending for these functional capabilities is 6

detailed further in the following paragraphs. 7

Cloud. The cloud migration and re-platforming costs of 8

$1.8 million from 2019-2020 include all project management, data migration services, application 9

development services, testing services, software, and hardware. The cost estimates for the operating 10

software tools with the required functionality according to SCE’s cloud strategy were derived using 11

vendor detailed quotes. Included is the build-out of a test environment using a subset (e.g., On-Prem 12

SQL Server to Azure SQL using Azure Synchronization) of the development-operations toolset for 13

workload migration to the cloud. Also included will be automation of cloud infrastructure provisioning 14

prior to production implementation in order to mitigate issues encountered in the migration of business 15

applications and associated data to our Cloud Software-as-a-Service. 16

Reliability. SCE will spend $3.9 million spread relatively 17

evenly over 2019-2022 to implement tools like Azure Disaster Recovery for automation of the failover 18

from SCE on-premise to our cloud provider and significantly improve system reliability and 19

recoverability, which will translate to improved business continuity. This will also include the 20

migration/re-platforming of IBM/Oracle and SAP applications from legacy IBM and Netapp hardware 21

(compute and storage) to a hyper-converged and Veritas Infoscale infrastructure in conjunction with our 22

Cloud Software-as-a-Service (SaaS) provider for enhanced business continuity and system reliability. 23

Business/disaster Recovery; Storage and Backup. 24

Implementation of Azure Co-lo Backup and Hana Large Instances (HLI) will be necessary to support 25

large projects like CSRP and Enterprise Platform Core Refresh (EPCR) and will lead to a larger than 26

typical expenditure on storage and backup of $10 million in 2023 and expenditures of $4.6 million over 27

2020-2023 for build-out of disaster recovery capability. 28

Networking; Security. Operational reliability and 29

complexity will be improved with network monitoring tools like Azure Network Monitoring with 30

Computer Associates Spectrum complimented by Cross-tenant Peer Connect. Needed improvement for 31

83

data security in the hybrid cloud environment will be provided with Azure Express Route which utilizes 1

private site-to-site communication routes instead of public internet links between SCE on-premise 2

systems and the Azure cloud services infrastructure. SCE will spend $168,000 over 2019-2020 for these 3

network management and security enhancements. 4

Identity Management. With respect to identity 5

management, SCE will spend $10.3 million75 over 2020-2023 to implement single-sign-on for cloud 6

applications and SailPoint IdentityIQ for an identity governance solution linking people, business 7

applications, data and devices, which will create an identity-enabled enterprise. Implementation of this 8

improved governance solution will mitigate support issues associated with the out-of-support IBM 9

Tivoli Access Management and Directory Server toolset. Utilizing industry guidelines, we broke down 10

SCE’s application portfolio into simple, medium and complex categories and developed an 11

implementation plan with forecast capital expenditures for the 2020-2023 timeframe.12

75 See WPSCE-06V01P01 pp. 125 – 127.

Appendix A

Witness Qualifications

A-1

SOUTHERN CALIFORNIA EDISON COMPANY 1

QUALIFICATIONS AND PREPARED TESTIMONY 2

OF ALBERT MA 3

Q. Please state your name and business address for the record. 4

A. My name is Albert Ma, and my business address is 2131 Walnut Grove Avenue, Rosemead, CA, 5

91770. 6

Q. Briefly describe your present responsibilities at the Southern California Edison Company. 7

A. I am Vice President of the Enterprise Services department for Southern California Edison’s 8

Information Technology Organizational Unit. In that capacity, I have the responsibility of 9

managing the planning, delivery, and running of information technology applications and 10

infrastructure. 11

Q. Briefly describe your educational and professional background. 12

A. I hold a Bachelor of Science degree in Biological Sciences from Stanford University. I joined 13

SCE in May 2018 as the Vice President of Enterprise Services. Before joining SCE, I served as 14

CIO of Toyota Financial Services, responsible for the planning, delivery, and operations of 15

technology solutions for the organization. I was also the Divisional Information Officer at 16

Toyota Motor Sales, responsible for driving the technology agenda for the After Sales business 17

including Parts Supply Chain, Corporate Accessories, Product Quality, Dealer Operations, and 18

Customer Services. I have over 17 years of experience in the area of IT over the course of my 19

career. 20

Q. What is the purpose of your testimony in this proceeding? 21

A. The purpose of my testimony in this proceeding is to sponsor the portions of Exhibit SCE-06, 22

Volume 1, Part 1, entitled Enterprise Support—Enterprise Technology and Exhibit SCE-06, 23

Volume 1, Part 2, entitled Enterprise Support—Enterprise Technology: OU Capitalized 24

Software, as identified in the Table of Contents thereto. Additionally, I am also witness to the 25

Amended Testimony to the above referenced Exhibits (SCE-06, Volume 1, Part 1A and Part 2A) 26

as identified in the Table of Contents thereto. 27

Q. Was this material prepared by you or under your supervision? 28

A. Yes, it was. 29

Q. Insofar as this material is factual in nature, do you believe it to be correct? 30

A. Yes, I do. 31

A-2

Q. Insofar as this material is in the nature of opinion or judgment, does it represent your best 1

judgment? 2

A. Yes, it does. 3

Q. Does this conclude your qualifications and prepared testimony? 4

A. Yes, it does. 5

6

7

A-3

1

Documents

General Rate Case · SERVICE MANAGEMENT OFFICE & OPERATIONS ... 14 three primary data centers.2 In addition, SCE-06, Volume 1, ... 24 SCE’s data center infrastructure and applications,