Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Application No.: A.16-09- Exhibit No.: SCE-04, Vol. 2 Witnesses: D. Bauder
D. Bernaudo T. Boucher J. Castleberry T. Condit G. Haddox T. Inlander P. Joseph
J. Kelly D. Kempf J. Lim S. Nagoshi D. Pierce M. Provenzano J.P. Shotwell J. Tran
(U 338-E)
2018 General Rate Case
Information Technology (IT) Volume 2 – Capitalized Software
Before the
Public Utilities Commission of the State of California
Rosemead, California
September 1, 2016
SUMMARY
• This Volume presents SCE’s request for $809.1 million in capitalized software expenditures for the
2016-2020 forecast period.1 These funds will support SCE’s core business processes, improve
customer transactions and outage notifications, and support the modernizing of the grid to improve
safety and reliability.
• This volume includes testimony on the following:
o An Operating System Software request for $82.6 million, which provides foundational software
that manages computer hardware and enables business applications to perform daily work
functions and to operate across multiple technology platforms, such as web and application
servers, storage, and personal computers.
o Cybersecurity & Compliance capitalized software project requests for $308.9 million that will
continue to protect critical SCE systems, the electric system, and customer information from
growing cyber threats, especially as access points to the system expand. This also includes
requests that will support compliance with current and future NERC CIP standards. Included in
this capitalized software forecast is $99.1 million for Grid Modernization Cybersecurity.
o Technology Consolidation & Optimization capitalized software project requests for $41.7
million that will help to optimize our existing technology portfolio to increase efficiencies and
reduce costs.
o SCE’s Operating Units have requested $376.0 million specific capitalized software projects to
support critical business operations. This request reflects SCE’s robust review, prioritization, and
selection, which allows us to put forth a technology portfolio that provides the most benefit to
the company and our customers.
1 Refer to WP SCE-04, Vol. 2 pp. 1-4.
Capitalized Software Expenditures 2016-2020 Forecast
CPUC-Jurisdictional Only (Nominal $Million)
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents
Section Page Witness
-i-
I. INTRODUCTION .............................................................................................1 T. Inlander
A. Summary of SCE’s Capitalized Software Request ................................1
B. Compliance Requirements .....................................................................1
C. 2015 Authorized versus Recorded .........................................................2
D. Capitalized Software Estimation............................................................3
II. OPERATING SYSTEM SOFTWARE..............................................................6 M. Provenzano
A. Operating System Software ...................................................................6
1. Project Description .....................................................................6
2. Need for the Project ...................................................................7
a) Business requirements ...................................................8
3. Scope and Cost Forecast ..........................................................11
a) Recorded Expenditures ................................................11
b) Forecast Expenditures ..................................................12
III. CYBERSECURITY & IT COMPLIANCE .....................................................15 G. Haddox
A. Perimeter Defense ................................................................................19
1. Program Description ................................................................19
2. Need for Program .....................................................................19
3. Scope and Cost Forecast ..........................................................20
B. Interior Defense ...................................................................................21
1. Program Description ................................................................21
2. Need for Program .....................................................................21
3. Scope and Cost Forecast ..........................................................22
C. Data Protection.....................................................................................23
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-ii-
1. Program Description ................................................................23
2. Need for Program .....................................................................24
3. Scope and Cost Forecast ..........................................................24
D. SCADA Cybersecurity .........................................................................25
1. Program Description ................................................................25
2. Need for Program .....................................................................26
3. Scope and Cost Forecast ..........................................................27
E. Common Cybersecurity Services for Generator Interconnections ...................................................................................28
1. Project Description ...................................................................28
2. Need for Project .......................................................................29
3. Scope and Cost Forecast ..........................................................29
F. Grid Modernization – Cybersecurity ...................................................30
1. Project Description ...................................................................30
2. Need for Project .......................................................................32
3. Scope and Cost Forecast ..........................................................33
a) Alternatives Considered ...............................................36
G. IT Support for NERC CIP Compliance ...............................................37
1. Project Description ...................................................................37
2. Need for Project .......................................................................38
3. Scope and Cost Forecast ..........................................................39
a) Alternatives Considered ...............................................42
IV. TECHNOLOGY CONSOLIDATION & OPTIMIZATION ...........................43 J. Kelly
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-iii-
A. Data Warehouse Consolidation ............................................................43
1. Project Description ...................................................................43
2. Need for Project .......................................................................44
3. Scope and Cost Forecast ..........................................................45
a) Alternatives Considered ...............................................46
B. Lotus Notes Migration .........................................................................46
1. Project Description ...................................................................46
2. Need for Project .......................................................................47
3. Scope and Cost Forecast ..........................................................48
a) Alternatives Considered ...............................................48
C. Backup and Disaster Recovery Optimization ......................................49
1. Project Description ...................................................................49
2. Need for Project .......................................................................50
3. Scope and Cost Forecast ..........................................................50
a) Alternatives Considered ...............................................51
D. Information Technology Projects less than $3 Million ........................51
V. OPERATING UNIT SOFTWARE PROJECTS ..............................................52
A. Customer Service Software Projects ....................................................52 J. Lim
1. SCE.com Strategic Upgrade/Stabilization ...............................52
a) Project Description .......................................................52
b) Need for Project ...........................................................53
c) Scope ............................................................................53
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-iv-
(1) Remaining Scope to be Completed in 2016..................................................................53
(2) Completed Scope .............................................54
d) Recorded and Forecast Expenditures ...........................55
2. Digital Customer Self-Service .................................................56
a) Project Description .......................................................56
b) Need for Project ...........................................................57
c) Scope ............................................................................57
(1) Device Support.................................................58
(2) Electronic Billing and Payment Transactions .....................................................58
(3) Security and Authentication .............................59
(4) Website Functionality ......................................60
d) Forecast Expenditures and Cost-Benefit Analysis........................................................................62
e) Alternatives Considered ...............................................63
3. Alerts & Notifications ..............................................................64
a) Project Description .......................................................64
b) Need for Project ...........................................................65
c) Completed Project Scope .............................................65
d) Remaining Project Scope .............................................66
e) Recorded and Forecast Expenditures ...........................68
4. Meter Data Management System (MDMS) Upgrade ..............68 D. Kempf
a) Project Description .......................................................68
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-v-
b) Need for Project ...........................................................69
c) Scope and Cost Forecast ..............................................70
(1) Phase 1 .............................................................70
(2) Phase 2 .............................................................70
d) Recorded and Forecast Expenditures ...........................71
5. Customer Service Projects Less than $3M ..............................72 D. Kempf, ...................................................................................................... D. Bernaudo ...................................................................................................... S. Nagoshi
B. Transmission & Distribution Software Projects ..................................72
1. Work Management Solutions ..................................................72 P. Joseph
2. WM - Portfolio Management ...................................................75
a) Project Description .......................................................76
b) Need for Project ...........................................................77
c) Scope and Forecast ......................................................78
(1) Alternatives Considered ...................................80
3. Scope Cost Management Tool (SCMT) ..................................80
a) Project Description .......................................................80
b) Need for Project ...........................................................82
c) Scope and Cost Forecast ..............................................83
(1) Alternatives Considered ...................................83
4. Work Management Dashboard ................................................84
a) Project Description .......................................................84
b) Need for Project ...........................................................85
(1) Benefits ............................................................86
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-vi-
c) Scope and Cost Forecast ..............................................86
(1) Alternatives Considered ...................................87
5. Transmission Telecommunications Work Order Lifecycle ..................................................................................88
a) Project Description .......................................................88
b) Need for Project ...........................................................89
(1) Benefits ............................................................89
c) Scope and Cost Forecast ..............................................91
(1) Alternatives Considered ...................................91
6. Click Schedule Refresh Release 1 & 2 ....................................92
a) Project Description .......................................................92
b) Need for Project ...........................................................93
(1) Benefits ............................................................94
c) Scope and Cost Forecast ..............................................94
(1) Alternatives Considered ...................................94
7. Vegetation Management ..........................................................95
a) Project Description .......................................................95
b) Need for Project ...........................................................96
c) Scope and Cost Forecast ..............................................96
(1) Alternatives Considered ...................................97
8. Pole Loading Application Replacement Tool ..........................97 T. Boucher
a) Project Description .......................................................97
b) Recorded Costs and Forecast .......................................99
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-vii-
9. Design Manager (DM) Refresh ...............................................99
a) Project Description .......................................................99
b) Recorded Cost and Forecast.......................................100
10. Graphic Design Tool (GDT) and Tract Deployment Refresh ...................................................................................101
a) Project Description .....................................................101
b) Need for Project .........................................................102
c) Scope and Cost Forecast ............................................103
(1) Alternatives Considered .................................103
11. Consolidated Mobile Solution (CMS) ...................................104
a) Project Description .....................................................104
b) Need for Project .........................................................104
c) Scope and Cost Forecast ............................................105
(1) Alternatives Considered .................................106
12. Field Tools Upgrade ..............................................................106
a) Project Description .....................................................106
b) Need for Project .........................................................107
a) Scope and Cost Forecast ............................................107
(1) Alternatives Considered .................................107
13. Enhanced Business Resiliency for Energy Management System (EBR) ..................................................108 G. Haddox
a) Project Description .....................................................108
b) Need for Project .........................................................108
(1) Benefits ..........................................................109
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-viii-
c) Scope and Cost Forecast ............................................109
(1) Alternatives Considered .................................110
14. Comprehensive Situational Awareness for Transmission (CSAT) Phase 1 ...............................................110 T. Boucher
a) Project Description .....................................................110
b) Need for Project .........................................................111
(1) Benefits ..........................................................112
c) Scope and Cost Forecast ............................................113
(1) Alternatives Considered .................................114
15. Centralized Remedial Action Scheme (CRAS) Project ....................................................................................115
a) Project Description .....................................................116
b) Need for Project .........................................................117
(1) Analysis of additional complex RASs ..............................................................119
(2) Generation Queue Completion ......................122
(3) Benefits ..........................................................124
c) Scope and Cost Forecast ............................................128
(1) Alternatives Considered .................................129
16. RGOOSE Project ...................................................................130
a) Project Description .....................................................130
b) Need for Project .........................................................131
(1) Benefits: .........................................................133
b) Scope and Cost Forecast ............................................134
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-ix-
(1) Alternatives Considered .................................135
17. Energy Management System (EMS) Refresh ........................136
a) Project Description .....................................................136
b) Need for Project .........................................................136
(1) Benefits: .........................................................137
c) Scope and Cost Forecast ............................................137
(1) Alternatives Considered .................................137
18. Outage Management System (OMS) Refresh ........................138
a) Project Description .....................................................138
b) Need for Project .........................................................139
(1) Benefits ..........................................................139
c) Scope and Cost Forecast ............................................140
(1) Alternatives Considered .................................140
19. Distribution Management System (DMS) Refresh ................141
a) Project Description .....................................................141
b) Recorded Costs and Forecast .....................................143
20. Grid Interconnection Processing Tool (GIPT) .......................144
a) Project Description .....................................................144
b) Need for Project .........................................................144
(1) Benefits ..........................................................147
c) Scope and Cost Forecast ............................................149
(1) Alternatives Considered .................................150
21. Grid Analytics Applications (GAA) ......................................150
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-x-
a) Project Description .....................................................150
b) Need for Project .........................................................151
(1) Benefits ..........................................................152
c) Scope and Cost Forecast ............................................154
(1) Alternatives Considered .................................154
22. Long-Term Planning Tools ....................................................155
a) Project Description .....................................................155
b) Need for Project .........................................................155
(1) Benefits ..........................................................159
c) Scope and Cost Forecast ............................................159
(1) Alternatives Considered .................................161
23. Grid Connectivity Model .......................................................161
a) Project Description .....................................................161
b) Need for Project .........................................................162
(1) Benefits ..........................................................163
c) Scope and Cost Forecast ............................................165
(1) Alternatives Considered .................................166
24. Transmission and Distribution Projects less than $3 Million....................................................................................167
C. Power Supply Software Projects ........................................................168
1. Generation Automation Upgrade & Control Systems Refresh .....................................................................168 T. Condit
a) Project Description .....................................................168
b) Need for Project .........................................................169
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-xi-
(1) Benefits ..........................................................169
c) Project Scope and Forecast ........................................169
(1) Alternatives Considered .................................170
2. Dam Monitoring and Surveillance .........................................170
a) Project Description .....................................................170
b) Project Need ...............................................................171
(1) Benefits ..........................................................171
c) Project Scope and Forecast ........................................171
(1) Alternatives Considered .................................172
3. CAISO Market Enhancements Program (IMEP) ...................173 J. Tran
a) Project Description .....................................................173
b) Need for Project .........................................................174
(1) Benefits ..........................................................174
c) Scope and Cost Forecast ............................................174
(1) Alternatives Considered .................................175
4. Energy Planning Platform Upgrade (EPP) ............................176
a) Project Description .....................................................176
b) Need for Project .........................................................176
(1) Benefits ..........................................................177
c) Scope and Cost Forecast ............................................177
(1) Alternatives Considered .................................177
5. PCI Replacement ...................................................................178
a) Project Description .....................................................178
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-xii-
b) Need for Project .........................................................179
(1) Benefits ..........................................................179
c) Scope and Cost Forecast ............................................179
(1) Alternatives Considered .................................181
6. Energy Trading and Risk Management (ETRM) System Replacement ..............................................................181
a) Project Description .....................................................181
b) Need for Project .........................................................182
(1) Benefits ..........................................................182
c) Scope and Cost Forecast ............................................182
(1) Alternatives Considered .................................183
7. Aggregated Demand Response (ADR) ..................................184
a) Project Description .....................................................184
b) Need for Project .........................................................185
(1) Benefits ..........................................................186
c) Scope and Cost Forecast ............................................186
(1) Alternatives Considered .................................186
8. Commodity Management Platform (CMP) ...........................187
a) Project Description .....................................................187
b) Need for Project .........................................................187
(1) Benefits ..........................................................188
c) Project Scope and Forecast ........................................188
(1) Alternatives Considered .................................189
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-xiii-
9. Generation Management System (GMS) Upgrade ................189
a) Project Description .....................................................190
b) Recorded Costs and Forecast .....................................190
10. Power Supply Projects less than $3 Million ..........................191
D. Ethics and Compliance ......................................................................192 J.P. Shotwell
1. Enterprise Content Management ............................................192
a) Project Description .....................................................192
b) Need for Project and Scope ........................................193
c) Cost Forecast ..............................................................198
(1) Alternatives Considered .................................199
2. Electronic Document Management / Records Management (eDMRM) .........................................................200
a) Project Description .....................................................200
b) Need for Project .........................................................201
c) Scope and Cost Forecast ............................................201
(1) Alternatives Considered .................................202
E. Finance Capital Projects ....................................................................203 D. Pierce
1. Plant Ledger Upgrade and Tax Module Installation ..............203
a) Project Description .....................................................203
b) Need for Project .........................................................204
c) Scope and Cost Forecast ............................................205
(1) Alternatives Considered .................................206
2. Corporate Projects less than $3 Million .................................207 J. Castleberry
SCE-04: Information Technology Volume 2 – Capitalized Software
Table Of Contents (Continued)
Section Page Witness
-xiv-
F. Operational Services Capital Projects ................................................207
1. C-CURE 9000 ........................................................................207 D. Bauder
a) Project Description .....................................................207
b) Need for Project .........................................................208
c) Scope and Cost Forecast ............................................208
(1) Alternatives Considered .................................209
2. Operational Services Projects less than $3 Million ...............210 J. Castleberry
1
I. 1
INTRODUCTION 2
A. Summary of SCE’s Capitalized Software Request 3
SCE is requesting $809.1 million from 2016 – 2020 to implement needed capitalized software to 4
support the business capabilities of SCE Operating Units and enterprise-level systems for SCE.2 5
Table I-1 Capitalized Software Forecast3
(Nominal $Millions)
B. Compliance Requirements 6
In D.12-11-051, SCE was directed to “establish that proposed capital projects are necessary and 7
that SCE has prudently examined alternatives for cost-effectiveness before seeking Commission 8
approval.”4 SCE has considered alternatives to projects and has detailed those evaluations in this 9
testimony. 10
In D.15-11-021, the CPUC required that SCE “include its own forecast and the Commission’s 11
adopted forecast from the previous GRC alongside historical costs, and brief explanations detailing any 12
changes in the scope of a category.”5 The capitalized software projects requested address this 13
requirement within their respective testimonies. 14
2 This does not include the $208.8 million in capitalized software requested in SCE-04, Vol. 3 for the Customer
Service Re-Platform. 3 SCE expects to avoid capitalized software expenditures of $5.46 million in 2020 related to avoided
development for legacy software, as a result of the implementation of the Customer Service Re-Platform (CSR). Consequently, we do not include these costs in our capitalized software forecast. These cost avoidances are contingent upon approval of the CSR project costs as defined in SCE-04, Vol. 3. Should the Commission not adopt the proposed CS Re-Platform costs, the corresponding legacy software development expenditures should be added to SCE’s capitalized software forecast. Refer to WP SCE-04, Vol 3 pp. 169-171 for details on these costs avoidances.
4 D.12-11-051; Conclusion of Law #4. 5 D.15-11-021; pp. 224.
Testimony Volume Year 2016 2017 2018 2019 2020 2016-20 Forecast
SCE-04, Volume 2 Total 151.662 213.382 202.868 135.965 105.230 809.108
2
C. 2015 Authorized versus Recorded 1
Figure I-16 2015 Requested, Authorized and Recorded
(Constant 2015 $000)
As shown in Figure I-1, SCE’s overall capital expenditures were approximately $8 million below 2
the Commission’s 2015 authorized level. SCE manages our portfolio of capitalized software work in 3
consideration of available resources, competing business priorities, and technology deployment 4
strategies. In 2015, SCE invested more in enterprise and operating system software to make needed 5
upgrades to our server, storage, and desktop operating systems environments prior to vendor-published 6
end-of-life support for Microsoft, Cisco, and VMware Enterprise License Agreements, and in 7
preparation for our migration to Microsoft Office 365. Additionally, SCE underspent its authorized 8
levels for OU capitalized software. This was largely due to the deferral of the Customer Data Warehouse 9
6 Refer to WP SCE-04, Vol. 2 Bk A pp. 5-6.
($16,846)
$209 $8,751
$187,551 $185,286 $177,400
0
20,000
40,000
60,000
80,000
100,000
120,000
140,000
160,000
180,000
200,000
2015 Request 2015 Authorized CybersecurityVariance
Enterprise &Operating Software
Variance
OU CapitalizedSoftware Variance
2015 Recorded
Cybersecurity & Compliance Enterprise & Operating Software OU Capitalized Software
3
project from its originally planned date, as SCE found synergies in merging this investment with our 1
efforts to migrate to HANA Hadoop technology in the future, as discussed in Section IV, Technology 2
Consolidation & Optimization. 3
D. Capitalized Software Estimation 4
The SCE capitalized software projects in this testimony include a combination of both in-flight 5
and future initiatives. For in-flight projects, estimates are based on a detailed scope of work definition 6
and are supported by detailed labor, licensing, and infrastructure estimates captured through SCE’s 7
project estimation and management processes. Project estimates are refined during the project life cycle 8
when more project specific data is available, such as having completed Request for Information (RFI) 9
and Request for Proposals (RFP) from vendors and completing solution analysis resulting in a system 10
architecture recommendation. 11
Many Operating Unit initiatives to be implemented between 2018 and 2020 are in early stages of 12
the project life cycle and only initial business requirements and scope definition has been performed. For 13
these, the Operating Unit and IT leverage a high-level estimation methodology, which involves 14
analyzing historical similar Operating Unit capitalized software projects with equivalent scale and 15
complexity. Complexity is determined by the functionality of the application being purchased 16
(sometimes referred to as COTS – Commercial-Off-The-Shelf) or developed, and by the number and 17
complexity of integration points and data being transferred between systems. 18
The COTS and In-house Application Development forecast methodologies use spreadsheet-19
based calculations.7 The methodology includes assigning standard assumptions by project type that are 20
adjusted to account for known non-standard differences from the generic estimates of the estimation 21
model. These adjustments are based on IT and the Operating Unit’s current understanding of the 22
complexity of the initiative, the likelihood of leveraging an existing product, the integration points, the 23
data transferred amongst systems, the volume of data transferred, the data validation required through 24
the transfer, and the transfer intervals and automation required. SCE uses historical data from the last 25
five years of capitalized software projects to derive the generic estimation template parameters based on 26
the following six categories: 27
1. Simple Commercial-off-the-Shelf (COTS) Product 28
7 Refer to WP SCE-04, Vol. 1, Bk A pp. 101-112.
4
2. Medium Complexity COTS Product 1
3. High Complexity COTS Product 2
4. Simple Software Development 3
5. Medium Complexity Software Development 4
6. High Complexity Software Development 5
The project and technology complexity is determined by the functionality of the application 6
being purchased or developed as well as by the integration points and data being transferred between 7
systems. 8
The estimation methodology includes several type of labor resources identified below. 9
• SCE Labor - This is an estimate of the total team size and uses a blended billing rate 10
based on SCE resources (SCE employees and Managed Service Providers). The duration 11
of the high-complexity (large) development project is assumed to be 2,000 hours or 12
approximately one year. Medium complexity and simple (small) development projects 13
are assumed to require approximately six months and three months, respectively. The 14
actual duration may be more and the number of SCE employees could be less, meaning 15
the base cases are estimating a total effort and not necessarily a fixed timeframe. 16
• Vendor Labor - This is an estimate for the cost of labor provided by the software 17
company who created the COTS package. The software vendors provide us with 18
technical expertise of their software products and expertise from experience in deploying 19
their products at other companies. 20
• Systems Integrator (SI) - This is an estimate of the work performed by a person or 21
company that specializes in bringing together component subsystems into a whole and 22
ensuring those subsystems function effectively together. 23
• Business Consultants - Business Consultants assist internal Operating Unit resources 24
with many of the business activities. The Business Consultants can support development 25
of business requirements with stakeholders, assist with design reviews, and support 26
overall project management. Business Consultants are also often used if SCE is seeking 27
to deploy new business capabilities and would benefit from the experience and guidance 28
of industry experts. 29
• Technical Consultants - Technical Consultants assist when new technology is being 30
introduced or complex real-time interfaces require specific expertise. The technical issues 31
5
are often fewer than the complexity inherent in the business processes and are therefore 1
are not typically required to the same extent as the business consultants. 2
The estimation methodology also provides an ongoing annual O&M labor support cost based on 3
a percentage of the overall labor costs of the completed project. SCE uses historical ongoing support 4
costs of previously implemented capitalized software projects as a way to estimate the anticipated future 5
support costs. 6
This estimation methodology has been used to develop the cost forecasts for the projects 7
discussed in this volume. Details of the cost estimate for individual projects are provided in 8
corresponding workpapers.9
6
II. 1
OPERATING SYSTEM SOFTWARE 2
A. Operating System Software 3
Table II-2 Operating System Software8
Work Breakdown Structure (WBS)9 Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
1. Project Description 4
SCE’s IT Operating Unit (IT) oversees an array of information technology infrastructure. 5
The computing systems composed of the computing hardware, such as servers in data centers and 6
laptops on end-user desks, and supporting software, are used by the SCE workforce and SCE customers. 7
Most of the supporting business software is Commercial-Off-The-Shelf (COTS) packages, which must 8
be updated on regularly with software patches and new releases. This section of testimony addresses the 9
core Operating System Software, which includes operating systems, business intelligence systems, 10
database management systems, cross-system integration tools, and the base platform of our Enterprise 11
Resource Planning system, SAP. This Operating System Software provides a stable and reliable 12
foundation for SCE’s critical business systems. 13
8 Refer to WP SCE-04, Vol. 2 Bk A pp 7-43. 9 Operating System Software: CIT-00-OP-CS-000030, CIT-00-DM-DM-000096, CIT-00-DM-DM-000064;
Database Platform Upgrade: CIT-00-OP-CS-000061, CIT-00-SD-PM-000246; Business Intelligence Tools Upgrade: CIT-00-ET-AE-000033; Enterprise Integration Tools Upgrade: CIT-00-ET-AE-000016; Enterprise Platform Core Refresh: CIT-00-CC-CC-000022.
ForecastOperating System Software 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 5Y TotalRecorded / Forecast 26.20 26.79 11.35 2.78 29.56 5.60 6.10 12.80 20.90 11.60 57.00
Database Platform Upgrade 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 3.00 - - 19.46 0.21 2.50 - - - 2.00 4.50
Business Intelligence Tools Upgrade 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 10.21 0.06 0.00 - - - 0.30 0.50 1.00 - 1.80
Enterprise Integration Tools Upgrade 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 21.39 3.11 3.11 2.46 0.17 - 0.30 1.00 1.00 0.30 2.60
Enterprise Platform Core Refresh 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 4.19 8.05 5.29 0.00 0.00 - 8.00 8.70 - - 16.70
OS Total 82.60
Recorded Forecast
7
2. Need for the Project 1
Operating System Software operates mainframe servers, midrange servers, and storage 2
devices in SCE’s data centers, customer call centers, and personal computers. This section presents the 3
business need to update Operating System Software, including any requisite testing and remediation 4
needed for applications to run on the updated operating system software. We have organized the 5
Operating System Software into five major areas: 6
• Windows 7, Windows Server 2012, AIX (Advanced Interactive executive product 7
from IBM), and Linux. This software is required to operate either our data center and 8
customer call center computer assets or end-user devices such as laptops and 9
desktops, and provide the foundation for the critical business applications operated by 10
SCE. Remote access software – such as Citrix and Virtual Private Networks (VPN) – 11
is also included. 12
• Database platforms are software products used to organize information so that it can 13
be easily accessed, managed, updated and analyzed. An example of a database 14
platform is the Oracle Database. The information derived from these databases 15
supports SCE’s critical business processes and capabilities, such as our asset 16
management, outage management, energy procurement, and financial management 17
applications and processes. It also supports SCE’s analytics platforms used to analyze 18
critical business information, such as Energy Price and Demand Forecasting, 19
Transformer Load Management, Voltage Management, and Storm Preparations. SCE 20
uses several database platforms, including Oracle, SQL Server, SAP Hana, and IBM 21
DB2. 22
• Business Intelligence (BI) is a set of methodologies, processes, architectures, 23
technologies, and services that help transform raw data into meaningful and useful 24
information for actionable business decision making. BI tools provide SCE the 25
essential capability to gain insight into many data to better translate that data into 26
actionable information and identify new opportunities and define and implement 27
strategies and services for cost, efficiency, process and operational improvements. 28
8
• SCE uses Enterprise Integration Tools to integrate and share data stored in various 1
applications, analytics systems, and between SCE and external third-party systems.10 2
These integration tools can also consolidate data into a data warehouse to provide a 3
unified view of data from different systems, which can allow for more complex data 4
mining and analytics to be completed to inform SCE business decisions. A few 5
examples of these tools include: 6
a. Application to Application integration; 7
b. IBM WebSphere ESB (Enterprise Service Bus) and SAP PI/PO ETL Tools 8
(Process Integration/Process Orchestration for Extract, Transform and Load); and 9
c. IBM Data Stage, SAP Data Services, etc. 10
• The Enterprise Platform Core Refresh will update our enterprise resource platform, 11
SAP, to a newer version supported by the vendor. SAP is the enterprise-wide system 12
used for our back-office financial management, human resources data, supply chain 13
management, work management, enterprise asset management, customer relationship 14
management, and governance, risk and compliance functions. The SAP system and 15
associated processes are essential for the operations of all the organizations within the 16
company. This refresh is not only critical for the ongoing operations of these business 17
processes, but it will also enable SCE to leverage its transition to SAP’s HANA 18
environment, which will increase performance while optimizing servers and storage 19
requirements. This SAP core refresh is also a key enabler to SCE’s gradual shift to 20
leverage cloud-based services maintained by vendors in the areas of Human Capital 21
Management and Supply Chain Management (Ariba), and the implementation of 22
additional customer relationship and customer care capabilities as they mature. 23
a) Business requirements 24
Operating System Software continues to be indispensable to SCE’s daily 25
operations. It allows SCE to operate effectively in highly complex and geographically diverse 26
computing environments. Software becomes obsolete during its lifecycle and requires updating on a 27
10 Examples of external 3rd party systems include CAISO (California Independent System Operator), Weather
Forecast Vendors, and JP Morgan Chase.
9
recurring basis as specified by the software manufacturers.11 A refresh of our Operating System 1
Software must manage and operate computing resources, while mitigating the ongoing potential risk to 2
critical business functions due to technology obsolescence and vendor obsolescence. 3
• Technology obsolescence results when the underlying technology that an 4
application uses is identified by its vendor to be phased out (no longer 5
approved for use). The reasons for technology obsolescence include, but are 6
not limited to, exposure to security vulnerabilities, unstable technology 7
platforms, and operating system changes. 8
• Vendor obsolescence (also known as Vendor Specified End of Life for 9
products) results when a COTS application or version of an application is no 10
longer supported by the vendor. 11
In the 2016-2020 period SCE will update the existing Windows 7, Windows 8, 12
and Windows 8.1 desktop operating systems to the current Microsoft platform, Windows 10. SCE has 13
approximately 17,500 end-point devices12 supporting operations across the company. The update to 14
Windows 10 will provide for improved identity protection, data protection, threat resistance, and device 15
security. The Operating System Software refresh will avoid vendor obsolescence, simplify and 16
modernize the desktop office experience,13 enhance our cybersecurity position,14 mobilize SCE’s 17
workforce with “anywhere access,” reduce long term investment and operational costs, and supports 18
more mobile technology capabilities. 19
11 For example, the Microsoft Support Lifecycle policy provides consistent and predictable guidelines for
product support availability when a product releases and throughout that product’s life: https://support.microsoft.com/en-us/lifecycle.
12 An endpoint device is an Internet-capable computer hardware device on a TCP/IP network. The term can refer to desktop computers, laptops, smart phones, tablets, thin clients, printers or other specialized hardware.
13 Windows 10 Simplification and Modernization the desktop experience: Windows 10 will simplify the desktop experience by providing a consistent experience across phones, tablets, and PCs; it will provide current enterprise-grade security to help protect against modern threats; and simplify the management of both corporate and personal devices.
14 Enhance Cybersecurity: Windows 10 offers key architectural changes such as Enterprise Data Protection (EDP) which provides a strong foundation for some of the key data loss prevention capabilities that SCE needs. Its capabilities will be extended with the Rights Management Services (RMS) that comes with Office 365. By using both EDP and RMS, SCE will be able to strengthen protection by limiting basic copy-and-paste when appropriate and prevent printing and forwarding of documents without authorization.
10
In addition, upgrading these operating systems to Windows 10 will avoid future 1
costs related to hardware refresh, software licenses, and maintenance upgrade costs by simplifying the 2
environment and avoiding expensive custom support agreements typically required for software that is 3
no longer supported under standard agreements. 4
Database platform updates will update the database software and perform related 5
application testing and remediation. Regular updates are required to address software compatibility 6
issues, to scale the installation of the database platform in consideration of data growth and usage 7
growth, and to comply with the vendor support life cycle for that product. This must maintain acceptable 8
performance according to product vendor requirements, which specify their supported operating system 9
versions and minimum server hardware capabilities. This will provide product stability, reduce costs by 10
avoiding the need for custom support agreements for software used beyond its supported life cycle, and 11
avoid technology obsolescence. 12
The Business Intelligence (BI) Tools Upgrade will update infrastructure and tool 13
software as required to continue operating our BI capabilities. Regular upgrades are required to address 14
software issues and scale the installation of the BI applications based on data growth and usage. This 15
will allow for optimal performance of the BI applications used across the organization. 16
As the BI software products and services continue to evolve, existing capabilities, 17
such as cloud, mobile, data discovery, data visualization, and predictive capabilities, are enhanced to 18
provide new and improved functionalities. As new functionalities are introduced, SCE must keep the 19
existing products current to meet the demands required for business decision support and analysis. 20
Failure to maintain the products with upgrades puts the organization at risk of technological 21
obsolescence, unsupported software versions and increased custom maintenance support agreements. 22
As integration tools are used by more applications, the size of the environments15 23
and product versions must be reviewed regularly to verify we have the right capacity. SCE needs to keep 24
the environment current to maintain acceptable performance, comply with vendor support life cycle 25
(avoiding higher maintenance costs), provide reliability and stability of the tools, avoid technology 26
obsolescence, and facilitate new innovations and capabilities to drive better decision making. 27
15 An “environment” refers to where computer users run application software.
11
In the Enterprise Platform Core Refresh we will conduct a periodic major version 1
update (also referred to as “refresh”), pursuant to our software license contract with SAP. The last 2
version update of the software was during 2012 and 2013. This refresh is planned for 2017 and 2018. 3
The Enterprise Platform Core Refresh will require system analysis, modifications to eight existing SCE 4
SAP related custom software packages, over 500 interfaces to other SCE systems, testing, and 5
implementation. 6
As with most COTS, SAP periodically updates its software by releasing support 7
packs and enhancement packs. SAP releases support packs and enhancement packs every year. Support 8
packs deliver fixes for problems in the software reported by SAP product customers and specialized 9
computational changes (e.g., legal-related updates needed to align with changes in the federal tax laws). 10
SAP recommends that customers apply support packs upon release to keep the installed software 11
current. Enhancement packs change the software system’s functions such as restructuring the system 12
software modules for technical efficiency (e.g., moving the employee succession planning function from 13
the recruiting application to the main application for human capital), adding functions to meet regulatory 14
demands (e.g., adding the international accounting standards required for U.S. businesses), and adding 15
new business functions based on client demand. The general software industry practice advises staying 16
within two versions of the currently-available product version to maintain support from the vendor, and 17
to mitigate risks due to security problems and failure because of vendor obsolescence and technology 18
obsolescence. 19
SCE has been implementing SAP support and enhancement packs together every 20
other year since completing our SAP installation in 2008. Although SAP releases new versions of its 21
software every year, SCE has implemented a support and enhancement pack update every other year for 22
minor enhancements and software fixes, and a major core refresh to a new version of the software every 23
four years. This strategy results in our not installing the newest versions, which may contain software 24
bugs (common with new software products) that will be fixed in later releases of these versions. This 25
also minimizes our risk of technology obsolescence and is a cost effective way to minimize labor 26
installation costs. 27
3. Scope and Cost Forecast 28
a) Recorded Expenditures 29
Figure II-2 shows our 2015 request, authorized and recorded amounts for 30
operating software. 31
12
Figure II-2 2015 Request, Authorized and Recorded Expenditures
Operating System Software (Nominal $Millions)
In 2015, SCE spent more than authorized on Operating System Software as a 1
result of needing to upgrade our database software to Hana. Procurement of Hana was needed to avoid 2
extending the life of Teradata beyond end of 2017, which would have resulted in increased O&M and 3
hardware expenses. These costs were in addition to our normal server, storage and desktop operating 4
systems environments prior to vendor published end-of-life support for the Microsoft, Cisco, and 5
VMware Enterprise License Agreements and in preparation for our migration to Microsoft Office 365. 6
b) Forecast Expenditures 7
Total forecast expenditures are $82.6 million for operating software updates in the 8
2016 – 2020 period. This forecast includes the cost to update the software, update the underlying 9
operating system software where required, and conduct application testing and remediation. This five-10
year forecast represents a substantial reduction from the previous five years (2011 – 2015) expenditure 11
level of $177 million. This results from a concerted effort to use more virtualization technologies,16 and 12
16 Using virtualization allows several operating systems running in parallel on a single central processing unit
(CPU). This parallelism tends to reduce overhead costs.
$15.67 $15.67 $15.67
$29.93
$14.27
2015 Request 2015 Authorized Variance 2015 Recorded
13
attempts to lower our overall cost structure by leveraging cloud and software-as-a-service subscriptions. 1
An example of this is the implementation of the Microsoft productivity software Office 365,17 which 2
replaced software previously running in our data centers. 3
The capital forecast for this refresh program was developed using SCE’s internal 4
cost estimation model. This model utilizes industry best practices and SCE subject matter expertise to 5
estimate project cost components. SCE’s forecast for this project includes costs for software, hardware, 6
licenses, vendor labor, and SCE IT labor. See this project’s workpaper for the cost breakdown 7
information. 8
For Operating System (OS) software, we forecast $5.6 million to update Windows 9
Server 2003 and software management tools in 2016. In 2017 we forecast $6.1 million to update server 10
operating system software, including Guardium and other Microsoft server products. In 2018 we 11
forecast $7.8 million to update server operating system software and $5 million to complete Windows 12
10 application remediation. In 2019 we forecast $10.9 million to update server operating system 13
software and $10 million for Windows 10 deployment. In 2020 we forecast $6.6 million to update server 14
operating system software and $5 million to complete Windows 10 deployment. 15
For database system software, we forecast $2.5 million in 2016 to update our 16
Oracle Exadata environment, and $2.0 million in 2020 to replace the System D data warehouse. 17
For business intelligence, we forecast $0.3 million in 2017 and $0.5 million to 18
update SAP BusinessObjects suite, IBM Data Stage, and Extract Transfer and Load tools used in SCE 19
analytics platforms for data ingestion and transformations. We forecast $1.0 million in 2019 to update 20
SAP Hana Geospatial Analytics and Predictive Analytics software. 21
For integration tools software, we forecast $0.3 million in 2017, $1.0 million in 22
2018, and $1.0 million in 2019 to update Managed File Transfer (MFT) software used for Business-to-23
Business (B2B) integration; IBM WebSphere Enterprise Service Bus used in SCE.com; update IBM 24
WebSphere Application server; and update IBM SAP Process integration software used for application-25
to-application integration. In 2020 we forecast $0.3 million to update IBM Data Power software used as 26
a security gateway for business-to-business integration. 27
17 Office 365 is the brand name used by Microsoft for a group of software and services subscriptions providing
productivity software and related services running in the Cloud.
14
For the SAP Enterprise Platform Core Refresh, SCE expects to perform analysis 1
in late 2016 and complete the installation of the major refresh over a 15-month period in 2017 and 2018. 2
We forecast $16.70 million to complete this update. 3
15
III. 1
CYBERSECURITY & IT COMPLIANCE 2
The importance of cybersecurity to the utility industry and SCE has expanded as systems and 3
data have become more integral to business operations and as the electric infrastructure has become 4
more essential to national commerce and communications capabilities. Cyber-attacks are continually 5
growing in number and sophistication, and the availability of cyber weapons is on the rise as well. 6
Therefore, maintaining a strong defense against cyber-attack requires a continually evolving set of 7
strategies. Recent examples of cyber-attacks are well documented in the news media and the intelligence 8
community, which include but are not limited to: 9
• Anthem data breach resulting in losing 80,000,000 sensitive personally identifiable 10
information (PII)18 data records (February 2015).19 11
• Sony Pictures America resulting in loss of systems, data, and services with remediation costs 12
estimated in the tens of millions (December 2014).20 13
• EBay loss of a large database of user credentials (March 2015).21 14
• US Office of Personnel and Management resulting in losing sensitive data related to security 15
clearance holders (June 2015). 16
• JP Morgan Chase loss of 76,000,000 personal information records (July 2014).22 17
• Disruption of Ukraine power grid causing over 225,000 customers to lose power. (March 18
2016)23 19
Cyber-attacks are being mounted against an array of organizations resulting in significant impact 20
to customer data privacy and system availability. To protect the data privacy of our five million 21
customers, and the integrity of our critical grid infrastructure, it is imperative that SCE implement strong 22
cybersecurity controls to mitigate risk. 23
18 Personally Identifiable Information (PII), as used in US privacy law and information security,
is information that can be used on its own or with other information to identify, contact, or locate a single person, or to identify an individual in context.
19 Refer to WP SCE-04, Vol. 2 Bk A pp. 44-46. 20 Refer to WP SCE-04, Vol. 2 Bk A pp. 47-50. 21 Refer to WP SCE-04, Vol. 2 Bk A pp. 51-53. 22 Refer to WP SCE-04, Vol. 2 Bk A pp. 54-57. 23 Refer to WP SCE-04, Vol. 2 Bk A pp. 58-63.
16
Figure III-3 SCE Detected Intrusion Attempts
Figure III-3 illustrates the number of intrusion attempts24 on the SCE network over the past five 1
years. As these threats significantly increase year over year, the SCE must develop stronger defense 2
strategies. SCE employs a defense-in-depth cybersecurity strategy,25 which uses multiple layers of 3
protection to prevent unauthorized access to its systems. SCE invests in several Cybersecurity capital 4
programs. These projects fall into three primary categories: Perimeter Defense, Interior Defense, and 5
Data Protection. 6
• Perimeter Defense: Perimeter Defense includes the processes, procedures, personnel, 7
hardware and software designed to protect SCE’s information and systems from external 8
attacks. Like security defense in a home, one can think of Perimeter defense as the locks on 9
doors and windows. Basic technologies include firewalls and intrusion detection systems. 10
24 An intrusion attempt is defined as an unauthorized attempt to access a system, network or endpoint. 25 Refer to WP SCE-04, Vol. 2 Bk A pp. 64-108.
17
The perimeter defense technology prevents, protects, and detects attacks reducing the risk to 1
critical back-end systems. Perimeter Defense is especially critical to systems that are 2
accessible via the Internet. 3
• Interior Defense: The goal of the Interior Defense program is to secure SCE’s internal 4
business systems from unauthorized users, devices, and software. Interior defense is like a 5
video surveillance system in a home, to know where people are and what they are doing in 6
case we need to respond to an emergency. Advanced and integrated real-time monitoring of 7
SCE’s internal business network makes it more difficult for unauthorized users to gain access 8
to our systems and for rogue devices or software to cause business disruption. 9
• Data Protection: The objective of the Data Protection program is to protect SCE customers, 10
employees, contractors, and other personnel from identity theft. The program also protects 11
confidential SCE information residing on all computing devices from unauthorized use, 12
distribution, reproduction, alteration, or destruction. 13
• SCADA Cybersecurity: SCE must enhance cybersecurity infrastructure services to address 14
emerging advanced cybersecurity threats against Industrial Control Systems.26 The objective 15
of this effort is to implement enhanced cybersecurity controls for SCADA systems and their 16
infrastructure to address modern cybersecurity threats targeting grid control systems. 17
Although in previous filings this section was titled “NERC CIP,” SCE implemented 18
additional SCADA protections with a portion of the approved NERC CIP funding. 19
26 Industrial Control System (ICS) is a general term that encompasses several types of control systems used to
automate industrial processes, including supervisory control and data acquisition (SCADA) systems, distributed control systems (DCS), and other smaller control system configurations such as programmable logic controllers (PLC). These systems are used in a variety of critical applications and industries including energy and utilities, transportation, health, manufacturing, food and water.
18
Figure III-4 Perimeter Defense, Interior Defense & Data Protection
Authorized vs Recorded27 (Nominal $Millions)
As shown in Figure III-4, the Commission authorized SCE to invest $26 million in these 1
cybersecurity programs. SCE recorded $28 million, $2 million above authorized levels, to implement 2
needed cybersecurity tools and controls. This advanced technology purchase replaced a less effective 3
incident response tool to help SCE analysts and Incident Response teams better detect campaigns or 4
attacks on SCE computing endpoints. 5
27 Totals include authorized expenditures for Solutions for Emerging Legislative Mandates program.
$2 $26 $26
$28
0
5
10
15
20
25
30
2015 Request 2015 Authorized Variance 2015 Recorded
19
A. Perimeter Defense 1
Table III-3 Perimeter Defense28
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
1. Program Description 2
Perimeter Defense is the first line of defense against cyber-attacks. It is the outer layer of 3
protection for our Defense in Depth approach to cybersecurity, which includes the processes, 4
procedures, hardware, and software to protect critical systems such as SAP, customer data, and 5
ultimately our grid. Today, perimeter security is used to protect our back-end systems from unauthorized 6
access. When properly configured, the perimeter defenses should only permit those activities required to 7
conduct business. Using a perimeter defense security model, the perimeter technology prevents, absorbs, 8
or detects attacks reducing the risk to critical back-end systems. Cybersecurity perimeter defenses 9
include technologies such as firewalls, intrusion detection systems (IDS), application proxies and virtual 10
private network (VPN) servers. Therefore the best network security based on best practice is a layered 11
Defense in Depth approach. This approach includes implementing security in layers with each layer 12
providing an increasing level of restrictive controls. Perimeter Defense is especially critical to systems 13
that are accessible via the Internet. 14
2. Need for Program 15
The energy sector is under continuous cyber-attack,29 and the attack methods, exploits, 16
and capabilities are constantly evolving as new types of attacks are discovered. As referenced in Figure 17
III-3, intrusion attempts against SCE continue to increase. Such attacks include computer viruses, 18
worms, phishing, spyware, and advanced persistent threats, any of which could cause significant damage 19
to SCE’s information systems, if successful. Security Magazine writes, “The modern enterprise network 20
has become expansive, porous, and completely blurred due to the large number of Internet-facing 21
28 Refer to WP SCE-04, Vol. 2 Bk A pp. 109-120. 29 Refer to WP SCE-04, Vol. 2 Bk A pp. 115-116.
CIT-00-TR-RM-000002 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
Recorded / Forecast 3.64 4.04 7.06 4.83 12.19 11.77 12.80 13.00 13.50 13.50 13.50
Recorded Forecast
20
applications that have been deployed and adopted. The number of potential entry points into the 1
enterprise network has proliferated uncontrollably.”30 As SCE’s enterprise network expands and 2
integrates with more and sophisticated technologies, SCE must deploy advanced perimeter defense 3
technology to keep pace. Without these defenses, systems could be vulnerable to a wide variety of zero-4
day infections (which are previously unknown malware for which an antivirus mitigation is not yet 5
available) and could be accessed by anyone on the internet with malicious intent. 6
3. Scope and Cost Forecast 7
This project will increase the security of our controls that prevent unauthorized access to 8
the business systems and data within our internal business network. This project will implement tools 9
that increase remote access security on SCE and employee-owned devices, such as cell phones, laptops, 10
etc. In addition, SCE will continue to implement next generation intrusion protection (such as firewalls), 11
and intrusion detection systems (such as advanced data analytics capabilities), to improve detection of 12
nefarious activity. This project will also integrate these new tools and controls into our existing 13
perimeter defense layer to create a common, unified monitoring platform that allows for rapid response 14
to security events. 15
SCE is requesting $66.3 million for the 2016 – 2020 period to enable this scope of 16
work.31 The capital forecast for this project was developed using SCE’s internal cost estimation model. 17
This model utilizes industry best practices and SCE subject matter expertise to estimate project cost 18
components. SCE’s forecast for this project includes costs for software, hardware, licenses, vendor 19
labor, and SCE IT labor. 20
30 Refer to WP SCE-04, Vol. 2 Bk A pp. 117-120. 31 SCE’s cybersecurity’s efforts are focused on the protection of critical infrastructure, therefore a secure
process for disclosing detailed tactics, techniques, and procedures is necessary to help ensure its protection. In an effort to provide the Commission access to the information needed to answer specific questions regarding the cybersecurity testimony, cost forecasts, and justification, SCE can provide an in-person briefing in a closed setting, and an optional electronic reading room to review documents, if needed.
21
B. Interior Defense 1
Table III-4 Interior Defense32
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
1. Program Description 2
Whereas Perimeter Defense acts as a home’s gates, alarm systems, and locks, Interior 3
Defense acts as a video surveillance system that is focused on where people are and what they are doing 4
in the home. Interior Defense is a set of protection controls that are necessary to secure SCE’s internal 5
business systems from unauthorized users, devices, and software attempting to access SCE’s business 6
systems, and to utilize analytics to prevent attacks from happening before they start. These efforts are 7
also focused on identifying and blocking security breaches from personnel with authorized access to the 8
systems. Users of SCE’s business systems can propagate and/or launch malware33 knowingly or 9
unknowingly. Without these controls, SCE could not identify or react to an infected or malicious 10
computer attempting to infect others on the network. Early identification of suspicious activity will 11
allow us to take quicker action to minimize any potential damage that may result from interior attacks. 12
2. Need for Program 13
There are several significant changes in the business environment that create new 14
cybersecurity risks that we must defend against. 34 These changes include: 15
• Growth in mobile technologies, which provides more entry points for vulnerabilities 16
to enter our environment. 17
• Growth in cloud computing, which expands our network to environments we do not 18
directly control and which could be hosted on someone else’s network. 19
32 Refer to WP SCE-04, Vol. 2 Bk A pp. 121-126. 33 Malware is software that is intended to damage or disable computers and computer systems. 34 Refer to WP SCE-04, Vol. 2 Bk A pp. 127-131.
CIT-00-TR-RM-000004 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
Recorded / Forecast 5.97 8.20 3.93 11.97 4.72 7.41 10.50 8.50 8.00 8.00 8.00
Recorded Forecast
22
• Use of business and personal social media platforms, which if not monitored could 1
prevent us from seeing insider threat behaviors, or from identifying and preventing 2
employees from providing information to potential adversaries who would use the 3
information to attack our network. 4
• Growth in the number of internet-connected devices (Internet of Things), which 5
increase the potential attack paths/exposures to our network. 6
• Integration of software platforms and modular applications, which can create new 7
gateways for malicious activities to infect devices. 8
• Growth of insider threats, which is an ever increasing risk and popular attack path 9
that requires SCE to expand our technology to assess user behaviors. 10
3. Scope and Cost Forecast 11
This program will enable advanced and integrated real-time monitoring of SCE’s internal 12
business network, which will make it difficult for unauthorized users to gain access to our systems, and 13
for authorized users to knowingly or unknowingly propagate cybersecurity attacks. It will also make it 14
more difficult for rogue devices or software to access SCE systems and confidential data or cause 15
business disruption. This program will also address Advanced Persistent Threats (APT)35 by using 16
advanced data collection and analysis technologies that can provide early detection of potential 17
questionable activity. 18
To accomplish this, the Interior Defense program will perform the following activities: 19
• Extend SCE’s Identity and Access Management system to newer generation security 20
technology; 21
• Enhance and expand SCE’s data collection capabilities to mine and potentially 22
connect disparate pieces of data to form a clear picture; 23
• Implement technology to allow SCE to analyze collected information for security 24
threats in a more automated and effective manner; 25
• Initiate automated alerts when questionable activity is detected to enable us to stay 26
ahead of possible threats and help prevent attacks from happening. 27
35 Advanced Persistent Threats (APT) are a network attack in which an unauthorized person gains access to a
network and stays there undetected for a long period of time. The intention of an APT attack is to steal data rather than to cause damage to the network or organization.
23
SCE is requesting $43.0 million for the 2016 – 2020 period to enable this scope of 1
work.36 The capital forecast for this project was developed using SCE’s internal cost estimation model. 2
This model utilizes industry best practices and SCE subject matter expertise to estimate project cost 3
components. SCE’s forecast for this project includes costs for software, hardware, licenses, vendor 4
labor, and SCE IT labor. 5
C. Data Protection 6
Table III-5 Data Protection37
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
1. Program Description 7
The Data Protection Program safeguards SCE’s core information computing 8
environment38 by introducing controls to protect critical business information. This program will protect 9
SCE customers, employees, contractors, and other personnel from identity theft, and protect confidential 10
SCE information residing on all computing devices from unauthorized use, distribution, reproduction, 11
alteration, or destruction. This program will improve the security of information stored within various 12
databases both within and outside of SCE’s computing environment. 13
The Data Protection Program will add specialized technology that will: 14
• Increase protection and encryption of data fields within files; 15
• Protect business information on mobile devices; 16
• Enhance access controls to protect sensitive business information; 17
36 SCE’s cybersecurity’s efforts are focused on the protection of critical infrastructure, therefore a secure
process for disclosing detailed tactics, techniques, and procedures is necessary to help ensure its protection. In an effort to provide the Commission access to the information needed to answer specific questions regarding the cybersecurity testimony, cost forecasts, and justification, SCE can provide an in-person briefing in a closed setting, and an optional electronic reading room to review documents, if needed.
37 Refer to WP SCE-04, Vol. 2 Bk A pp. 132-137. 38 SCE’s core information computing environment includes customer data in Customer Service databases, back-
end systems for SCE.com, and SAP, which contains employee data.
CIT-00-TR-RM-000003 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020
Recorded / Forecast 1.39 3.64 0.44 3.10 8.18 9.00 3.50 6.00 6.00 6.00 6.00
Recorded Forecast
24
• Control the replication of business information to personal removable storage media; 1
and 2
• Protect business information stored at external sites hosting SCE business systems. 3
2. Need for Program 4
As discussed at the start of Chapter III, several large scale data breaches impacted large 5
and established companies in 2014, including Sony, JP Morgan Chase, Target, and eBay. These data 6
breaches resulted in significant financial, reputational and proprietary impacts to these companies. These 7
companies will be forced to spend hefty funds on improved security measures by way of consultants, 8
security vendors, and test runs—not to mention the fees for lawyers, pending lawsuits, and paying fines 9
from data protection authorities. Given the magnitude of these threats, it is imperative for SCE to 10
implement ongoing cybersecurity improvements to data protection. 11
3. Scope and Cost Forecast 12
This program will perform the following activities to achieve the objectives described in 13
the Project Description section: 14
• Implement enhanced controls for granular data protection by deploying Data Loss, 15
Categorization, and Identification tools; 16
• Automate data classification by tying together the different systems that contain data 17
and the ability to classify them; 18
• Monitor and alert unauthorized access to business information by leveraging the 19
monitoring and data analysis environment with new toolsets; 20
• Manage business information saved on personal devices through implementation of 21
more robust endpoint and mobile device tools; and 22
• Control the copying of business information to removable devices such as memory 23
sticks and DVDs. 24
SCE forecasts $27.5 million for the 2016 – 2020 period to enable this scope of work.39 25
The capital forecast for this project was developed using SCE’s internal cost estimation model. This 26
39 SCE’s cybersecurity’s efforts are focused on the protection of critical infrastructure, therefore a secure
process for disclosing detailed tactics, techniques, and procedures is necessary to help ensure its protection. In an effort to provide the Commission access to the information needed to answer specific questions regarding
(Continued)
25
model utilizes industry best practices and SCE subject matter expertise to estimate project cost 1
components. SCE’s forecast for this project includes costs for software, hardware, licenses, vendor 2
labor, and SCE IT labor. 3
D. SCADA Cybersecurity 4
Table III-6 SCADA Cybersecurity40
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
1. Program Description 5
This project extends security measures beyond the NERC CIP requirements governing 6
the bulk electric system by implementing additional risk-reduction methods specifically for SCE’s 7
Supervisory Control and Data Acquisition (SCADA) systems. SCE’s SCADA systems provide remote 8
control and monitoring capabilities of the electric grid. While NERC CIP requirements establish a 9
security-control baseline, SCE believes that the rise of electric grid threats and a strong understanding of 10
SCE’s specific environment warrants implementing further security control measures. 11
SCADA Cybersecurity should not be confused with SCE’s Grid Modernization 12
Cybersecurity project presented later in this testimony. SCADA Cybersecurity involves protections for 13
legacy industrial control systems that are currently connected via routable networks. Improved visibility, 14
detection, and protection controls are needed to secure these environments from the evolving threats that 15
continue to propagate against the utility industry. Grid Modernization Cybersecurity efforts are centered 16
on implementing security controls for systems that are interconnected to each other or to external 17
devices such as distributed energy resources (DERs). 18
Continued from the previous page the cybersecurity testimony, cost forecasts, and justification, SCE can provide an in-person briefing in a closed setting, and an optional electronic reading room to review documents, if needed.
40 Refer to WP SCE-04, Vol. 2 Bk A pp. 138-140.
CIT-00-TR-RM-000017 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 1.06 0.09 0.15 1.26 0.38 - 8.80 8.95 2.50 2.50
Recorded Forecast
26
SCE has invested in visibility tools and firewalls designed for grid segmentation to create 1
choke points that enable us to close off certain areas of the network in the event of an attack. It also 2
allows us greater visibility. These firewalls act like a watch post at a guarded facility checking 3
identification to validate that you are who you say you are. The goal of this current effort is to focus on 4
additional SCADA protection needs, taking a programmatic approach to reduce the risk. The approach 5
includes performing an assessment of current state cybersecurity controls, establishing desired target 6
states based on advanced cybersecurity risks, and implementing enhanced controls commensurate with 7
system risk. Changes to these systems will be used to update and enhance cybersecurity incident 8
response and investigation capabilities by improving and maturing controls or adding new ones. In some 9
cases, military grade technology will be needed to defend against nation-state attacks, which may 10
require additional implementation costs. 11
2. Need for Program 12
The energy sector has become a major focus for targeted attacks and is now among the 13
top five most targeted sectors worldwide.41 SCADA systems are increasingly becoming the target of 14
sophisticated cybersecurity attacks as evidenced by recent Black Energy attacks on Ukrainian power 15
grids.42 These systems are more vulnerable to cyber-attack as they have longer refresh cycles, fewer 16
security updates, 100% expectation of reliability to our customers, and require significant coordination 17
to test and implement security upgrades. 18
In the last three years SCE has seen a ten-fold rise in attempted cybersecurity intrusions. 19
Coupled with warnings from James Clapper, Director of National Intelligence, that advanced attacks 20
against the electric grid are an imminent threat, SCE believes that its customers are best served by 21
implementing additional SCADA security controls. Beginning in 2011, while implementing NERC CIP 22
controls, SCE began enhancing SCADA security controls as SCADA concerns arose, and resources 23
permitted. 24
The threats and challenges against the grid underscore the need to further enhance 25
cybersecurity for SCE’s SCADA systems. Below are infrastructure services that SCE believes the 26
SCADA systems should leverage to garner more robust security controls. 27
41 Refer to WP SCE-04, Vol. 2 Bk A pp. 141-170. 42 Refer to WP SCE-04, Vol. 2 Bk A pp. 171-173.
27
3. Scope and Cost Forecast 1
The SCADA cyber security project scope includes the following: 2
• Build a secure network to protect the administrative interfaces of critical tools; 3
• Develop device access controls to secure how operators interact with control systems; 4
• Develop user access controls to secure role-based access to least-required 5
privileges,43 which is a more secure profile for user access; 6
• Implement next generation malware protections to identify malware in an 7
environment; 8
• Deploy vulnerability management tools to scan the environment looking for known 9
vulnerabilities; 10
• Provide data encryption services to encrypt data at transit and at rest; 11
• Develop system monitoring services to provide greater visibility to the network; 12
• Implement threat intelligence integration tools that can automatically ingest 13
intelligence to monitor and analyze the environment; and 14
• Procure government-sponsored secure technology to defend against advanced attacks. 15
SCE is requesting $22.6 million for the 2016 – 2020 period to enable this scope of 16
work.44 The capital forecast for this project was developed using SCE’s internal cost estimation model. 17
This model utilizes industry best practices and SCE subject matter expertise to estimate project cost 18
components. SCE’s forecast for this project includes costs for software, hardware, licenses, vendor 19
labor, and SCE IT labor. 20
43 The Principle of Least Privilege is the idea that only the most minimum number of people should have access
to information and resources that are necessary for its legitimate purpose. 44 SCE’s cybersecurity’s efforts are focused on the protection of critical infrastructure, therefore a secure
process for disclosing detailed tactics, techniques, and procedures is necessary to help ensure its protection. In an effort to provide the Commission access to the information needed to answer specific questions regarding the cybersecurity testimony, cost forecasts, and justification, SCE can provide an in-person briefing in a closed setting, and an optional electronic reading room to review documents, if needed.
28
E. Common Cybersecurity Services for Generator Interconnections 1
Table III-7 CCS for Generator Interconnections45
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
1. Project Description 2
In SCE’s 2015 GRC, the Commission adopted the Common Cybersecurity Services 3
(CCS) for Generator Interconnections project.46 This project enables the design and enforcement of 4
policies that can be configured for a type of SCADA system in the electric grid. Each device on the 5
electric grid secured by CCS will have a unique key to enable secure communications with its control 6
system. This approach mitigates the risk that an attacker can seize control of the electric grid from an 7
individual device, such as a relay or capacitor bank controller, and provides the ability to rapidly 8
respond to a cybersecurity event. 9
This project will deploy a Central Security Services engine47 and Edge Security Services 10
systems48 to protect critical electricity generator interconnections. The Central Security Services engine 11
consists of two services that are physically located at SCE’s grid control centers. These include: 12
• Central Security Configuration Services, which manages secure encrypted 13
connections and system health checks on CCS member systems. 14
• Automated Security Services are control actions defined in the configuration that 15
execute when a given set of events occur. 16
45 Refer to WP SCE-04, Vol. 2 Bk A pp. 174-179. 46 This project was titled “Common Cybersecurity Services (CCS)” in SCE’s 2015 GRC. 47 A Central Security Services engine is the central system used to manage the secure connections created by
CCS. 48 Edge Security Services systems are located at the edge of grid networks that provide measurement or control
data back to central SCADA systems.
Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - 6.86 8.15 9.78 7.36 1.00 3.40 5.90 - - 42.46 2015 GRC Authorized/Request* - 6.86 8.15 5.34 8.23 8.46 8.68 45.72 2015 GRC - Original Request - 6.86 9.90 5.34 8.23 8.46 8.68 47.47 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized 2014 and 2015 levels. 2016 and 2017 reflect forecast amounts from SCE's 2015 GRC.
Multiple CITs: CIT-00-DM-DM-000141, CIT-00-SD-PM-000103, and CIT-00-SD-PM-000175
Recorded Forecast
29
The Edge Security Services systems consist of services that provide distributed 1
enforcement of cybersecurity on devices at or near the perimeter of a SCADA system. This protects 2
edge configurations from being altered in a manner that introduces cybersecurity risk to central control 3
SCADA systems. These systems are also supported by the Automated Security Services engine to take 4
automated responses when configuration events are triggered that are deemed a risk to the overall 5
system. 6
2. Need for Project 7
The Common Cybersecurity Services (CCS) project will provide enhanced cybersecurity 8
protections for critical generator interconnections. The applications on these interconnection paths 9
require low latency49 to transmit data to back-office systems. It is critical to maintain assurance over 10
these network paths as these systems make automated control decisions on the electric grid. The CCS 11
system is specially designed to provide cybersecurity assurance over these paths while maintaining the 12
minimum performance requirements to enable the functionality of low latency control systems. This 13
system provides controls to meet critical NERC CIP compliance requirements as it relates to electronic 14
security perimeters. 15
3. Scope and Cost Forecast 16
CCS for SCE’s Phasor system was implemented in 2015. This implementation used a 17
proprietary vendor solution. After conducting an assessment on the cost effectiveness of the current 18
version of CCS on the Phasor system, SCE decided that due to the high device integration costs, CCS is 19
too cost prohibitive to scale in its current form. Consequently, SCE will invest in a non-proprietary, 20
standards-based version of CCS that can scale to meet the security needs of the bulk electric system. 21
SCE believes that this revised approach is in the best interests of our customers and the security of the 22
electric grid. 23
SCE’s original, adopted forecast for this project was $47.5 million. Our revised total 24
project forecast through the end of 2017 is $42.5 million, of which $33 million has been incurred 25
through the end of 2015. This total project cost reduction results from scope modifications that include: 26
49 Low latency refers to systems that require having a very low time interval between when a message is sent
and when it is received.
30
• CCS security for EMS and SCADA will be scheduled separately as part of the 1
network upgrades and cybersecurity implementations associated with SCE’s Grid 2
Modernization program;50 and 3
• The refresh of CCS central services servers that can be deferred until 2021. 4
SCE is requesting $10.3 million for the 2016 – 2020 period to complete this project.51 5
The capital forecast for this project was developed using SCE’s internal cost estimation model. This 6
model utilizes industry best practices and SCE subject matter expertise to estimate project cost 7
components. SCE’s forecast for this project includes costs for software, hardware, licenses, vendor 8
labor, and SCE IT labor. 9
F. Grid Modernization – Cybersecurity 10
Table III-8 Grid Modernization Cybersecurity52
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
1. Project Description 11
Grid Modernization enables new capabilities to support the evolving use of the 12
distribution system. This will require many new applications that extend grid networks in a two-way 13
50 Refer to SCE-02, Vol. 10 for more information on SCE’s Grid Modernization program. Cybersecurity related
efforts during the 2016-2020 period will be addressed in the Grid Modernization Cybersecurity program within this testimony.
51 SCE’s cybersecurity’s efforts are focused on the protection of critical infrastructure, therefore a secure process for disclosing detailed tactics, techniques, and procedures is necessary to help ensure its protection. In an effort to provide the Commission access to the information needed to answer specific questions regarding the cybersecurity testimony, cost forecasts, and justification, SCE can provide an in-person briefing in a closed setting, and an optional electronic reading room to review documents, if needed.
52 Refer to WP SCE-04, Vol. 2 Bk B pp. 180-186.
CIT-00-TR-RM-781701 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 5.28 16.19 24.44 29.15 24.80 99.86
Previous GRC Request - - - - - - - -
Recorded Forecast
31
relationship with customers and third parties. The distributed intelligence53 from grid modernization 1
presents new cybersecurity challenges to the grid of the future: 2
• Two-way communications with edge devices open new avenues of attack as edge 3
computing devices will communicate with control centers over routable connections 4
in near real-time. 5
• Implementation of secure network segmentation and survivability strategies must 6
protect grid reliability if a cyber-attack occurs. 7
• Integration of new technologies with cybersecurity infrastructure. 8
• Significant upscaling of cybersecurity service layers to automate controls, 9
monitoring, and management for new grid operations. 10
• Protection of legacy systems that do not support modern security protocols. 11
• Enhanced security for system monitoring and management networks with privileged 12
access to control systems. 13
Addressing these cybersecurity challenges requires a combination of infrastructure, 14
application, and threat intelligence initiatives. Infrastructure service layers54 are needed to extend strong 15
cybersecurity controls to edge networks. New grid applications must be designed with cybersecurity 16
controls throughout their lifecycle by integrating strong access controls, secure communications, and 17
secure programming code. Integration of cybersecurity operations with external threat intelligence 18
sharing organizations55 will enable more robust incident response and investigation capabilities. 19
Cybersecurity needs to be integrated into each grid modernization component throughout its lifecycle to 20
provide a strong framework against a cyber-attack. Therefore, this project will implement the requisite 21
cybersecurity controls for the various Grid Modernization related hardware and software applications 22
discussed in SCE-02, Volume 10 – Grid Modernization.56 23
53 Distributed intelligence refers to gathering distributed energy resource data points to make power control
decisions on the electric grid. 54 Infrastructure service layers are the systems used to provide cybersecurity device access management, user
access controls, malware protection, vulnerability management, system monitoring, incident investigation and response, and data protection.
55 These organizations include government law enforcement and intelligence, private sector intelligence feeds, and electric industry specific cybersecurity intelligence sharing organizations.
56 See SCE-02, Vol. 10 – Grid Modernization for detailed descriptions of these hardware and software applications.
32
This Grid Modernization Cybersecurity project is distinct from the SCADA 1
Cybersecurity project requested in preceding testimony. Grid Modernization Cybersecurity focuses on 2
extending cybersecurity controls to distribution system substations as they join a routable wide area 3
network.57 SCADA cybersecurity is focused on enhancing cybersecurity controls for critical control 4
center systems performing central management of the bulk electric system. 5
The Grid modernization project is also distinct from the Substation Automation 3 project. 6
The SA3 project leverages existing cybersecurity services to provide enhanced security to managing 7
intelligent electronic devices. SA3 is specifically focused on secure access and management of 8
Intelligent Electronic Devices (IED). It leverages a central cybersecurity service layer to provide 9
enhanced protection to IED access controls and management. 10
2. Need for Project 11
New grid capabilities with additional communication channels increase the potential for 12
cyber-attacks. Enabling a distributed intelligence system requires real-time communications from edge 13
distribution systems to central control centers. This upstream data flow will be used to make automated 14
control system decisions that can significantly impact reliability on the electric grid. These systems can 15
be used as a foothold by an attacker to attempt to compromise various layers of the grid network. 16
However, the new communication paths provided by the wide area network will enable a centrally 17
managed cybersecurity controls designed in a more preventative and automated architecture. This 18
architecture will be designed to provide layered defense-in-depth cybersecurity controls while enabling 19
distributed intelligence systems. 20
Despite the implementation of strong preventative controls, cybersecurity design must 21
account for the possibility that a compromise of a system on the distribution network will occur. A 22
compromised system on the grid enables an avenue of attack to escalate privilege, launch malware 23
attacks, or render a grid system inoperable. Preventative controls alone will not adequately mitigate the 24
potential cybersecurity risk of a malicious insider or sophisticated attacker. The cybersecurity 25
protections must be able to identify when a compromised system behaves anomalously and execute an 26
automated response to isolate the system to minimize its potential impact to the grid operations. 27
57 A routable wide area network is a geographically dispersed telecommunications structure that allows
communications between a series of local networks. I.E. network connectivity between substations and control centers that can be routed per the needs of the grid modernization applications.
33
Implementation of these types of controls requires advanced monitoring and behavior analysis to 1
identify and prevent system exploitation. A concerted initiative is required, so cybersecurity technology 2
and best practices are implemented throughout our grid modernization effort. New two-way 3
communication paths and grid management applications must operate on secure cybersecurity 4
infrastructure and be designed with cybersecurity capabilities and requirements throughout their 5
lifecycle. 6
3. Scope and Cost Forecast 7
SCE forecasts $99.9 million for the 2016 – 2020 period to complete this project. The 8
capital forecast for this project was developed using SCE’s internal cost estimation model. This model 9
utilizes industry best practices and SCE subject matter expertise to estimate project cost components. 10
SCE’s forecast for this project includes costs for software, hardware, licenses, vendor labor, and SCE IT 11
labor. See this project’s workpaper for the cost breakdown information. This forecast will support the 12
implementation of the functions and technologies discussed below.58 13
Cybersecurity for the power grid must be carefully engineered not to interfere with 14
energy delivery functions. For instance, our power grid has legacy devices that are decades old, with 15
limited computational resources and communications bandwidth to support cybersecurity protections. 16
Control and protection devices are widely distributed; some are in unmanned, remote substations or on 17
top of poles in publicly accessible areas. Cybersecurity controls are important and have the potential to 18
disable critical grid systems if configured in an overly strict manner without proper reduction of system 19
false positives.59 Operation of cybersecurity controls must not jeopardize normal operations or 20
emergency responses. Thus, cybersecurity controls must be designed to provide the appropriate response 21
to an alert without disabling critical energy delivery systems. Grid modernization will require significant 22
redesign of cybersecurity architecture, concept of operations, and operationalization planning to 23
facilitate the organizational change management to operations technology and information technology. 24
58 SCE’s cybersecurity’s efforts are focused on the protection of critical infrastructure, therefore a secure
process for disclosing detailed tactics, techniques, and procedures is necessary to help ensure its protection. In an effort to provide the Commission access to the information needed to answer specific questions regarding the cybersecurity testimony, cost forecasts, and justification, SCE can provide an in-person briefing in a closed setting, and an optional electronic reading room to review documents, if needed.
59 System false positives are a test result which incorrectly indicates that a particular condition or attribute is present.
34
New Grid Modernization applications will enable integration with distributed energy 1
resources and communication relationships with third parties. These new interactions necessitate that 2
development efforts perform specialized secure application coding reviews throughout the development 3
lifecycle to minimize the possibility of introducing new vulnerabilities to grid. This project will 4
thoroughly vet application code through a multitude of secure code review services to identify and 5
remediate vulnerabilities in the architectures, system inputs, cryptographic implementations, access 6
controls, database security, memory management, communication sessions, and system configurations. 7
Secure coding is more challenging as it introduces limitations to application functions that could 8
otherwise be used for malicious purposes. This process will be conducted for the Grid Modernization 9
software applications mentioned above. 10
Grid Modernization also necessitates changes to the distribution grid cybersecurity 11
service infrastructure. This project will provide enhanced segmentation and inspection between 12
distributed energy resources and third parties must be implemented to enforce system separation 13
engineering principles. This must provide maximum isolation of critical service layers and systems from 14
new avenues of communication originating from untrusted systems and networks. 15
Cybersecurity control sensors placed throughout the system will need to work in concert 16
to limit any potential attack. The ability to identify the origin of a breach and segment these systems 17
from the network is paramount to limiting the impact of any given attack. Network segmentation of the 18
system must be implemented in a manner that provides segmentation between control center and bulk 19
systems communications while enabling the new communication paths required for grid modernization. 20
Systems must enforce non-repudiation60 of all user activity to deter insider threat and 21
track system usage. Access controls must be strong enough to uniquely identify each user of a system 22
without preventing access to the system in the event of connectivity loss to central authentication 23
systems. This requires implementation of distributed privileged access management systems to enable 24
auditing of shared control system credentials. This data must all be integrated into centralized security 25
operations monitoring to perform cybersecurity analytics and facilitate incident response. Upon 26
detection of suspicious access behavior the system must notify incident response teams to mobilize a 27
response to potential system access loss. This requires baselining access behavior and alerting upon 28
60 Non-repudiation refers to the concept that a user of a system is unable to deny that they performed a specific
action on that system, i.e., a log attributable to a person’s name is generated for all system activity.
35
unauthorized or irregular use of privileged system access. The system will then be able to create alerts 1
based upon unauthorized user activity. Balancing the benefits of central access control with the needs for 2
local support of systems in the event of network loss is a critical control required to protect the 3
cybersecurity of the system without compromising its reliable operation. 4
This project will also tune monitoring systems to detect anomalous and suspicious 5
activities while permitting new and existing grid applications. Electric grid controls systems operate in 6
predictable communication patterns using a limited number of communications protocols. This project 7
will implement systems that are able to aggregate logs and communication data to detect when a system 8
or systems begin communicating in an anomalous way to detect potential attackers. For example, if a 9
compromised system attempts to launch a denial-of-service attack on the grid it would generate 10
significantly more network traffic than is typical during normal operation. Maintaining an automated 11
baseline of system configurations and communication behavior will enable an immediate and 12
accelerated response to such an attack upon detection of anomalous traffic. It would isolate the origin of 13
the attack, enable automated blocking of denial of service traffic, and segment this portion of the grid 14
from critical systems until remediation is complete. This would significantly impede the effect of this 15
attack and prevent the compromise from spreading to other portions of the grid network. 16
Detection and management of authorized and unauthorized systems and software must be 17
strengthened. Advanced malware detection systems must be tuned to whitelist61 permitted applications 18
and autonomously detect known bad malware and the behavior of unknown malware. Electric grid 19
systems typically do not require frequent software changes. This simplifies the implementation of 20
whitelisting technology to prevent unknown malicious or unauthorized software from executing on the 21
system. This would require an attacker to first circumvent the whitelisting protection prior to executing 22
any malicious code on the system. This significantly increases the difficulty of an attack on a critical 23
system as protection software is integrated into central monitoring systems to detect tampering. 24
Additionally, in the event that malicious code does execute on a system despite whitelisting controls, 25
advanced malware detection systems will be implemented as part of this project to detect malicious 26
behavior and remove the malware from compromised systems. This project will also implement 27
61 Whitelisting involves controls within software that permit known good applications and code to run while
denying all other applications and code from running on a system.
36
malware protection into network security devices to prevent the spread of malware through the network 1
upon detection. 2
The infrastructure changes require tight integration with operations teams to support 3
system availability and coordinated incident response. This project will enable the convergence of 4
operations technology with IT cybersecurity controls for awareness and coordination between system 5
stakeholders to protect the grid from cyber-attack. System behavior data and alerts will be shared with 6
correct stakeholders to enable operational awareness and response. These protections will ingest system 7
configuration and log data into usable analytics dashboards tuned to alert on potential incidents in a 8
timely and accurate manner. These response procedures must be regularly tested with incident exercises 9
and penetration tests to verify the effectiveness of cybersecurity operations. 10
This project will implement secure communication protocols between grid systems by 11
using secure encryption technologies protecting the integrity of control system data. Grid modernization 12
systems will enable automated actions to be taken on the grid based on data from distributed intelligence 13
systems, so cybersecurity infrastructure services will be implemented to provide the services needed to 14
facilitate strong authenticated encryption and key management to enable these new protection 15
mechanisms. The lack of integrity protections on this control data could result in man-in-the-middle 16
attacks where systems take actions based on forged messages lacking cryptographic protection. If the 17
system is responsible for managing power it could be exploited to disable power delivery. This type of 18
protection will also apply to sensitive data leaving the grid. Protections must be put in place to verify 19
that meter and measurement data sent to outside parties has not been tampered with so it may be trusted 20
as it is passed to system stakeholders. This is a critical function to protect power delivery and 21
information sharing of grid data. Grid modernization aims to change the way power is managed and 22
delivered, necessitating adapting robust cybersecurity controls to the changes in grid modernization’s 23
electricity management model. 24
a) Alternatives Considered 25
Grid modernization systems are highly customized and adapted to specific 26
applications using communications protocols that are not traditionally employed in IT environments. 27
Due to this complexity, development of a custom-built cybersecurity control suite was considered to 28
provide a central management and monitoring for all cybersecurity controls. This approach was deemed 29
difficult to scale as no single vendor or software suite was able to provide all of the necessary controls to 30
address the risks. While custom cybersecurity solutions will need to be developed for special purpose 31
37
cases, cybersecurity controls will primarily adapt commercial off the shelf technology to support grid 1
modernization applications. 2
G. IT Support for NERC CIP Compliance 3
Table III-9 NERC CIP Compliance62
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
1. Project Description 4
This project will continue the on-going implementation of systems and processes that will 5
help SCE maintain compliance with the evolving cybersecurity-related NERC Critical Infrastructure 6
Protection (CIP) requirements. These systems and processes will improve facility access management, 7
asset change control maintenance, and physical access control. In addition, this project anticipates future 8
expenditures associated with emerging mandatory requirements. This project includes cost forecasts for 9
work important to complying with the following FERC-approved requirements: 10
1. NERC CIP-006 – Physical security perimeter requirements for low-impact facilities; 11
2. NERC CIP-014 – Physical security requirements; and 12
3. NERC CIP-010-2 and NERC CIP-007-6 – Change control requirements for numerous 13
asset types. 14
This project forecasts costs required to comply with anticipated FERC requirements 15
applicable to this rate case period. 16
62 Refer to WP SCE-04, Vol. 2 Bk A pp. 187-208.
Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 6.18 1.00 0.75 11.79 10.42 14.37 12.92 5.97 3.42 2.47
Previous GRC Request* 6.18 1.00 4.80 14.09 11.00 4.80 5.10 *The Commission adopted SCE's request for 2014 and 2015 expenditures for this program in D.15-11-021.
Multiple CITs: CIT-00-OP-NS-000377, CIT-00-OP-NS-000446, CIT-00-TR-RM-000001, CIT-00-TR-RM-000013, CIT-00-TR-RM-000014, CIT-00-TR-RM-000015,
CIT-00-TR-RM-000016, and CIT-00-DM-DM-000081
Recorded Forecast
38
2. Need for Project 1
As discussed in more detail in SCE-08, Volume 1, SCE must adhere to FERC-approved 2
CIP Reliability Standards, or risk being levied penalties or fines.63 Since 2006, NERC has developed and 3
FERC has approved five CIP versions of standards, and subsequently issued several substantive 4
revisions, such as the Revised Critical Infrastructure Protection Reliability Standards often referenced as 5
“CIP v6 Reliability Standards.”64 This project will implement technology-related solutions to enable 6
SCE to maintain compliance with these NERC CIP requirements. 7
As identified in SCE’s NERC Compliance Program section (Article II), SCE must 8
implement approved NERC CIP v5 standards, while FERC continues to evolve the Critical 9
Infrastructure Protection (CIP) mandatory standard requirements. FERC has approved recent NERC CIP 10
v5 revisions and issued orders for CIP modifications. The three NERC CIP requirement areas listed 11
above require SCE’s response to remain in compliance. CIP requirement updates and modifications have 12
been issued consistently since 2006, and SCE has responded in kind. 13
As the past ten years have demonstrated, SCE expects CIP requirements will continue to 14
evolve in this GRC period: requirement gaps will materialize, and threats and technologies will evolve. 15
The following bullets illustrate the continuous pattern of CIP requirement change and substantiate 16
SCE’s reasonable expectation of future changes: 17
• NERC CIP v5 requirements, enforceable beginning July 1, 2016, address the NERC 18
CIP Standard Requirements CIP-002 through CIP-011. The requirements include 19
work such as implementing access controls, change control, and grid system 20
perimeter controls. 21
• FERC approved NERC CIP v5 revisions, from January 21, 2016, are mostly 22
enforceable on July 1, 2016, and largely impact IT.65 Some CIPs have later 23
enforcement dates, such as CIP-003-6, which requires, for low impact Bulk Electric 24
System (BES) Cyber Systems, implementing security plans or plans that address 25
security awareness practices, physical security controls, electronic access controls, 26
and incident response plans; and CIP-010-2, which requires for transient cyber assets 27
63 See SCE-08, Vol. 1, Chapter IV, Section C, O&M for a discussion on applicable fines. 64 See FERC Order No. 822, issued on January 21, 2016, 154 FERC 61,037, Docket No. RM15-14-000. 65 See FERC Order No. 822, issued on January 21, 2016, 154 FERC 61,037, Docket No. RM15-14-000.
39
and removable media, implementing a plan or plans to include controls over software 1
vulnerabilities mitigation, introduction of malicious code mitigation, removable 2
media authorization, and unauthorized use of transient cyber assets. 3
• FERC ordered NERC CIP modifications that have yet to be drafted and approved.66 4
These modifications include: 5
o Protection of transient electronic devices used at Low Impact BES Cyber System; 6
o Protections for communication network components and data communicated 7
between BES Control Centers tailored to be commensurate with the risk posed to 8
the bulk electric system; and 9
o Modifications to the definition for Low Impact External Routable Connectivity. 10
• FERC created a new CIP, CIP-014, which addresses physical security requirements.67 11
The physical security requirements, however, depend on information technology. 12
• FERC directed NERC to conduct a study that assesses the effectiveness of the CIP 13
remote access controls, the risk posed by remote access-related threats and 14
vulnerabilities, and the appropriate mitigating controls. The results could cause 15
further mandated standard requirements.68 16
• FERC explored the need to develop mandatory requirements for cyber controls in the 17
supply chain, including during a Technical Conference on January 28, 2016. 18
3. Scope and Cost Forecast 19
This project includes cost estimates for complying with FERC-approved requirements 20
that remain to be implemented, and cost estimates for complying with anticipated FERC requirements 21
applicable to this rate case period. The FERC-approved controls require work covering the years 2016, 22
2017, and 2018. These forecasts are summarized in Table III-10 below, and follow with a description of 23
the work these forecasts will support. 24
66 See FERC Order No. 822, issued on January 21, 2016, 154 FERC 61,037, Docket No. RM15-14-000. 67 See FERC Order No. 802, issued on November 20, 2014, 149 FERC 61,140, Docket No. RM14-15-000. 68 See FERC Order No. 822, issued on January 21, 2016, 154 FERC 61,037, Docket No. RM15-14-000.
40
Table III-10 Capital Forecast by NERC CIP Standard
(Nominal $Millions)
First, to support NERC CIP-006 compliance, SCE identified 90 low-impact facilities 1
needing physical security perimeter protection as stated in SCE-07, Volume 5, Chapter V. As NERC 2
CIP required and as the historical expenditures reflect, physical access to high-impact and medium-3
impact facilities has already been addressed. For low-impact facilities, information technology 4
commensurate with the requirements can be leveraged to manage physical access. SCE plans to use 5
cost-effective smart keys to manage this physical access. Similar to hotel card key systems, SCE can 6
program the keys and remotely enable and disable key access. The solution requires performing 7
telecommunications work, which includes installing kiosk and renewal stations in various locations that 8
can connect to SCE’s network, and implementing a smart key application. Capital expenditure is 9
estimated to be $5 million in 2016. 10
Second, to support NERC CIP-014 compliance, SCE has identified nine facilities 11
requiring physical security controls as stated in SCE-08, Volume 1, Chapter V. SCE plans to execute the 12
following related to these nine sites at an estimated cost of $16.39 million covering years 2016 and 13
2017: 14
• Implement new monitoring systems to detect intrusions, including gunshots, seismic 15
events, and other activities that could pose a threat to the facility or BES. 16
• Enhance the Physical Security Information Management system to provide a secured 17
and integrated application allowing Corporate Security personnel to quickly identify 18
and respond when a threat is detected. 19
2016 2017 2018NERC CIP-006 - Physical security perimeter requirements for low impact facilities
5.00$
NERC CIP-014 - Physical security requirements 8.87$ 7.52$
NERC CIP 010-2 and NERC CIP-007-7 - Change control requirements for numerous asset types
3.00$ 1.00$
41
• Upgrade the telecommunications infrastructure to support the new monitoring 1
systems, enabling the transfer of data, alarms, and alerts between the originating sites 2
and the monitoring locations. 3
Third, to support NERC CIP-010-2 and NERC CIP-007-6, SCE must replace the current 4
and interim Test Smart Form Tool (TSFT) tool with a production tool. TSFT is used to collect evidence 5
for all work performed on T&D managed BES Cyber Assets and associated Protected Cyber Assets 6
(PCA). The current tool was never designed to be used in production. Rather, TFST accommodated 7
SCE’s needs while developing new business processes for NERC CIP compliance and to help define 8
requirements for a production tool. The current TSFT has minimal security, does not integrate with other 9
systems, and cannot accommodate the required user base. Due to the temporary nature of the current 10
tool, these shortcomings create an unnecessary compliance risk. SCE estimates capitalized software 11
costs of $3 million in 2017 and $1 million in 2018 for a production tool. 12
Regarding work required for complying with anticipated FERC requirements, IT will 13
continue to have significant responsibility for NERC CIP evolving mandated standard requirements. IT 14
is estimating costs for standard requirements being drafted and for standard requirements FERC will 15
order. To forecast the anticipated costs associated with these future requirements, SCE considered the 16
total IT expenditures for NERC CIP over the 2014 – 2016 timeframe, to forecast future NERC CIP 17
expenditures for 2018, 2019, and 2020. This is shown in Table III-11 below. 18
Table III-11 Forecast Methodology for Emerging IT NERC CIP Requirements
(Constant $Millions)
With evolving mandated standard requirements for NERC CIP, history is the most 19
reliable predictor for estimating future costs during this rate case period. Two types of historical IT-20
spend were excluded when creating the forecast, based on reasonable assumptions: 21
(1) Foundational telecommunication expenditures are excluded because SCE assumes 22
that during this GRC period the NERC CIP requirements will not have the same 23
Expected2014 2015 2016 2018 2019 2020
NERC CIP IT $3.60 $6.76 $0.50 $4.97 $3.42 $2.473-Yr Total
Recorded Forecast
$10.86$10.86
42
magnitude of foundational change affecting telecommunications as they did in the 1
past. 2
(2) CIP-014 expenditures are excluded because a limited spend history exists related to 3
CIP-014, which is a new CIP. 4
Rather than equally allocate the forecasted dollars over 2018, 2019, and 2020, we 5
allocated based on anticipated standards effective dates. 6
a) Alternatives Considered 7
For standards drafted and approved, SCE believes the best option is to forecast 8
spend based on the amount and types of work needed to meet requirements. SCE considered whether to 9
forego estimating required expenditures on NERC CIP standard requirements not drafted and approved. 10
SCE considered the probability that standards will evolve and be approved during this General Rate 11
Case cycle to be high. Therefore, SCE has embraced better planning by anticipating that mandatory 12
standard requirements will be drafted, and using historical spend as a predictor of cost, which is 13
illustrated in Table III-11. 14
43
IV. 1
TECHNOLOGY CONSOLIDATION & OPTIMIZATION 2
The Technology Consolidation and Optimization project opportunities include work to 3
standardize and reduce applications across SCE, reduce non-critical hardware, and prioritize IT 4
investments based on business outcomes. The Technology Consolidation and Optimization (TC&O) 5
section includes three enterprise-level projects and six smaller enterprise technology projects under $3 6
million. The following testimonies provide the project description, need, scope, and forecast for 7
consolidation and application rationalization opportunities that will lead to future operating cost savings. 8
A. Data Warehouse Consolidation 9
1. Project Description 10
In 2015, SCE adopted a refreshed and comprehensive enterprise analytics strategy with 11
the goal of simplifying and reducing costs in the SCE technology environment, while providing robust 12
capabilities for data-driven strategic decisions in the fast changing utility analytics landscape. At the 13
core of this strategy is the consolidation of the current SCE data warehouse environments onto an 14
enterprise analytics platform consisting of a foundation of SAP HANA and Cloudera Hadoop 15
technologies. SAP HANA is an in-memory database technology that allows the processing of large 16
quantities of data quickly. Hadoop is an open source software that provides massive storage of any data 17
and the ability to process that data at low costs. With these two technologies as a foundation, the 18
capabilities of the enterprise analytics platform include: 19
• Ability to maximize the use of Hadoop for storage and analytics by leveraging the 20
ability to store large amounts of data at low cost; 21
69 Refer to WP SCE-04, Vol. 2 Bk A pp. 209-217.
Table IV-12 Data Warehouse Consolidation69
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 9.40 4.70 2.00 0.20 0.50 16.80 Previous GRC Request - - - - - - - - Forecasts include CIT-00-DM-DM-000070 and CIT-00-OP-CS-000074
Recorded Forecast
44
• Ability to leverage HANA’s in-memory capabilities for high-performance analytics; 1
• Improved accessibility, consistency, security, and scalability of data in the platform; 2
• Flexibility to cater to different levels of analytics from reporting to statistical analysis 3
to predictive & prescriptive modeling; 4
• Integration with a comprehensive set of existing analytical tools that include the SAP 5
Business Objects Suite, SAS and ESRI to enable ad-hoc analytics, visualization, 6
reports & dashboards, search, data exploration, advanced analytics, etc.; and 7
• Ability to deliver analytics via email or through a web portal on smart phone, tablet, 8
laptop or desktop computer. 9
This project will consolidate the current SCE data warehouse environments onto an 10
enterprise analytics platform and includes the migration of all data and capability off of the current 11
Teradata platform onto the enterprise analytics platform. This includes ensuring all data integration into 12
and out of Teradata are moved to the new platform. Consolidation of other data warehouse platforms 13
onto the enterprise analytics platform will be part of other projects. 14
2. Need for Project 15
The migration off of the Teradata Data Warehouse was prompted by the need for further 16
investment in the Teradata platform, requiring a hardware refresh and software upgrade in 2016 to keep 17
the system operational and supportable. This initiative will resolve the issue with the aging Teradata 18
platform by moving to the new platform and will provide additional capability and reduced on-going 19
capital and O&M costs associated with the maintenance of the Teradata platform. It also provides the 20
foundation for the full enterprise analytics platform strategy (consolidation of additional data 21
warehouses onto the enterprise platform), enabling several additional key analytics initiatives. The 22
additional data warehouse platforms planned for consolidation onto the enterprise platform (as part of 23
separate projects) include the Customer Service Data Warehouse (on DB2 mainframe) as described in 24
SCE-04, Vol. 3—CS Re-Platform, Power Procurement Data Warehouse (on Oracle), and the SAP BW 25
Data Warehouse (on a separate instance of SAP HANA). This consolidation is illustrated in Figure IV-5 26
below: 27
45
Figure IV-5 Consolidation onto Enterprise Analytics Platform
3. Scope and Cost Forecast 1
To enable this move, SCE made a strategic purchase of SAP HANA in 2015 to support 2
this project and several others. The strategic SAP HANA software contract negotiation allows SCE to 3
purchase 12TB of HANA software and use up to 42TB within 4 years at no additional cost. The benefit 4
case for the strategic purchase of HANA was based on implementing two key projects: (1) The 5
migration from Teradata to the Enterprise Analytics Platform (this project) as part of the overall Data 6
Warehouse Consolidation, and (2) The Core Refresh of the SAP transactional systems, which includes a 7
move onto HANA as the transactional database. While this is the first implementation project to realize 8
the benefits of the strategic purchase of SAP HANA, there are other initiatives that will leverage this 9
HANA capability, including SAP Core Refresh, Grid Modernization, and Customer Service Re-10
Platform. 11
The total project cost forecast is $16.8 million. The capital forecast for this project was 12
developed using SCE’s internal cost estimation model. This model utilizes industry best practices and 13
SCE subject matter expertise to estimate project cost components. SCE’s forecast for this project 14
46
includes costs for SCE employees, supplemental workers, and consultants, software and vendor costs, 1
and hardware costs. See this project’s workpaper for the cost breakdown information.70 2
a) Alternatives Considered 3
SCE could potentially continue investing in the Teradata data warehouse 4
platform. This alternative was not chosen due to: 5
• Higher on-going capital and O&M costs for the Teradata platform; 6
• Higher project costs for future analytics initiatives due to the need for 7
increased integration and management of data across multiple platforms; 8
• Reduced capability for analytics, resulting in more complexity and less 9
efficiency for solutions for end-user analytics needs; and 10
• The benefits of moving to the enterprise analytics platform outweighed 11
continued investment in the Teradata platform. 12
B. Lotus Notes Migration 13
Table IV-13 Lotus Notes Migration71
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
1. Project Description 14
The Lotus Notes Migration project will migrate and consolidate the application 15
functionality or retire 230 Lotus Notes complex and custom business solutions to achieve cost savings. 16
This project will eliminate the overall Lotus Notes environment, and eliminate its associated software 17
license, server hardware, server software, and storage costs. This project will also eliminate operational 18
issues associated with using obsolete Lotus Notes software and maintaining co-existence of Lotus Notes 19
and Office 365. These 230 Lotus Notes business solutions support OU operations in T&D, Customer 20
70 Refer to WP SCE-04, Vol. 2 Bk A p. 217. 71 Refer to WP SCE-04, Vol. 2 Bk A pp. 218-223.
CIT-00-OP-CS-000067 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Total
Recorded / Forecast - - - - - - 3.90 3.00 - - 6.90
Previous GRC Request - - - - - - - -
Recorded Forecast
47
Service, HR, Legal, IT, Power Supply, Operational Services, Finance, and Corporate Communications. 1
These Lotus Notes solutions will be migrated to existing SCE standard tools such as SAP, Microsoft 2
SharePoint, and BMC Remedy. In doing so, this project will provide for the following operational 3
benefits: 4
1. Simplify SCE’s technology tools by reducing the number of email systems we 5
maintain and support; 6
2. Maximize SCE’s investment in Microsoft Office 365; 7
3. Eliminate the need for a major upgrade to Lotus Notes in 2018 due to technology 8
obsolescence; 9
4. Eliminate operational issues associated with using obsolete Lotus Notes software and 10
maintaining co-existence of Lotus Notes and Office 365; 11
5. Reduce ongoing maintenance costs by decommissioning all remaining components of 12
Lotus Notes at SCE; and 13
6. Improve data governance and data monitoring capabilities through adoption of Office 14
365 functionalities. 15
2. Need for Project 16
In 2014, SCE migrated email, desktop and collaboration tools from Lotus Notes to 17
Microsoft Office 365. The key deliverables of the Office 365 project were the migration of 17,500 email 18
boxes, 1,500 document libraries, and the update of the office productivity tools. The successful 19
completion of the Office 365 project in 2015 now enables SCE to simplify our Lotus Notes business 20
solutions by migrating them to Office 365 or decommissioning them. 21
There are 230 complex Lotus Notes business solutions and 202 shared Lotus Notes mail 22
boxes operating at SCE. As part of this project, SCE will migrate approximately 44% of these 23
applications and decommission the remaining 56% of applications. Of the applications to be migrated, 24
40% will be migrated to Microsoft SharePoint, 22% will be migrated to existing business applications 25
such as OpenText or Remedy, 21% will be migrated to third-party or custom applications, and 17% will 26
be migrated to SAP. Migration or decommissioning of these business solutions and shared mail boxes 27
will facilitate savings in hardware refresh, software license, and maintenance/upgrade costs estimated at 28
$9.66 million over 5 years once the project is implemented. 29
48
3. Scope and Cost Forecast 1
Project costs include all project management, organizational change management, data 2
migration services, application development services, testing services, and software and hardware 3
components.72 This estimate is based on analysis and estimation of migration costs associated with 4
existing business solutions. In addition, costs for decommissioning of all remaining Lotus Notes 5
components have been included. The project forecast is included in Table IV-13. 6
a) Alternatives Considered 7
Alternative: Continue to use Lotus Notes as a business solutions technology. This 8
alternative is not recommended due to the following reasons: 9
1. This alternative would cost SCE an additional $5.64 million over 5 years 10
between 2017 and 2021 to maintain both Lotus Notes and Office 365 email and application services. 11
These costs include an additional $4.960 million for Lotus Notes email software licenses over 5 years 12
between 2017 and 2021 and approximately $0.7 million to upgrade Lotus Notes software, servers, and 13
storage to maintain continued vendor support, operations, and cybersecurity compliance in 2017; 14
2. This alternative would not remediate the operational and usability issues 15
associated with using obsolete Lotus Notes software and maintaining co-existence of Lotus Notes and 16
Office 365; and 17
3. This alternative would prevent SCE from fully realizing the value in our 18
investment and use of Office 365. 19
72 Refer to WP SCE-04, Vol. 2 Bk A p. 223.
49
C. Backup and Disaster Recovery Optimization 1
Table IV-14 Backup and Disaster Recovery Optimization73
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
1. Project Description 2
Disaster Recovery (DR) includes the strategies, plans, and computing infrastructure to 3
minimize interruption and recover business application systems if a disaster occurs. SCE currently 4
maintains DR capabilities through redundant computing capabilities at SCE’s two data centers, in 5
Alhambra and Irvine, such that when a Mission Critical Application (MCA)74 fails at one location, the 6
failure is mitigated by the resumption of operations by its redundant counterpart in the other location. 7
This project will mitigate deficiencies in the current disaster recovery environment and replace existing 8
backup systems with newer technologies. Improvements in DR will improve SCE’s ability to enable 9
restoration of business functions, associated with mission critical systems, at either data center during a 10
disaster. 11
This project will implement new technology to improve DR capabilities in the following 12
areas: 13
• Install DR hardware and software in each data center to provide adequate support 14
should a critical application need to operate from a non-primary data center location 15
in the event of a disaster. DR hardware and software is composed of data storage, 16
network equipment, and computer processing units. 17
• Conduct controlled DR scenarios in the production environment to provide data 18
replication and system recovery for all required components. 19
73 Refer to WP SCE-04, Vol. 2 Bk A pp. 224-229. 74 SCE’s Mission Critical Applications (MCA) include applications that support power procurement,
transmission and distribution, and customer service business processes, such as the Outage Management System, Energy Management System, Power Costs Inc. (energy trading), and Customer Service System Account Management.
CIT-00-DM-DM-000140 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 2.00 2.30 1.00 0.50 5.80 Previous GRC Request - - - - - - - -
Recorded Forecast
50
• Implement new technical methods in which DR strategies are implemented for 1
applications, leveraging cloud computing concepts. 2
2. Need for Project 3
As technology advances in data center management, SCE has identified opportunities to 4
improve and advance the data recovery capabilities in the event of a disaster. SCE’s MCAs currently 5
leverage DR capabilities, but there is a lack of consistency in how each MCA uses technologies to 6
execute DR capabilities. This lack of consistency and standardization creates higher and unnecessary 7
complexity in the maintenance and execution of DR capabilities for these applications, which results in 8
longer recovery times in the event of a disaster, and increased costs to operate the DR solutions relative 9
to the solution this project will provide. Additionally, newer technologies have come to market in recent 10
years that enable increased reliability in methods of data recovery. Failure to remediate and improve DR 11
capabilities of these systems will impact our ability to restore business operations during a disaster. In 12
the event of an outage on a critical application, service to our customers, such as billing operations, 13
would be impacted. 14
3. Scope and Cost Forecast 15
This project is scheduled to begin in 2017 and will be completed in 2020. Total project 16
costs are forecast to be $5.8 million. The forecast for this project includes procurement and installation 17
of new hardware along with project team costs for SCE employees, supplemental workers, and 18
consultants to implement and test the upgraded DR environments.75 19
The first phase of this project will install storage, network, and computing resources for 20
DR capabilities associated with MCAs. SCE will transition from traditional magnetic tape backup 21
technology to newer spinning disk and cloud-computing-based approaches. To enable this transition 22
from magnetic tape, new computer processing units (also known as “appliances”) must be installed. In 23
addition to this, disaster recovery software is required to orchestrate the transfer of information and 24
recovery process across various data centers. The end result of this updated design will provide faster 25
data backup and restoration times in the event of a disaster. 26
The second part of this initiative involves migrating the DR capabilities of identified 27
MCAs to leverage the new disaster recovery approach. This transition will be completed in a phased 28
75 Refer to WP SCE-04, Vol. 2 Bk A p. 229.
51
manner based on prioritization of MCAs. The work activities during each migration will require SCE 1
employees, supplemental workers, and consultants to design, build and test the new DR functionality 2
leveraging infrastructure installed during the first phase. 3
a) Alternatives Considered 4
Alternative 1: Do not plan for automated disaster recovery capabilities across 5
data centers. In the absence of automated disaster recovery, SCE would rely on responding to a disaster 6
by identifying interim computing resources at an alternative site and restoring data from physical backup 7
media. This approach would add days or weeks to the recovery process and restoration of critical 8
services. If the target computing resources are not available, resources might have to be procured, which 9
can also add to the length of an application recovery. 10
Alternative 2: Plan for all MCA DR to be provided by an offsite service provider 11
and rely on this service provider to provision and test DR functions. This is not currently being 12
considered due to the data vulnerability risk associated with transitioning to a traditional public cloud 13
service. 14
D. Information Technology Projects less than $3 Million 15
Table IV-15 Information Technology Projects less than $3M76
Work Breakdown Structure (WBS) Forecast Capital Expenditures (Nominal $Millions)
Table IV-15 lists capitalized software projects with total project costs less than three million 16
dollars in capital funding and will start and finish within the years 2016 to 2020. Please see the 17
workpapers for these projects for detailed information on each project and its associated costs. 18
76 Refer to WP SCE-04, Vol. 2 Bk A pp. 230-262.
WBS Project Description 2016 2017 2018 2019 2020 TotalCIT-00-DM-DM-000072 Enterprise Schedulers Consolidation - - 2.25 - - 2.25 CIT-00-DM-DM-000073 Database Backup Optimization - 0.40 1.50 - - 1.90 CIT-00-OP-CS-000065 CITRIX VDI Capacity Increase 1.00 - - - - 1.00 CIT-00-DM-DM-000144 User Experience Technologies - 0.50 0.80 1.00 0.30 2.60 CIT-00-DM-DM-000145 Application Distribution - 0.40 1.20 0.80 - 2.40 CIT-00-DM-DM-000146 Modernize Tools for Software Development - 0.50 1.50 - - 2.00
Total 1.00 1.80 7.25 1.80 0.30 12.15
52
V. 1
OPERATING UNIT SOFTWARE PROJECTS 2
A. Customer Service Software Projects 3
This section addresses SCE’s 2016-2020 forecast for capitalized software projects to support the 4
Customer Service OU. This request includes the following four projects, each over $3 million: (1) 5
SCE.com Strategic Upgrade/Stabilization, (2) Digital Customer Self-Service, (3) Alerts & Notifications, 6
and (4) Meter Data Management System (MDMS) Upgrade. As described in SCE-3 and SCE-4, Volume 7
3, a key focus for SCE is modernizing our customer care technology portfolio and systems to deliver 8
simple and efficient solutions. This includes providing greater digital capabilities for customers, 9
managing customers’ contact and alert preferences, and maintaining current software to deliver accurate 10
customer bills. As further described below, the Customer Service capitalized software projects requested 11
in this rate case are proposed in support of meeting these objectives. 12
1. SCE.com Strategic Upgrade/Stabilization 13
Table V-1677 SCE.com Strategic Upgrade/Stabilization
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 14
The Commission approved the SCE.com Strategic Upgrade project in SCE’s 2012 15
GRC, and subsequently in SCE’s 2015 GRC.78 While SCE has completed the majority of work 16
approved in previous GRC applications, there is remaining work to be completed in 2016. This 17
77 Refer to WP SCE-04, Vol. 2 Bk B pp. 6-8. 78 Please see D.12-11-051, p. 859, for the authorized forecast costs in 2011 and 2012 with a 10% reduction for
all SAM projects including SCE.com and D.15-11-021 for the authorized forecast for the remaining scope of work for this project.
Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast 8.66 13.52 22.64 9.17 9.24 6.15 - - - - 69.38 2015 GRC Authorized* 8.66 13.52 22.64 13.80 - - - 58.62 2015 GRC - Original Request 8.66 13.52 30.63 13.80 - - - 66.61 * In D.15-11-021, the Commission adopted SCE's 2013 recorded costs and 2014 request.
Multiple CITs: CIT-00-DM-DM-000022, CIT-00-DM-DM-000024, CIT-00-OP-CS-000022.
Recorded Forecast
53
testimony updates the Commission on the remaining scope of work to be completed in this rate case 1
period and a summary of the completed scope of work. 2
The SCE.com Strategic Upgrade/Stabilization project replaces existing SCE.com 3
legacy web platform with a modernized IBM WebSphere web platform, and it includes the migration of 4
existing core applications to a new platform. This enables SCE to support the continued use of the 5
website as a robust source of information for rates, programs, and services, while enabling new and 6
contemporary self-service tools for customers to complete core transactions online.79 The project was 7
originally intended to span four years, beginning in 2011 and ending by 2014; but due to program design 8
changes, and longer periods required to test and stabilize functional and technical capabilities, the 9
project completion date has been extended to 2016. 10
b) Need for Project 11
SCE.com’s prior platform and architecture had reliability risks and could not fully 12
support the changing business needs of SCE and its customers. The SCE.com applications lacked a 13
standards-based design, rendering the entire system difficult to maintain and scale. If SCE did not 14
implement this project, extensive reprogramming efforts would have been needed to accommodate 15
increases in the types and volumes of transactions SCE expects in the near future. For these reasons, the 16
Commission approved SCE’s 2012 and 2015 GRC projects to replace the current SCE.com 17
infrastructure. 18
c) Scope 19
(1) Remaining Scope to be Completed in 2016 20
The remaining scope of this project includes migration of the existing My 21
Account and Billing and Payment options to the new platform, which is targeted for completion in 2016. 22
While many of the My Account information items were provided under the prior platform, this project 23
will migrate these same offerings to the new platform so customers can access these features through the 24
79 SCE considers core transactions to include Service Enablement: allowing customers to submit their electric
service turn on / off / transfer requests; Billing: access to current bills and billing history; Payment (credits / arrangements): ability for users to submit payments or set up payment arrangements; Outage: allowing customers to quickly report an outage, and receive updates about outages via the channel of choice (email, text, or voice message); and Usage Data: easy access to energy usage information to help customers understand their bills.
54
updated WebSphere platform. A high-level summary of the completed project scope for migration of 1
core legacy .NET applications is provided below. 2
(2) Completed Scope 3
The following scopes of work have been completed through 2015: 4
Online Turn-On, Turn Off, or Transfer (Move Center): Customers are 5
now able to initiate a turn-on, turn-off, or transfer-service-location request any time online.80 Additional 6
foundational capabilities have also been implemented with the SCE.com Move Center such as 7
“shopping cart” and “check out” functionality for enrolling in SCE Products/Services. 8
Device Flexibility/Responsive Design: By auto-detecting device access 9
format requirements, the webpages resize and present content appropriate for the device. This is known 10
as adopting a mobile-first design approach, which provides a multiplatform foundation that delivers 11
content parity for any device. Additionally, this eliminates the need to create separate websites with 12
content optimized for desktop, tablet, and mobile device access. The remaining programs to be migrated 13
to the mobile-first design approach include My Account and Billing and Payment options. The project 14
scope to complete this work is a part of the Digital Customer Self-Service project. 15
Outage Center: Customers have been provided with a stand-alone mobile-16
first outage center website where customers can report outages and street-light outages through their 17
mobile devices. Through Google Maps, customers have an at-a-glance view of current outages across 18
SCE’s territory with one-click access to additional details on the outage, such as the cause of the outage, 19
the status of the repairs, and the estimated restoration time. 20
Stabilization: Upon implementation of the initial phase in 2013 and 21
subsequent releases of SCE.com on the new IBM WebSphere platform in 2015, a period of stabilization 22
was required to focus on improving site performance. This effort also included identification and 23
resolution of underlying platform issues created by the necessity of having both SCE’s legacy web 24
platform and the new WebSphere platform in co-existence during this transition. The primary objectives 25
during this stabilization period were to reduce the number and severity of Business Impact Events 26
80 Customers who opt out of the Edison SmartConnect® program or non-residential customers can also use the
online process; however, the service transaction does not currently use the Remote Service Switch (RSS).
55
(BIEs);81 address the root cause of recent BIEs; understand and reduce the risk of all deployments that 1
impact SCE.com; and improve the process for future releases. Each stabilization period was a six-month 2
effort requiring all in-progress work to be temporarily placed on hold until the achievement of functional 3
and technical stability. 4
d) Recorded and Forecast Expenditures 5
The capital forecast for this project was developed using SCE’s internal cost 6
estimation model along with the recorded expenditures for 2011-2015. This model utilizes industry best 7
practices and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this 8
project includes costs for labor, hardware, licensing, and other costs. See this project’s workpaper for the 9
cost breakdown information.82 10
SCE forecast $6.15 million in 2016 to complete the remaining scope of work for 11
the SCE.com Strategic Upgrade/Stabilization project. The recorded and forecast project expenditures 12
from 2011-2016 total $69.38 million, which is $2.77 million above our 2015 GRC forecast of $66.61 13
million. The increase in project expenditures is due to the need to resolve technical and stabilization 14
issues beyond the original project scope and forecast that occurred in 2013. These challenges delayed 15
completion of the planned and previously authorized original scope of work from 2013 to 2014 and 16
2015. The difference between the updated total project forecast and the 2015 GRC revised authorized 17
project costs is discussed below. 18
In SCE’s 2015 GRC, SCE forecast project expenditures of $66.61 million to 19
complete the scope of work under a revised timeline. In D.15-11-021, however, the Commission 20
authorized 2013 recorded costs for all capitalized software projects. For the SCE.com Strategic 21
Upgrade/Stabilization Project, SCE spent approximately $8 million less in 2013 than originally forecast. 22
The Commission authorized $58.62 for this project in the 2015 GRC, as shown in Table V-16, which 23
reflected the $8 million underspend in 2013. 24
81 A Business Impact Event (BIE) is an incident that causes disruption to an IT service (e.g., application failure),
which ultimately impacts the business user. BIEs track and monitor the incident through its lifecycle and regularly communicate the status and mitigation plans to SCE business users.
82 Refer to WP SCE-04, Vol. 2 Bk B pp. 6-8.
56
2. Digital Customer Self-Service 1
Table V-17 Digital Customer Self-Service83
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditure(Nominal $Millions)
a) Project Description 2
The purpose of the Digital Customer Self Service (DCSS) project is to expand the 3
customer self-service capabilities of SCE.com by implementing solutions that make it quick and easy for 4
our customers to take action regardless of the device they are using. This project will make possible new 5
electronic billing and payment transactions, will upgrade customer security specifically for program 6
enrollment, and will make website functionality improvements. The project will create O&M savings by 7
streamlining new ebilling enrollments and will result in avoided cost savings related to customer 8
security and authentication. 9
Our customers expect simple, easy, and intuitive interactions with us through a 10
variety of methods, with an increasing reliance on mobile connectivity. Investing in digital as the 11
channel of choice for customer transactions is a primary focus for SCE. SCE aspires to have digital as 12
the channel of choice84 to serve the online needs of our customers. We expect digital core transactions to 13
increase to over 60% by 2020.85 To serve this growth in self-service transactions, SCE is undergoing a 14
major transition in how we view and manage our digital interactions. 15
83 Refer to WP SCE-04, Vol. 2 Bk B pp. 9-21. 84 SCE defines digital or digital as the channel of choice as web and mobile. 85 SCE considers core transactions to include Service Enablement: online ability of customers to submit their
electric service turn on / off / transfer requests; Billing: access to current bills and billing history; Payment (credits / arrangements): ability for users to submit payments or set up payment arrangements; Outage: allowing customers to quickly report an outage, and receive updates about outages via the channel of choice (email, text, or voice message); and Usage Data: easy access to energy usage information to help customers understand their bills.
CIT-00-SD-PM-000237 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Total
Recorded / Forecast - - - - - 3.10 7.50 4.00 2.50 1.50 18.60 Previous GRC Request - - - - - - - -
Recorded Forecast
57
b) Need for Project 1
Customers are becoming more technologically savvy and expect to be able to 2
complete day-to-day transactions online themselves such as receiving and paying bills, turning service 3
on and off, and accessing news and information.86 Easily transacting with their electricity provider 4
should be no exception. Failed digital transactions result in dissatisfied customers and, more often than 5
not, a higher-cost transaction to SCE.87 6
Part of the Customer Service Strategy is to make it easy for customers to use 7
channels like digital. Enabling and optimizing digital features allow customers to self-serve at their 8
convenience. When a digital transaction is properly designed and reliably built, it will shift and maintain 9
volume to the digital channel, meeting basic customer needs and leading to higher customer 10
satisfaction.88 11
Mobile devices are driving the growth in the digital marketplace, and the 12
transition to mobile is affecting how SCE designs our future digital interactions with our customers and 13
how we provide that service on their preferred channels. Sixty-five percent of smartphone users say 14
paying and receiving bills on a smartphone increases their satisfaction.89 On SCE.com, over 40% of our 15
traffic today is generated by mobile devices and is forecast to grow to 70% by 2020.90 For SCE to meet 16
customer expectations we must develop solutions that work on mobile devices and make completing 17
transactions secure and easy. 18
c) Scope 19
The scope of this project will leverage the enhancements established by the 20
WebSphere Platform discussed in the SCE.com Upgrade project. The project scope includes an 21
integrated set of solutions: Device Support, Electronic Billing and Payment transactions, Security and 22
Authentication, and Website Functionality Improvements. These project scope components are 23
discussed below. 24
86 Refer to WP SCE-04, Vol. 2 Bk B pp. 22-24. 87 Refer to WP SCE-04, Vol. 2C pp. 1-16. 88 Refer to WP SCE-04, Vol. 2 Bk B pp. 23-30. 89 Eric Leiserson, “Mobile Billing and Payment: Consumer Preferences and Billers’ Strategic Response,”
Fiserv, p. 12. 90 Refer to WP SCE-03, Ch. VII-X, p. 148.
58
(1) Device Support 1
The project scope for Device Support provides for the continued 2
implementation of our mobile-first framework, which auto-detects device access format requirements 3
and resizes webpages to present appropriate content, making it quick and easy for our customers to take 4
action regardless of the size of the screen. By allowing customers to use their device of choice when 5
transacting, we will increase customer usage of SCE.com and the volume of self-service transactions. 6
This device-support framework, which starts with responsive design, 91 is applied to each new and 7
enhanced digital feature, including development of conceptual wireframes, clickable prototypes, 8
execution of customer research, user experience implementation, cross-browser and device testing. 9
(2) Electronic Billing and Payment Transactions 10
The project scope includes My Account / Billing and Payment 11
optimization that is focused on improving customers’ ability to easily view and pay bills and on 12
improving the main Billing & Payment webpages. These capabilities will be enabled once the migration 13
to the WebSphere platform is complete. This project will consolidate existing websites, webpages, and 14
mobile applications into one integrated experience to simplify and make consistent the online customer 15
experience. The Electronic Billing & Payment (B&P) scope is a key enabler for increasing Paperless 16
Billing adoption. Increased customer participation in electronic billing is forecast from enrolling 17
customers who choose to pay electronically (e.g., credit card, EFT, SCE ePay) automatically in the 18
Paperless Billing program. 19
The increased customer participation in electronic billing will save 20
postage expense and billing production costs (e.g., paper, ink, printing costs). Additionally, other 21
electronic billing enrollment initiatives including marketing efforts will require SCE.com programming 22
and modifications to achieve the forecast number of new ebilling customer enrollments. The annual 23
forecast for 2018-2020 in O&M reductions for increased customer participation in ebilling is $4.839 24
million for Postage Expense (FERC 903.100) and $1,257 million for billing (903.500).92 The forecast 25
savings in postage expense and billing expenses depend on approval for this capitalized software project 26
or the reductions will need to be removed from the O&M forecast. Table V-18 below shows the forecast 27
91 Responsive web design is a webpage approach aimed at allowing webpages to function on multiple devices,
independent of display device (e.g., desktop monitors, mobile phones, tablets). 92 See SCE-03, Chapter IV. B. for Billing and Chapter IV. D. for Postage testimony.
59
number of new ebilling transactions for 2016-2018 and the 1.25 million in additional customers 1
expected to participate in ebilling by the Test Year. 2
Table V-18 SCE Planned Electronic Billing Program Initiatives
Forecast New Customer Ebilling Transactions and Enrollments 2016-2018
(3) Security and Authentication 3
This project will also enhance our existing Authentication framework 4
while upgrading our customer-facing identity management security. Authentication is one of the largest 5
hurdles for self-service transactions for SCE customers expecting to access and complete core 6
transactions without a lengthy, complex registration and log-in process. The project scope for Security 7
and Authentication includes simplifying account set-up and password resets that allow for guest 8
transactions through Two-Factor authentication.93 Online Two-Factor authentication will increase the 9
security of authentication and increase the success rate of customers signing on. Currently, SCE.com 10
requires an account number to register as a user, which can be inconvenient for customers unable to 11
93 Two-Factor Authentication is a security process in which the user provides two forms of identification from
separate categories of credentials.
Line No.
Description 2016 2017 2018
Estimated New Ebilling Transactions1 Default SCE ePay Customers 2,201,859 1,169,289 - 2 EV Sweepstake - 960,000 480,000 3 Default Credit Card Customers 379,512 212,796 - 4 Default DP Customers 486,918 462,306 - 5 Offer Paperless to remaining - 210,000 126,000 6 iPad Promotion 187,150 88,850 - 7 Marketing 537,207 500,000 8 New Initiatives 997,580 1,045,129 544,871 9 Default EFT Customers 345,000 1,635,000 - 10 Total Yearly New Ebilling Transactions 5,135,226 6,283,370 1,150,871 11 Total Cumulative New Ebilling Transactions 5,135,226 11,418,596 12,569,467 12 Weighted Average Postage Rate $0.385 $0.385 $0.38513 Cumulative Postage Savings $1,977,062 $4,396,159 $4,839,24514 Postage Study Savings in $000 $1,977 $4,396 $4,839
15Estimated Number of New Ebilling Customer Enrollments based on 10.05 mailings per year 510,968 1,136,179 1,250,693
60
quickly find their bill and locate their account number. The project will allow SCE.com users to choose 1
from multiple options to authenticate (e.g., phone number, zip code, address, and social sign-on). This is 2
expected to avoid increasing the number of calls placed to our Customer Call Center (CCC) to assist 3
with these requests. Additionally, the project scope includes providing different levels of account access 4
through roles/permissions to further the customer experience and protect the account holder from 5
unauthorized changes to their SCE account. 6
Table V-19 below provides the forecast avoided cost savings from the new 7
Authentication framework. The forecast is based on estimated number of calls to the CCC by transaction 8
type, historical percentage of customers who failed to complete authentication and then exited SCE.com 9
or clicked on “Contact Us,” and forecast increase of customers using SCE.com post-authentication 10
improvements. The forecast avoided cost savings to be realized with implementing the Authentication 11
initiative is forecast to be $3.5 million over 2016-2020 period.94 12
Table V-19 Digital Customer Self-Service Project
Authentication Framework Forecast Avoided Costs 2016-2020
(4) Website Functionality 13
The Project Scope for Website Functionality centers on targeted SCE.com 14
user experiences and capability improvements by using data-driven customer research, key performance 15
94 Refer to WP SCE-04, Vol. 2 Bk B pp. 18-20.
Line No. Description 2016 2017 2018 2019 2020 Total
1Projected Increased SCE.com Transaction Growth 48% 54% 59% 63% 66% n/a
2 Password Reset 58$ 246$ 259$ 268$ 278$ 1,109$ 3 User ID Recovery 10$ 42$ 44$ 46$ 48$ 190$ 4 My Account Registration 325$ 456$ 472$ 488$ 1,741$ 5 Summer Discount Program 12$ 26$ 27$ 28$ 93$ 6 CARE Enrollments 43$ 91$ 94$ 97$ 325$ 7 Other Program Enrollments 3$ 6$ 6$ 6$ 21$
8Total avoided cost of customer calls related to authentication 68$ 671$ 882$ 913$ 945$ 3,479$
61
indicators, and metrics. Web intelligence is a combination of digital and business metrics that examines 1
the performance of a website and visitor’s interactions with site features. SCE can then measure the 2
impact of these interactions across the enterprise. SCE’s ability to leverage improved web intelligence to 3
make customer-focused, data-driven decisions along with customer testing and internal performance 4
testing will be a key shift in how SCE sets its website priorities. Improved website data and metrics 5
coupled with knowledge from visitor interactions will allow us to present and offer the right site features 6
and content, and measure the impact of these interactions across the enterprise. The following are 7
examples of website functionality improvements: 8
• Optimizing the Customer Support pages based on the results from data 9
driven metrics and analytics; 10
• Evaluating the “Pay as Guest” vs. “Log In & Pay” changes from the 11
previous year to further optimize to drive customer satisfaction; 12
• Strengthening metrics to allow us to better pinpoint user experience 13
problems and improve customer satisfaction ratings and increase 14
online adoption; 15
• Updating the Billing & Payment pages to promote Paperless Billing 16
and streamline content so it is easier to identify additional options; and 17
• Updating the Home Page, Move Center, and Outage Center pages to 18
promote key features customers care about most and streamline 19
content so it is easier to identify additional options. 20
Along with Web Intelligence functionality, digital support services can 21
assist us in retaining customers in the digital channel. Digital support services include the integration of 22
intelligent virtual assistance for customers with a constantly improving knowledge base, resulting in 23
seamless interactions that increase customer satisfaction and conversion rates. Forrester states: 24
62
“Web chat will grow because it preserves web self-service context. 1
Online chat adoption among US online adults has significantly risen in 2
the last five years — from 38 percent in 2009 to 43 percent in 2012 to 3
58 percent in 2014. Chat offers several benefits to the customer: Firms 4
can quickly connect customers to an agent with the right skills to 5
answer the question without having to navigate an arduous interactive 6
voice response (IVR); questions can be succinctly resolved in near 7
real-time; and agents can leverage customer behavior on the website 8
to move the conversation forward instead of rediscovering information 9
that has already been communicated to the customer.” 95 10
d) Forecast Expenditures and Cost-Benefit Analysis 11
The Digital Customer Self-Service project is a five-year project beginning in 2016 12
and is planned to be completed in phases through 2020 with a forecast total expenditures of $18.6 13
million.96 The capital forecast for this project was developed using SCE’s internal cost-estimation 14
model. This model utilizes industry best practices and SCE subject matter expertise to estimate project 15
cost components. SCE’s forecast for this project includes costs for labor, hardware, licensing, and other 16
costs. See this project’s workpaper for the cost breakdown information.97 17
The Digital Customer Self-Service project operational benefits result in a benefit-18
to-cost ratio of 1.21.98 As discussed above, Customer Service operational savings relate to ebilling 19
customer enrollments that reduce both postage and billing expenses. IT operational savings of 20
approximately $1.155 million will be achieved by decommissioning servers and lowering licensing costs 21
once the Authentication framework is placed into service. SCE expects to incur project-related expenses 22
that will not be capitalized. These expenses include IT O&M expenses of $1.1 million for vendor 23
authentication software licenses and other IT support and maintenance expenses as well as one-time 24
marketing costs of $750,000. The marketing costs will be incurred to increase customer awareness of the 25
new functionality to conduct core transactions through SCE.com. 26
95 Refer to WP SCE-04, Vol. 2C pp. 17-30. 96 Refer to WP SCE-04, Vol. 2 Bk B p. 21. 97 Id. 98 Refer to WP SCE-04, Vol. 2 Bk B pp. 18-20.
63
The benefit-to-cost ratio analysis assumed an overall project life through 2022 1
(five years after 2017). This is appropriate when the majority of the capital expenditures are in service. 2
The benefit-to-cost analysis also escalated all future costs and benefits to year-of-expenditure, then 3
discounted to 2015 NPV using a 10% discount rate. Finally, the sum of the present-value benefits was 4
divided by the sum of the present-value costs to derive the benefit-to-cost ratio for the project. The 5
benefits and ongoing costs from this project that occur from 2021-2022 will be reflected in SCE’s next 6
2021 GRC Test Year. Table V-20 shows the forecast project capital expenditures and O&M savings 7
from 2016 through 2020. 8
Table V-20 Digital Customer Self-Service Project
Benefit to Cost Ratio ($ in Millions)
e) Alternatives Considered 9
The alternative considered for this project was to maintain the current state of 10
SCE.com without the necessary improvements, increased functionality, and O&M savings that this 11
project is designed to deliver through the IBM WebSphere Portal framework. The result of maintaining 12
the current state of SCE.com without this project would be removal of the forecast benefits of over $25 13
million from 2016 through 2020 for new ebilling enrollments. Additionally, the avoided cost of $3.5 14
million would need to be added to the O&M forecast as an increased number of customer calls would 15
occur without the improved security and authentication that this project will put in place. Lastly, 16
increased customer usage of SCE.com as a key tool to improve customers’ management of energy 17
consumption and participation in utility programs is less likely to be achieved. 18
Revenue RequirementNominal Present Value
1 Project Capital 18.60$ 2 Total O&M 9.37$ 3 Total Costs 27.97$ 26.21$ 4 Benefits 43.51$ 31.84$ 5 Benefit to Cost Ratio 1.21
Line No.
Description
64
3. Alerts & Notifications 1
Table V-21 Alerts & Notifications99
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
The Alerts and Notifications project will provide timely and accurate digital 2
communications to customers, maintain customer data privacy, and facilitate compliance with the 3
Telephone Consumer Protection Act (TCPA) and Controlling the Assault of Non-Solicited Pornography 4
and Marketing Act (CAN-SPAM) federal laws. This project is the consolidation of two capitalized 5
software projects that were authorized in our 2015 GRC: the Alerts & Notification project and Outage 6
Communication project.100 This testimony will update the Commission on the completed scope of work, 7
remaining scope of work, and recorded and forecast costs to complete the project. 8
a) Project Description 9
The Alerts & Notification project will provide a new centralized and flexible 10
platform of systems to store customer contact information and alert preferences, and tools to manage 11
those preferences. The new platform will support, among other things, the following system capabilities: 12
(1) centralized storage of all customer contact information and digital alert preferences, (2) functionality 13
to auto-enroll customer contacts into various alerting services (e.g., outage alerts), (3) customer ability to 14
directly manage their contact information and alert preferences (e.g., outage, demand response, billing 15
and payment) using self-service tools (e.g., web preference center and Interactive Voice Response 16
(IVR)), (4) delivery of alerts and notifications including the packaging, scheduling, and messaging 17
99 Refer to WP SCE-04, Vol. 2 Bk B pp. 31-36. 100 See D.15-11-021, pp. 249-254.
Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - 0.50 7.64 0.60 4.30 4.90 0.60 - 18.54 Previous GRC Request* - - - 3.40 12.30 4.10 - 19.80 *The Commission adopted SCE's request for this project in D.15-11-021.
Multiple CITs: CIT-00-SD-PM-000107, CIT-00-SD-PM-000238, and CIT-00-SD-PM-000171
Recorded Forecast
65
alignment to enrolled customers, and (5) improved reporting on customer contact enrollment, customer 1
type, alert preferences, and unenrollment reasons. 2
b) Need for Project 3
In 2014 and throughout much of 2015, SCE used multiple isolated systems to 4
store contact and alert preferences, and multiple notification systems to send alerts to customers. The 5
new platform will consolidate multiple core and auxiliary support systems into two systems: a contact 6
and preference management system (single source of accurate information) and a notification system. 7
The new platform reduces the complexity of SCE’s technology portfolio. It also increases customer 8
adoption by allowing customers to enroll and receive timely and accurate alerts for the programs and 9
services they choose, and to select the digital channel they prefer most (email, voice or text).101 Digital 10
alerts will include core electricity service alerts such as outage alerts and final call notices, and optional 11
alerts and notification such as bill ready notification, demand response program events, and energy 12
efficiency recommendations.102 13
The new Alerts & Notifications platform is necessary to provide integrated and 14
scalable systems to support compliance with TCPA and the CAN-SPAM federal laws. The new platform 15
will store customer consent to use their landline, mobile phone, or email address for automated digital 16
communications, which will reduce SCE’s risk of TCPA and CAN-SPAM non-compliance. This portion 17
of the project scope will develop and maintain a centralized repository to capture customer “Do Not 18
Contact” preferences for meeting the “Do Not Call” requirements set forth in various federal statutes.103 19
c) Completed Project Scope 20
SCE is using a phased development approach for the Alerts and Notifications 21
project and has implemented Releases 1 and 2a. These two releases are discussed below: 22
Project Release 1 – Implemented in September 2015 23
Release 1 provided the capability for approximately 300,000 eligible small 24
business customers to optionally enroll and receive proactive maintenance and repair outage alerts via 25
101 Refer to WP SCE-04, Vol. 2 Bk B pp. 35-36. 102 There are approximately 3.5 million maintenance or repair outages each year that impact residential and non-
residential customers. 103 E.g., Mobile Marketing association and Direct Marketing Association Code of Conduct, California’s Shine
the Light Law, FCC Telephone Consumer Protection Act, Telemarketing and Consumer Fraud and Abuse Prevention Act, CAN-SPAM.
66
their channel of choice (email, voice, or text message). This release improved and simplified outage alert 1
message content and format for all enrolled customer types and facilitated self-service opt-out of outage 2
alerts when desired. The opt-out process also identifies when an outage alert delivery failed due to 3
inactive phones and email addresses, which enables SCE to discontinue sending unnecessary outage 4
alerts. The total cost to implement Release 1 was $1.3 million. 5
Prior to Release 1, small business customers only received initial maintenance 6
outage alerts via U.S. Mail or door hangers. When there were outage schedule changes a few days 7
before the outage there was rarely enough time to inform impacted customers before the original outage 8
date. Release 1 addressed this issue by enabling digital maintenance outage updates throughout the 9
lifecycle of the outage so small business customers receive sufficient time to prepare and help ensure 10
their employees and customers remain informed. 11
Project Release 2a – Implemented in December 2015 12
Release 2a established a new SAP CRM platform for outage alert preferences that 13
stores and manages small business and residential customer contact information and digital outage alert 14
preferences. The new SAP CRM platform delivers the following functionality: (1) enabled residential 15
customers to enroll and receive proactive maintenance and repair outage alerts using self-service tools, 16
(2) enabled new customer self-service tools to manage their outage alert preferences on SCE.com, (3) 17
established auto-enrollment for eligible customer contacts in outage alerts, (4) automated the processing 18
of outage alert customer opt-outs and unenrollment for inactive phones and email addresses from 19
required systems, and (5) provided new and improved outage and alert enrollment reporting. Release 2a 20
established the base framework of the new platform that will help SCE create a simple and efficient 21
outage experience for our customers. For Release 2a, the recorded costs to implement this project were 22
$6.8 million in 2015. 23
d) Remaining Project Scope 24
Project Release 2b – Planned Implementation in Early 2017 25
The next phase of this project is Release 2b, which will refine our customer 26
contact and outage alert preference self-service tools and the actual content of the outage alerts based on 27
customer feedback. Release 2b scope includes (1) simplifying the customer self-service alert-preference 28
management online experience by making it easier to enroll in outage alerts, (2) introducing new 29
customer self-service alert preference management tools using Interactive Voice Response (IVR), and 30
67
(3) refining outage alert preparation and send rules to improve the timeliness and accuracy of outage 1
alerts sent to enrolled contacts. 2
Release 2b is expected to improve our customers’ outage experience satisfaction. 3
This Release will also help to avoid increasing customer calls to the Customer Contact Center (CCC) to 4
update their contact information and outage alert preferences, by providing customers with effective and 5
easy-to-use self-service tools along with timely and accurate outage alert communications. 6
Future Project Releases – Planned Implementation in 2018 and 2019 7
Additional releases are needed to continue expanding and enhancing SCE’s 8
digital Alerts and Notifications platform to realize all the business capabilities described above. The 9
remaining scope includes the following: 10
• Migrating contact and alert preference information from multiple systems 11
(i.e., legacy Customer Service System (CSS) and SCE.com) into the new 12
platform; 13
• Decommissioning of multiple legacy and aging contact management and 14
notification systems where possible; 15
• Providing new customer self-service and employee tools, such as: 16
o Customer contact information management (e.g., name, phone, phone 17
type, email address); 18
o Enrollment and preference management for migrated alert programs (e.g., 19
Critical Peak Pricing, Summer Discount Plan, Budget Assistant); 20
o Enrollment in new alert programs; 21
o Preference management for core utility alerts (e.g., final call, pending 22
discount, rotating outages); 23
o Other communication preferences management (e.g., marketing 24
communication opt-in/opt-out, Do Not Contact, and mobile consent); 25
• Enabling lower cost digital alert capabilities for migrated and new alert 26
programs; and 27
• Expanding and improving contact information, digital alert participation, and 28
alert reporting. 29
68
e) Recorded and Forecast Expenditures 1
As discussed above, the total cost to implement Release 1 was $1.3 million 2
($500,000 in 2014 and $800,000 in 2015). These costs included SCE and vendor labor costs to support 3
the design, construction, testing, implementation, and warranty support. For Release 2a, the recorded 4
costs to implement this project were $6.8 million in 2015, which included costs for SCE and vendor 5
labor to support the design, construction, testing, and implementation, as well as for new hardware and 6
software licenses. The overall forecast for the Alerts & Notifications program is $18.54 million, which 7
includes the recorded costs for Release 1 and 2a and forecast expenditures for Release 2b and future 8
releases from 2016 through 2019.104 9
4. Meter Data Management System (MDMS) Upgrade 10
Table V-22 Meter Data Management System Upgrade105
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 11
In SCE’s 2015 GRC, the Commission authorized upgrades to the Meter Data 12
Management System (MDMS) and the Network Management System (NMS) as part of the Enhanced 13
Metering and Usage project. SCE completed the upgrade of the NMS in 2014. This testimony explains 14
why SCE deferred implementing the MDMS upgrade from 2016 to 2017 and describes the current 15
schedule and forecast for implementation. 16
The MDMS is the repository for the meter and event data collected from the 17
Edison SmartConnect® meters, which is used to bill over five million customers. Because Itron, SCE’s 18
vendor, committed to support the existing MDMS software through 2016, the planned MDMS upgrade 19
104 Refer to WP SCE-04, Vol. 2 Bk B p. 34. 105 Refer to WP SCE-04, Vol. 2 Bk B pp. 37-43.
CIT-00-SD-PM-000236 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - 5.43 1.40 6.70 - - - 13.53 Previous GRC Request* - - - 6.70 - 9.90 - 16.60 *The Commission adopted SCE's request for this project in D.15-11-021.
Recorded Forecast
69
did not needed to be implemented in 2016 as previously anticipated. In 2013, Itron released its latest 1
version of Itron Enterprise Edition (IEE) MDMS software, which results in SCE being one version 2
behind the current product release. If SCE falls behind over two product releases, SCE may lose Itron 3
product maintenance support. Upgrading to the latest Itron IEE MDMS version will also improve 4
functionality and speed of data collection used for billing. To maintain compatibility with the new 5
product release, the new MDMS software requires updating Windows 2003 and Oracle 10.2 software. 6
b) Need for Project 7
Stabilizing and maintaining the MDMS environment by implementing the Oracle 8
EXADATA106 solution will minimize unbilled revenues and reduce customer complaints regarding late 9
billing. Further, this project will improve MDMS system reliability such that the Edison SmartConnect® 10
infrastructure can minimize the volume of billing disruptions and reduce customer impacts. The reduced 11
processing time reduces the risk of billing delays, minimizing impacts and complaints from customers 12
who may also rely on the availability of daily data on SCE.com. 13
Edison SmartConnect® relies on COTS products from SCE’s vendor for meter 14
usage data collection from the NMS and the MDMS. To keep the software up-to-date for support, 15
regular vendor upgrades are required (18-24 month cycle). The new enhancement packages are available 16
through SCE’s licensing agreements with the vendor. These enhancements must be made to maintain the 17
reliability of the meter data collection system. 18
The daily collection of meter data is used for bill preparation and made available 19
daily on SCE.com for use by customers, such as our customers participating in Net Energy Metering 20
(NEM) and demand response, who view and track their daily energy use. Also, customers who have 21
enrolled in online account management can view their daily usage on SCE.com. Failures in the data 22
collection process can cause processing delays beyond 24 hours, impacting the following day’s data-23
collection process. These failures can result in late billing and delays in the upload of daily usage data 24
for viewing on SCE.com. The MDMS Stabilization Plan will reduce the daily processing time from 20-25
24 hours to 12-14 hours, and improve the recovery time from system failures. With a shortened 26
processing time, unexpected failures could be resolved within 24 hours and avoid impacting the next 27
106 Oracle Exadata Database is a combined computing and storage system that runs Database software.
70
data-collection cycle. Implementing the upgrade to EXADATA solution will improve the MDMS 1
stability, allowing for faster data loading and recovery times. 2
Besides the Oracle Exadata implementation needed to stabilize the MDMS, the 3
project includes Windows and Itron software upgrades, which are needed for compatibility to support 4
the MDMS. This will also extend product maintenance support by the vendor. If SCE falls behind over 5
two product releases, SCE may lose Itron product maintenance support affecting the end-to-end 6
reliability of the application, affecting the downstream billing processes. SCE resources are not trained 7
to support and maintain proprietary Itron products, so retaining Itron product maintenance support is 8
essential. 9
c) Scope and Cost Forecast 10
The project will be implemented in two phases. Phase 1 includes stabilization of 11
the current MDMS environment and will be completed in 2016. Phase 2 will include software upgrades 12
to preserve vendor support of the system and devices and will be implemented in 2017. 13
(1) Phase 1 14
Phase 1 consists of a hardware upgrade to the Oracle Exadata appliance. 15
This includes replacing the existing IBM AIX hardware with an EXADATA platform. This will upgrade 16
the Oracle version for better management of Oracle processes and improve stability and reliability of the 17
system improving throughput within the MDMS database. The upgrade will improve the data load and 18
batch processes to reduce customer impacts and prevent billing delays. The project focuses on Customer 19
Service databases,107 which will be migrated through the MDMS Stabilization project. 20
(2) Phase 2 21
Phase 2 consists of a Windows upgrade and the IEE 8.x software upgrade 22
to the latest version of IEE, making it compatible with the upgraded versions of Windows and Oracle as 23
required to maintain vendor support. Edison SmartConnect® relies on COTS products from the vendor 24
for meter data collection using the MDMS for managing metering data. The vendor’s products are 25
critical infrastructure components for the SCE-integrated systems, and SCE has a contractual obligation 26
to maintain the software within the latest two release versions to preserve vendor support of the system 27
and devices. These commercial products are enhanced and upgraded by the vendor regularly. Because 28
107 Network Management System (NMS), Cell Relay Configuration Management System (CGS), SCE.com, and
Event Notification System.
71
the system and the process are new, technology and COTS enhancements will occur more frequently 1
than standard. Once the system is in steady state, the duration between upgrades will become longer. 2
This is a common way for software vendors to upgrade their systems as new offerings are developed, 3
improvements in the speed and efficiency of managing data are developed, and problems identified 4
since the previous upgrade are fixed. The new enhancement packages are available to SCE via its 5
licensing agreements with the vendor. 6
d) Recorded and Forecast Expenditures 7
SCE forecasts $1.40 million in 2016 and $6.70 million in 2017 to complete the 8
remaining scope of work for the MDMS project. These expenditures are composed of labor, systems 9
hardware, and licensing costs. The forecast is based upon previous experience with similar COTS 10
projects and work performed on MDMS, labor rates, and cost components. Table V-22 above illustrates 11
the forecast expenditures of the MDMS Stabilization and Upgrade project by year. The overall project 12
forecast of $13.53 million is $3.07 million below the original request of $16.60 million.108 The 13
decreased forecast expenditures are a result of more refined and increased confidence level in the 14
estimates based on more recent information. Furthermore, the product version to be implemented 15
includes SCE requirements in the base software and hardware that will reduce the overall cost. 16
108 Refer to WP SCE-04, Vol. 2 Bk B p. 43.
72
5. Customer Service Projects Less than $3M 1
Table V-23 Customer Service Projects less than $3M109
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
Table V-23 above lists other capitalized software projects for Customer Service with total 2
project costs less than three million dollars in capital funding and will start and finish within the years 3
2016 to 2020. Please see the workpapers for these projects for detailed information on each project and 4
its associated costs.110 5
B. Transmission & Distribution Software Projects 6
1. Work Management Solutions 7
As part of SCE’s Operational Excellence (OpX) efforts to reduce operating expenses and 8
increase productivity, SCE’s Transmission & Distribution Operating Unit (T&D) identified Work 9
Management process improvement opportunities across key Work Management areas of the Initiate, 10
Plan, Schedule, Execute and Close (IPSEC) process.111 While T&D plans to improve performance with 11
process changes only, technology solution investments are required to fully achieve all Work 12
Management process efficiencies. This section, and the projects that follow, extend the T&D Work 13
Management OpX efforts. 14
T&D’s investment in maintaining and improving the grid has grown over the last 3 years. 15
Managing such a large work portfolio has depended on employees such as planners, schedulers, project 16
managers, analysts, and managers to closely monitor workloads, track task completion, and balance 17
109 Refer to WP SCE-04, Vol. 2 Bk B pp. 44-73. 110 Id. 111 SCE-02, Volume 1, Operational and Service Excellence, Work Management.
WBS Project Description 2016 2017 2018 2019 2020 TotalCIT-00-SD-PM-000170 2015 GRC Rate Changes 1.10 - - - - 1.10 CIT-00-SD-PM-000239 2018 GRC Rate Changes - 1.00 1.00 - - 2.00 CIT-00-SD-PM-000245 Contact Center Optimization - - 2.90 - - 2.90 CIT-00-DM-DM-000060 Itron 3.9 NMS Upgrade project - - - 1.00 - 1.00 CIT-00-SD-PM-000172 SmartConnect Monitor&Analysis (SCMAS) - - 1.00 - - 1.00
Total 1.10 1.00 4.90 1.00 - 8.00
73
work against resources. Employees across departments and business functions perform these work 1
management activities using independent processes and tools. They do so leveraging a blend of core IT 2
systems (i.e., SAP and Design Manager), outdated Project Management solutions, and standard 3
Microsoft applications such as Excel and Project. Processes, where similar, do not allow for data across 4
departments to be easily integrated. Processes for T&D internal and contract resources are dissimilar, 5
making data sharing and data integration virtually impossible. 6
Work Management within T&D can be improved by optimizing work schedules and 7
resource assignments across the IPSEC process. To meet the increasing demands of the grid and our 8
customers, SCE performed two process assessments with separate independent partners. These 9
assessments resulted in recommendations for improving the way SCE manages its work across the 10
IPSEC model. 11
The “Work Management Assessment” looked at organizations, people, processes, and 12
technology to address key objectives. The objectives include: (1) Increase the accuracy of project scope 13
estimating, (2) Reduce the technology footprint and redundancies by using common tools, (3) Improve 14
and standardize IPSEC workflow processes, and (4) Improve performance through integrating enterprise 15
solutions across the company. The Project Controls Improvement Initiative (PCII) performed a gap 16
analysis between SCE and industry best practices in the areas of project scope, costs, and expenditure 17
controls.112 This initiative identified gaps in the current scope development, cost estimating tools and 18
processes, while also noting the benefits of integrating the scope estimating process with our current 19
enterprise system (SAP). 20
Based on these assessments the scope definitions for each of the IT capital projects 21
(supporting T&D Work Management) were refined to align the business capabilities being sought with 22
the appropriate technical solution. Portfolio management needs originally authorized in the 2015 GRC 23
as the “Integrated Portfolio Management for MPO” were identified to extend beyond MPO.113 This 24
project scope expanded to include project management capabilities. This change was made based on 25
project management capabilities being more aligned with portfolio management than with work 26
management dashboards. The Work Management Dashboard project was reduced accordingly. Another 27
112 Refer to WP SCE-04, Vol. 2 Bk B pp. 77-101. 113 Major Projects Organization (MPO) integration details can be found in the assessments further described in
the subsequent Work Management projects.
74
example of how project scopes were redefined is the Scope Cost Management Tool (SCMT). Project 1
forecasting capabilities were originally planned to be delivered through the SCMT application. The 2
assessments determined that project forecasting capabilities were better suited to be delivered with 3
portfolio management than with SCMT. The scope and costs for these projects were then revised to 4
reflect this change. The scope refinement outcomes are included in the project descriptions that follow. 5
The assessments resulted in a proposed system architecture provided in Figure V-6 below. 6
The series of projects based on the assessments and that are included within the suite of 7
Work Management Solutions include: 8
1. Portfolio Management; 9
2. Scope Cost Management Tool; 10
3. Work Management Dashboard; 11
4. Transmission Telecom Work Order Lifecycle; and 12
5. Click Schedule Refresh Release 1 & 2. 13
These projects will deliver to T&D the increased work management capabilities in 14
portfolio management, resource and budget forecasting, project scope estimating and scheduling, and 15
reporting and analytics. By leveraging standard industry work management tools in a consistent manner, 16
the improved work management capabilities also enable T&D to integrate oversight of work assigned to 17
contract resources. 18
75
Figure V-6 Work Management Assessment
1
2. WM - Portfolio Management 2
Table V-24 WM - Portfolio Management114
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
114 Refer to WP SCE-04, Vol. 2 Bk B pp. 102-110.
CIT-00-DM-DM-000077 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 1.80 6.00 6.20 - - 14.00 Previous GRC Request* - - - 0.40 2.90 1.30 - 4.60 *The Commission adopted SCE's request for this project in D.15-11-021.
Recorded Forecast
76
a) Project Description 1
D.15-11-021 adopted the project as part of SCE’s 2015 GRC, where it was 2
included as part of the “Integrated Portfolio Management for MPO” project. The project was delayed to 3
complete the Work Management Assessment (WMA) described above. Besides delaying the start of this 4
project until late 2016, the WMA confirmed the gaps and benefits were not isolated to “Integrated 5
Portfolio Management for MPO” but all Work Management projects across T&D. This broadened the 6
effort to include all T&D capital projects. 7
T&D capital projects are planned and managed as a portfolio. Each portfolio 8
component represents a collection of projects similar in nature. The similarity is usually based on the 9
asset being worked on, or the construction being performed. For example, substation circuit breaker 10
infrastructure replacement projects follow a common process, similar project timeline, and require 11
essentially the same project management, engineering, electrician, and field construction skillsets. 12
Likewise, transmission reconductor projects have commonalities when compared to each other. 13
Resource requirements and project schedules for transmission reconductor work would not be the same 14
as those for substation circuit breaker infrastructure replacement work. The same analogy would be true 15
for Distribution overhead conductor programs, and all other portfolio components. 16
Conflicts within different projects across the portfolio can occur due to resource, 17
scheduling, or operating constraints. It may not be feasible to perform Distribution overhead conductor 18
replacement concurrent with Transmission reconductoring on the same overhead pole line, from a field 19
coordination perspective. Nor may it be feasible from a grid operations perspective to perform 20
Transmission reconductoring concurrently with Substation Infrastructure Replacement, if the outages 21
required to support each activity cannot be simultaneously managed. Potential resource constraints may 22
also exist while attempting to simultaneously execute Distribution 4kV substation elimination work and 23
Substation Infrastructure Replacement work if both projects depend on the same substation apparatus 24
personnel. The Portfolio Management solution identifies these potential conflicts early to resolve the 25
operational impacts and maximize productivity. 26
By building a long-term plan for the portfolio based on standardized project 27
schedules, the Portfolio Management solution will allow T&D to quantify demands across T&D by 28
work group, asset, circuit or system, and geography. This will allow SCE to assess the organizational, 29
customer and financial implications related to “what if” scenarios or changes in investment plans. If 30
several projects within the portfolio were tracking under budget or behind schedule, the Portfolio 31
77
Management solution would allow the organization to assess various mitigation alternatives. By 1
managing all large projects as a portfolio, schedule or cost deviations of individual projects from their 2
respective project plans can be balanced by adjusting plans for other projects if feasible. 3
Project schedules will be developed based on the standard project template for 4
each area of the portfolio. This will allow SCE to monitor actual project performance against the 5
standard portfolio schedule template and will improve visibility to project risk. Actual project 6
performance, across many projects over time, can be aggregated for each area of the portfolio. The 7
aggregated data can be fed back to improve the corresponding standard portfolio schedule template. This 8
performance-based feedback loop will drive continuous improvement of the portfolio standard schedule 9
and maximize long-term planning efforts. 10
The capabilities delivered would be focused on large capital projects that typically 11
last from a few months to multiple years. The final solution is not designed to address short-duration 12
work (typically from a few days to a few weeks). 13
b) Need for Project 14
The level of infrastructure replacement and modernization that SCE expects to 15
execute within the next few years requires effective and efficient long-term planning and project 16
management. Ineffective planning and project scheduling can cause project delays, project over runs, 17
write offs, or frequent cycles of work ramp-up and ramp-down. These results can cascade into 18
overloaded workloads for employees as work spikes and deadlines accelerate. 19
Portfolio and project management capabilities within T&D are performed to 20
various degrees of maturity and across a variety of applications. Substation projects are submitted for 21
portfolio review within a custom-built solution and managed using several older versions of a 22
scheduling tool, which cannot work together. Conversely, Distribution employees rely on Excel and 23
Access to perform reviews of the portfolio. Distribution capital projects are managed using a 24
combination of multiple tools (e.g., SAP, Design Manager, Access, and Excel). The MS Office-based 25
tools are not integrated with SCE’s enterprise applications. Therefore, data for reporting within 26
Distribution are not integrated. Integrating project information between Transmission, Substation, and 27
Distribution is even more problematic. This lack of integration creates an organizational blind spot for 28
resource managers and employees within Substation and Operations. These groups regularly support 29
work plans being driven by Transmission and Distribution projects. Transmission reconductor and 30
distribution 4kV circuit elimination projects may depend on support from grid operators and substation 31
78
apparatus employees. However, the resource managers for these employees do not have access to the 1
project plans or visibility to the timing for the planned work. Instead, they rely on email communications 2
and Excel files for this information. This information can be unreliable and becomes difficult to manage 3
when project dates change. 4
The Portfolio Management project will enable resource managers to balance 5
demand and capacity more efficiently by providing a long-range overview of project needs. This will 6
drive faster project execution. Roles and hand-offs will become more defined and standardized across 7
the system. Implementing Portfolio Management will provide focus on critical activities. It will also 8
minimize rework and false starts resulting from a lack of overall job coordination. The tool will allow 9
resource constraints to be identified earlier to better assess the ramp-up needs of resources (internal or 10
contract) or the resequencing of work, before capital charges record. 11
A common project-management solution that all employees utilize will provide 12
portfolio and project managers a comprehensive view of project status and risk. This will cause drive 13
project delivery or reduced project costs as project risks are mitigated in advance. By meeting these 14
objectives, SCE will improve its overall ability to deliver capital projects as measured by quicker project 15
execution, reduced project delays, timely and effective scope on-ramp/off-ramp, and improved resource 16
and contractor management. Project plan progress can be consistently and accurately monitored, and 17
project plan risk can be mitigated. 18
c) Scope and Forecast 19
The Portfolio Management project will implement a COTS Project and Portfolio 20
Management Solution (PPM) integrated with the balance of T&D’s existing work management 21
solutions. Project scope includes: 22
1. Portfolio Planning & Forecasting—Provide a centralized tool for all large 23
capital programs where program scope, costs,115 resources and schedules are 24
prioritized and approved, as the baseline portfolio investment plan. Provide a 25
tool where project data related to schedule, costs and resources can be 26
aggregated and summarized across capital programs, resource pools and cost 27
categories. Provide the ability to generate forecasts (schedule, budget, and 28
115 Portfolio and project costs would be calculated within the application provided as part of the SCMT project
and input as a project cost estimate by the portfolio or project manager.
79
resource) for capital projects, the programs the capital projects belong to, and 1
the portfolio of capital programs. 2
2. Project Scheduling—Provide a standard tool for all large capital work to be 3
planned and managed allowing for consistent management of schedules and 4
resources and project monitoring against portfolio baselines. Integration of 5
project schedule data into Portfolio Planning solution for risk management, 6
change control governance, and project oversight. 7
3. SAP Integration—Align project schedule structure with recorded costs within 8
SAP work order operations. Provide integration to allow cost data to be 9
provided to project and portfolio managers. 10
4. Contractor Enablement—Provide a tool where project data for project work 11
assigned to contractors can be integrated into the view of the overall portfolio, 12
reducing the specialized hand-offs between groups and eliminating the 13
translation of contractor data into project status views on a delayed basis. 14
Provide a means for contractor collaboration through a standard project 15
documentation exchange and project documentation retention solution. 16
5. Reporting Data—Provide portfolio and project data to enterprise reporting 17
databases. This would be for integration into the centralized reporting and 18
analytics solution with data for work outside the Portfolio Management 19
solution.116 20
The total project cost forecast is $14 million.117 The capital forecast for this 21
project was developed using SCE’s internal cost estimation model. This model utilizes industry best 22
practices and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this 23
project includes costs for SCE employees, supplemental workers and consultants, software and vendor 24
costs, and hardware costs. See this project’s workpaper for the cost breakdown information. This 25
116 Examples of work not included with the Portfolio Management solution include Distribution Inspection,
O&M and New Service Connections. The reporting needs from this integrated reporting database would be the scope of the Work Management Dashboard project.
117 Refer to WP SCE-04, Vol. 2 Bk B p. 110.
80
includes the implementation of COTS Portfolio Management and Scheduling solutions across T&D and 1
integration of these solutions with our SAP enterprise system. 2
(1) Alternatives Considered 3
Alternative 1: SCE considered replacing the existing solution by building 4
a customized solution using SCE and consultant resources or contracting with a third party to build it. 5
We did not pursue this option because it would require SCE and contracted resources to identify the 6
required skills and to set up the development and testing environments. This alternative was also not 7
pursued because it would not be cost-effective to create such a solution because it would require 8
significant future maintenance costs. 9
Alternative 2: SCE considered replacing the existing solution by 10
procuring a new commercially available tool or service. We did not pursue this alternative because it 11
would require SCE to go through the competitive bid process for both the tool and additional support 12
services to integrate it with the existing tools in our IPSEC model. This option was also not pursued 13
because implementing a new tool would also carry additional costs for training and other organization 14
change management activities. 15
3. Scope Cost Management Tool (SCMT) 16
Table V-25 Scope Cost Management Tool118
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 17
The Scope and Cost Management Tool (SCMT) is foundational to the Work 18
Management process as it will provide a standard method of estimating the scope and costs of a project 19
throughout its lifecycle. An improved scope estimating solution is required for SCE to accurately 20
estimate the cost for all T&D and FERC 1000 projects, while managing to forecasts (via Portfolio 21
118 Refer to WP SCE-04, Vol. 2 Bk B pp. 111-118.
CIT-00-DM-DM-000093 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 2.00 3.00 - - 5.00
Previous GRC Request - - - - - - - -
Recorded Forecast
81
Management) in a consistent manner. SCMT will allow for estimation accuracy commensurate with the 1
phase of the project and level of engineering estimate available. As large capital projects progress from 2
initial template-based estimates, to post job walks, or to detailed design,119 SCMT will provide the 3
ability to conduct scope and cost estimates with more detail and improved accuracy. 4
SCMT is intended for larger, non-programmatic projects120 that are included with 5
the Portfolio Management project, described above. SCMT will allow SCE to perform accurate scope 6
and cost estimating earlier in the project planning process. As an input into the Portfolio Management 7
solution, it will enable SCE to produce a comprehensive project forecast from inception to completion. 8
SCE’s existing process develops scope based on conceptual engineering prior to work order creation.121 9
However, revisions to scope and costs can also occur later in the planning and development phases of a 10
project. These revisions can cause impacts to the project schedule and budget when not properly re-11
estimated. 12
The SCMT project implements a solution that will: 13
1. Provide an accurate initial estimate of scope and cost using standard project 14
templates, then modified and revised as project scope is refined; 15
2. Provide a standardized platform for capital project scope estimation so they 16
are consistent and repeatable; 17
3. Expand the range of projects that can be estimated (e.g., major capital project, 18
programmatic project, interconnection, FERC 1000 projects, etc.); and 19
4. Assist in improved cost controls by providing a standard cost estimating 20
methodology. 21
SCMT will replace the disparate set of tools in use today with a single user-22
friendly platform. SCE uses a combination of software tools developed in-house, along with 23
commercially available products. As these products age, their performance diminishes. Their inability to 24
119 Detailed design estimates for work orders managed in SCE’s Design Manager would be produced from
within the Design Manager application. 120 Non-Programmatic work refers to projects whose scope is unique in nature and not repeatable. The
construction of a new large transmission line would be an example. 121 SCE’s Design Manager performs CU bases cost estimation for sub-transmission and distribution work, after
the work order has been pulled and engineered.
82
handle new types of work make them less useful. Replacing them with up-to-date technology will 1
alleviate the issues described above and provide a platform capable of meeting future needs. 2
b) Need for Project 3
Implementation of an industry-standard Scope and Cost Management Tool will 4
allow SCE to cost out the scope included in our current capital operating roadmap plans and more 5
accurately determine future spending needs. This allows SCE work managers and planners to perform 6
more accurate estimates for increasingly complex projects. This is accomplished by providing 7
predictable and consistent scope and cost estimating tools integrated to other enterprise work 8
management tools. Additionally SCMT will help enable SCE management to better analyze variances 9
between planned and actual project costs and scope, across all large capital projects. This analysis will 10
provide opportunities to improve the cost basis for future projects and improve our future project scope 11
costing and project controls capabilities. The estimates that the SCMT solution will provide are expected 12
to be an input into the Portfolio Management solution described above. 13
This project modernizes SCE’s tools and processes in use today and improves our 14
current capabilities in several major areas: 15
1. Ease of use – Effective project scoping and costing is an involved and often 16
complex process. The solution will allow for a simpler user experience by embedding complex rules and 17
processes into the workflow. It will also provide project scope and cost templates easily adjusted for a 18
project. 19
2. Consistency – A standardized tool will improve consistency of estimates and 20
therefore will cause improved accuracy. It will also provide more intelligence for users to understand the 21
variances between estimated and actual costs. 22
3. Performance – Besides not meeting SCE’s functional needs, the tools in use 23
are slow and cumbersome. The current tool can take up to 10 minutes or longer to calculate and display 24
results for large projects. Due to the architecture of the current tool, a user’s computer and its processor 25
are wholly occupied when running the SCMT analysis. The employee is left with little else to do other 26
than to wait for the transaction to complete. By using contemporary tools and technology with modern 27
architecture, the new solution will be more responsive and scalable. 28
4. Better Integration – As SCE moves toward a more robust EPC (engineering, 29
procure, construct) model, the project scope and costing tools must integrate with both internal and 30
external systems. This is especially true for major capital construction projects. 31
83
5. Flexibility – The current tools are limited in functionality to Transmission and 1
Distribution projects only. Since SCE is expanding the need for project scoping and costing accuracy 2
into other areas (e.g., Transmission Telecom), a more extensible and scalable tool will be required. As 3
other project types are incorporated into SCE’s IPSEC model, we can leverage the new Scope and Cost 4
Management Tool for them as well. By addressing the areas mentioned above, SCE will improve overall 5
capital project efficiencies, while providing more accurate capital spend plans. 6
c) Scope and Cost Forecast 7
In this rate case period, the total project costs are $5.0 million.122 The capital 8
forecast for this project was developed using SCE’s internal cost estimation model. This model utilizes 9
industry best practices and SCE subject matter expertise to estimate project cost components. SCE’s 10
forecast for this project includes costs for SCE employees, supplemental workers, and consultants, 11
software and vendor costs, and hardware costs. See this project’s workpaper for the cost breakdown 12
information. 13
(1) Alternatives Considered 14
Alternative 1: SCE considered enhancing the existing solution by 15
addressing the performance issues and extending the capabilities to include all project types. We did not 16
pursue this option because it would not be practical to meet our needs for better performance and would 17
require extensive work to integrate with other future systems. As stated above, the performance issues 18
are inherent to the system architecture and data model. Any enhancements would require significant 19
changes in the code and database, which would not be practical. Additionally, since the data model is 20
very different from other systems used in the IPSEC model, integration would be especially difficult and 21
prone to errors as there is no effective way to directly map across the data models. 22
Alternative 2: SCE considered replacing the existing solution by building 23
another customized solution to meet the needs of the SCMT scoping and costing functionality. We did 24
not pursue this option because it would not meet our requirements to use an industry-standard platform 25
and to minimize future maintenance costs. As stated previously, the existing tool was a custom-built 26
122 Refer to WP SCE-04, Vol. 2 Bk B p. 118.
84
solution that was internally developed by leveraging the existing database from the MDI123 tool, and it 1
did not perform adequately. 2
Alternative 3: SCE considered leveraging its existing portfolio and 3
performing an integrated build by distributing the current functionality across other IT work 4
management initiatives, such as P6, SAP Project Systems, and Design Manager upgrades, if possible. 5
This would include a detailed gap analysis of the capabilities of these tools to the business requirements 6
and then develop processes for leveraging them. As reporting and notifications are major components of 7
SCMT, other systems must also be leveraged (e.g., Business Intelligence (BI) tools, eDMRM,124 8
SharePoint, Outlook). Similar to Alternative 1 above, modifying other systems to meet the requirements 9
was not pursued as there are significant gaps in functional requirements as well as dependencies on 10
systems and work processes that would require major changes (e.g., earlier creation of SAP work orders, 11
even for conceptual/unapproved project analysis). 12
4. Work Management Dashboard 13
Table V-26 Work Management Dashboard125
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 14
D.15-11-021 adopted the Work Management Dashboard project as part of SCE’s 15
2015 GRC. This project was delayed initially while the team assessed the project scope, business 16
impacts, and method of deployment. During the delay, it was discovered that additional business lines 17
123 MDI (Multiple Document Interface) is a Microsoft Windows programming interface for creating an
application that enables users to work with multiple documents at the same time. 124 eDMRM is Enterprise Document Management Records Management, a central records repository where
certain archived documents are stored. 125 Refer to WP SCE-04, Vol. 2 Bk B pp. 119-127.
CIT-00-SD-PM-000155 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 0.25 1.00 0.50 - - 1.75 Previous GRC Request* - - - 1.90 1.70 - - 3.60 *The Commission adopted SCE's request for this project in D.15-11-021.
Recorded Forecast
85
within T&D were also using Access and Excel as business tracker databases. Using these tools require 1
extensive manual efforts, limit department-wide consistency, and introduce a large potential for errors. 2
The team considered various implementation approaches to minimize the organizational disruption that 3
would result from data conversion, interim parallel (redundant) processes, and a phased versus “big 4
bang”126 deployment. The project was subsequently delayed to accommodate the Work Management 5
Assessment (WMA) described in earlier sections. The WMA recommendation resulted in the scope for 6
the T&D Work Management (WM) Dashboard project being modified. Project Management capabilities 7
associated with project schedules for capital programs are now expected to be delivered through the 8
Portfolio Management solution. The Portfolio Management solution will capture major capital project 9
task information managed through several of the Access tracker databases and transition that data 10
collection as part of its project scope. The T&D WM Dashboard project will transition all other key 11
performance data into the proper enterprise tool to decommission the Access and Excel tracker 12
databases. The T&D WM Dashboard remaining project scope will focus primarily on reporting and 13
analytics. 14
The T&D WM Dashboard project will create an integrated work management 15
reporting and analytics solution to enable consistent planning, forecasting, and reporting across all T&D 16
capital and O&M work. This project will provide comprehensive reporting by integrating information 17
from the large capital work orders, capital maintenance work orders, customer-driven capital orders, and 18
all O&M orders. 19
To accomplish this, the WM Dashboard project will establish the reporting 20
database structure from which all integrated T&D management reports will be generated. The project 21
will select and implement existing enterprise technology, from which the standard reports will be 22
generated. This project will also provide training required by business analysts in order perform ad-hoc 23
reporting. 24
b) Need for Project 25
T&D does not have a reporting and analytics solution that provides the 26
organization with a comprehensive view of performance across all resources and types of work. The 27
organization relies on 18 Access databases and numerous Excel workbooks to fill this gap. These 28
126 “Big bang” deployments are deployments that typically include all users and all capabilities at one time.
86
databases are used by planners, project managers, and Resource Planning and Performance Management 1
(RPPM) to track the status of work order tasks (e.g., planner-related tasks such as design and 2
permitting), key work-order dates (e.g., customer commitment, regulatory compliance dates, and 3
material delivery dates), and contractor crew scheduling status, among other tasks. The current tools and 4
processes have the following issues: 5
• There is not a comprehensive view of T&D’s construction and maintenance 6
work, requiring multiple tools to gather and reconcile resources and work status. 7
• Resource forecasting and demand management across all T&D capital and 8
O&M work is therefore manually intensive. 9
• The current independent trackers have no automated work management 10
capabilities, are not integrated with core systems such as Design Manager, SAP or Click Schedule, and 11
require employees to manage work in multiple systems. This results in extensive duplicative data entry, 12
as the same data fields are entered into multiple trackers. This manual process of redundant data entry 13
can cause data quality and accuracy issues. 14
• The siloed tools are not developed according to standards and are difficult to 15
maintain. They are prone to lock up and crash, which results in lost time while the system and data are 16
restored. 17
(1) Benefits 18
The WM Dashboard provides an integrated system that will update 19
automatically to address the current data quality and accuracy issues, while eliminating duplicate data 20
entry. The WM Dashboard will also provide a comprehensive view of T&D’s construction and 21
maintenance work by replacing multiple tools with one integrated tool. The Dashboard tool will improve 22
management of resource demand and capacity planning for all capital and O&M work. The solution will 23
provide near real-time access127 to data; allow reporting based on current data; and provide enterprise 24
standard technology to support operation activities. 25
c) Scope and Cost Forecast 26
The first component of the WM Dashboard project is the development of a work 27
management dashboard that provides an integrated view of work across departments at a level of detail 28
127 Near real-time in this case pertains to the ability to report on new contractor-provided data multiple times
daily as needed.
87
that is not available today. Through this functionality, SCE can report on department employee 1
utilization, department employee capacity, work progress, work volume by status (completed, in-flight, 2
etc.), work volume by department, and by work type. 3
The second component will decommission the Access databases by replacing 4
tracker task data with equivalent data from core enterprise applications. This will include further 5
leveraging applications within the existing SCE software portfolio. The second component will evaluate 6
and select the best appropriate technologies to address any remaining capability or performance 7
requirements. These will be integrated with the enterprise work and portfolio management solutions. 8
This phase will also deliver improved forecasting and demand management 9
capabilities and resource capacity planning. 10
This project is scheduled to begin in October 2016 and be completed in June 11
2018. In this rate case period the total project costs are $1.8 million.128 The capital forecast for this 12
project was developed using SCE’s internal cost estimation model. This model utilizes industry best 13
practices and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this 14
project includes costs for SCE employees, supplemental workers, and consultants, software and vendor 15
costs, and hardware costs. See this project’s workpaper for the cost breakdown information. 16
(1) Alternatives Considered 17
Alternative 1: SCE considered not pursuing this project and instead 18
continuing to use the current Access and Excel tools to manage workflow. However, if we chose not to 19
pursue this project, then this would address none of the existing issues or achieve the benefits described 20
above. Continued use of the Access database would result in continued disruptions to the business, as 21
the systems are unstable and experience frequent outages due to the volume of users trying to work in 22
them. Additionally, due to the instability of the databases, SCE is not building new data fields into the 23
systems. This is resulting in the creation of even more disparate methods of tracking additional data and 24
perpetuating the operational inefficiencies of managing information and inability to share common data 25
across organizations. 26
Alternative 2: SCE considered procuring a COTS solution to meet the 27
transmission and distribution work management needs. We did not pursue this option for the reason that 28
128 Refer to WP SCE-04, Vol. 2 Bk B p. 127.
88
it would not meet our requirements, nor leverage the current technology footprint. Since there is no 1
commercially available product that currently meets all of SCEs business requirements, any selected tool 2
would still require effort to configure and develop the code necessary to integrate with SCE’s existing 3
enterprise work management tools. In addition, adding a new tool into SCE’s portfolio would add new 4
complexity and additional cost to maintain. 5
5. Transmission Telecommunications Work Order Lifecycle 6
Table V-27 Transmission Telecommunication Work Order Lifecycle129
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 7
The Transmission Telecommunication (Transmission Telecom) group within 8
T&D is responsible for the design, construction, and operations of an extensive telecommunications and 9
fiber-optic network that supports critical electric utility operations and business systems. The processes 10
used by the Transmission Telecom group are similar to those used throughout T&D. However, the 11
Transmission Telecom group does not use the same project and work-management tools used by the rest 12
of T&D for end-to-end planning, design, scheduling, field execution, and work order closure. This 13
Transmission Telecom Work Order Lifecycle project will extend existing work management tools used 14
by the rest of T&D (e.g., Design Manager, Graphical Design Tool, Click Schedule, and Powerplan) to 15
the Transmission Telecom group. 16
The Transmission Telecom group will leverage the existing Design Manager 17
(DM) core platform to: (1) Manage the assignment of work to the planning staff and measure 18
corresponding employee performance, (2) Apply the proper pricing methods to determine work order 19
estimates, (3) Verify that work order designs are completed and that material is appropriately selected 20
129 Refer to WP SCE-04, Vol. 2 Bk C pp. 128-136.
CIT-00-DM-DM-000095 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - - 2.00 2.00 - 4.00 Previous GRC Request - - - - - - - -
Recorded Forecast
89
according to job type, and (4) Confirm that the proper documentation, permits, and notices are achieved 1
and secured prior to construction. 2
b) Need for Project 3
Current Transmission Telecom processes consume significant effort and time 4
because data and information for work management processes are dispersed across several computer 5
systems. These systems are maintained by the business unit and therefore the required manual 6
configurations and regular maintenance must be performed by a few individual employees within the 7
Transmission Telecom team. This results in staffing inefficiencies as employees balance core business 8
functions with additional system support duties. It also presents risks to data quality and consistency 9
when the organization experiences employee turnover. Because the systems used by the Transmission 10
Telecom group are not consistent with the rest of the T&D organization, the data and information the 11
systems produce is also inconsistent. This can create duplication, operational misalignment, and 12
additional cost to how we approach work management across the T&D organization. 13
(1) Benefits 14
The Transmission Telecom Work Order Lifecycle project is key to the 15
organization’s ability to operate efficiently and productively. The project will align work order processes 16
and information with the rest of T&D by having work created and managed consistently across the 17
organization. Use of these standard technology platforms will better enable SCE to manage cost, 18
schedule, and risks through accurate initiation, planning, scheduling, and closure of work performed by 19
Transmission Telecom. Moving Transmission Telecom work onto a common T&D technology platform 20
also enables the Portfolio Management and Work Management Dashboard solutions to be extended to 21
include Transmission Telecom without customization. 22
This effort will reduce reliance on manual processes and provide an 23
integrated system for Transmission Telecomm processes. The effort will also leverage current enterprise 24
technology to capture work order transaction information (e.g., transaction time stamps, 25
created/modified employee data) that must otherwise be manually tracked and managed. 26
i. Work Order Planning 27
This project will extend the Design Manager (DM) application, 28
discussed further in this testimony, to the Transmission Telecom group. The Transmission Telecom 29
group will leverage the existing DM core platform to: (1) Manage the assignment of work to the 30
planning staff and measure corresponding employee performance, (2) Apply the proper pricing methods 31
90
to determine work order estimates, (3) Verify that work order designs are completed and that material is 1
appropriately selected according to job type, and (4) Confirm that the proper documentation, permits, 2
and notices are achieved and secured prior to construction. Use of DM by this group will provide 3
visibility of in-progress and completed work. The assignment and tracking of work orders and requests 4
through DM will provide a platform for communication and collaboration between Transmission 5
Telecom and other T&D departments. 6
ii. Graphical Design 7
SCE’s current Graphical Design Tool (GDT) is used by T&D 8
distribution and sub-transmission planners and provides a graphical design platform upon which these 9
individuals draft work order construction diagrams and maps. GDT also provides a planner or designer 10
the ability to capture and relay information related to safety, standards, and construction methods to the 11
field crew executing the work. Use of GDT by Transmission Telecom will: (1) Provide a standard 12
platform for the drafting and design of work orders, (2) Verify that drafts and designs comply with 13
standards and policy, (3) Confirm that appropriate material is selected for efficient construction, and (4) 14
Enable integration of work order designs with the Comprehensive Geographical Information System. 15
iii. Scheduling 16
SCE’s use of an automated scheduling tool for T&D work order 17
field execution aligns resources to planned units of work. The resulting work assignments are then 18
dispatched to crews. This capability allows the T&D organization to optimize work schedules for field 19
crews. Transmission Telecom has no similar scheduling system and relies on manual management of 20
resources to accomplish work. This project will extend SCE’s scheduling COTS tool, discussed later in 21
this testimony, to the Transmission Telecom group. Providing Transmission Telecom a standard COTS 22
platform for the scheduling and dispatch of work will make the scheduling process more efficient. The 23
ability to match available resources to planned work will cause the establishment of full and stable work 24
order schedules, which will bring productivity gains in the field execution of the work order. 25
iv. Execution & Closure 26
Leveraging much of the existing configuration for T&D work 27
orders will allow Transmission Telecomm to follow the standardized work order closure process and 28
take advantage of the existing integration between SAP and Powerplan. This approach will also result in 29
91
improvements to the closure process, as the work order accountants will no longer have to accommodate 1
exception processes for telecomm work orders. 2
SCE’s Field Accounting Organization (FAO) continues to pursue 3
opportunities to close work orders faster and more efficiently, while maintaining quality and completing 4
other core duties. By using the same technology as the other organizations, SCE can reduce process 5
inefficiencies resulting from different technology and manual processes.130 6
c) Scope and Cost Forecast 7
This project will extend existing technology platforms used by T&D to support 8
the Transmission Telecom group manage their end-to-end work processes. Transmission Telecom will 9
leverage DM and GDT for the Initiation and Planning phases. The tools will improve the work order 10
management process, work order creation, work order design drafting, and the pricing of work to be 11
executed. Click Schedule will support the scheduling aspect of the work order process. This will support 12
field crews to be appropriately equipped and available for the execution of the scheduled work. SAP and 13
Powerplan will verify proper completion and closure of the work order, and maintenance and archival of 14
related documentation. This project will implement system configuration changes to the existing 15
architecture already in place and develop software to support regulatory and compliance-related 16
activities. 17
The project is scheduled to start in the second quarter of 2018 and be completed 18
by the fourth quarter of 2019. The capital forecast for this project was developed using SCE’s internal 19
cost estimation model.131 This model utilizes industry best practices and SCE subject matter expertise to 20
estimate project cost components. SCE’s forecast for this project includes costs for SCE employees, 21
supplemental workers, and consultants, software and vendor costs, and hardware costs. See this project’s 22
workpaper for the cost breakdown information. 23
(1) Alternatives Considered 24
Alternative 1: SCE considered not pursuing this project. However, if we 25
chose not to pursue this project, it would result in significant levels of redundant work across multiple 26
departments, higher operating costs for projects requiring collaboration, inability to streamline work 27
130 Field Accounting Organization (FAO) can now receive inputs from one consistent process, rather than a
separate process unique to Transmission Telecomm. 131 Refer to WP SCE-04, Vol. 2 Bk B pp. 136.
92
orders, inability to maintain and archive designs and records, and the inability to support audit-related 1
activities in a timely manner. 2
Alternative 2: SCE considered procuring a COTS solution to meet the 3
needs of the Transmission Telecom group. We did not pursue this option because it would not meet our 4
needs unless it was customized to specifically address the Transmission Telecom requirements. In 5
addition, this would require introducing another application into our environment, require new 6
interfaces, and also add further costs for licenses and on-going maintenance. Due to the high complexity, 7
cost, and risk, SCE elected not to pursue this option. 8
6. Click Schedule Refresh Release 1 & 2 9
Table V-28 Click Schedule Refresh Release 1&2132
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 10
D.15-11-021 adopted the Scheduling Refresh Project to refresh the outdated 11
scheduling application and extend the scheduling capabilities to other T&D business lines. 12
Subsequently, the project scope was divided into two releases to minimize the daily operational impact 13
of new functions being introduced to the Transmission and Distribution organization. 14
The Scheduling application was initially deployed in 2007 to T&D Distribution 15
groups including Resource Managers, Schedulers, and their support staff as part of the Enterprise 16
Resource Planning (ERP) program.133 In 2010, also as part of the ERP program, the user base was 17
expanded to include the Transmission, Substation, Grid Operations, and Power Delivery Resource 18
Planning and Performance Manager (RPPM) organizations. Additional planned users include 19
132 Refer to WP SCE-04, Vol. 2 Bk B pp. 137-140. 133 SCE’s Enterprise Resource Planning (ERP) program was implemented in 2010.
Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - 0.56 1.52 2.50 3.50 0.50 - 8.58 2015 GRC Authorized* - - - 5.40 2.50 - - 7.90 2015 GRC - Original Request - - 0.38 5.40 2.50 - - 8.28 *The Commission adopted SCE's request for this project in D.15-11-021. Multiple WBS elements include: CIT-00-DM-DM-000030 and CIT-00-SD-PM-000232.
Recorded Forecast
93
construction coordinators, damage assessment teams, Troublemen, dispatch coordinators, and mutual 1
assistance crews. 2
SCE started the project in 2014 using O&M funds to perform a preliminary 3
impact assessment. SCE engaged an experienced scheduling consultant to recommend an 4
implementation approach. SCE then undertook a competitive selection process to select the most 5
qualified implementation vendor. The assessment took longer than anticipated and delayed the start of 6
the project until 2015. 7
Release 1 refreshes hardware, updates the scheduling software version and 8
database, and updates interfaces to SCE systems. This release provided “like for like”134 functionality to 9
the users. Release 2 delivers additional capabilities such as automated dispatch, crew boards, integration 10
with crew call-out system, and integration with SCE’s Comprehensive Geographical Information 11
System (cGIS). 12
b) Need for Project 13
T&D’s scheduling application is four versions behind the current vendor product 14
and is no longer supported due to its obsolete technology foundation. Our contract with the vendor 15
requires us to stay within two major versions of the vendor’s most recent commercial release. If we fail 16
to stay within two major versions, our contract requires that we purchase new licenses if we refresh the 17
application. SCE did not complete the required major version upgrades of the software due to the lack of 18
additional features that would provide additional operational improvements for T&D and because the 19
newer versions would require substantial effort to install. But the vendor is waiving this license 20
repurchase provision if we refresh now. Refreshing the scheduling application now will allow us to 21
continue having vendor support. 22
The Scheduling Project Release 2 is scheduled to start in 2017 and will complete 23
in 2018. The project will address additional gaps we have found with the current version and provide 24
additional capabilities. 25
134 “Like for Like” is a term used within SCE to describe when an older or unsupported application is replaced by
a newer, more technologically advanced application, architecture, and interface, that provides the same functionality, use, and results.
94
(1) Benefits 1
By completing Scheduling Project Release 2, SCE will be positioned to 2
gain additional benefits by: (1) Reducing multiple short-term scheduling tools across T&D, (2) 3
Providing a scheduling solution for work groups in other SCE Operating Units, (3) Improving 4
integration with work management, outage, storm, mapping, and ERP solutions, and (4) Eliminating 5
custom code “add-ons.” SCE will also avoid a larger and more costly project by leveraging the 6
Release 1 investment (see Alternative 2 below). 7
c) Scope and Cost Forecast 8
The first release was started mid-2015 and is targeted to complete in 2017. The 9
second release is scheduled to start in 2017 and complete in mid-2019. In this rate case period the total 10
project costs are $8.6 million.135 The capital forecast for this project was developed using SCE’s internal 11
cost estimation model. This model utilizes industry best practices and SCE subject matter expertise to 12
estimate project cost components. SCE’s forecast for this project includes costs for SCE employees, 13
supplemental workers, and consultants, software and vendor costs, and hardware costs. See this project’s 14
workpaper for the cost breakdown information. 15
(1) Alternatives Considered 16
Alternative 1: SCE considered keeping the existing solution without 17
modification. We did not pursue this option because we would have to continue with the current labor-18
intensive and inflexible process, work around technology obsolescence, and continue with limited 19
visibility to field work orders status and crew availability and with high maintenance and support 20
overheads due to process gaps. This option also leaves SCE without necessary vendor support. 21
Alternative 2: SCE considered replacing the existing solution by 22
procuring another vendor’s COTS product. The original scheduling project was an approximately $34 23
million effort and spanned five years. A new COTS system would require an investment in time and 24
money to provide new license purchases, integration, and build interfaces for the new application with 25
our other existing software applications. 26
Neither of these options address the current issues described above in the 27
most cost-effective way. 28
135 Refer to WP SCE-04, Vol. 2 Bk B p. 140.
95
7. Vegetation Management 1
Table V-29 Vegetation Management136
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 2
D.15-11-021 adopted the Vegetation Management (VM) project as part of SCE’s 3
2015 GRC. SCE subsequently changed our contractor sourcing strategy and developed new vendor 4
management requirements. To address the changes in strategic direction, SCE conducted research to 5
determine the best approach and software tools to enable the new VM capabilities. To minimize the risk 6
of developing a solution that did not support the new processes, the project did not begin development 7
until the new processes were fully defined in 2015. 8
The VM project will consist of three phases: (1) deployment of a mobile software 9
and hardware solution to enable data collection in the field, (2) implementation of COTS VM software, 10
and (3) integration to SCE systems such as SAP and cGIS. The project is scheduled to be completed in 11
2017. 12
The VM project implemented a new mobile field collection device in late 2015. 13
This phase of the project was funded with O&M. This device is in use by pre-inspectors, tree-trimmers, 14
and SCE VM employees. In the upcoming phases, the VM project will also implement new systems and 15
analytic tools used to plan and analyze VM work processes such as tree trimming, tree removal, and 16
weed abatement. The VM project will replace the current manual process, which is paper-intensive and 17
outdated. 18
Since the last GRC, SCE has revised its implementation approach, which has 19
resulted in a reduced forecast compared to SCE’s 2015 request.137 SCE recognized that some relatively 20
straight-forward changes in current processes would allow for a COTS solution, without compromising 21
136 Refer to WP SCE-04, Vol. 2 Bk B pp. 141-148. 137 Refer to WP SCE-04, Vol. 2 Bk B p. 147.
CIT-00-SD-PM-000154 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Total
Recorded / Forecast - - - - - 2.00 5.70 - - - 7.70
Previous GRC Request* - - - 0.90 4.00 4.80 - 9.70 *The Commission adopted SCE's request for this project in D.15-11-021.
Recorded Forecast
96
the safety and reliability objectives. This also allowed SCE to avoid the costs of customizing a COTS 1
product. In addition, SCE elected to use a cloud-hosted solution, instead of building up the additional 2
infrastructure within SCE’s environment. Both these decisions resulted in reduced project costs. 3
b) Need for Project 4
The current VM process involves manually entering information from paper 5
forms into an Access database. The current Access front end and the SQL database used for VM are 6
antiquated and are operating near peak capacity. The VM group manages regulatory compliance of 1.5 7
million trees and 600,000 to 700,000 annual tree trim records. Managing this large volume of work 8
within the current process and tools has inherent risks. These risks include keeping data current while 9
processing a large backlog of paperwork, and controlling data errors resulting from manual data entry. 10
By replacing the current paper-intensive process with a digitized, map-based 11
system, and providing the data and tools to perform better analytics and work management, the VM 12
project will: (1) Increase SCE’s ability to track and report on the quality of its contractors’ performance, 13
(2) Increase the ability to prioritize work, including the ability to prescribe work proactively instead of 14
reactively, which will help expedite the removal of potentially dangerous trees, and (3) Reduce the risk 15
of non-compliance with requirements from federal, state, local and environmental agencies (including 16
General Order (GO) 95, Rules 35 and 37, and Public Resources Code (PRC) 4292 and 4293). This will 17
be achieved by integrating the VM with our Comprehensive Geographic Information System (cGIS) to 18
enable monitoring of trees in areas with drought conditions, and align prioritization of tree trimming 19
regarding the distribution and transmission facilities. 20
c) Scope and Cost Forecast 21
The project will be implemented in three phases. Phase One (deployed in 2015) 22
focused on the deployment of the field data collection software and hardware.138 Phase Two will 23
configure and deploy a COTS VM system, and Phase Three will complete the project integration to SCE 24
systems such as SAP and cGIS. Total project capital costs are $7.7 million.139 25
The capital forecast for this project was developed using SCE’s internal cost 26
estimation model. This model utilizes industry best practices and SCE subject matter expertise to 27
138 The preliminary planning activities of this VM project were funded through O&M costs, and therefore are
requested for recovery in this GRC. 139 Refer to WP SCE-04, Vol. 2 Bk B p. 148.
97
estimate project cost components. SCE’s forecast for this project includes costs for SCE employees, 1
supplemental workers, and consultants, software and vendor costs, and hardware costs. See this project’s 2
workpaper for the cost breakdown information. 3
(1) Alternatives Considered 4
Alternative 1: SCE considered keeping the existing solution without 5
modification. We did not pursue this option because it would not address the current issues or achieve 6
the benefits described above. 7
Alternative 2: SCE considered building a customized solution to meet the 8
needs of the VM group. We did not pursue this option because this would require significant investment 9
of time and effort to build and test a new application from scratch, which would not be cost effective for 10
SCE to develop. This alternative would subsequently add additional costs and complexity to the 11
maintenance of the application. 12
8. Pole Loading Application Replacement Tool 13
Table V-30 Pole Loading Application Replacement Tool140
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 14
D.15-11-021 authorized the Pole Loading Application Replacement (PoLAR) 15
project. General Order 95 (GO 95) sets forth rules for the design, construction, and maintenance of 16
overhead lines. GO 95 specifies that, at the time of installation, a newly constructed utility pole must be 17
built with an acceptable safety factor. Calculations are used to determine what the safety factor is—this 18
calculation is called pole loading (i.e., determining the load and its effect on the pole). In 2012, SCE 19
began the PoLAR project to deliver a long-term solution that enables T&D personnel to perform pole 20
140 Refer to WP SCE-04, Vol. 2 Bk B pp. 149-153.
CIT-00-SD-PM-000137 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - 1.09 3.39 4.28 3.64 2.44 - - - - 14.83 2015 GRC Authorized* - 1.09 3.39 2.74 - - - 7.22 2015 GRC - Original Request - 1.09 4.33 2.74 - - - 8.16 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized 2014 costs.
Recorded Forecast
98
loading analysis that meets safety and regulatory requirements, while performing effectively within a 1
dynamic technical landscape. Besides meeting core requirements to enable regulatory compliance, the 2
PoLAR project will also improve the end-to-end work order process through integration of several 3
workflow management applications (Design Manager/SAP) to optimize data transfer, pole load control 4
processes, and reporting capabilities. 5
In December 2013, the PoLAR scope was increased to deliver additional 6
capabilities to enable SCE’s new Pole Loading Program (PLP). Through PLP, SCE is performing a 7
systematic assessment of all existing poles in SCE territory, as described in SCE-02 Volume 9.141 8
Because of this new organization and process, PoLAR expanded from replacing the pole-loading tool, to 9
a larger effort to enable the PLP. This project has implemented several new processes and technologies 10
to enable this significant increase in the volume of all pole-related transactions (e.g., inspection through 11
construction and closure). 12
While a significant portion of the PoLAR project has been implemented and is 13
operational, certain business critical capabilities remain to be delivered. Emergent business priorities, 14
such as the initiation of the PLP, resulted in shifts of the original implementation schedule, resulting in 15
the delay in delivering the capabilities. These requirements were originally written in 2012, and the 16
development work that needs to occur to implement them has increase, because the business processes 17
have grown in complexity over the past 4 years. SCE has incorporated these additional technical 18
requirements into the project’s scope and determined that additional costs would be required to support 19
these new requirements. Additional capabilities included in the revised scope include new automation 20
for certain steps in our PLP quality control processes, and integration with cGIS to automate pole-related 21
data submissions to the PLP assessment vendor. SCE is also supplementing the original scope of work 22
to add more integration, data touch points, and other system-related requirements to accommodate these 23
additional capabilities. SCE will complete the final phase of the PoLAR project in 2016. 24
141 The Commission adopted SCE’s Pole Loading Program in D.15-11-021, p.123.
99
b) Recorded Costs and Forecast 1
SCE has recorded $12.4 million through the end of 2015 and expects to spend an 2
additional $2.44 million in 2016.142 This increase above authorized levels is a direct result of the 3
increase in scope described above. 4
The remaining capital forecast for this project was developed using SCE’s 5
internal cost estimation model. This model utilizes industry best practices and SCE subject matter 6
expertise to estimate project cost components. SCE’s forecast for this project includes costs for SCE 7
employees, supplemental workers, and consultants, software and vendor costs, and hardware costs. See 8
this project’s workpaper for the cost breakdown information. 9
9. Design Manager (DM) Refresh 10
Table V-31 Design Manager Refresh143
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 11
D.15-11-021 approved the Design Manager (DM) Refresh project. The current 12
forecast of $3.7 million is an increase of $1.0 million relative to our 2015 GRC. This project began in 13
August 2013 and will be completed in 2016. The capital forecast for this project includes $1.0 million in 14
project team costs for SCE employees, supplemental workers, and consultants, and $2.7 million in 15
software and vendor costs. No hardware costs are included for this project. 16
SCE’s DM application is used by planners for distribution, transmission, and 17
substation work. DM acts as the intermediary between the Graphical Design Tool (GDT) application 18
and SAP, and it is the pricing and estimating tool used by SCE planners in both transmission and 19
distribution and by substation engineers. DM is an eight-year-old, developed-in-house system. DM is 20
142 Refer to WP SCE-04, Vol. 2 Bk B p. 153. 143 Refer to WP SCE-04, Vol. 2 Bk B pp. 154-158.
CIT-00-SD-PM-000152 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - 0.90 1.15 1.62 - - - - 3.68 2015 GRC Authorized* - - - 2.20 - 2.20 Previous GRC Request* - - 0.50 2.20 - - - 2.70 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized 2014 costs.
Recorded Forecast
100
used by planners in work creation, reporting, compatible unit selection, and cost estimating. As the 1
usage of this tool has expanded across multiple business lines, several gaps have been identified within 2
the existing functionality. The current DM refresh project will update the software and hardware 3
infrastructure, and it will add functionality to enable better overall reporting and usability. 4
The DM project will change the current application from a front-end application 5
loaded on employee computers to a web-enabled application. This tool provides the ability to work from 6
mobile devices, such as tablets, reduces contractor connectivity issues by freeing them to upgrade their 7
hardware, while using more advanced processing. In addition, this project will improve the integration 8
between DM and Graphic Design Tool (GDT) project discussed in the following section. 9
Besides updating the DM application, the DM refresh project will replace the 10
current reporting functionality of DM with SAP Business Intelligence (BI) reporting to provide 11
improved reporting. This will give SCE the ability to provide employee performance reports, ad-hoc 12
reports, weekly status reports, and work order aging reports. In summary, the DM Refresh will address 13
the issues with the current system that include: (1) lack of mobility while using the DM application 14
described above, (2) system performance issues described above, (3) limited reporting capabilities 15
described above, (4) lack of ability to support other projects including WM Dashboard and Electronic 16
Work Order Package, (5) limited visibility of work management and tracking, (6) contractor 17
connectivity issues due to requirements to use older hardware and software platforms, (7) the 18
requirement for the user to create notifications for system issues by calling SCE’s help desk and 19
explaining the issue, and (8) limited ability to perform root-cause analysis for system defects. 20
DM Refresh will reduce risk by providing the technology updates needed to 21
mitigate information security exposures, and it will reduce the possibility of application failures 22
resulting from outdated or unsupported vendor technology. The DM Refresh project will provide 23
solutions for many of the existing architectural and usability issues that slow down or cause repetitious 24
tasks for SCE planners and designers. 25
b) Recorded Cost and Forecast 26
When the DM Refresh project began in 2013, SCE selected the vendor that 27
provided the best proposal for executing the project, considering cost and schedule. During the analysis 28
period of the project, however, several issues and gaps were discovered in the vendor’s understanding of 29
the application and the scope of the project. At this time, SCE was at a decision point to either continue 30
with the current vendor, or to close the analysis period and engage a different vendor for the project. 31
101
SCE elected to engage a different vendor for a couple reasons: (1) the new vendor we engaged was the 1
same supplier that constructed the original version of Design Manager and was already an expert in the 2
system, and (2) this vendor was also recently selected as one of SCE’s managed service providers 3
(MSPs). By using one of SCE’s MSPs for the project effort, it would minimize the effort of transitioning 4
the project (once completed) to the maintenance mode, since it is the same vendor providing both 5
services. While this change in vendor partner resulted in the identification of additional costs during the 6
analysis phase, it also mitigated the risk of finding more impactful gaps much later in the development 7
processes. 8
The remaining capital forecast for this project was developed using SCE’s 9
internal cost estimation model.144 This model utilizes industry best practices and SCE subject matter 10
expertise to estimate project cost components. SCE’s forecast for this project includes costs for SCE 11
employees, supplemental workers, and consultants, software and vendor costs, and hardware costs. See 12
this project’s workpaper for the cost breakdown information. 13
10. Graphic Design Tool (GDT) and Tract Deployment Refresh 14
Table V-32 Graphic Design Tool and Tract Deployment Refresh145
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 15
The Commission adopted the Graphical Design Tool (GDT) Refresh, and the 16
GDT for Tract Planners, Underground Drafting and Design Group (UGDDG) projects,146 as part of 17
SCE’s 2015 GRC. The GDT Refresh was originally planned to begin in 2015. During our planning 18
process it was determined that delaying the effort to incorporate a later vendor product release would 19
144 Refer to WP SCE-04, Vol. 2 Bk B p. 158. 145 Refer to WP SCE-04, Vol. 2 Bk B pp. 159-168. 146 See SCE-05, Vol. 2 from A.13-11-003.
CIT-00-DM-DM-000023 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 1.50 3.50 1.60 - 6.60 Previous GRC Request* - - - - 0.95 3.25 - 4.20 *The Commission adopted SCE's request for this project in D.15-11-021.
Recorded Forecast
102
allow for SCE to combine the two projects thus reducing change impacts to our systems and business 1
processes. The revised approach modified the project plan to begin in late 2016. 2
The GDT application is a system used by SCE’s distribution and sub-transmission 3
planners. GDT was originally implemented with the Design Manager tabular application to provide 4
graphical design capability to planners. GDT was first implemented in 2009 for distribution planners, 5
and subsequently integrated in 2010 with SAP, which expanded the tool’s availability to transmission 6
estimators. 7
During the original GDT project in 2008-2010, SCE partnered with the software 8
vendor to develop several customized capabilities to enhance the COTS software application. The 9
vendor has since recognized the value of these capabilities and has incorporated them into its newest 10
commercial product. By adopting this new version, SCE will eliminate the additional effort required to 11
maintain the customized software and will benefit from the capabilities now supported by the vendor. 12
SCE has also determined that combining the GDT Refresh and GDT for Tract Planners and UGDDG 13
would be a least-cost option for adopting the newest vendor product and would expand its use to new 14
user groups. 15
Combining these two efforts into a single project will reduce the risk of managing 16
and implementing multiple system changes and complex integrations simultaneously under separate 17
initiatives. The single project planning approach will provide consistency of resources and project 18
management activities across both phases while optimizing efforts to communicate and train employees 19
on the new tools. 20
b) Need for Project 21
The GDT Refresh and deployment of the GDT to the UGDDG is essential for the 22
continued use of the existing software. The current version is no longer supported by the vendor. In 23
addition, deployment of the application to the Tract and Underground Drafting planners is necessary to 24
move all planners and designers onto a centralized common design platform. The anticipated new user 25
groups use an antiquated, non-vendor-supported system to complete their design work. These groups 26
support other design groups by producing graphical designs for their work. The inconsistency of 27
applications used between multiple groups and among various resources within the design organization 28
causes difficulties in data sharing and limits the capacity to develop consistent work practices and 29
metrics. A common platform promotes efficient data sharing with no duplicate data entry or re-work 30
when projects are transitioned across groups within the same organization. 31
103
c) Scope and Cost Forecast 1
SCE forecasts $6.6 million to complete this project. The capital forecast for this 2
project was developed using SCE’s internal cost estimation model.147 This model utilizes industry best 3
practices and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this 4
project includes costs for SCE employees, supplemental workers, and consultants, software and vendor 5
costs, and hardware costs. See this project’s workpaper for the cost breakdown information. 6
(1) Alternatives Considered 7
Alternative 1: SCE considered not pursuing this project, and instead 8
continuing on the current version and delaying the refresh. Further delay is not recommended based on 9
the age of the existing application. Incorporating and maintaining old architecture creates complexities 10
when integrating into modern platforms and can add costs and delays to efforts such as operating 11
systems and server upgrades. The current version will be difficult for SCE to maintain, as it is no longer 12
supported by the product vendor. Delaying further will not address the business needs explained above. 13
Additionally, further delay poses a risk to procuring new licenses for a product upgrade. Our contract 14
requires that we purchase new licenses if we fail to maintain product versions within agreed-to limits. 15
Re-licensing risks additional increases to our forecasted costs. 16
Alternative 2: SCE considered separating the GDT Refresh and Tract 17
deployment efforts. This is not recommended because it will increase the cost and effort of redundant 18
training and communication activities. This approach would also increase significant risk of failure, due 19
to applying system changes multiple times, within the same application. 20
147 Refer to WP SCE-04, Vol. 2 Bk B p. 168.
104
11. Consolidated Mobile Solution (CMS) 1
Table V-33 Consolidated Mobile Solution148
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 2
D.15-11-021 adopted the Commission adopted the Consolidated Mobile Solution 3
(CMS) project. CMS encompasses three components: (1) a collection of field computing devices (field 4
tools), (2) a mobile software application, and (3) required infrastructure that allows SCE’s field 5
employees to manage work even while not connected to the SCE network. The field tools consist of 6
ruggedized portable laptop computers and automated vehicle locating devices, all of which incorporate a 7
global positioning system. The mobile software application, which is installed on the ruggedized laptop, 8
enables the tool to function as a remote device with real-time capabilities to access asset and map 9
information. It also provides the capability to create edits for map updates, which are then processed by 10
back-office personnel using SCE’s Enterprise Resource Planning (ERP) system, Outage Management 11
System (OMS), and the Comprehensive Geographical Information System (cGIS) database. The 12
infrastructure required for the project includes telecommunications networks, servers, databases, and 13
system interfaces that pass information between the mobile application and supporting office systems 14
(e.g., ERP, cGIS, and OMS). CMS enables field personnel, system operators, and office workers to 15
work more efficiently, resulting in productivity benefits, enhanced employee safety, improved outage 16
responsiveness, and improved ability to meet compliance obligations. 17
b) Need for Project 18
To improve safety and reliability in maintaining our electrical system, SCE field 19
personnel must have access to the same applications, data, and maps as our office staff. The existing 20
148 Refer to WP SCE-04, Vol. 2 Bk B pp. 169-174.
CIT-00-SD-PM-000041 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast 3.74 10.73 15.65 10.82 6.33 5.54 5.45 0.37 - - - 58.64
2015 GRC Authorized 3.74 10.73 15.65 10.82 5.42 - - - 46.37 2015 GRC - Original Request 3.74 10.73 15.65 9.00 7.03 - - - 46.16 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized 2014 costs.
Recorded Forecast
105
eMobile field tool is technically obsolete and is difficult to maintain. eMobile was an in-house-1
developed solution that consolidated several field tools. In eMobile, information transfer is a batch-2
process, not real-time, which leads to inconsistencies between field conditions and back-office data. It is 3
programmed in an older version of the programming language and operates on Windows XP. Windows 4
XP is out of normal support by Microsoft; this lack of support requires SCE to pay an additional support 5
cost of $0.59 million per year. The additional support costs are adjusted upwards each year. 6
c) Scope and Cost Forecast 7
The CMS project has deployed to approximately 700 T&D field users. 8
Deployment to the remaining 700 users’ devices is underway and planned to be completed by Q1 2017. 9
The CMS project has completed the application configuration, implementation, and deployment to T&D 10
Grid Operations (Streetlight team in 2013 and Substation Operators in 2015), Distribution organization 11
(Electrical System Inspections and Quality Assurance teams in 2014), and Substation Construction & 12
Maintenance organization (from 2015 to 2016). 13
In 2016 and 2017, the CMS project requires $5.82 million to complete the SCE 14
configuration of the application, implementation, training development, deployment and stabilization of 15
the remaining user groups (Grid Ops Troubleman, Distribution Construction & Maintenance (DC&M) 16
users, and Apparatus and Transmission). This will fully decommission the legacy eMobile application 17
and eliminate the continuing Windows XP support requirements, significantly reducing the operational 18
risks of relying on outdated and unsupported hardware and software for this highly utilized application. 19
SCE used a phased approach to roll out the solution to field users. Each roll out 20
was allowed to stabilize, while the lessons learned were applied to subsequent releases. One of the 21
lessons learned was that more end-user involvement in determining the functional look and feel of the 22
application was needed. This approach also resulted in the reworking of requirements to clarify 23
functions and a subsequent redesign of the software. The changes were larger than anticipated, resulting 24
in deployment delays and higher development costs. The software vendor had difficulty developing 25
solutions to meet the new requirements, which also contributed to schedule delays and resulted in 26
additional costs. However, the resulting delay has allowed SCE to deploy a solution that is higher 27
quality and rapidly accepted by the field users. 28
D.15-11-021 authorized total CMS project costs of $46.4 million. The total 29
project costs are now forecast to be $58.6 million. This increase in expenditures is a result of the 30
increase in scope described above. The remaining capital forecast for this project was developed using 31
106
SCE’s internal cost estimation model. This model utilizes industry best practices and SCE subject matter 1
expertise to estimate project cost components. SCE’s forecast for this project includes costs for SCE 2
employees, supplemental workers, and consultants, software and vendor costs, and hardware costs. See 3
this project’s workpaper for the cost breakdown information.149 4
(1) Alternatives Considered 5
SCE considered stopping development of this in-flight project due to the 6
extended schedule and higher than forecasted costs. From a technical risk assessment, the option to stop 7
the project and consider an alternate solution is not viable. While solutions are available in the market, 8
none acceptably meet SCE’s requirements without a significant amount of customization. When the 9
project started, none of the existing market vendors had a viable solution. Over time, these vendors 10
developed a comprehensive workforce solution. However, none of these vendors provide an integrated 11
work management and mapping capability similar to what is provided by CMS. 12
12. Field Tools Upgrade 13
Table V-34 Field Tools Upgrade150
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 14
D.15-11-021 adopted the completion of the Consolidated Mobile Solution (CMS) 15
project, to deploy approximately 1,400 field tools to T&D,151 as part of SCE’s 2015 GRC. The Field 16
Tools Upgrade builds on the CMS platform and enables SCE’s field personnel, system operators, and 17
office workers to improve employee safety, outage responsiveness, and SCE’s ability to meet 18
compliance obligations. 19
149 Refer to WP SCE-04, Vol. 2 Bk B p. 174. 150 Refer to WP SCE-04, Vol. 2 Bk B pp. 175-180. 151 See Consolidated Mobile Solution (CMS) Testimony for foundational project description.
CIT-00-DM-DM-000047 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Total
Recorded / Forecast - - - - - - - 1.00 6.00 3.00 10.00 Previous GRC Request - - - - - - - -
Recorded Forecast
107
b) Need for Project 1
SCE’s focus on maintaining and improving the grid has grown over the last three 2
years. This growth has increased the need for SCE’s field personnel and system operators to work more 3
effectively and efficiently, and it has highlighted opportunities to improve how work is performed. 4
Work performance can be improved by optimizing work and resource 5
assignments across the IPSEC (Initiate, Plan, Schedule, Execute, and Close) process. This initiative 6
supports delivery of recommendations from the “Work Management Assessment” as described in the 7
work management testimony presented previously in this volume, and it improves work execution by 8
SCE’s field workers. 9
c) Scope and Cost Forecast 10
The Field Tools Upgrade project builds on the platform delivered by the CMS 11
project. It will enhance the solution to: (1) enable access of information from mobile devices such as 12
smartphones, iPads, and tablet devices, (2) enable location-based services to reduce the data footprint in 13
the Field Tool device, (3) allow contractors and mutual assistance crews to use the Field Tool in case of 14
a storm, (4) expand use of the Field Tool to other SCE Operating Units instead of buying or building a 15
comparable system, and (5) improve data quality, data consistency, and interface between the Field Tool 16
and SCE supporting office systems such as SAP and Comprehensive Geographical Information System 17
(cGIS). 18
The total project forecast is $10.0 million.152 The capital forecast for this project 19
was developed using SCE’s internal cost estimation model. This model utilizes industry best practices 20
and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this project 21
includes costs for SCE employees, supplemental workers, and consultants, software and vendor costs, 22
and hardware costs. See this project’s workpaper for the cost breakdown information. 23
(1) Alternatives Considered 24
Alternative 1: SCE considered not pursuing this project. While the 25
current Field Tool (CMS) is operational, our experience suggests that as additional assets are added to 26
the existing system and the user base expands, the potential for the system performance to degrade 27
increases. In addition, the enhanced mobility options such as tablets and smartphones could not be 28
152 Refer to WP SCE-04, Vol. 2 Bk B p. 180.
108
provided to SCE’s field personnel. Mobile devices can improve efficiency, usability and user safety for 1
certain tasks such as inspections. 2
Alternative 2: SCE considered replacing the existing field tool solution 3
by building or buying a customized solution. This option was rejected because it would not be cost-4
effective to create such a solution, it would cause a redundant application, and it would create additional 5
maintenance costs to the existing system. 6
13. Enhanced Business Resiliency for Energy Management System (EBR) 7
Table V-35 Enhanced Business Resiliency for Energy Management System153
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 8
SCE’s Energy Management System (EMS) supports the reliable operation of the 9
entire SCE power system including the distribution and transmission networks, and is itself supported by 10
primary and back-up systems at two separate SCE locations. This configuration provides for redundancy 11
and protection from physical concerns such as an earthquake or other conditions that would require a 12
back-up site to be activated and staffed. However, both EMS systems are connected to, and run off of, 13
the SCE IT network, which may pose a cybersecurity risk. The Enhanced Business Resiliency for EMS 14
Project establishes a third, off-network back-up site to protect against threats, primarily cyber-security 15
threats, to the existing two EMS systems. 16
b) Need for Project 17
A complete loss of EMS availability would affect the Grid Control Center and the 18
ability of SCE’s 14 switching centers to monitor, control, and optimize the power grid and detect and 19
respond to outages. Any outages while EMS is not available pose a significant risk to public safety, SCE 20
personnel, and external SCE customers, including the CAISO and Peak RC (Reliability Coordinator). 21
153 Refer to WP SCE-04, Vol. 2 Bk B pp. 181-186.
CIT-00-DM-DM-000051 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 3.00 4.00 - - 7.00 Previous GRC Request - - - - - - - -
Recorded Forecast
109
The system and the aforementioned configuration do not provide complete protection from new and 1
emerging security threats. These threats present a potential vulnerability. Besides having other security 2
controls, the additional site would be off of the network for a majority of the time, which reduces the 3
risk of being compromised if the primary and secondary on-network locations are attacked. The tertiary 4
EMS instance could start up within minutes of the primary and secondary locations becoming 5
unavailable. 6
(1) Benefits 7
EMS is used by over 200 system operators and engineers, both at the Grid 8
Control Center (GCC) and SCE’s Switching Center. These personnel rely on the EMS to provide 9
monitoring and controlling functionality and to provide analytical capabilities and reliability solutions. 10
The EMS Real Time Contingency Analysis screens outages and outage combinations and will alert 11
operations if any of the outages being prescreened will have a reliability impact on the system. This 12
critical analysis capability would not be possible if EMS were not available. In addition, it would not be 13
possible for the CAISO or the Peak RC154 to perform this analysis on the SCE grid. SCE outages 14
comprise approximately 50% of CAISO’s contingencies and 30% of Peak RC, which illustrates the 15
potential impact to the western grid. 16
SCE’s EMS system is critical to the reliability of not only the SCE system, 17
but the Western Interconnection. A third instance of EMS that is not attached to the network would 18
increase SCE’s ability to function in certain threat scenarios (e.g., cyber-attack, malware infection). SCE 19
customers and system reliability would benefit from the availability of a third back-up system. The 20
approach of supplementing an online configuration with an instance that stays disconnected from the 21
network for a majority of the time is consistent with cybersecurity best practices, in which high-value 22
targets like “Key Infrastructure” stay disconnected to keep the attack surface as small as possible. 23
c) Scope and Cost Forecast 24
This project will provide a third back-up, standalone system that will not be part 25
of the site synchronization scheme between the existing EMS Primary and Backup systems. The system 26
will receive configuration updates and manual entries when it connects to the network periodically. 27
Once activated, the operators could start the system up with current configuration and synchronize with 28
154 PEAK RC (Peak Reliability Coordinator) provides real time monitoring of the Western Interconnection.
110
manual updates within a reasonable time (expected to be within an hour). This is the preferred method, 1
as it poses the least risk of being compromised by cyber events, and with proper site selection could 2
provide the additional protection against major catastrophic events such as earthquakes. 3
The total project forecast is $7 million.155 The capital forecast for this project was 4
developed using SCE’s internal cost estimation model. This model utilizes industry best practices and 5
SCE subject matter expertise to estimate project cost components. SCE’s forecast for this project 6
includes costs for SCE employees, supplemental workers, and consultants, software and vendor costs, 7
and hardware costs. See this project’s workpaper for the cost breakdown information. 8
(1) Alternatives Considered 9
Alternative 1: SCE considered building a network-isolated mobile system 10
housed in a vehicle, potentially a van or an RV. The vehicle would have an independent source of 11
electrical power (generator) and the means to communicate. The system would be isolated and 12
periodically synchronized with the active EMS system. We did not pursue this option due to the inherent 13
space limitations of a mobile option. Due to the limited space, the system would have to be a scaled-14
down version of the primary/backup EMS with limited functionality. 15
14. Comprehensive Situational Awareness for Transmission (CSAT) Phase 1 16
Table V-36 Comprehensive Situational Awareness for Transmission156
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 17
D.15-11-021 adopted the Comprehensive Situational Awareness for Transmission 18
(CSAT) project in SCE’s 2015 GRC, where the project was referred as “Advanced Phasor Data 19
155 Refer to WP SCE-04, Vol. 2 Bk B pp. 186. 156 Refer to WP SCE-04, Vol. 2 Bk B pp. 187-193.
CIT-00-DM-DM-000085 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 2.00 4.00 8.00 8.00 22.00 Previous GRC Request* - - - 1.20 7.70 4.20 - 13.10 *The Commission adopted SCE's request for this project in D.15-11-021 under the Advanced Phasor Analytics project name.
Recorded Forecast
111
Analytics” under the Phasor program. The objective of the CSAT program is to provide the ability for 1
Grid Control Operators to assess the status of the entire transmission system at a glance and provide 2
quick access to detailed data and robust analytics to make more informed decisions during critical 3
operational periods. This project was scheduled to be deployed from 2014 through 2016. 4
SCE did not launch the Phasor Analytics project as proposed in our 2015 GRC 5
Application. The delay in the CSAT project launch was a result of the extended deployment and 6
stabilization157 of the Phasor project. CSAT requires Phasor data, and the effectiveness of the analytics 7
engine depends on the deployment of several strategically located time-synchronized Phasor 8
Measurement Units (PMUs) to provide improved wide area situational awareness data. SCE will have 9
deployed a sufficient number of PMUs by 2018 to enable the analytics engine to deliver reliable and 10
accurate real-time information. 11
b) Need for Project 12
SCE requires an Advanced Analytics platform to provide improved visibility of 13
grid conditions by utilizing EMS and PMU data. CSAT will use this data to display a graphical, visual 14
representation of the transmission grid in its current state. Operators can overlay circuit information with 15
other key data elements such as asset location, weather, traffic, and fire. These capabilities will 16
collectively give the operators early warning of developing events or impending faults to enable a 17
proactive mitigation strategy and faster, more targeted responses to those events. 18
This proposed solution will follow the recommendations outlined in the Federal 19
Energy Regulatory Commission (FERC)/North American Reliability Council (NERC) report on the 20
2011 San Diego blackout. The report concluded that improved real-time situational awareness would 21
have allowed the system operators both to take proactive action and to take timely restoration measures 22
to operate the system in a secure state without affecting millions of customers on a hot summer day, 23
potentially endangering public safety.158 Implementing the CSAT solution will significantly improve 24
SCE grid dispatcher situational awareness of fast moving abnormalities in the electric grid and will 25
provide targeted mitigation procedures to combat grid instability conditions. 26
157 Stabilization is a common IT process to monitor initial operations of new systems/equipment and optimize
their use in the production environment. 158 Refer to WP SCE-04, Vol. 2 Bk B pp. 194-349.
112
SCE continues to procure renewable power to meet the needs of our customers 1
and make progress toward State policy goals.159 As this happens, SCE’s grid control centers and 2
personnel must be equipped with upgraded tools to enable them to respond effectively to challenges 3
posed by higher penetration of renewables on our grid. Renewable generation sources produce 4
electricity intermittently, not on a predetermined schedule as is the case with more traditional power 5
sources. Both solar and wind power are accompanied by similar variability and reliability challenges. 6
This variability in generation from renewable sources makes it difficult to accurately forecast the 7
amount of expected output, which is a new challenge for Grid Operators who currently rely on 8
scheduled and accurate projections of generation output to maintain the grid in a balanced state. Other 9
challenges related to renewable generation include power quality, contingency planning, and predictive 10
analytics. These factors combine to increase the complexity of operating the grid and drive the need for 11
accurate tools to equip Grid operators with actionable information in a timely manner, improving 12
Situational Awareness in the Grid Control Center. 13
(1) Benefits 14
The CSAT project will provide benefits in three major areas listed below: 15
• Avoidance of Major Outages: Certain fast moving system disturbances 16
and resulting voltage instability could cause outages over a wide area 17
before mitigation actions can be taken. With real-time situational 18
awareness through PMU data and advance analytics, such disturbances 19
could be responded to and the impacts reduced. 20
• Improved Utilization of select Stability Limited Transmission Paths: 21
The analytics will enhance transmission models, specifically the 22
refinement of the dynamic models, to increase power load while still 23
meeting safety and operational requirements. These changes must be 24
coordinated and agreed to by the CAISO & Peak RC.160 25
159 State policy goals related to increasing the percentage of power deliveries from renewable resources can be
found in legislation such as Assembly Bill 32 and Senate Bill 350. 160 PEAK RC is Peak Reliability Coordinator provides real time monitoring of the Western Interconnection.
113
• Energy Procurement Benefits: SCE has conducted a study161 to 1
forecast benefits from increased utilization of transmission line 2
capacity, providing a potential opportunity to reduce energy 3
procurement costs. An average of $1.44 million and $2.36 million 4
annual savings on procurement cost of energy are estimated for a 5% 5
and 10% transmission capacity increase scenario, respectively. 6
c) Scope and Cost Forecast 7
Phase 1: CSAT – Real Time Monitoring and Analytics 8
In this phase, a new analytic platform consisting of the additional hardware and 9
software needed to run the application will be implemented. The system will integrate with SCE’s 10
Geographic Information System (GIS) and Energy Management System (EMS) for PMU data, and 11
provide the visualization capability to monitor real-time grid dynamics, such as phase angles, 12
oscillations, damping, and intelligent alarms. This phase will provide the foundational capabilities for 13
real-time monitoring. 14
Phase 2: CSAT – Enhanced Multi-layer Displays 15
In Phase 2, multi-layer displays will be implemented to enable effective 16
presentation of large amounts of information from various sources and applications, such as geographic 17
layers, electric circuit data, environment & weather data, and synchronized phasor data. The system can 18
diagnose and assess the system events and stressed conditions, plan actions, create preventative analytics 19
using the enhanced state, produce analytic compliance reports, and interface with multiple real-time 20
feeds from multiple sources.162 21
Phase 3: CSAT – Intelligent Displays, Events and Alarms for post-event analytics 22
This phase will provide the Grid Operators with trend displays, adding the 23
capability to view collected data plotted against time (e.g., the operator could observe a downward trend 24
in voltage that could lead to system instability). Operators can also play back and perform post-event 25
analysis. In addition, the CSAT system will enhance the prioritization criteria applied to incoming 26
alarms, improving the operator’s ability to separate and focus on high-priority alarms. The system will 27
display historical records and events using geospatial maps, present alarms on the geographical map, 28
161 Refer to WP SCE-04, Vol. 2 Bk B pp. 350-353. 162 Id.
114
assign priorities, and record change of priority. These features will benefit Grid Operators by providing 1
early warning of risks posed by environmental factors such as temperature, wind, wildfires, and changes 2
in renewable generation levels due to weather changes, etc. 3
The preferred solution is to purchase a COTS application to deploy the 4
capabilities mentioned above. The COTS application will be capable of integrating with SCE’s Grid 5
Analytics platform, which will enable seamless access to data from applications that directly monitor 6
and/or control the grid, as well as from other enterprise applications such as the GIS or services that 7
deliver environmental data. The integration of data from various sources will be implemented using a 8
data lake technology aligning with SCE’s analytical platform. 9
In our 2015 GRC, we forecast $13.1 million for this project. We now forecast 10
total project costs to be $21.8 million.163 The increase in cost is due to additional scope items (explained 11
above), such as integration with other enterprise data sets (e.g., geographical data, environmental and 12
weather data), post-event analysis, and intelligent alarms and events. The capital forecast for this project 13
was developed using SCE’s internal cost estimation model. This model utilizes industry best practices 14
and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this project 15
includes costs for SCE employees, supplemental workers, and consultants, software and vendor costs, 16
and hardware costs. See this project’s workpaper for the cost breakdown information. 17
(1) Alternatives Considered 18
Alternative 1: SCE considered building a customized solution on its 19
standardized data analytics platform. We did not pursue this option since the subject matter expertise 20
required for this endeavor is scarce and expensive in the industry. There are COTS applications 21
available today that will meet the business needs with some additional enhancements and integration. In 22
SCE’s experience, long-term support and maintenance of a custom application will generally cost more 23
than vendor costs to support a licensed COTS application. 24
163 Refer to WP SCE-04, Vol. 2 Bk B p. 193.
115
15. Centralized Remedial Action Scheme (CRAS) Project 1
Table V-37 Centralized Remedial Action Scheme164
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
D.15-11-021 partially adopted the Centralized Remedial Action Scheme (CRAS) project. 2
SCE implemented CRAS in April 2016. The CRAS project included CRAS application development, 3
implementation of central control telecommunication infrastructure, and deployment of two RASs 4
(Remedial Action Schemes, defined below) to validate full capabilities of CRAS on SCE’s transmission 5
grid. As part of the final testing and implementation of CRAS, we have been able to observe that the 6
inability of GOOSE165 messages to be routable validated the need to upgrade the communication 7
capability of the architecture. This upgrade is discussed in the subsequent RGOOSE Project request.166 8
Remedial Action Schemes (RASs) are designed to detect predetermined system 9
conditions on the transmission grid and automatically take specific corrective actions such as tripping 10
generation or shedding load. When problematic system conditions are detected via RAS monitoring 11
relays, the RAS system arms and prepares to take protective action. This is our protection against 12
cascading outages and similar system conditions. If system conditions worsen, system load is shed (in 13
cases where load exceeds the remaining transmission capacity resulting in overloads or voltage collapse) 14
or generation is tripped (in cases where generation exceeds transmission capacity). This isolates the 15
system emergency, and prevents it from causing widespread services outages. 16
164 Refer to WP SCE-04, Vol. 2 Bk C pp. 3-91. 165 GOOSE refers to Generic Object Oriented Substation Events. 166 RGOOSE Project is a network upgrade project that improves the accuracy, failover capability, and
traceability of CRAS communications by upgrading the communications from current protocol (called GOOSE) to RGOOSE (Routable GOOSE). Please see the RGOOSE Project, presented below, for more information.
Forecast
CIT-00-SD-PM-000102 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast 0.34 9.48 15.88 8.12 4.82 9.54 0.95 - - - - 49.13 2015 GRC Authorized* 0.34 9.48 15.88 8.12 33.82 2015 GRC - Original Request 0.34 9.48 15.88 13.92 9.77 - - - 49.39
* In D.15-11-021, the Commission adopted 2013 recorded expenditures and allowed SCE to reapply for capital expenditures in later years.
Recorded
116
As an example, a RAS might sense an overload on a transmission line it monitors. Based 1
on that reading, the RAS will send a signal to a generator, taking the generator offline to avoid 2
overloading lines and damaging equipment. Most RASs are far more complex than this simple example, 3
with many monitoring points and numerous conditions under which an action to either curtail generator 4
output, drop generation, or shed load would be initiated. Although technically complex, RAS systems 5
are significantly less expensive than constructing redundant transmission lines. Also, RASs can be 6
implemented much faster and avoid the extensive licensing processes required for new transmission 7
lines. RAS installations are approved by the Western Electricity Coordinating Council’s (WECC’s) RAS 8
Reliability Subcommittee (RASRS).167 9
a) Project Description 10
In the 2015 General Rate Case, SCE proposed its CRAS (Centralized Remedial 11
Action Scheme) project at a cost of $49.392 million. In D.15-11-021, the Commission partially 12
approved the CRAS funding request by authorizing expenditures through 2013.While the Commission 13
Decision cited CRAS’s “intuitive appeal,”168 it disallowed recovery of costs for 2014 and 2015. In its 14
Decision, the Commission allowed SCE to reapply for the denied capital expenditures in our next GRC, 15
if we provided a detailed cost-benefit analysis in support of that request. We are providing that analysis 16
in this filing. The balance of unrecovered amounts between 2014 and 2016 is $15.31 million. 17
Since SCE’s transmission system is networked, we must protect against a single 18
transmission line outage resulting in a cascading blackout and wide-scale interruption of service. Our 19
transmission system network connects large, remotely located generators to major load centers and 20
connects our system to neighboring utilities. The reliability of the grid must be preserved as new 21
generation comes online. One factor we consider when a new generator interconnects is how the power 22
from that generator will flow under normal and contingency conditions. 23
167 “The purpose of the RASRS is to review the reliability aspects of existing and planned Remedial Action
Schemes (RAS) and to enhance grid performance within the Western Interconnection by providing a uniform review process.” https://www.wecc.biz/OC/Pages/RASRS.aspx
168 See D.15-11-021, pp. 43-44 (“[T]he intuitive appeal of the CRAS benefits that SCE describe are strong and the outcome of any effort to quantify them at this time may be primarily driven by preliminary assumptions (number of interconnections, policies on economic curtailment, etc.). As a matter of policy, this Commission supports a future with renewable generation resources operating efficiently on the grid and seeks opportunities to improve grid operations with respect to such resources. CRAS appears to be such an opportunity, and may be cost-effective in some scenarios. . . .”).
117
When new generators request interconnection and the existing transmission 1
system is not capable of handling all required contingencies, SCE has two choices. We can either build 2
redundant transmission lines or install a transmission system RAS. A redundant transmission line 3
provides the alternate pathway for energy during an outage and prevents other networked lines from 4
overloading. On the other hand, RASs work by shedding generation and/or load when a load flow 5
congestion condition occurs that risks system reliability or stability. This reduces power flows on the 6
system and prevents overloads that could lead to cascading outages. 7
Building many redundant transmission lines takes years to license and construct 8
and can be very expensive. These lines can cost as much as $9.5 million per mile and run dozens of 9
miles or more. SCE prudently pursues the construction of transmission line in some situations, but we 10
also rely on RASs to protect system reliability. To date, SCE has implemented 17 RASs across its 11
service territory. These systems are designed to automatically disconnect generators, load, or both under 12
certain identified system conditions. Prior to implementing CRAS, each RAS was designed as a 13
separate, self-contained system (e.g., a stand-alone system), including all of the relays, instrumentation, 14
and other equipment necessary for the RAS to function as designed. 15
SCE’s existing RASs are field-based (“stand-alone”) and have logic controllers in 16
the substations. CRAS, in contrast, has centralized decision-making and trip settings (“analytics”) 17
running on high-speed servers that are considerably more powerful than the field-based logic processors 18
in stand-alone RASs. Unlike a stand-alone RAS logic processor, CRAS is scalable by using additional 19
high-speed servers. This is important because the field-based logic processors have hard constraints 20
inherent in their design that limit the level of RAS complexity that can be accommodated. In contrast, 21
SCE designed and tested the initial hardware capacity of the CRAS system to be scalable to meet five-22
year “worst case” expected growth (30 RASs). SCE cannot envision a scenario where CRAS analytics 23
processing capacity would be exceeded by RAS data demands because CRAS blade servers are scalable 24
in processing power. SCE would either add additional blade servers, or upgrade existing servers, or 25
both. 26
b) Need for Project 27
The increasing RAS complexity that SCE described in its 2015 GRC application 28
occurred sooner than anticipated. The interconnect tariff requires four RASs in 2018-2020. These RASs 29
have a high complexity level. That complexity is composed of a higher number of generators, 30
118
substations, and lines to protect and, therefore, more contingency conditions to evaluate and on which to 1
take action. 2
The best measure of RAS capacity and complexity combines these factors in what 3
are called “arming points.” Arming points are the specific thresholds where a RAS will take action for a 4
particular generator as it evaluates a specific contingency (problematic system condition) related to that 5
generator or load. A generator might have one or more arming points based on the number and types of 6
contingencies involved. These arming points are defined by flow on transmission lines, in combination 7
with generation output that require mitigation if critical contingencies occur. Regarding specific arming 8
points for a generator, the contingency involved (such as a line being overloaded) might also apply to 9
other generators with other arming points. Arming points requirements, because they relate numbers of 10
contingencies and the number of generators, are a reasonable measure of a stand-alone RAS’s capacity. 11
Current stand-alone RAS logic capacity, without CRAS, is limited to 48 arming points due to limitations 12
in the relay hardware and firmware. 13
The technical problem that SCE faces is the complex stand-alone RASs needed in 14
the 2018-2020 timeframe would run out of capacity to add generators. In other words, a stand-alone 15
RAS would run out of available arming points. For example, one of these RASs (Northern Area RAS, a 16
new RAS) is significantly complex, requiring an estimated 76 arming points total to accommodate 17
Queue Cluster 8 (QC-8) or 124 arming points total when factoring in Queue Cluster 9 (QC-9).169 18
Northern area RAS cannot be accommodated as a stand-alone RAS.170 This is supported by SCE’s 19
planning and protection study. Therefore Northern RAS must be served by CRAS. The other alternative 20
to CRAS for the Northern Area RAS is building new transmission lines. This would be prohibitively 21
expensive,171 and the licensing and construction timeline would cause significant delays to two large 22
queue clusters in process (QC-8 and QC-9). Alternative design considerations of trying to “dumb down” 23
169 These queue clusters are groupings of projects in the interconnection queue that fall into similar geographic
areas and timeframes as describes further in the Generation Queue Completion section. 170 See testimony section Generation Queue Completion for additional details, which shows that most future
generation projects come to completion. 171 For more information on alternatives, refer to WP SCE-04, Vol. 2 Bk C pp. 9-56 and refer to WP SCE-04,
Vol. 2 Bk C pp. 57-71 (Northern Area Transmission tab), which shows transmission build timeline). In addition, the cost study compares CRAS for Northern Area versus a hypothetical stand-alone RAS for comparison purposes.
119
a stand-alone RAS approach (e.g., compress the required logic tables) have many undesirable effects, as 1
explained below. 2
In the past, SCE has compressed the logic tables for a stand-alone RAS (as 3
described in the 2015 GRC CRAS testimony), using Devers stand-alone RAS as an example.172 4
“Compression” is the grouping together of multiple contingencies on each logic step that would result in 5
sufficient corrective action. SCE considered using this approach for Northern Area, but there are two 6
potential problems: 7
1. Combining contingencies and/or generation would result in consistently 8
tripping more generation than needed for mitigation,173 which could expose the system to unwarranted 9
risks and potential incremental costs. Over-tripping uses more system spinning reserves than necessary, 10
which could cause the system to operate closer to stability limits. 11
2. Tripping more generation than needed for mitigation has impacts on the 12
generators. The more interruptions experienced by a generator, the more wear and tear occurs on the 13
mechanical and electrical systems of the generator; interruptions also cause an overall decrease in the 14
generator’s power production and associated revenues. 15
CRAS is needed to mitigate the potential issues caused by the compression of 16
contingencies/logic steps using a traditional stand-alone RAS. CRAS is designed to allow for 17
significantly more arming points, thereby permitting improved corrective action options that are tailored 18
to the specific needs of the grid, and avoiding over-tripping generation that is unnecessary. As 19
demonstrated in the workpaper entitled “Northern Area RAS Arming Points Comparison,” CRAS would 20
provide up to 189 arming points compared to the 45 provided by traditional RAS, providing a significant 21
increase in operational flexibility 22
(1) Analysis of additional complex RASs 23
Arming points requirements have been increasing significantly in 24
renewables-rich areas, as shown in the Figure V-7 below.174 25
172 See SCE-03, Vol. 2 of SCE’s 2015 GRC Application. 173 Refer to WP SCE-04, Vol. 2 Bk C pp. 72-74. 174 Refer to WP SCE-04, Vol. 2 Bk C pp. 75-78 and WP SCE-04, Vol. 2 Bk C pp. 79-80.
120
Figure V-7 Arming Points (Current & Future)
In addition, the diagram below Figure V-8 “RASs Nearing or Exceeding 1
Capacity Limits” provides a schedule of upcoming RASs, including Northern Area RAS in 2018. These 2
are the upcoming complex RASs in renewables-rich areas that are driving the need for CRAS.175 3
175 Refer to WP SCE-04, Vol. 2 Bk C pp. 81-85.
121
Figure V-8 RASs Nearing or Exceeding Capacity Limits
As a further illustration of the need for CRAS, please refer to Figure V-9 1
below on Northern Area RAS, which shows the numerous generators in queue. SCE is seeing large 2
queue numbers in renewables-rich areas, with still more expected as a result of the State’s 50% 3
renewables mandate by year 2030. 4
The challenge is more than just a RAS being complex at a certain point in 5
time. RASs in renewables-rich areas have many generators in queue, and this means a high rate of 6
change to the RAS. This has two undesirable effects concerning stand-alone RASs: (1) Any attempt to 7
increase capacity, such as by changing the relay CPUs (central processing units) and firmware, merely 8
creates another hard limit in arming points capacity that is likely to be exceeded in the future, and 9
(2) Changes to a stand-alone RAS are much more expensive than changes to a RAS in CRAS due to the 10
ease of making programming changes in CRAS centrally as well as considerably more streamlined test 11
procedures in the field to validate the RAS functionality. The CRAS Cost Study Overview and 12
Reference176 has a detailed explanation of the differences in testing methods and levels of testing 13
automation available in CRAS versus a stand-alone RAS which drive large differences in required labor 14
between the two approaches. 15
176 Refer to WP SCE-04, Vol. 2 Bk C pp. 9-56.
122
Figure V-9 177 Northern Area RAS Area Diagram with Generators in Queue
Regarding the method used to tally Northern Area arming points, when 1
QC-8 cluster additions are accounted for, 76 arming points are estimated for Northern Area. This is the 2
trigger for realizing we must use CRAS, as a stand-alone approach will not work. When we look further 3
out and add QC-9 cluster, total arming points for Northern Area climbs to 124 arming points, which far 4
exceeds the current maximum limit of 48 arming points for traditional stand-alone RAS, as well as the 5
66 arming points total offered by a stand-alone RAS capacity upgrade.178 6
(2) Generation Queue Completion 7
In the last two SCE GRC decisions, the Commission raised questions 8
about the percentage of queue projects that actually reach completion. A recent study on queue data 9
performed by SCE using CAISO queue data shows that a large majority of these projects do in fact 10
complete, and that the rate of completion is generally increasing over time. Table V-38 below shows 11
available CAISO queue data. As the numbers for individual years indicate, more projects can come on-12
177 Refer to WP SCE-04, Vol. 2 Bk C pp. 86-87. 178 Refer to WP SCE-04, Vol. 2 Bk C pp. 88-89.
123
line in a particular year because prior projects have slipped from their original proposed operating dates 1
to subsequent years. Also, it is not uncommon for more projects to come on-line in any given year 2
compared to what was expected at the beginning of a queue cluster study cycle. CAISO’s generator 3
interconnection study process is ongoing, with each queue cluster taking up about two and one-third 4
years from inception to completion. In any given year, there are typically two clusters that overlap each 5
other in a staggered manner. For instance, Queue Cluster 8 (QC-8) is in the Phase II study stage 6
concurrently with Queue Cluster 9’s Phase II study, with various reassessments in progress as well. 7
Detailed information on the application and study processes, timing, posting requirements, etc., can be 8
found at the CAISO link.179 9
Table V-38 Queue Completion Table
As shown in the Queue Completion Table, from 2006 to 2016, there were 10
41 queue applications. Thirty of these applications actually went into operation, which is an overall 11
completion rate of 73% over 11 years. The completion percentage has also increased significantly since 12
the CRAS project was launched, which is why the growth in complexity of RASs was greater in 13
magnitude and occurred sooner than originally expected. SCE must prepare to interconnect most of the 14
179 http://www.caiso.com/planning/Pages/GeneratorInterconnection/InterconnectionStudy/Default.aspx
124
generation that seeks interconnection. With the limitations inherent in stand-alone RASs, SCE cannot 1
rely upon local RAS control to achieve the interconnection requirements needed by the currently active 2
queue clusters.180 3
(3) Benefits 4
CRAS offers an easier, quicker, and less expensive solution than stand-5
alone RASs.181 Two key attributes that CRAS provides to achieve these advantages are a standardized 6
approach and automated test capabilities. These powerful features were designed to replace the 7
cumbersome, one-off customization needed for stand-alone RASs that increase the stand-alone RAS 8
costs at every stage—from design through installation, testing, and completion of commissioning. For 9
example, there are only three basic types of CRAS relays with identical settings for each type: 10
monitoring relays, customer-side mitigation relays, and utility-side mitigation relays. Stand-alone RAS 11
relays, in contrast, are individually customized. This customization drives increases in labor in all phases 12
of work, placing greater demand on SCE resources. 13
With detailed cost data for (1) the CRAS Project in hand, (2) the 14
underlying stand-alone RASs before they were brought into CRAS, and (3) many changes to stand-alone 15
RASs since 2013, SCE now has the data available to perform a more detailed cost-benefit comparison of 16
CRAS to stand-alone RASs. The cost-effectiveness for CRAS can be demonstrated for all new RASs 17
and certain types of existing RASs planned to be brought into CRAS. These RASs are existing complex 18
RASs in renewable-rich areas (e.g., Colorado River RAS, Whirlwind RAS, and Mojave Desert RAS) 19
that are running out of logic capacity, as explained above. 20
SCE considered a stand-alone RAS relay capacity upgrade to increase the 21
approximate number of arming points from 48 to 66.182 This would still leave the drawback of a hard 22
limit in arming points in place in the future for the stand-alone RASs, albeit a higher hard limit. SCE’s 23
cost study shows that such a capacity upgrade is prohibitively expensive due to high implementation and 24
change costs (as future generators will be added each year). CRAS addresses the expensive nature of a 25
stand-alone RAS installation through the following cost efficiencies: 26
• Reducing labor for each round of test. 27
180 Refer to WP SCE-04, Vol. 2 Bk C pp. 90-91. 181 Refer to WP SCE-04, Vol. 2 Bk C pp. 9-56 and WP SCE-04, Vol. 2 Bk C pp. 57-71. 182 Refer to WP SCE-04, Vol. 2 Bk C pp. 72-74.
125
• Reducing field crew requirements by eliminating the need for physical 1
inputs from field test sets in the final and most time-consuming phase 2
of testing (end-to-end field testing). 3
• Reducing the rounds of tests triggered by different design and logic 4
capabilities. 5
The complex RASs in generation growth areas brought into CRAS have 6
long test plans in terms of the number of test steps needed to validate the RAS’s functionality (250 – 7
1,000 or more lines of code). Stand-alone RASs and CRAS have very different test methodologies. 8
CRAS has automated test capabilities, which leads to dramatic differences in field labor requirements 9
for the same basic type of RAS validation. 10
The test methodology for a stand-alone RAS involves providing physical 11
inputs (currents and voltages) to field relays from test sets in the substations. Each test set typically can 12
only be hooked up to four relays at a time, depending on the scenario being tested. Frequently during the 13
lengthy test plans (250 – 1,000 or more lines of test steps) the test sets must be moved and connected to 14
different field relays. The substation environment is hazardous, making fast changes of temporary test 15
lead wiring unsafe. In summary, stand-alone RAS testing is cumbersome and slow, using significant 16
labor to validate the RAS. 17
By contrast, CRAS first validates physical inputs during the individual 18
testing that all relays must have, to make sure the relay is correctly sending and receiving data, and then 19
uses virtual inputs and automated test scenarios that can be loaded and activated centrally—a much 20
faster process than using the test sets with physical inputs as the stand-alone RASs do. 21
An additional benefit CRAS provides is that it eliminates the need to 22
repetitively enter physical inputs, which reduces the number of crews required per substation during 23
field end-to-end testing. In the end-to-end phase of testing, eliminating this need for physical inputs 24
reduces or eliminates the need for test sets, which are limited to about four relay connections per test 25
scenario. This in turn reduces the number of test technician hours needed, and represents a significant 26
reduction in required labor as RASs get more complex with larger test plans (e.g., more scenarios and 27
more line items in the test plan). 28
In addition, CRAS is more efficient in its test methodology and flexible in 29
its logic. As a result, we can consolidate designs and redesigns as generators are added, in order to 30
minimize the number of rounds of field end-to-end testing compared to stand-alone RAS testing. This 31
126
means that labor for CRAS testing is much more efficiently used, with fewer test hours per round of test 1
and significantly fewer rounds of end-to-end testing per year for the complex RASs ahead. In qualitative 2
terms, CRAS’s more efficient testing approach can cut times for scenarios (groups of test plan lines in a 3
test plan) from “hours” to “minutes” and time for a complete round of end-to-end testing in a complex 4
RAS from “weeks” to “days.”183 5
An additional CRAS benefit falls more into the area of risk mitigation than 6
direct cost avoidance: As part of the logic testing that SCE does centrally for CRAS logic validation 7
(e.g., the logic and algorithms present in the CRAS central controller servers), CRAS can centrally 8
execute more tests to validate logic, detect failures, and monitor communication volume and speed. 9
These additional tests, especially in the area of additional logic validation, would be prohibitively 10
expensive to perform for a stand-alone RAS because of the physical test set requirements described 11
above. The reason is that additional logic validation involves testing a range of increasingly unlikely 12
combinations of system conditions that would be high impact if they occur. This additional logic 13
validation is practical to test when high-speed CRAS automated testing is available. It is not practical 14
when stand-alone RAS test methods are used because field test plans would increase to thousands or 15
tens of thousands of lines. For a stand-alone RAS, this would result in a large increase in labor that is 16
beyond field crew availability constraints. In addition, it is impractical to schedule outages (with CAISO 17
and the generator customers involved) of the required length to do such additional testing for stand-18
alone RASs. 19
To clarify SCE’s plans to use CRAS going forward, not all existing stand-20
alone RASs will be converted into CRAS. We do not expect that all stand-alone RASs will reach the 21
threshold values of complexity or number of generators as the RASs in the cost study. The majority of 22
SCE’s existing RASs generally do not meet these criteria for conversion because of the conversion costs 23
to CRAS. It is the RASs in high renewables growth areas that typically meet the criteria for inclusion 24
within CRAS. Any RAS with arming points growth could meet criteria for inclusion even if their current 25
level of complexity is not that high because costs to accommodate growth in number of generators or 26
additional transformer banks are very high for stand-alone RASs and moderate with CRAS. SCE’s cost 27
study also confirms the benefits of deploying all new RASs into CRAS. 28
183 Refer to WP SCE-04, Vol. 2 Bk C pp. 9-56.
127
i. CRAS Cost Avoidance Summary 1
Table V-39 below shows the sum of expected cost avoidances for 2
CRAS for years 2018 – 2030 (in nominal dollars) in the more detailed cost study. A new transmission 3
build for Northern Area, as the only feasible alternative to CRAS, is a significant cost-avoidance driver. 4
In addition, the table contrasts stand-alone RAS implementation and change costs versus the 5
significantly lower implementation and change costs for CRAS. Designing and implementing stand-6
alone RASs is significantly more expensive than CRAS. Much of the difference is caused by high end-7
to-end field crew testing requirements for stand-alone RASs. These field testing requirements are much 8
more streamlined as explained above because a different test methodology is feasible and CRAS has 9
automated testing capabilities after the relays are given certain initial testing.184 10
Regarding Whirlwind RAS, at first glance it would appear that 11
there is lower than normal cost avoidance versus CRAS. This is due to the unusually high number of 12
existing generators served by the Whirlwind RAS. SCE tariffs require SCE to pay for CRAS change 13
costs in customer facilities. These costs are factored into the results of the cost study. 14
ii. CRAS and RGOOSE Cost Recovery 15
RGOOSE, as explained further in the RGOOSE section, is a 16
network communications protocol upgrade that serves as an enabler for further CRAS deployment. As 17
shown in SCE’s cost studies, CRAS appears to be more cost-efficient than alternatives. Therefore CRAS 18
cost avoidances (covered in detail in this CRAS section and CRAS Cost Study work papers) should 19
apply to both CRAS remaining unrecovered project costs and RGOOSE project costs. 20
184 Refer to WP SCE-04, Vol. 2 Bk C pp. 9-56.
128
Table V-39 CRAS Expected Cost Avoidance
(in Nominal Dollars)
c) Scope and Cost Forecast 1
The CRAS project consisted of system deployment work, including procuring, 2
developing, and testing software applications for communications processing, grid monitoring, 3
performance self-monitoring, security monitoring, security monitoring and controls, and protection logic 4
algorithm management. The project scope also included procuring, configuring, testing, and installing 5
servers and racks for these applications. Project implementation included converting two RASs into 6
CRAS as part of the deploying and demonstrating CRAS. SCE tested these two RASs in parallel with 7
the existing stand-alone RAS’s, and they are currently operational as part of the CRAS system. 8
Telecommunications work included fiber modifications, microwave link modifications, new switches, 9
and new routers. Substation equipment and telecommunication equipment within the substation must be 10
physically and electronically protected from unauthorized access to comply with NERC/CIP standards. 11
SCE included the construction of these physical and electronic security perimeters as part of the project 12
deliverables. 13
SCE is estimating revised total CRAS project costs of $49.13 million from 2010 14
through 2016. In our 2015 GRC, SCE requested $49.39 million for total CRAS project costs. D.15-11-15
129
021 allowed recovery of recorded costs from 2010 through 2013, which totaled $33.82 million. This 1
leaves remaining unrecovered CRAS Project costs at $15.31 million.185 2
(1) Alternatives Considered 3
Alternative 1: SCE considered not implementing this “project” (in this 4
sense, by “project” we mean further deployment of CRAS, not the base CRAS project already 5
implemented). Instead, for the three RASs out of the four under consideration where a stand-alone RAS 6
capacity upgrade is possible, we considered increasing the capacity of stand-alone RAS relays instead of 7
converting the RASs into CRAS. This alternative is not recommended because it is cost-prohibitive and 8
merely sets another hard arming-points limit that will be exceeded in the future, eventually causing the 9
same capacity shortfall. In addition, a critical deficiency of this alternative is that Northern Area RAS 10
(the remaining RAS out of the four under consideration) cannot be accommodated as a stand-alone RAS 11
because it would not meet grid protection requirements186 even if there is an increase in stand-alone 12
RAS relay capacity (see Alternative 2). Alternative 1 provides no long-term solution to arming-point 13
capacity and is not cost-efficient. 14
Alternative 2: SCE considered not implementing this “project” (in this 15
sense, by “project” we mean further deployment of CRAS, beyond the base CRAS project already 16
implemented) and instead, build a transmission line to meet the Northern Area protection needs. This 17
alternative is not recommended due to costly infrastructure and lead time requirements that would 18
significantly impact the feasibility of QC-8 and QC-9 renewables generation projects. 19
185 Refer to WP SCE-04, Vol. 2 Bk C p. 8. 186 Refer to WP SCE-04, Vol. 2 Bk C pp. 88-89.
130
16. RGOOSE Project 1
Table V-40 RGOOSE Project187
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 2
The RGOOSE Project is a communications network upgrade project. It takes 3
advantage of the significant communication improvements offered by a new industry standard for 4
network communication and security.188 This industry standard improves the existing GOOSE (Generic 5
Object Oriented Substation Events) protocol used in substations to Routable GOOSE (RGOOSE). 6
The GOOSE protocol sends data packets (like small messages) in broadcast 7
fashion, much like a radio network. The packets go to all network points and all devices “hear” the 8
messages whether they need to or not. This broadcast approach puts a substantial burden on a large 9
network.189 RGOOSE significantly improves performance by making the protocol “routable” by giving 10
each packet precise targeting to go only where it needs to go, to a specific device or set of devices. As 11
networks expand beyond the substation (such as a large network combining many substations and 12
control centers), this precise targeting is important for the capabilities of the system and for effective 13
troubleshooting. The RGOOSE communication protocol will first be used by SCE to meet the growing 14
data communications needs of CRAS project. As described in the following sections, RGOOSE is 15
required for CRAS moving forward to accommodate the high data volume and complexity of network 16
communications associated with upcoming RASs that contain significantly greater arming point 17
requirements. 18
187 Refer to WP SCE-04, Vol. 2 Bk C pp. 94-106. 188 Industry standard IEC 61850 90-5. Background information: https://www.pacw.org/no-
cache/issue/december_2013_issue/network_architecture/microgrids_integrating_renewable_energy_sources_res_to_improve_reliability/complete_article/1/print.html.
189 Refer to WP SCE-04, Vol. 2 Bk C pp. 103-106.
CIT-00-SD-PM-000243 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Total
Recorded / Forecast - - - - - 4.50 5.90 - - - 10.40 Previous GRC Request - - - - - - - -
Recorded Forecast
131
b) Need for Project 1
The following diagram provides a simplified view of the CRAS system and 2
indicates the high flow of data at all points in the communication structure. For example, monitoring 3
relays send information about the status of transmission lines to the central controllers, and mitigation 4
relays receive instructions about the generation tripping to be executed through opening circuit breakers 5
at the generation plant interconnection. 6
Figure V-10 CRAS Simplified System Diagram
In Figure V-10, “CRAS Simplified System Diagram,” blue lines indicate high-7
speed diverse path communication links with switches and routers. A and B side monitoring and 8
mitigating data go to and from control centers. Per WECC requirements, RASs have A and B sides and 9
diverse paths for redundancy/reliability purposes. The small arrows represent a high volume of 10
messages. CRAS central controllers are servers in control centers that evaluate monitoring data and send 11
mitigation signals. Mitigation relays trip generation or shed load as needed to protect the transmission 12
grid (via circuit breakers not shown). 13
RGOOSE is the communication architecture that enables high speed and reliable 14
communication between the substation and control center. Its central value is the ability to target 15
messages to the specific devices that need them, providing greater accuracy and traceability for CRAS 16
communications, as well as adding communications failover capabilities that a GOOSE network cannot 17
provide. 18
Our experience during late phases of testing and since CRAS implementation has 19
validated the capacity of relay equipment to emit extremely high volumes of data, as this characteristic 20
is required to meet the control functions of CRAS. The high data traffic in the current GOOSE network 21
132
raises concerns that, due to the required GOOSE tunnel architecture, the communication equipment may 1
not be capable of handling the expected increase in transmitting all data packets reliably in the current 2
design.190 With the current GOOSE deployment, the system will not be able to satisfactorily handle the 3
increased data flow for the more complex RASs such as Colorado River, Northern Area, Mojave Desert, 4
and Whirlwind. 5
The only way to effectively use GOOSE protocol on a large network is to limit 6
where it can go. To accomplish this communication, tunnels are created on the network. These tunnels 7
force communication from substation A to go only to a limited number of devices (such as the central 8
controllers in the control center) but not get mixed with communication from substation B on the wider 9
network. Each substation requires multiple tunnels for GOOSE to be operating effectively. 10
Unfortunately, the settings for the communications devices to create the tunnels are one-off, customized, 11
and cumbersome to configure. The proliferation of these cumbersome tunnels as a GOOSE network 12
grows in size raises performance risks as more substations are added. 13
Two additional challenges exist with GOOSE: First, there is a concern that as 14
more relays are “talking” and the tunnel architecture is overwhelmed, critical data packets can be 15
dropped. This problem can occur when tunnels fail. Second, as additional tunnels are created to enable 16
more communications, diagnosing failures becomes extremely challenging. Each device in the network 17
is individually customized and is not addressable on an individual basis, making it considerably more 18
difficult to track and troubleshoot problems. To use an analogy, the monitoring relays speaking GOOSE 19
can be viewed as a group of radio stations emitting different radio programs. Each mitigation relay must 20
listen to all of the programs on a particular tunnel, but only operate when it hears a particular song. In 21
contrast, RGOOSE architecture is more akin to a telephone exchange where relays are called and given 22
instructions without every relay having to listen to the phone call targeted for only one relay. 23
With the more complex RAS requirements SCE now faces, RGOOSE is needed 24
so that the functional requirements of CRAS can be fulfilled and the full benefits of CRAS are realized. 25
The first system to benefit from the use of the RGOOSE communication protocol will be CRAS. The 26
description, business need, and benefits of the CRAS solution are provided in the section above. To 27
move beyond the initial substation deployment of CRAS and support the full deployment of this 28
190 Refer to WP SCE-04, Vol. 2 Bk C pp. 103-106.
133
solution, we need to use RGOOSE. We need to adopt this more advanced protocol because of 1
significantly greater number of arming points191 in upcoming RASs. 2
The most significant arming point challenge SCE faces is with the Northern Area 3
RAS, which requires 76 arming points in the near term and 124 in the longer term, as explained further 4
in the CRAS project presented earlier. Each arming point drives continuous messaging traffic on the 5
network about the conditions affecting that arming point, such as electric current flow or the status of a 6
transmission line. This data volume requires the certainty of message delivery as well as the rapid 7
diagnostics of communication failure. We can obtain these capabilities by deploying RGOOSE on the 8
CRAS network for future RAS deployments. 9
The RGOOSE Project has been developed to take advantage of the now-available 10
standard (IEC 61850 90-5) to overcome the limitations inherent in GOOSE. The RGOOSE standard was 11
not available when CRAS project was launched in 2011; it was necessary to use the protocol available at 12
the time (GOOSE) to design, implement, and test the central controllers for CRAS. The industry has 13
been developing the RGOOSE protocol for several years due to increasing data needs on the electric 14
grid. SCE is able to upgrade to RGOOSE as it is now technically and commercially feasible to 15
implement. In the meantime, a great increase in RAS complexity occurred as the rate of generation 16
queue project completion increased to substantially greater levels compared to when the CRAS project 17
was first launched. This increased complexity leads to the need for the RGOOSE network upgrade 18
shortly after CRAS project implementation. 19
(1) Benefits: 20
The primary benefits of RGOOSE are: 21
• More accurate, stable, and secure high-speed communication on the 22
wide area network, regardless of the volume of communication. This 23
reduces the risk of outages on the transmission grid. 24
191 As explained in the CRAS section, the increasing RAS complexity that SCE described in its 2015 GRC
Application is being experienced, with four RASs required in 2018-2019 that have a high complexity level. That complexity represents a higher number of generators, substations, and lines to protect—and therefore more contingency conditions to evaluate and take action on. The best measure of RAS capacity and complexity combines these factors in what are called “arming points.” Arming points are the specific thresholds where a RAS will arm for a particular generator as it evaluates a specific contingency (problematic system condition) related to that generator.
134
• Better capabilities in monitoring internal communications and 1
troubleshooting. This leads to quicker restoration if a communication 2
problem does occur. 3
Adopting RGOOSE brings the network capability up to match the 4
advanced capabilities of the CRAS platform central controllers and field relays. 5
b) Scope and Cost Forecast 6
Upgrading network gear and central controller software to the RGOOSE protocol 7
to prepare for upcoming RAS deployments, consists of four main areas: 8
1. Upgrading relays. General Electric Company will deliver RGOOSE-9
compatible test relays. This is a relatively minor cost component (General Electric incurred the design 10
and development costs to convert their relays to RGOOSE). 11
2. Upgrading to RGOOSE-capable networking gear (switches and routers for 12
test purposes), including communications network design costs. 13
3. Upgrading the CRAS central controller software to use RGOOSE, which 14
includes costs for designing and developing the upgrade. 15
4. Testing key aspects of the overall CRAS system using RGOOSE in Site 16
Acceptance Test - Cycle 1 to make sure the communications changes will succeed. See cost workpaper 17
for additional details. 18
The actual rollout of communications networking for a particular RAS and the 19
detailed testing of that particular RAS are not included in the RGOOSE project. Instead, these items are 20
covered as part of infrastructure work performed regardless of the conversion to RGOOSE. The network 21
gear will be at similar cost to the current CRAS network gear (or, communications gear for stand-alone 22
RASs). The testing for CRAS deployment for particular RASs will be much more comprehensive, 23
automated, and efficient than for stand-alone RASs and will therefore be highly cost-effective.192 24
SCE launched the RGOOSE project in June 2016, and it is expected to be 25
completed in Quarter 2 of 2017 (with completion of the first cycle of Site Acceptance Testing). SCE will 26
then deploy CRAS using RGOOSE for upcoming RAS infrastructure investments. The net present value 27
of the T&D capital cost avoidances more than offset the CRAS Project and RGOOSE Project costs, as 28
192 Refer to WP SCE-04, Vol. 2 Bk C pp. 9-56 and WP SCE-04, Vol. 2 Bk C pp. 57-71.
135
discussed in the CRAS Project section of the filing and associated work papers. RGOOSE is a key 1
enabler for further CRAS deployment, as described in this section. Therefore CRAS cost avoidances 2
(covered in CRAS section and CRAS Cost Study) should cover both CRAS remaining unrecovered 3
project costs and RGOOSE project costs. 4
The capital forecast for this project was developed using SCE’s internal cost 5
estimation model. This model utilizes industry best practices and SCE subject matter expertise to 6
estimate project cost components. SCE’s forecast for this project includes costs for SCE employees, 7
supplemental workers, and consultants, software and vendor costs, and hardware costs. See this project’s 8
workpaper for the cost breakdown information. 9
(1) Alternatives Considered 10
Alternative 1: SCE considered not implementing this project. In other 11
words, we considered not implementing the RGOOSE protocol193 and not deploying CRAS further due 12
to the risks of using GOOSE protocol with significantly more complicated RASs. Instead, we could 13
implement stand-alone relay capacity increases on three of the four RASs (Colorado River, Mojave 14
Desert, and Whirlwind) and a new transmission build for Northern Area (i.e., the remaining RAS that is 15
too complex for a stand-alone RAS architecture). This combined alternative is not recommended, as 16
described above in the CRAS testimony. For three of the four RASs, this alternative does not provide a 17
long-term solution to capacity issues, and it is not cost efficient. For Northern Area RAS, the only other 18
alternative besides CRAS with RGOOSE is building out very expensive and redundant transmission. 19
Alternative 2: SCE considered not implementing the RGOOSE Project 20
and instead deploying CRAS with the existing GOOSE network approach. This alternative is not 21
recommended, as complex data volumes make it technically difficult and cumbersome to determine 22
where a failure occurs without using RGOOSE protocol. Though it was necessary to have a wide area 23
network approach to properly develop and test the CRAS platform, GOOSE is less stable than 24
RGOOSE, and its non-standard usage on the wide area network would be imprudent, given that a better 25
standardized approach is now available. SCE has seen a large increase in RAS and required arming 26
point complexity. This increased complexity drives higher data volume requirements and the need for 27
the stronger, more stable, and more precise RGOOSE communications network protocol. 28
193 Refer to WP SCE-04, Vol. 2 Bk C pp. 103-106.
136
17. Energy Management System (EMS) Refresh 1
Table V-41 Energy Management System Refresh194
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditure(Nominal $Millions)
a) Project Description 2
The Energy Management System (EMS) is operated by the Grid Control Center 3
(GCC). The GCC monitors and controls the bulk power system 24/7, using SCE’s EMS, and its output is 4
displayed on a video wall that graphically depicts the status of SCE’s and neighboring utilities’ bulk 5
power systems. 6
This project will refresh the aging hardware and software components of the EMS 7
system to maintain system availability. In addition to a technical refresh of the EMS hardware and 8
software, this project will consolidate the EMS and Phasor systems. SCE’s Phasor system collects, 9
stores, and shares Phasor Measurement Units (PMU)195 data. Consolidating EMS and Phasor onto a 10
single platform will reduce hardware and software costs and simplify the maintenance of both systems. 11
It will also provide a unified operator view of system conditions, eliminating the need for two separate 12
but related systems. 13
b) Need for Project 14
The primary driver for the EMS Refresh Project is the aging EMS system, since 15
the vendor will not support the current version after 2017. The risk of hardware failure continues to rise 16
as the system ages. By the time of the refresh, the hardware will be 7 years old. Refreshing EMS will 17
allow SCE to maintain system reliability at 99.95% or greater and reduce the risk posed by components 18
reaching the end of life. System reliability of EMS is important to SCE Grid Operations because EMS 19
194 Refer to WP SCE-04, Vol. 2 Bk C pp. 107-115. 195 A Phasor Measurement Unit is a device installed in the field that monitors & reports electrical signal on the
grid.
CIT-00-SD-PM-000231 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 6.21 7.22 2.67 - - 16.10 Previous GRC Request - - - - - - - -
Recorded Forecast
137
system failures could impair the system operators’ ability to safely operate the grid. Besides the 1
reliability needs, the current EMS configuration is also nearing its maximum data storage and processing 2
capacity, and requiring expansion. 3
(1) Benefits: 4
In addition to the operational benefits of the technical refresh described 5
above, consolidating EMS and Phasor systems will eliminate the following: 6
• The need to maintain dual systems, providing savings on Compliance 7
and Operational support activities. 8
• The need to refresh each of the EMS and Phasor systems 9
independently, resulting in a cost avoidance of $7.0 million. 10
c) Scope and Cost Forecast 11
The EMS Refresh scope will replace aging computing hardware and network 12
equipment critical to maintaining communications with field devices. It will also enhance existing test 13
environments to provide more effective testing prior to deployment. In addition, the refresh project will 14
upgrade system and application software and expand user capacity to meet projected future growth. 15
The EMS and Phasor consolidation will migrate the separate instances of the GE 16
solution to a shared infrastructure. This consolidation will substantially reduce the total number of 17
components in the system, reducing maintenance and support costs. 18
This project was launched in June 2016 and is expected to complete in mid-2018. 19
The capital forecast for this project was developed using SCE’s internal cost estimation model. This 20
model utilizes industry best practices and SCE subject matter expertise to estimate project cost 21
components. SCE’s forecast for this project includes costs for SCE employees, supplemental workers, 22
and consultants, software and vendor costs, and hardware costs. See this project’s workpaper for the cost 23
breakdown information.196 24
(1) Alternatives Considered 25
Alternative 1: SCE considered keeping the existing solution without 26
modification. We did not pursue this option as the current version of EMS will not be supported by the 27
vendor, and it does not provide on-going savings realized from consolidation of EMS and Phasor. 28
196 Refer to WP SCE-04, Vol. 2 Bk C p. 115.
138
Alternative 2: SCE considered enhancing the existing solution by 1
performing a refresh of EMS hardware and software now, but not consolidating the Phasor and EMS 2
systems. We did not pursue this option because conducting the EMS Refresh and Phasor Refresh 3
projects independently will lead to higher costs, both in terms of project and maintenance costs. This 4
alternative is estimated at $23.04 million as compared to the selected approach, estimated at $16.1 5
million. 6
Alternative 3: SCE considered enhancing the existing solution by 7
performing an EMS hardware refresh without upgrading software and not consolidating the Phasor and 8
EMS systems. We did not pursue this option for the same reason as alternative two. In addition, this 9
option would not address the risk posed by third-party-software components also nearing end of life. 10
This solution is also expected to have costs higher costs ($20.5 million) compared to the selected option 11
($16.1 million). 12
18. Outage Management System (OMS) Refresh 13
Table V-42 Outage Management System197
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditure(Nominal $Millions)
a) Project Description 14
D.15-11-021 adopted the Outage Management System (OMS) Refresh project as 15
part of SCE’s 2015 GRC. OMS is used by SCE’s Grid Operations division to monitor, identify, and 16
operate electrical network and associated assets during electrical system outages on SCE’s distribution 17
network. 18
We have made cost and schedule adjustments to the project since originally 19
proposed. The project was originally scheduled to begin in July 2014 and complete in June 2016. The 20
project start date was delayed to November 2014 and will now end in December 2017. The schedule 21
197 Refer to WP SCE-04, Vol. 2 Bk C pp. 116-123.
CIT-00-DM-DM-000050 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - 1.05 1.94 6.80 3.50 - - - 13.28 Previous GRC Request* - - - 1.60 5.10 3.10 - 9.80 *The Commission adopted SCE's request for this project in D.15-11-021.
Recorded Forecast
139
changes were necessary to accommodate an extended pre-engineering phase due to scope increase and 1
related vendor scope re-definition. 2
The addition to scope from the previous filing can be categorized into two main 3
areas aligned with the release schedule of the Project. Those two areas, further described in the scope 4
and forecast section below, are: (1) a Transmission Substation-level (Trans-Sub) Connectivity Model; 5
and (2) additional OMS enhancements, including further integration with Smart Meter functionality. 6
b) Need for Project 7
A refresh of the OMS system is required to maintain system availability, usability, 8
and reliability, as the current version will no longer be supported by the vendor. This creates a 9
significant operating risk, as SCE’s Grid Operators cannot identify and/or respond to customer outages 10
effectively if OMS is not available. This refresh will also enhance system performance and the ability to 11
report and assist in mitigating customer outages during peak system use periods. 12
(1) Benefits 13
First, refreshing the OMS system will retain the vendor support required 14
for on-going operations and help deliver more reliable system operations and performance. The OMS 15
technical refresh will also improve availability and reliability of interfaces with other key SCE operating 16
systems that enable OMS to reduce system down time and meet business objectives. 17
Second, the current system has a stand-alone Distribution circuit model, 18
and it does not show the connections between the Distribution system and the Transmission system. This 19
lack of visibility to the Transmission network results in analysts having to do time-consuming manual 20
analyses to fully determine outage impacts. The availability of the Trans-Sub Connectivity model in 21
OMS will provide direct visibility to Transmission level outages, allowing better grouping of affected 22
distribution circuits, reducing manual analysis, and help streamline dispatch decision-making. The 23
Trans-Sub Connectivity model implementation in OMS is a building block toward establishing an end-24
to-end Grid Connectivity model.198 25
Third, we have identified a set of enhancements to the current OMS 26
functionality. Implementation of automated methods to prioritize outage incidents will lead to better 27
decision-making during storm response and help reduce outage response times. In addition, the 28
198 See Grid Connectivity Model in SCE-02, Vol. 10.
140
capability to seamlessly “ping” customer meters from OMS, both individually and by groups of meters, 1
to confirm an outage or restoration in near real-time will assist in timely decision-making by Grid 2
Operations. 3
c) Scope and Cost Forecast 4
The project will be executed in a three-phase approach: (1) upgrade the current 5
application (CGI) to its latest version, (2) implement enhancements, and (3) implement a Trans-Sub 6
Connectivity model. 7
The OMS System Refresh scope as defined in our 2015 GRC testimony included 8
the implementation of the vendor’s OMS version 6.5. This will be carried out in the first phase. 9
Capabilities available in the new OMS software include a web-based version of OMS for users other 10
than system operators to reduce any performance impact of their use on the primary system. Phase 2 will 11
implement (1) enhanced graphical functions including improved circuit and substation visualization that 12
will greatly benefit the end users, (2) a dynamic scoring model for improved outage prioritization, and 13
(3) improvements through closer integration of OMS with Smart Meters. Phase 3 will implement 14
changes to OMS and its interfacing systems to deploy the Trans-Sub Connectivity Model capability. 15
The project cost proposed in our 2015 GRC application was $9.8 million. The 16
revised project cost for this rate case request is $13.3 million199 to complete application development, 17
training development and delivery, and implementation. The increase of $3.5 million is attributed to (1) 18
an increase in scope and (2) an increase in vendor cost estimates due to a three-phase implementation as 19
opposed to a higher-risk, single-phase approach as proposed in the 2015 GRC. 20
The capital forecast for this project was developed using SCE’s internal cost 21
estimation model. This model utilizes industry best practices and SCE subject matter expertise to 22
estimate project cost components. SCE’s forecast for this project includes costs for SCE employees, 23
supplemental workers, and consultants, as well as software and vendor costs. See this project’s 24
workpaper for the cost breakdown information. 25
(1) Alternatives Considered 26
Alternative No. 1: SCE considered keeping the existing solution without 27
modification. We did not pursue this option because it would lead to significant risk to (1) system 28
199 Refer to WP SCE-04, Vol. 2 Bk C pp. 123.
141
availability as the product will be unsupported by the vendor, and (2) reliability and public safety while 1
SCE operates manually during times of OMS system outage. Any associated benefits related to 2
enhancements and the Trans-Sub Connectivity model would not be realized under this alternative. 3
Alternative No. 2: SCE considered enhancing the existing solution by 4
refreshing the current application to its latest version (i.e., technology upgrade only). We did not pursue 5
this option as it would prohibit SCE from realizing previously identified benefits from storm response 6
and Smart Meter integration, and any associated benefits related to enhancements and the Trans-Sub 7
Connectivity model. 8
19. Distribution Management System (DMS) Refresh 9
Table V-43 Distribution Management System200
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 10
D.15-11-021 adopted a revised forecast for the DMS project. DMS is the 11
centralized computing system used by SCE to gather data from various distribution automation devices 12
and facilitate automated operation and control of the distribution system. DMS is an essential 13
distribution system for operating our system and maintaining its reliability. More automated equipment 14
and devices with remote-control capability are added to the distribution system every year to test current 15
conditions, isolate problems, and restore service more quickly. 16
The DMS project was proposed to be implemented in two phases. Phase 1 was 17
implemented in 2012 and provided the following benefits: 18
• A custom-built distribution management system was replaced by a COTS system, 19
which provided full vendor support for defect resolution and future upgrades; 20
200 Refer to WP SCE-04, Vol. 2 Bk C pp. 124-130.
CIT-00-SD-PM-000140 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast 7.74 6.23 5.51 5.62 1.73 5.09 1.50 - - - - 33.41 2015 GRC Authorized* 7.74 6.23 5.51 5.62 4.84 - - - 29.93 2015 GRC - Original Request 7.74 6.23 5.51 8.30 4.84 - - - 32.61 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized our 2014 request.
Recorded Forecast
142
• The ability to handle new automated devices added to the distribution network to 1
better analyze circuits, leading to better planning and remediation of system problems; and 2
• Improved capabilities of meeting security requirements for authentication of user-3
command and control data. 4
The Phase 2 schedule was revised due to technical challenges involved with 5
enhancing a COTS solution to meet operational requirements. These operational requirements were 6
associated with user interfaces, system alarm settings and notifications, and reliable and safe operation 7
of field devices. The COTS product required multiple iterations of enhancements and testing to meet 8
these operational requirements which resulted in a rescheduling of implementation and rollout activities. 9
Phase 2 was planned to be implemented in three releases: 10
• Release 1 was implemented in 2014 and deployed AVVC (Advanced 11
Voltage VAR control) capability; 12
• Release 2 was implemented in 2015 and deployed Operation Training 13
Simulation capability; and 14
• Release 3 will be implemented in 2016 and will deploy advanced 15
control and analysis capabilities. 16
Phase 2 will provide following benefits: 17
• AVVC (Advanced Voltage VAR control) is a method of Conservation 18
Voltage Reduction (CVR) where automated field capacitors on distribution circuits optimize 19
overall customer voltage and reduce energy use for distribution purposes. This advanced control 20
allows SCE to provide the required quality of service (i.e., voltage level) to its customers, but 21
with less purchased power.201 22
• An operator training-simulation feature for all switching center 23
operators, which provides a realistic behind-the-wheel training experience that tests the 24
operator’s capabilities to perform under a real-world stress situation; and 25
201 See SCE 2015 GRC, SCE-05, Vol. 2, Pt.2, Table IV-27 “Estimated DMS Avoided Purchase Power Benefits.”
143
• An advanced control and analysis capabilities allowing SCE to 1
implement self-healing security,202 which provides automated responses to system problems. 2
b) Recorded Costs and Forecast 3
SCE has recorded $31.9 million through 2015 and expects to spend an additional 4
$1.5 million to complete the project in 2016. This total project cost of $33.4 million is above the 5
Commission’s 2010 – 2015 authorized expenditures of $29.9 million. This is primarily due to additional 6
testing needed to meet operational requirements. 7
In our 2015 GRC, SCE estimated that the total cost of completing the DMS 8
project would be $32.6 million, as reflected in Table V-43. SCE is requesting expenditures in this GRC 9
to complete the project at a total cost of $33.4 million. This increase relative to our original request and 10
our authorized amount from the GRC is due to two primary factors described below. 11
First, in D.15-11-021, the Commission authorized 2013 recorded costs for this 12
project. SCE underspent in 2013 relative to our requested amount by $2.7 million, primarily due to the 13
technical challenges in aligning the commercial product to SCE’s operational needs. These challenges 14
delayed a portion of the original scope of work for 2013 to future years. As this scope of work is integral 15
to DMS performance, SCE subsequently completed this work to enable the project to provide its 16
intended benefits. Since the scope that the Commission adopted for this project remains through this 17
request, the original forecast from the 2015 GRC is still the appropriate benchmark of costs required to 18
achieve the adopted project scope. 19
Second, the complexity of the DMS project required longer periods to test, 20
stabilize, and implement the functional and technical capabilities of the system. As the DMS is 21
responsible for core aspects of our electric system operations, this extended effort was critical to our 22
ability to introduce DMS upgrades in a phased approach that could handle the large volume and 23
diversity of data, while maintaining system and operational performance standards. 24
The capital forecast for this project was developed using SCE’s internal cost 25
estimation model. This model utilizes industry best practices and SCE subject matter expertise to 26
estimate project cost components. SCE’s forecast for this project includes costs for SCE employees, 27
202 Self-healing is the ability of the system to detect that it is not operating correctly, and without human
intervention make the necessary adjustments to restore itself to normal operations. This feature will provide capability to detect faults and restore the grid to normal operations.
144
supplemental workers, and consultants, software and vendor costs, and hardware costs. See this project’s 1
workpaper for the cost breakdown information.203 2
20. Grid Interconnection Processing Tool (GIPT) 3
Table V-44 Grid Interconnection Processing Tool204
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 4
The Grid Interconnection Processing Tool (GIPT) is a suite of integrated 5
workflow and project management software tools that will allow customers to more quickly and 6
efficiently connect electrical generation and load to the grid. This project will implement a software 7
solution to handle various types of interconnection requests such as Net Energy Metering (including 8
NEM 2.0), Rule 21 (Export and Non-export), Wholesale Access Distribution Tariff (WDAT), and 9
Transmission Operated Tariff (TOT). GIPT will speed up the interconnection project lifecycle, improve 10
the customer experience, and provide SCE with more timely and accurate interconnection data. The 11
GIPT will be a suite of software applications, composed of commercially available products and custom 12
built tools integrated together by a single workflow engine. The custom built tools will be a co-13
development project between SCE and a leading software integration vendor. 14
b) Need for Project 15
Regulatory drivers,205 customer choice, and technological advancement are 16
driving increased adoption of distributed energy resources (DERs) in SCE’s service territory and the 17
resulting need to efficiently process interconnection requests. We predict the current and forecasted 18
future volume of DER interconnection requests will be constrained by SCE’s existing tools and 19
processes. Customers can submit digital interconnection requests online for Net Energy Metering and 20
Rule 21 Non-export projects. Customers submit hard copy and/or soft copy interconnection requests 21
203 Refer to WP SCE-04, Vol. 2 Bk C p. 130. 204 Refer to WP SCE-04, Vol. 2 Bk C pp. 131-133. 205 See D.16-06-052, pg. 25, 32, 37.
Recorded Forecast
CIT-00-OP-NS-000520 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 0.46 6.90 6.32 - - 13.68 Previous GRC Request - - - - - - - -
145
through multiple mechanisms for the remaining tariffs and programs (WDAT, Rule 21 export, TOT, 1
Queue Cluster, and others). Forms filled out by hand cannot be automatically validated and require 2
manual verification and processing. This creates delays, as SCE personnel must manually substantiate 3
the technical details based on information that may lack data accuracy and consistency. 4
Grid interconnection data (such as technology type, size, location, status, 5
contracts, ownership, and study reports) are stored in a variety of disparate databases. System planners 6
and other technical stakeholders manually import generation data to perform time-consuming technical 7
studies. Given there are multiple interconnection request entry points, all interconnection projects are 8
managed with independent trackers maintained by different organizational units, and do not integrate 9
with the intake pathways. 10
These multiple disparate databases/trackers often contain conflicting records and 11
pose significant challenges with system planning and operations, and publishing system information 12
through external portals in the public domain. Customer satisfaction is affected by the inability to 13
quickly present customers with potential system constraints and associated interconnection costs. 14
Various entry points for generation and load interconnection requests can pose a communication gap 15
between the organizations receiving the requests and the organizations responsible for planning the 16
system (and those creating and managing the interconnection contracts), complicating the roles and 17
responsibilities of lifecycle project management. Finally, legacy payment mechanisms (e.g., check, wire 18
transfer) combined with manual work order creation can also cause delays throughout the 19
interconnection process. 20
146
Figure V-11 NEM Application Growth
2008-2016
Note: 2016 is a forecast. Prior years are actual
As shown in Figure V-11, SCE is experiencing an increased volume in 1
interconnection requests and greater diversity in technology types being deployed. These factors further 2
increase the level of complexity required to perform system analysis. If SCE does not improve the tools 3
that are available to handle and manage these requests, there may be interconnection delays. 4
Load interconnection activity largely influences SCE’s system planning and 5
forecasting. Today, customers typically submit a new load service request or an increase of existing load 6
service in hard copy or soft copy format through local planning offices. Phone calls, meetings, and email 7
are the primary means of communicating these requests to planning organizations, which are then stored 8
in planning tools to be included in system forecasting and capacity planning. Implementing GIPT will 9
help improve data quality and interconnection work process management across organizations. 10
Impacts to Interconnection Customers: 11
Generators and demand-side resources connect to SCE’s system through many 12
tariffs and programs. Some generators interconnect by mailing paper applications, while others can 13
access online forms. While SCE’s NEM program now offers the ability to complete an interconnection 14
request online, many other Rule 21 and wholesale generators interconnecting under SCE’s WDAT 15
147
submit applications in paper or paper scanned to email. Customers often sign up for distributed energy 1
resource programs online or submit paper applications to local planning authorities. SCE staff store and 2
process these applications in various unlinked databases. Depending on how those applications are 3
processed, an SCE system engineer evaluating the impact of an interconnection on the system will not 4
have access to the most up-to-date information about other pending or recently-completed 5
interconnections. 6
GIPT will serve as a single, uniform repository for all interconnections to SCE’s 7
grid. GIPT will enable all interconnection requests, contracts, and related information to be digitized and 8
available in a common repository. 9
Impacts to SCE’s internal processes: 10
To perform accurate and efficient system planning and operations, SCE personnel 11
will need a Grid Connectivity Model (GCM), described later in this volume, which depicts an accurate 12
representation of the system. The GIPT is the GCM’s primary source of DER technical and contractual 13
information. Absent the GIPT, the GCM will contain an incomplete set of DER technical and 14
contractual information and therefore cascade inaccuracies to dependent systems. 15
(1) Benefits 16
The goals of GIPT are to improve the overall customer experience 17
throughout the interconnection project lifecycle by reducing study costs, providing a higher certainty of 18
interconnection viability, and accelerating interconnection-request turn-around times. 19
The GIPT is a single, web-based user interface that allows customers to 20
submit interconnection requests for generation, load, and combinations thereof. The system will present 21
users with help dialogues and interactive tools to enable a simple step-by-step process, offering 22
customers tailored information based on their inputs to provide accurate information in the 23
interconnection request process. In this manner, GIPT will not only guide customers through a self-24
selected tariff application, it will also provide options as to which tariff would be best suited for their 25
needs. GIPT performs fast-track or detailed analysis during the customers’ interconnection request 26
submittal process, which greatly reduces the duration for the Permission to Operate (PTO) to be issued. 27
Based on immediate feedback, customers can make informed revisions to their project configuration 28
prior to submitting an interconnection request and payment, resulting in an increased interconnection 29
success rate. Digital payments combined with work management process and account receivables 30
process integration create a seamless financial pipeline so SCE’s processing work can begin more 31
148
quickly. Transparency in lifecycle project management and communication keeps all internal and 1
external stakeholders informed throughout the interconnection process. These efforts result in further 2
improvements to system forecasting, capacity planning, operations, public data publishing, Distribution 3
Resource Plan requirements, and customer satisfaction. 4
The GIPT, when combined with the other work streams listed above, will 5
provide more accurate interconnection responses in a shorter period and reduce the growing backlog of 6
interconnection requests. Once fully implemented, the GIPT has the potential to significantly decrease 7
the cycle time, or time between application submission and PTO for DER interconnection and 8
potentially reduce project costs. The GIPT will provide the following benefits: 9
• More accurate DER information will lead to a more accurate load 10
forecast, DER forecast, and DER adoption model. This will allow for 11
more accurate capacity planning. 12
• Improved load flow analysis, system impact studies, and integration 13
capacity analysis. A more complete representation of NEM generators 14
in the Grid Connectivity Model will improve results from the System 15
Modeling Tool.206 16
• Better DER locational data. System operators will have a clearer and 17
more comprehensive understanding of where DERs are on the grid. 18
Routine switching procedures will cause more predictable changes to 19
voltage and loading. An improved ability to accurately monitor and 20
enforce contractual operating parameters of DERs will support more 21
optimal grid conditions. 22
• Publishing up-to-date integration capacity analysis results will better 23
lead customers to areas of the grid better able to support their proposed 24
DER interconnection. This improved certainty of interconnection 25
viability can be obtained with a more comprehensive source of DER 26
data. 27
206 Please refer to SCE-02, Vol. 10 for a detailed description of the System Modeling Tool.
149
• By integrating with the System Modeling Tool, GIPT will decrease the 1
time required to perform a technical evaluation, generate reports, and 2
contracts. 3
• Reduced distribution upgrade costs. When customers leverage the 4
Integration Capacity Analysis results from SCE, a higher cost certainty 5
is achievable. The Integration Capacity Analysis describes the DERs 6
the system can handle in its existing configuration, without additional 7
distribution upgrades. Comprehensive DER data provided by GIPT 8
will enhance the accuracy of Integration Capacity Analysis. 9
c) Scope and Cost Forecast 10
SCE has conducted a pilot project to test the value of automating portions of the 11
interconnection process. This pilot has proven successful in digitizing and accelerating the application 12
intake process for behind-the-meter NEM interconnections. Through the pilot, SCE has also assessed its 13
current and desired interconnection lifecycle processes. 14
The GIPT Project Plan will prioritize functionality roll-out in the following 15
manner: 16
• 2016: Prepare multiple databases and validate the information. 17
• 2017: Deploy base product, configure for NEM and Rule 21 non-export to 18
prepare to retire pilot solution. 19
• 2018: Integrate with System Modeling Tool for automated analysis; leverage 20
ICA results to streamline interconnection process; configure tool to support 21
load interconnection, WDAT, and TOT to include cluster study analysis. 22
SCE forecasts $13.68 million to complete this scope of work. The capital forecast 23
for this project includes project team costs for SCE employees, supplemental workers, and consultants, 24
software and vendor costs, and hardware costs. The project estimation is based on our internal SCE IT 25
cost estimation process. See the workpaper for more detail.207 26
207 Refer to WP SCE-04, Vol. 2 Bk C p. 133.
150
(1) Alternatives Considered 1
Alternative 1: SCE considered procuring a COTS solution for standalone 2
project management solutions and standalone workflow tools. We did not pursue this option as none of 3
the COTS products appear capable of dealing with the level of complexity, reducing the manual steps, or 4
providing the accurate information needed by SCE and customers. Most COTS products (unless 5
modified) would increase the number of manual work process steps due to the way they manage 6
workflow processes. These standalone tools are not expected to solve data integration challenges related 7
to data validation and accuracy issues because some of the third-party installers may need to request 8
additional information such as service account, address clarification, or verification of application 9
information (e.g., location, valid DER information) from customers. 10
Alternative 2: SCE also considered expanding the functionality of the 11
existing pilot application to support the full GIPT scope. We did not pursue this option as there are 12
multiple limitations of the pilot program, both in terms of functionality and in terms of technical 13
viability. After considering the dependent requirements in various other initiatives such as the System 14
Modeling Tool and the Long Term Planning tool, SCE determined this option is not viable. 15
21. Grid Analytics Applications (GAA) 16
Table V-45 Grid Analytics Applications208
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 17
The Grid Analytics Applications (GAA) will provide SCE engineers, operators, 18
and distribution grid designers with improved analytical, visualization, and decision-support capabilities 19
required to plan and operate the grid of the future. Investment in GAA is necessary to evaluate the 20
effects of higher DER penetration, accurately predict future conditions, and develop plans to better 21
208 Refer to WP SCE-04, Vol. 2 Bk C pp. 134-136.
Recorded Forecast
CIT-00-SD-PM-000247 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 3.74 12.73 0.36 0.37 0.39 17.59 Previous GRC Request - - - - - - - -
151
accommodate DERs into system operations. GAA will provide SCE personnel with the capability to 1
perform analytics on large data sources, including smart meter data, weather data, outage data, and 2
electrical network field measurement data (e.g., Supervisory Control and Data Acquisition, or SCADA, 3
data). In addition, the GAA will allow engineers to further analyze field measurements of the electrical 4
network for abnormal grid conditions and augment it with Smart Meter data to accurately predict asset 5
load profiles, which can then be used for long-term system-planning analysis. These applications also 6
allow SCE personnel to visualize the analytics on geographic information system maps and asses circuit 7
voltage degradation, line transformer utilization, and accuracy of transformer to meter relationships. 8
SCE personnel will also be able to perform more granular post-mortem analyses of outages to improve 9
current and future reliability management. For the sake of clarity, GAA will provide capabilities that are 10
separate and distinct from the CSAT project described above. Moreover, GAA will provide unique 11
analytics capabilities associated with planning and operating the distribution system, whereas the CSAT 12
project will provide unique analytics capabilities associated with the bulk power transmission system. 13
b) Need for Project 14
Implementation of Smart Meters has provided a significant opportunity to 15
leverage the historical hourly customer meter information (e.g., hourly peak demand, average voltage 16
reads, and energy demand usage) provided by these devices to better understand customer energy usage 17
patterns and overall status of the grid. This smart meter data, when integrated with electrical network 18
connectivity information and status of the grid assets, provides the most accurate information for 19
operators and engineers to make proactive and reactive decisions. This integrated data, made available 20
through GAA, will better inform SCE personnel on troubleshooting power outages and restorations and 21
power quality issues. Analytics can also be performed to better understand the asset load conditions to 22
proactively predict loss of life or catastrophic failure of assets before it occurs, enabling a proactive 23
corrective action. 24
To perform these types of analytics, key applications will be procured to enable 25
further detailed analysis of the electrical grid. A voltage analytics tool will be developed to provide a 26
more granular view into voltage characteristics across the distribution system, which will enable 27
enhanced system planning and operation of the grid. Load profile analytics using raw field measurement 28
data would be provided to system planners to develop foundational circuit load profiles. These profiles 29
can then develop system planning forecasts to analyze and compare DER-based solutions to traditional 30
wire-based solutions. In addition, load profile analytics can be leveraged to develop more realistic DER 31
152
device profiles. System planning requires comprehensive Photovoltaic (PV) profiles for determining 1
output capacity based on attributes such as local geographical solar insolation, local historical cloud 2
cover, and historical PV performance to supplement distribution system forecasts. DER profiles (from 3
GIPT data), and load profiles acquired from field measurement data, enable the ability to operate and 4
plan a dynamic distribution system capable of integrating large numbers of DERs and their impacts to 5
system planning performing a comprehensive capacity analysis. 6
The GAA enables the ability to develop analytics around assets to understand 7
characteristics such as historical load and DER adoption rates and operational asset overloads. Localized 8
load growth and DER adoption rates (using information from systems such as GIPT) can be integrated 9
to inform system planning forecasts to better understand future distribution grid needs by analyzing 10
historical installations throughout the SCE territory. In addition, during storm conditions (i.e., peak 11
conditions) grid operations could use the GAA in evaluating historical line transformer overloads, circuit 12
overloads, substation overloads, and at any aggregate level within the distribution grid using smart meter 13
data, and field measurement data to prioritize and communicate those overloads that may cause large 14
outages. 15
Today, SCE can perform grid analysis using distribution circuit and substation 16
data points to evaluate utility assets for overloading and voltage concerns for substations and circuit 17
mainlines. However, this analysis is conducted without the benefit of leveraging load and voltage 18
profiles available from customer smart meters. These raw data reads are used in system planning to 19
perform peak load forecasting for circuit and substation assets. In addition, availability of circuit and 20
substation level data points only provides a high-level overview of distribution grid peak demand, and 21
does not provide insight to individual customer’s peak demands and localized issues such as low 22
voltage. Manual labor-intensive processes are required to include customer meter information in the 23
analysis. Customer meter information only allows for a “point in time” view of customer behavior and 24
may not completely reflect the history required to both inform and proactively assess customer electrical 25
service and power quality issues. 26
(1) Benefits 27
Once the Grid Analytics Applications are functional, the following 28
benefits can be realized: 29
1) Analytics on customer meter data can be automated to perform routine 30
calculations such as voltage degradation (i.e., low/high voltage 31
153
evaluation), accurate line transformer loading, and historical outage 1
conditions. These capabilities proactively enable system planners and 2
operators to quickly evaluate portions of the grid that may be at risk of 3
exceeding acceptable operating thresholds and conditions. A system 4
planner can accurately assess the current loading of a line transformer 5
serving multiple customers (based on an automated aggregate service 6
customer meter data) and can adequately predict the effect of load and 7
generation increases on these transformers based on a new customer 8
request of installing a large motor, air conditioning unit, or DER. In 9
addition, grid operators during outages can assess the electrical service 10
status for groups of individual customer meters by evaluating meter 11
statuses through interactive GIS maps. 12
2) An analytics application using customer meter data can supplement 13
and automate missing historical circuit and substation operating data 14
points. This is required during abnormal circuit re-configurations and 15
when SCE initiated demand response events occur. The ability to 16
aggregate and blend collections of customer meter data with circuit 17
and substation load trends allows for more accurate system planning 18
during peak conditions. These newly blended historical load profiles 19
for SCE’s 4600+ circuits can then be leveraged by other tools such as 20
a Long Term Planning Tool (LTPT) to perform load and generation 21
forecasts to then evaluate mitigation for potential infrastructure 22
overloads. 23
3) The ability to develop time series profiles, such as load and DER 24
profiles, enables system planning and grid operations the insight to 25
further integrate DER installations with a future dynamic distribution 26
system. These profiles will accurately communicate how DER and 27
load demand may coincide to benefit or create issues on the 28
distribution system that will require the grid solutions to mitigate (e.g., 29
traditional capital projects). 30
154
c) Scope and Cost Forecast 1
The GAA will deliver the functional capabilities and will have the following 2
implementation plan: 3
• 2016-2019: Foundational Asset Analytics – Automated customer meter and 4
field measurement asset voltage and current analytics to inform SCE 5
distribution engineers, distribution grid designers, and grid operators of asset 6
violations. 7
• 2017-2020: Engineering Analytics – Ability to perform statistical analytics on 8
power system data (i.e., voltage deterioration and circuit load profiles) both 9
manually and as an automatic service. 10
• 2017-2018: Operational Analytics – Near real-time streaming analysis of grid 11
asset overloads and customer outage playback behavior to further enhance 12
operating the distribution grid through proactive mitigation. 13
SCE forecasts $17.59 million to complete this scope of work. The capital forecast 14
for this project includes project team costs for SCE employees, supplemental workers, and consultants, 15
software and vendor costs, and hardware costs. The project estimation is based on our internal SCE IT 16
cost estimation process. See the workpaper for more detail.209 17
(1) Alternatives Considered 18
Alternative 1: SCE considered not pursuing this project and continuing to 19
use existing manual data analysis methods for planning and operating the grid. We did not pursue this 20
option due to the vast amounts of electrical and asset data types, sources, and formats that SCE 21
maintains that require significant time and resources to access, consolidate, validate, and then analyze. 22
This approach does not allow SCE personnel to fully utilize SCE’s smart meter data to support grid 23
analytics at the customer and distribution circuit level. 24
209 Refer to WP SCE-04, Vol. 2 Bk C p. 136.
155
22. Long-Term Planning Tools 1
Table V-46 Long-Term Planning Tools210
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 2
The Long-Term Planning Tools (LTPT) is a set of software tools that will 3
facilitate integrated planning and forecasting over a five-to-ten-year horizon to identify optimal solutions 4
to system planning challenges. These challenges include poor reliability and loading conditions created 5
when installed assets are expected to exceed design limits because of forecasted changes in loading and 6
generation. Functions of the LTPT include advanced circuit and substation modelling to support 7
distributed energy resources (DER) integration, power flow and system planning analyses, calculation of 8
load blocks at circuit and substation levels, and capacity planning analyses. LTPT is not a single stand-9
alone tool; rather, it is a compilation of several software products integrated together into a single 10
platform. LTPT will replace the existing suite of outdated software tools incapable of performing the 11
analyses needed for the modern electric grid. 12
b) Need for Project 13
Due to the limited capabilities of the current analysis and planning tool set, there 14
is the potential risk of introducing errors into SCE’s long-term plans resulting in over- or under-15
estimation of load and generation capacity. SCE must forecast the load and DER growth accurately and 16
quantify and rank all elements of project risk. Therefore, SCE must develop advanced system planning 17
methodologies, which require the implementation of new tools. 18
The need to replace the current suite of tools with a more complete solution 19
includes: technical obsolescence of the current tool, inability of current tools to integrate with other SCE 20
systems, and changes in the business needs and legislation that cannot be met with the existing tools. 21
210 Refer to WP SCE-04, Vol. 2 Bk C pp. 137-139.
156
Obsolescence 1
The current tools are approaching the end of their lives and are degrading rapidly 2
in terms of usefulness and maintainability. Current issues include: 3
• Difficulty to maintain. The current tools were built in-house during the last 4
decade using a complex architecture of custom-built code and business logic. 5
Over time, it is becoming increasingly difficult to maintain the skill sets 6
required to support the ongoing operation of this aging technology. 7
• Increased user base. The current tools were designed and built specifically for 8
the distribution circuit and substation projects. However, as SCE realized 9
operational efficiencies in integrating the work groups (e.g., system planning, 10
district (load growth) planning, programmatic projects, project management), 11
the number of users has increased by over 50 times. The underlying 12
architecture of the tool cannot handle this many users, which is impacting 13
system performance. 14
• Inability to scale. Besides the dramatic increase in users, the current system’s 15
database has grown to almost five times its original design limit. This is a 16
major architecture problem as the current tool maintains all projects (and their 17
alternatives) indefinitely. As the user base and number of projects expanded, 18
the database grew exponentially. 19
Inability to integrate with other systems 20
As SCE moves to an integrated approach to grid planning, our systems and 21
software need to interoperate with each other seamlessly. Examples include a unified analysis of 22
transmission, sub-transmission, and distribution needs from a “bottoms-up” and “top-down” analysis. 23
Functions such as risk-based analysis, protection engineering, and reliability studies need to be 24
integrated with the other analyses. The current tool lacks the ability to integrate with other important 25
systems (e.g., CYME211, PSLF212) and cannot be made to integrate without completely rebuilding much 26
of the architecture or using extensive manual processes. 27
211 The CYME software package, acquired in 2008, is used to calculate power flow and short-circuit duty for actual or simulated configurations and generation/loading conditions on distribution circuits.
157
• Sub-transmission and transmission analyses and project simulations are 1
performed independently using a separate set of software tools. In the LTPT 2
all will be performed in one integrated platform where manual re-entry is 3
eliminated and personnel can perform multiple iterations for optimizing their 4
solutions. 5
• The current tool does not have the capability of fully evaluating and 6
quantifying risks associated to each condition or proposed project. Since it 7
cannot integrate with the risk analysis tools, this function must be done 8
outside of the tool and the results analyzed independently. This lack of 9
integration creates inefficiencies as proposed projects must have their risk 10
assessments performed outside the tool and subsequently have the results 11
brought back into the tool. This problem is magnified by the iterative 12
approach required in analyzing risk for a large portfolio of projects. 13
• The current tool cannot integrate with other project management and financial 14
systems. 15
Changes in business needs. The increasing complexity of the electric grid 16
demands new cutting-edge methodologies to identify capital projects. The current tool lacks several 17
major required functions, including: 18
• The growth of DERs on our system will require us to evaluate the grid based 19
on load profiles as opposed to a single peak load value as done today. The 20
current tool is incapable of providing this functionality. 21
• There are multiple load forecasts used for different purposes at SCE. 22
Historically, these forecasts include a distribution forecast, and transmission 23
forecast, and a system-level forecast. These forecast are primarily developed 24
independently by different organizations and are not aligned. SCE must align 25
these forecasts to understand the implications that DERs may have on the 26
Continued from the previous page 212 Positive Sequence Load Flow (PSLF) is a software package that is used to calculate power flows on transmission and subtransmission systems.
158
distribution and bulk electric systems. To coordinate planning and 1
identification of optimal grid solutions and to align with the vision in the 2
Distribution Resources Plan (DRP), SCE must utilize a load and DER forecast 3
coordinated with the CEC Integrated Energy Policy Report (IEPR) and across 4
all internal planning organizations. 5
• The current planning tool performs system analysis assuming one-way power 6
flow. In this manner it can forecast potential problems such as circuit and 7
substation overloads based on loading limits and forecasted demand. The 8
current tool cannot perform analyses resulting from bi-directional flow213 and 9
cannot evaluate overloads at a more granular level such as a line segment. 10
• The current tool cannot evaluate the risk types associated with identifying 11
optimal grid projects/solutions. To enable a process for managing the elements 12
of the risks identified, LTPT will facilitate the concurrent evaluation of 13
multiple types of projects for risk mitigation, including utilizing DERs to defer 14
or replace traditional capital upgrades. System Planners can evaluate the 15
project driver and evaluate multiple project alternatives. System Planners can 16
better determine the value that DER can add to the distribution and 17
transmission systems by creating load profiles for DERs as well as customer 18
demand and evaluating the profiles over an annual period. 19
In conclusion, a profile-based, coordinated load and DER forecast will enable 20
SCE to determine where affected areas will be. A risk-based, coordinated planning process will enable 21
SCE to determine risks on the distribution grid. A methodology to develop optimal grid solutions to 22
address the identified risks will enable SCE to fully utilize DERs while developing a safe, reliable and 23
affordable future distribution grid. 24
213 Bi-directional power flow: Historically, power has flowed from the source (transmission system) to the load
(customers). In the present system, the high penetration of distributed generation can cause power to flow in the “reverse” direction, from the customer to upstream portions of the system. The existence of power flow in both directions is referred to as “two-way” or “bi-directional” power flow.
159
(1) Benefits 1
The LTPT will provide better response to customer requests, with the 2
GIPT discussed previously in this volume, allowing customers to get faster, more accurate answers, 3
reducing power quality issues, and increasing the ability to use the full hosting capacity of the 4
distribution system. The LTPT will also integrate with the Work Management’s Portfolio Management 5
project, described previously, for effective bundling of work to be performed in the field. This will allow 6
SCE to better utilize its field resources while maintaining reliable grid system. 7
i. Forecasting 8
The LTPT will increase the accuracy of SCE’s Load and DER 9
Forecast by incorporating load profiles versus a single point representation of load and generation. As 10
described above, this may cause better identifying mitigations, and may cause a reduction of the capital 11
upgrades identified using a point based forecast. 12
ii. System Analysis 13
The LTPT will expand SCE’s capability to perform analysis to 14
allow SCE engineers to proactively address forecasted grid concerns. This may cause reduction in 15
customer minutes of interruption (CMI). 16
The LTPT will allow planners to address multiple risks within a 17
project. This may prevent multiple projects in the same geographic area, which may cause fewer 18
customer outages, less cost due to coordinating resources, less cost due to avoiding duplicative scope. 19
The LTPT will automatically determine circuit reconfigurations 20
resulting in a time savings to distribution planners. 21
iii. Optimal Grid Solutions 22
The LTPT will allow planners to create optimal grid solutions. For 23
example, the LTPT may recommend the full utilization of available capacity, or the utilization of DER 24
to defer grid upgrades. 25
c) Scope and Cost Forecast 26
The Long Term Planning tool(s) will have the following implementation plan, 27
which includes the following functional capabilities to be provided to the Electric System Planning, 28
Long-Term Demand Forecasting, Strategy, Integrated Planning and Performance, and Grid Operations 29
departments: 30
160
• 2016-2017: Load and DER profile-based Forecasting, Aggregation, 1
Distribution System Analysis, Capacity Analysis, Protection Analysis, Basic 2
Distribution Reliability Analysis, Optimal Grid Solutions, Basic Risk Model, 3
Basic Geographical Modeling, and Basic Integration with internal tools, Basic 4
Real Time Analysis. 5
• 2018-2019: Sub Transmission Analysis, Transmission Analysis, Advanced 6
Distribution System Analysis, Advanced Risk Model, Advance Geographical 7
Modeling, Full Integration with internal tools, Full Integration with external 8
tools. 9
The LTPT must be fully integrated with the planning tools and operational tools 10
described in this GRC, including the Grid Analytics Application, Grid Connectivity Model, System 11
Modeling Tool,214 Grid Interconnection Processing Tool, and the Grid Management System.215 The tool 12
must be integrated with the Grid Connectivity Model, the Grid Interconnection Planning Tool, and the 13
System Modeling Tool to minimize the amount of rework that must be done for each project and reduce 14
the amount of field data collection. The LTPT supports adding extensions into the Grid Connectivity 15
Model, again reducing the manual rework required and helping to keep the connectivity model accurate. 16
The LTPT solution will allow SCE to develop a load, DER and generation 17
forecast, perform system analysis to identify all grid risks, and determine optimal solutions to reliability 18
risks for the entire power grid from the lower end of distribution to the Bulk Transmission system. This 19
allows SCE to fully evaluate risks and opportunities related to existing and forecasted DERs in its long-20
term system planning process. The Long Term Planning Tool will allow SCE to develop and continue to 21
shape a safer, more reliable and affordable grid while encouraging DERs and enabling customer choice. 22
SCE forecasts $15.07 million to complete this scope of work. The capital forecast 23
for this project includes project team costs for SCE employees, supplemental workers, and consultants, 24
software and vendor costs, and hardware costs. The project estimation is based on our internal SCE IT 25
cost estimation process. See the workpaper for more detail.216 26
214 Please refer to SCE-02, Vol. 10 for a detailed description of the System Modeling Tool. 215 Please refer to SCE-02, Vol. 10 for a detailed description of the Grid Management System. 216 Refer to WP SCE-04, Vol. 2 Bk C p. 139.
161
(1) Alternatives Considered 1
Alternative 1: SCE considered pursuing the enhancement of an existing 2
distribution system planning COTS solution to meet LTPT functional requirements. We did not pursue 3
this option because no product in the marketplace possesses the full suite of capabilities required by 4
LTPT (i.e., capacity planning, project alternatives analysis for DER versus wires-based solutions, risk 5
analysis). Due to the level of complexity required to fully implement the LTPT solution, a complete re-6
design of both the technology and planning methodologies would be necessary to fully address 7
enterprise scalability and utility system planning requirements. Based on the large amount of 8
customization expected, SCE does not recommend this solution approach. 9
Alternative 2: SCE considered modifying the existing planning tool 10
currently used by SCE and enhance it to meet the new requirements. We did not pursue this option 11
because the current software solution cannot meet the existing system planning requirements and has 12
numerous performance issues. This option would require SCE to locate and identify qualified resources 13
and onboard them, as the technology is seldom used in practice in today’s industry technology 14
landscape. Based on our research, the LTPT project as presented here is the most responsive option that 15
SCE has found. 16
23. Grid Connectivity Model 17
Table V-47 Grid Connectivity Model217
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 18
The Grid Connectivity Model (GCM) is a software model of SCE’s complete 19
electrical grid. This model replaces existing disparate and disconnected models and will serve as the 20
single, centralized source of connectivity information for all assets—from bulk generation down to the 21
217 Refer to WP SCE-04, Vol. 2 Bk C pp. 140-142.
Recorded Forecast
CIT-00-OP-NS-000521 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 3.44 5.02 5.05 1.43 - 14.95 Previous GRC Request - - - - - - - -
162
end-consumer meters. This new software model will support other enterprise tools that require the use of 1
SCE’s electric system connectivity information and the operational configuration of devices.218 To 2
provide up-to-date information about the grid, this connectivity model will receive near real-time 3
information about device settings219 through multiple operational systems, such as the Distribution 4
Management System (DMS). 5
b) Need for Project 6
SCE’s current grid connectivity information is stored across different software 7
solutions, including separate electrical-based and structural-based solutions. Business processes around 8
these datasets vary across the owners of data, and automated synchronization of datasets is limited. In 9
the current state of the network model, gaps exist in three broad areas: 10
• No existing systems of record, 11
• Lack of information and inconsistent data, and 12
• Business process gaps in maintaining and synchronizing information across 13
systems. 14
The GCM represents the network of electrical devices and structural components 15
connected by conductors, in terms of nodes and links. The model considers various device statuses and 16
the electrical characteristics to represent the power flow network.220 The model will maintain various 17
perspectives of the grid, such as the as-built configuration, which represents the grid in its permanent 18
static configuration, and the as-operated configuration, which accounts for operational deviations from 19
the as-built permanent configuration of the circuit due to abnormal conditions or switching. Other 20
perspectives include historical as-operated views, and the as-planned view, which depicts the long-term 21
configuration of the distribution systems. 22
Grid connectivity information is currently scattered across multiple systems such 23
that no unified end-to-end connectivity model exists. SCE’s mapping organization uses a legacy system 24
for maintaining the electrical connectivity model for the schematic view of the distribution circuits. The 25
218 The GCM will support many tools and systems, including: planning tools (like CYME), analytical systems
(like Hadoop), and operational systems, such as the Outage Management System (OMS). 219 Examples of these device settings include capacitor bank settings and auto-recloser settings. 220 A distribution circuit model is composed of the following: connectivity model, asset data (nameplates,
ratings), device settings (capacitor bank operating limits, circuit breaker trip settings, automatic recloser trip settings), and planning and operation data (such as forecast electrical models).
163
transmission group uses the Energy Management System (EMS) system for static and dynamic views of 1
the transmission connectivity model. Grid operators use the distribution connectivity model in the 2
Outage Management System (OMS) system for managing outages and determining the extent of an 3
outage. Customer Service uses pieces of distribution secondary circuit connectivity information in 4
Customer Service System (CSS) and information from OMS for identifying the list of customers 5
impacted by an outage. The above business functions and roles will benefit from having a centralized 6
and unified grid connectivity model. 7
Engineering, design, construction, operations, maintenance, and customer service 8
departments rely on various network models manually created by various users for different analysis 9
required to operate the electric system. It is becoming more resource-intensive for users of network 10
models to manually gather data from different sources and analyze the distribution system. Today, the 11
attributes of various distribution assets and the grid require significant manual validation to verify that 12
the electric system models represent actual field conditions. There is no automated and reliable way of 13
inputting the actual device settings from the field into the connectivity model. These device settings are 14
important in simulations because the results should represent what is achievable in the electric system. 15
Verifying this information for one distribution circuit takes an engineer roughly five hours to complete. 16
It would be labor intensive and expensive for SCE to manually verify and model each of its distribution 17
circuits for the entire territory. The GCM will be integrated with operational systems like the 18
Distribution Management System (DMS) so near real-time data from DMS will provide actual device 19
settings information to the Grid Connectivity Model. 20
(1) Benefits 21
This project will provide several benefits by addressing the gaps in the 22
current state of the network model. It will eliminate the need to manually update various connectivity 23
models by interfacing directly with operational systems for device setting information. This results in 24
consistent connectivity data across systems and user groups. The GCM will also contain information 25
about distributed energy resources (DERs), collected from the customer-facing Grid Interconnection 26
Processing Tool (GIPT). Both grid-connected and behind-the-meter DERs will be modeled with their 27
interconnection information (e.g., generating capacity, technology, contractual parameters) in the overall 28
connectivity model. 29
The GCM will provide the system of record for reliable, up-to-date, and 30
end-to-end connectivity model of the entire SCE grid such that various stakeholders (including system 31
164
planners, distribution engineers, and system operators) will have consistent information. The model will 1
be the system of record for all the connectivity information and allow users to effectively and reliably 2
manage the grid across planning, analysis, and operations. The project will leverage the existing GIS 3
system to provide an end-to-end view of the grid: from transmission to the distribution secondary. 4
Two models will be included as part of the GCM: the electrical 5
connectivity model and a structural connectivity model. The electrical connectivity model will provide 6
various business capabilities including tracing of the circuits, simulation capabilities, topology of the 7
entire grid, end-to-end connectivity information, schematic diagram generation capability for circuits, 8
geographic view of the grid, etc. The structural connectivity model represents various structures located 9
overhead (e.g., poles, towers, vaults) and underground (e.g., ducts and trenches). The structural 10
connectivity model will include information like the length of the conductor, impedance calculation of 11
segments, capacity of the underground ducts, location of electrical assets for maintenance, etc. The 12
structural connectivity model will complement the electrical connectivity model and enable various 13
electrical planning and analysis capabilities. 14
The Grid Connectivity Model will provide various perspectives of the 15
grid: as-planned, as-designed, as-built and as-operated. The as-planned perspective will provide the 16
long-term (5 to 10 years) view of potential future system configurations. The as-designed perspective 17
will provide the approved construction designs of the grid. The as-built perspective will be the base 18
foundational model that will provide a view of the grid, as it is constructed in the field. The as-operated 19
perspective will provide the current configuration of the grid. 20
The GCM will be the system of record for providing complete, correct, 21
and current information of the electrical and structural network connectivity, therefore enabling users to 22
perform more efficient, intelligent system operations. The Grid Connectivity Model can represent the 23
network connectivity in multiple perspectives, such as the as-built perspective and the as-operated 24
perspective. By looking at multiple system perspectives, grid operators can decide that are more 25
efficient. 26
Grid Connectivity Model will provide the connectivity information to the 27
rest of the enterprise using industry standards like Common Information Model (CIM). This will 28
promote the reuse of interfaces for GCM across the enterprise and eliminate the need for developing 29
custom interfaces for each new system. Web-based user interfaces will be provided to enable users to 30
165
view the schematic diagrams, perform tracing functions, perform simulations, and view the geographic 1
location of circuits. This information will be easily accessible to multiple user groups. 2
c) Scope and Cost Forecast 3
Phase I Scope 2016: 4
Phase I of the project will focus on addressing the needs of the Integration 5
Capacity Analysis (ICA), where the Grid Connectivity Model will provide connectivity information and 6
load information for performing ICA on distribution circuits as mandated by the CPUC. This will 7
include providing the as-built connectivity information and field device setting information for capacitor 8
banks and automatic re-closers from DMS. Load information will include monthly load profiles from 9
AMI data, DER generation data and circuit SCADA data for demand information. 10
Phase II Scope 2017: 11
Phase II of the project will focus on developing the substations’ internal 12
connectivity model and the foundational framework for the end-state design of the solution. We will 13
develop basic functionality for tracing and initial version of core engine for the Grid Connectivity 14
Model. This phase will also provide CIM-based connectivity files for the usage across SCE. For the sake 15
of clarity, the GCM will leverage the foundational work on the Trans-Sub Connectivity model 16
performed under the OMS Refresh project discussed above. 17
The scope will also include supporting various Grid Connectivity-related use 18
cases for outage management enhancement work, System Modeling Tool and ICA phase 2, Long Term 19
Planning Tool and the GIPT. 20
Phase III Scope 2018: 21
Phase III will include modeling the mesh networks and micro grids. It will 22
address the gaps in the secondary circuit connectivity information for transformer-to-meter connectivity. 23
A validation engine will be developed for verifying the connectivity model 24
changes. Phase III will also provide standards-based data services for the connectivity information to be 25
used by SCE. It will also provide schematic generation capabilities to address the requirements for as-26
operated, as-planned and as-designed perspectives. 27
SCE forecasts $14.95 million to complete this scope of work. The capital forecast 28
for this project includes project team costs for SCE employees, supplemental workers, and consultants, 29
166
software and vendor costs, and hardware costs. The project estimation is based on our internal SCE IT 1
cost estimation process. See the workpaper for more detail.221 2
(1) Alternatives Considered 3
Alternative 1: SCE considered procuring a COTS solution to meet the 4
needs of the GCM project. We did not pursue this option because there is no commercially-available 5
product available that can address the needs of Grid Connectivity Model comprehensively. The market 6
is not mature enough to provide a commercial product that can cover multiple perspectives of the 7
network. Various commercial products are specialized for a specific segment of the grid with a purpose. 8
This approach does not meet the requirement of a unified model where different users can benefit from 9
the information beyond their organizational or functional boundaries. 10
Alternative 2: SCE considered not pursuing this project and continuing to 11
use existing grid connectivity information stored across different software solutions. We did not pursue 12
this option because it will not address the various business capabilities that SCE planners and grid 13
operators need in response to the more complex grid and increasing number of DERs on the grid. This 14
approach will require significant time and resources to manually collect, consolidate, and validate the 15
combination of electrical and structural connectivity information before it can be used to support grid 16
planning and operations. 17
221 Refer to WP SCE-04, Vol. 2 Bk C p. 142.
167
24. Transmission and Distribution Projects less than $3 Million 1
Table V-48 Transmission & Distribution Projects less than $3M222
Work Breakdown Structure (WBS) Forecast Capital Expenditures (Nominal $ Millions)
Table V-48 above lists capitalized software projects with total project costs less than 2
three million dollars in capital funding and will start and finish within the years 2016 to 2020. Please see 3
the workpapers for these projects for detailed information on each project and its associated costs. 4
222 Refer to WP SCE-04, Vol. 2 Bk C pp. 143-192.
WBS Project Description 2016 2017 2018 2019 2020 TotalCIT-00-SD-PM-000187 Substation Health Assessment Tool (previously As - - 2.60 - - 2.60 CIT-00-SD-PM-000206 Substation 3D Design - - 1.26 0.72 - 1.98 CIT-00-SD-PM-000233 Electronic WO Package Automation (e-WOP) - - - - 1.00 1.00 CIT-00-SD-PM-000234 Fast Response Energy Storage 0.63 - - - - 0.63 CIT-00-DM-DM-000076 Transient Devices (HW for TSFT) 1.20 0.33 - - - 1.53 CIT-00-DM-DM-000083 High-Z Impedance Fault Detection - - - 1.00 1.00 2.00 CIT-00-DM-DM-000084 Secure DNP Ver5 Support for EMS - - - 1.00 - 1.00 CIT-00-DM-DM-000086 Grid Management Dashboards - 2.00 - - - 2.00 CIT-00-DM-DM-000090 PSMP 2.0 - - 1.00 1.00 - 2.00
Total 1.83 2.33 4.86 3.72 2.00 14.74
168
C. Power Supply Software Projects 223 1
1. Generation Automation Upgrade & Control Systems Refresh 2
Table V-49 Generation Automation Upgrade & Control Systems Refresh224
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 3
SCE owns and operates 33 hydroelectric (Hydro) generating power plants, which 4
can be divided into two groupings, Big Creek (formally known as Northern) and all Other (formally 5
known as Eastern) assets. Big Creek assets are the larger group, encompassing all SCE Hydro facilities 6
in the upper San Joaquin River watershed. These assets are in the western Sierra Nevada Mountains, 7
across an area centered approximately 50 miles northeast of Fresno. SCE’s Other Hydro assets comprise 8
the remaining facilities and are in the Bishop and Mono Basin areas of the eastern Sierra Nevada 9
Mountains, the Kern River, Kaweah, and Tule River areas in the southern Sierra Nevada Mountains, and 10
the Ontario, San Bernardino, and Banning areas in the San Bernardino and San Gabriel Mountains, 11
located in Southern California. Each power plant is equipped with various control systems to perform 12
and supervise the safe operation of the plant, and capture various data about the operation of the plant. 13
The control systems at the Hydro facilities have historically been sourced from 14
multiple vendors, and the lack of equipment standardization has caused problems with inter-operability 15
between the different Hydro operating regions and maintainability as components approach the end of 16
their life-cycle. SCE’s plans to standardize future plant control system equipment and software from a 17
single vendor. We have begun this process already. Standardization will eliminate the need to store parts 18
from multiple vendors and will improve communications between equipment. 19
223 Refer to WP SCE-04, Vol. 2 Bk C pp. 193-195. 224 Refer to WP SCE-04, Vol. 2 Bk C pp. 196-199.
CIT-00-SD-PM-000227 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - 1.00 3.00 2.00 1.50 2.50 10.00 Previous GRC Request - - - - - - - -
Recorded Forecast
169
b) Need for Project 1
Replacing aged equipment must keep pace with current and future demands of the 2
system. Following replacement, the new automation system will provide vital information required for 3
operations, work management planning, preventative maintenance, regulatory compliance, and other 4
needs that cannot be conveyed with the existing communication system. This data can be captured, 5
transmitted, and retained with the planned automation upgrades. 6
(1) Benefits 7
Performing this Automation upgrade will add additional business 8
capabilities to SCE’s Hydro facilities, which include: 9
1. Enhancing the responsiveness of the Big Creek plant to CAISO control 10
signals by moving existing custom-developed code into the control systems. This will enable Big Creek 11
to offer more capacity into the CAISO regulation market, which will benefit the integration of renewable 12
generation into SCE’s grid; 13
2. Upgrading the control systems at Big Creek to avoid generator rough 14
loading zones (i.e., output ranges where the mechanical construction of the generating unit causes 15
excessive vibration); and 16
3. Providing a common test environment and enabling the interoperability 17
of the control systems in the different hydro operating regions to facilitate data exchange and 18
preparedness for enhanced disaster recovery capabilities between the regions. 19
Upgrades to the existing Automation system will enable Hydro to meet 20
mandates from FERC and other governmental entities to monitor water release capabilities and also 21
maintain dam surveillance. The overall benefit will be increased water control, system reliability, and 22
regulatory compliance. The Dam sites are important because they are points of mandated FERC 23
Instream Releases critical to FERC license compliance. 24
c) Project Scope and Forecast 25
This project continues the Automation standardization efforts, which include 26
implementing enhanced business capabilities and replacing existing automation systems as they become 27
obsolete. The Automation project will deploy advanced data archiving, monitoring, reporting, and 28
remote diagnostics with secure, NERC Critical Infrastructure Protection (CIP) compliant software and 29
servers for viewing of Ovation screens, data trending, and reporting on the administration network. This 30
project is driven by three business needs: (1) standardizing equipment, (2) replacing aged equipment in 31
170
use, and (3) adding new equipment to those areas not automated, which will allow for remote 1
monitoring and control of Hydro facilities. 2
The forecast project costs are $10.0 million.225 The capital forecast for this project 3
was developed using SCE’s internal cost estimation model. This model utilizes industry best practices 4
and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this project 5
includes costs for SCE employees, supplemental workers, and consultants, software and vendor costs, 6
and hardware costs. See this project’s workpaper for the cost breakdown information. 7
(1) Alternatives Considered 8
Alternative 1: SCE considered not pursuing this project. However, if we 9
chose not to pursue this project, then we would have to keep the existing equipment in-service. This is 10
not a long-term viable option as software upgrades for the existing equipment are no longer being 11
supported by the vendor and will soon cease to meet current NERC CIP requirements. 12
Alternative 2: SCE considered enhancing the existing solution by 13
upgrading the equipment using the existing vendor to meet the needs of the generation business. We did 14
not pursue this option because it would not meet our needs of providing standardization across SCE’s 15
generation assets while meeting NERC CIP requirements. 16
2. Dam Monitoring and Surveillance 17
Table V-50 Dam Monitoring and Surveillance226
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 18
SCE Hydro operates and maintains 22 dams considered by FERC to be large 19
and/or high-hazard dams, and many are in mountainous terrain at elevations over 7,000 feet above sea 20
225 Refer to WP SCE-04, Vol. 2 Bk C p. 199. 226 Refer to WP SCE-04, Vol. 2 Bk C pp. 200-203.
CIT-00-SD-PM-000228 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 1.00 2.00 1.50 1.00 5.50 Previous GRC Request - - - - - - - -
Recorded Forecast
171
level. These locations are often remote and difficult places to work. SCE’s dams are under FERC 1
jurisdiction, and FERC has an ongoing effort to increase the safety of dams across the U.S. by 2
improving the monitoring of the dams and by improving the dams’ ability to withstand natural disasters, 3
including seismic events (earthquakes) and severe storms. 4
Several of our dams are both high-risk and time-sensitive. Time-sensitive means 5
there is not enough time to notify emergency services in case of a dam failure with sufficient lead time 6
to evacuate downstream towns. SCE monitors its dams through remote sensors that measure water levels 7
and water flows. When a sensor returns an unexpected or abnormal reading (which frequently occurs), 8
an operator must travel to the dam to perform a visual inspect. Often the travel time required can exceed 9
two hours, which in an emergency condition is not adequate. 10
This project captures the IT components of the Dam Monitoring and Surveillance 11
(e.g., cameras and communications equipment) Project. Supporting testimony for SCE’s risk analysis, 12
construction of field towers, permitting, and related building materials is captured within the SCE-05, 13
Volume 3 - Hydro O&M and Capital testimony. Two site surveys performed in 2015 assessed 14
equipment needs and were utilized to develop the initial project cost estimates. The total capital forecast 15
for the project described in the SCE-05, Volume 3 is $7.90 million, and the IT portion of this project is 16
$5.50 million for 2016-2020. 17
b) Project Need 18
SCE’s dams are under FERC jurisdiction, and FERC has an ongoing effort to 19
increase the safety of dams across the U.S. through improved monitoring and improvements to a dam’s 20
ability to withstand natural disasters, including seismic events (earthquakes) and severe storms. 21
(1) Benefits 22
The primary benefit for this project is improved response time to a 23
pending or developing dam failure by adding visual surveillance capabilities to high-risk dams, and to 24
remove the need to send operations personnel to perform visual inspections when abnormal sensor 25
readings are received. 26
c) Project Scope and Forecast 27
The project scope at each dam will be similar and includes the purchase and 28
installation of cameras and communications infrastructure. Cameras require the ability to capture 29
periodic still images (e.g., 5 frames per minute) and remotely capture a still image on demand 30
(sometimes infrared vision will be required to capture images at night or in snowy conditions). Cameras 31
172
and communications equipment must remain operable in remote areas and under harsh weather 1
conditions. In addition, not all dams have electric power and necessary communications infrastructure so 2
this will be provided via solar and/or wind generation and satellite and/or microwave communications. 3
Installation will occur according to the risk priority assigned to each dam and some installations may 4
span multiple years (i.e., project may commence in the fall of the first year and conclude in the second). 5
The actual cost per dam will vary across different dams depending on dam size 6
(larger dams will require more cameras, more cabling and/or wireless communication links between the 7
cameras and satellite dish), communications availability (some dams have existing microwave 8
communications equipment with capacity, other dams have no communications infrastructure and 9
require a satellite link due to their remote location), power availability (some dams have electric power 10
at or near the dam, other dams have no power infrastructure and will need to be powered via solar and/or 11
wind power generators with batteries for energy storage). 12
The capital forecast for this project was developed using SCE’s internal cost 13
estimation model. This model utilizes industry best practices and SCE subject matter expertise to 14
estimate project cost components. SCE’s forecast for this project includes costs for SCE employees, 15
supplemental workers, and consultants, software and vendor costs, and hardware costs. See this project’s 16
workpaper for the cost breakdown information.227 17
(1) Alternatives Considered 18
Alternative 1: SCE considered installation of fixed power and fiber-optic 19
communications links to each dam. Initial cost estimates performed in 2012 showed that the fixed power 20
and fiber-optics communications link would not be a lower-cost option. Communications equipment 21
costs were $2.4 million for these seven dams; assuming similar complexity for the other dams, the total 22
project cost estimate was approximately $8.2 million. 23
Alternative 2: SCE considered the use of remote-control unmanned 24
aircraft (i.e., drones) fitted with cameras to monitor the dams. Use of drones was deemed to not be a 25
viable option due to it being an unproven technology for this application. Currently, FAA regulations 26
require drones to be flown only within line-of-sight, which limits their usefulness in remote areas. In 27
addition, a drone may not be able to fly (or survive the flight without crashing) in harsh weather 28
227 Refer to WP SCE-04, Vol. 2 Bk C p. 203.
173
conditions, including heavy rain, cold, and snow. It is also unclear if commercially available drones 1
exist that could satisfy the business needs listed above (and public safety requirements). In addition, 2
some of the SCE dams are located on land managed by the U.S. Forest Service, and the regulatory 3
environment for drones on federally managed lands is evolving. For these reasons, this alternative was 4
not pursued. 5
3. CAISO Market Enhancements Program (IMEP) 6
Table V-51 CAISO Market Enhancement Program (IMEP) 228
Work Breakdown Structure (WBS) Authorized Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 7
D.15-11-021 adopted the CAISO Market Enhancement program as part of SCE’s 8
2015 GRC. For 2013 and 2014 this project was funded through the MRTU Memorandum Account and 9
not through the GRC. In 2015 the program was funded through the GRC. This program was previously 10
presented as two projects in SCE’s 2015 GRC application: (1) CAISO Market Enhancements Project 11
(2013); and, (2) CAISO Market Enhancements Project (2014 – 2017).229 Each CAISO release 12
implements several market enhancements. 13
CAISO conducts an annual process to identify and prioritize future market 14
initiatives. Once identified, each of the initiatives is ranked based on two criteria: (1) benefits, including 15
grid reliability, overall market efficiency, and desire by market participants; and (2) feasibility, including 16
market participants’ implementation impact (costs and resources) and CAISO implementation impact 17
(costs and resources). 18
228 Refer to WP SCE-04, Vol. 2 Bk C pp. 204-211. 229 Please see SCE’s 2015 GRC Application – SCE-05, Volume 2, Part 1, pp. 189-200.
Various WBS IDs 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast - 1.42 1.60 2.16 2.16 3.95 4.00 4.00 4.00 4.00 Previous GRC Request* - - 5.30 6.79 7.00 7.00 6.00 *The Commission adopted SCE's 2013 recorded costs and 2014-2015 request for this project in D.15-11-021.
Various WBS IDs include: CIT-00-SD-PM-000225, CIT-00-SD-PM-000162, CIT-00-SD-PM-000069, CIT-00-SD-PM-000164, CIT-00-SD-PM-000212
Recorded Forecast
174
The CAISO Stakeholder Initiatives Catalog230 outlines the priority issues 1
identified by the CAISO and stakeholders that may require enhancements to the energy market. This list 2
is used by SCE to identify market initiatives that have a high likelihood of being addressed in the 2016-3
2020 timeframe. Although the exact timelines and requirements are still being defined by CAISO, SCE 4
must be able to respond and implement required system enhancements once these market initiatives are 5
mandated by CAISO. 6
Based on the catalog, CAISO plans to make significant enhancements to their 7
Day Ahead and Real Time Markets during 2016-2020 timeframe. CAISO is also planning to enhance 8
some of their major processes including Residual Unit Commitment, Ancillary Services procurement, 9
Convergence (Virtual) Bidding, Congestion Revenue Rights procurement, and Resource Adequacy. 10
Other initiatives on the roadmap will enhance the Capacity Procurement 11
Mechanism, implement the Reliability Services initiative, replace the existing Operational Meter 12
Analysis and Reporting (OMAR) meter data management system, and enhance the way CAISO handles 13
Multi-Stage Generators. The CAISO multi-state Energy Imbalance Market (EIM) will be enhanced, and 14
new participants are expected to join the EIM market. 15
The objective of the CAISO Market Enhancements Program is to enhance 16
existing systems and processes to implement CAISO market initiatives to meet SCE’s CAISO 17
Scheduling Coordinator obligations. This implements ongoing CAISO market changes that happen 18
every year. 19
b) Need for Project 20
SCE is a CAISO participant and subject to the CAISO’s tariff and operating 21
procedures. Therefore, SCE must update existing systems to implement CAISO market changes to 22
remain compliant with CAISO tariffs and participate in the CAISO market. 23
(1) Benefits 24
This project will support compliance with the CAISO tariff provisions. 25
c) Scope and Cost Forecast 26
In 2015, SCE recorded lower expenditures than authorized primarily because 27
CAISO requirements changed between our GRC application and when the project was implemented. 28
230 Catalog available at
http://www.caiso.com/Documents/Final_2016StakeholderInitiativesCatalog_Roadmap.pdf.
175
These changes resulted in a lower-than-expected project complexity. The project spend varies year by 1
year depending on the size and complexity of the CAISO changes implemented. 2
Since implementing the CAISO Market Redesign and Technology Upgrade 3
(MRTU) project in 2009, the CAISO energy market continues to evolve. SCE will need to modify its 4
existing systems to handle the required changes according to the latest CAISO roadmap. 5
SCE forecasts the IMEP project to be $19.8 million for the 2016-2020 period.231 6
The capital forecast for this project was developed using SCE’s internal cost estimation model. This 7
model utilizes industry best practices and SCE subject matter expertise to estimate project cost 8
components. SCE’s forecast for this project includes costs for SCE employees, supplemental workers, 9
and consultants, software and vendor costs, and hardware costs. See this project’s workpaper for the cost 10
breakdown information. 11
Based on an analysis of the information from CAISO about market changes in 12
this period, SCE has estimated this project to be a high complexity COTS software implementation 13
requiring significant upgrades to existing applications, the system interfaces that connects the systems 14
and databases, and hardware infrastructure. 15
(1) Alternatives Considered 16
This project implements tariff changes CAISO, which are required for 17
regulatory compliance. Therefore, no alternatives have been considered. 18
231 Refer to WP SCE-04, Vol. 2 Bk C p. 211.
176
4. Energy Planning Platform Upgrade (EPP) 1
Table V-52 Energy Planning Platform Upgrade232
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 2
This project will upgrade the existing Energy Planning Platform application to 3
incorporate new market requirements and reporting functionalities. SCE implemented this application in 4
2013 to support portfolio analysis and regulatory reporting and to provide advanced analytics for SCE’s 5
energy hedging program. The original application has grown from a single risk-reporting module to 6
several complex analytical modules. This project will provide the EPP application with the upgrades 7
required in 2018-2019. 8
b) Need for Project 9
As SCE’s power procurement portfolio continues to change, so does its reporting 10
requirements. As Community Choice Aggregators (CCAs) are established in the SCE service territory 11
there is a need to map and track the energy and customers served by these CCAs. SCE also expects the 12
need to track resources installed on the distribution grid, usually referred as distributed energy resources 13
(DERs). 14
SCE requires a refresh of its analytic platform to accurately capture SCE’s market 15
position (i.e., how much supply is available vs. the how much demand is projected at different times and 16
locations). Additionally, SCE must be able to track and report on CCAs and expected future distributed 17
energy resources, and to be able to report on how different forecasts for future market prices will impact 18
the cost of supplying power to customers. 19
232 Refer to WP SCE-04, Vol. 2 Bk C pp. 212-218.
CIT-00-DM-DM-000078 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020Recorded / Forecast 1.13 1.19 0.58 0.29 - - - 2.00 2.00 - *2011-2014 recorded costs pertain to previous phases of the EPP application, and do not pertain to this GRC's EPP Upgrade project.
Recorded Forecast
177
Furthermore, SCE is also seeking to retire some of its older applications and move 1
the functionality to the new EPP system. This will create a more robust data management process and 2
will also avoid future IT maintenance costs for the systems that are replaced by the new EPP system. 3
(1) Benefits 4
This project will enhance SCE’s energy portfolio reporting capabilities. 5
c) Scope and Cost Forecast 6
SCE forecasts total project costs of $4.0 million over the 2018-2019 period to 7
upgrade the EPP.233 The capital forecast for this project was developed using SCE’s internal cost 8
estimation model. This model utilizes industry best practices and SCE subject matter expertise to 9
estimate project cost components. SCE’s forecast for this project includes costs for SCE employees, 10
supplemental workers, and consultants, software and vendor costs, and hardware costs. See this project’s 11
workpaper for the cost breakdown information. 12
(1) Alternatives Considered 13
Alternative 1: SCE considered not pursuing this project. However, if we 14
chose not to pursue this project, then we would not be able to accurately monitor SCE’s market position, 15
track and report on CCAs and expected future distributed energy resources, and report on how different 16
forecasts for future market prices will impact the cost of supplying power to customers. 17
233 Refer to WP SCE-04, Vol. 2 Bk C p. 218.
178
5. PCI Replacement 1
Table V-53 PCI Replacement234
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 2
D.15-11-021 adopted the Power Costs, Inc. (PCI) Replacement project, where it 3
was referenced as the Market Systems Replacement project, with a forecast of $5.8 million. The project 4
start was delayed due to dependencies on the Commodity Management Platform (CMP) project that 5
implements a new system to manage renewable power contracts. The CMP project is scheduled to be 6
fully implemented in 2016. 7
The existing system is used to enable SCE to participate in the California 8
Independent System Operator (CAISO) energy market. The system was installed at SCE in 2009 and 9
manages generator outages, captures power trades, creates and submits energy bids to the CAISO energy 10
markets to purchase and sell electric power, and downloads market awards and market prices from 11
CAISO. 12
This project will replace the existing system with new third-party vendor 13
solutions selected through a competitive solicitation process. Since the installation of the existing 14
system, the software vendor landscape has changed significantly; several vendors have merged and 15
continued product development has been done by multiple vendors to enhance the integration, 16
performance, and analytics offered by their systems. 17
This project is related, yet distinctly different, to the CAISO Market Enhancement 18
(IMEP) project described above. While both projects relate to CAISO, the IMEP project implements 19
ongoing market changes mandated by CAISO, whereas this project is a one-time effort to replace the 20
system platform used for day-to-day operations. 21
234 Refer to WP SCE-04, Vol. 2 Bk C pp. 219-224.
CIT-00-DM-DM-000042 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 3.00 3.50 1.00 - 7.50
Previous GRC Request - - - - - 3.00 2.80 5.80
Recorded Forecast
179
b) Need for Project 1
There are issues with the current system that must be addressed by replacing this 2
system with a newer platform. These issues include a lack of ability to keep up with CAISO market 3
changes, system performance, and limited functionality for analytics. In addition, several vendors now 4
offer hosted solutions that provide the same or better functionality at a lower cost, compared to 5
traditional on-premises solutions. 6
SCE requires enhanced system functionality in the following areas: 7
• Ability to participate in CAISO market simulations. CAISO market changes 8
are implemented through a cycle consisting of stakeholder processes to design the market changes, 9
workshops to review the detailed technical design of the market changes, market simulation of the 10
changes, and finally go-live where the changes are implemented into the production environment. The 11
purpose of market simulation is for both CAISO and market participants (including SCE) to test the new 12
market changes using systems and business processes that closely, if not entirely, resemble those 13
actually used to perform the transactions in production. In the past, SCE has frequently had to perform 14
market simulations using manual workarounds instead of the actual software since our vendor could not 15
deliver software updates in time for the CAISO market simulations. 16
• Adequate performance in the SCE environment with data volumes as required 17
by CAISO. CAISO data volumes have increased dramatically after the Market Redesign and 18
Technology Upgrade (MRTU) market went live in 2009. SCE has seen issues where our current system 19
struggles to retrieve and process the CAISO data volumes in a timely manner. 20
• Better functionality for analytics. Analytics is increasingly important as the 21
CAISO market grows more complex and the portfolio of renewable power continues to grow. Adequate 22
analytics require access to updated and relevant data using a variety of data reporting and analysis tools. 23
SCE has experienced issues where data is stored in the current vendor database in a format either 24
inaccessible or hard to retrieve and analyze using industry-standard reporting and analysis tools. 25
(1) Benefits 26
This project will replace the existing system with a new system that will 27
offer similar or better functionality at lower total cost of ownership. 28
c) Scope and Cost Forecast 29
When the existing system was installed at SCE in 2009, the industry-standard 30
approach to implement new software was to run the software on servers in SCE’s data centers. Since 31
180
2009, significant technological development has taken place among software vendors and one of the 1
most important evolutions has been the introduction of hosted or cloud solutions. 2
Hosted solutions offer several advantages to SCE for this project: 3
• Lower initial installation costs since computer hardware does not have to be 4
purchased and the software does not have to be manually installed on the new 5
hardware. 6
• Avoided ongoing operating costs since the vendor will operate SCE’s system 7
in parallel with systems from other clients, which offers opportunities for 8
automating similar maintenance procedures across different client systems. 9
• Faster and more effective response to CAISO market changes. Today SCE 10
must receive new software releases from the vendor, test the software, and 11
implement it into our computer environment before the users can perform 12
their testing and/or participate in CAISO market simulations for new energy 13
market products. In a hosted solution the vendor tests and installs the new 14
software release on their servers, which reduces the overall cycle time. This 15
allows SCE more time to participate in market simulations. 16
The market survey and analysis that SCE conducted in 2014 showed that vendor 17
solutions are available that offer similar or better functionality at lower total cost of ownership than 18
today’s solution.235 19
In the 2018 GRC the total project costs are $7.5 million. The capital forecast for 20
this project was developed using SCE’s internal cost estimation model. This model utilizes industry best 21
practices and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this 22
project includes costs for SCE employees, supplemental workers, and consultants, software and vendor 23
costs, and hardware costs. See this project’s workpaper for the cost breakdown information.236 24
235 Refer to WP SCE-04, Vol. 2 Bk C pp. 225-236. 236 Refer to WP SCE-04, Vol. 2 Bk C p. 224.
181
(1) Alternatives Considered 1
Alternative 1: SCE considered not pursuing this project. However, if we 2
chose not to pursue this project, then we would incur higher total five-year ownership costs based on 3
vendor assessment and analysis performed by SCE in 2014 and 2015. 4
Alternative 2: SCE considered custom development to meet the business 5
needs. We did not pursue this option since the third-party vendor offerings are mature and will meet our 6
business needs. If we developed a custom solution we would need to perform ongoing development to 7
implement planned CAISO changes and would not benefit from sharing these costs with multiple other 8
customers, as we would be able to do with a third-party vendor. 9
6. Energy Trading and Risk Management (ETRM) System Replacement 10
Table V-54 Energy Trading and Risk Management237
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 11
D.15-11-021 adopted the Energy Trading and Risk Management (ETRM) System 12
Replacement project as part of SCE’s 2015 GRC. The ETRM System Replacement project will replace 13
the existing ETRM system installed in 2008. It is used to store and process all conventional bilateral 14
power contracts and physical and financial power and gas transactions. 15
The project start was delayed due to dependencies on the CMP project. The 16
ETRM replacement project builds on the functionality being implemented as part of the CMP project, so 17
the CMP project must be completed and stable in production before the ETRM replacement project can 18
begin. The CMP project is scheduled to be completed in 2016.238 19
237 Refer to WP SCE-04, Vol. 2 Bk C pp. 237-242. 238 Refer to testimony for Commodity Management Platform later in Section C – “Power Supply Software
Projects.”
CIT-00-SD-PM-000226 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 3.00 2.40 - - 5.40 Previous GRC Request - - - - - 6.00 6.00 12.00
Recorded Forecast
182
b) Need for Project 1
The existing ETRM system is obsolete and is not being enhanced by the vendor. 2
It runs on older versions of Microsoft Windows Server and Oracle database server, which are no longer 3
supported with software upgrades and security fixes by the vendors. 4
SCE has already selected a new energy trading and contract management 5
platform, which is being used to implement the CMP project. By replacing the existing ETRM system 6
with this new system, SCE can handle conventional, renewable, and financial transactions in a single 7
system. This will simplify the accounting and reporting processes since data from only a single system 8
must be extracted. 9
The existing ETRM system requires a multitude of spreadsheet workarounds to 10
support the current business needs. Moving to the new system will reduce the need for these manual 11
spreadsheet workarounds. 12
(1) Benefits 13
This project will consolidate all of SCE’s wholesale energy trades and 14
bilateral contracts into a single system. It will reduce the need for manual spreadsheet workarounds and 15
streamline the capture and management of energy transactions. It will also simplify accounting, internal, 16
and regulatory reporting processes. 17
c) Scope and Cost Forecast 18
We reevaluated the scope of this project as part of this 2018 GRC application, and 19
the total cost estimate was reduced. In addition, we enhanced the cost estimates based on experience 20
gained from the CMP project. 21
The forecast total project costs are $5.4 million.239 The capital forecast for this 22
project was developed using SCE’s internal cost estimation model. This model utilizes industry best 23
practices and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this 24
project includes costs for SCE employees, supplemental workers, and consultants, as well as software 25
and vendor costs. See this project’s workpaper for the cost breakdown information. 26
239 Refer to WP SCE-04, Vol. 2 Bk C p. 242.
183
(1) Alternatives Considered 1
Alternative 1: SCE considered not pursuing this project. However, if we 2
chose not to pursue this project, then we would have to rely on the existing system, which is not being 3
enhanced by the vendor going forward and relies on obsolete operating system and database software. 4
Based on this we concluded that keeping the existing system is not a viable solution. 5
Alternative 2: SCE considered enhancing the existing solution by 6
upgrading the existing system to the latest product from the existing vendor to meet the needs of the 7
energy trading and risk management functions. We did not pursue this option because it would not meet 8
our needs. The implementation costs for the latest product would be similar to the new SCE system that 9
is being implemented for the CMP project since the latest vendor product requires a new system 10
implementation rather than an upgrade. In addition, we would have significant integration expenses 11
since half of our portfolio (renewable power) would be in the new SCE system and the other half 12
(conventional power and financial contracts) would be in the latest vendor product. 13
Alternative 3: SCE considered replacing the existing solution by 14
procuring an existing COTS solution to meet the needs of the energy trading and risk management 15
functions. We did not pursue this option because it would not meet our needs. We have already 16
purchased the license for the new system, which is a class-leading product. It is unlikely SCE could find 17
another vendor that can provide better functionality and/or lower price. In addition, implementation 18
expenses for another vendor would be higher than for the new vendor, since for the new vendor we are 19
starting from a system already implemented (through the CMP project) for renewable contracts. We 20
would also have additional integration expenses since half of our portfolio (renewable power) would be 21
in the new system and the other half (conventional power and financial contracts) would be in the 22
second new system. 23
184
7. Aggregated Demand Response (ADR) 1
Table V-55 Aggregated Demand Response240
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 2
D.15-11-021 adopted the Aggregated Demand Response project, as part of SCE’s 3
2015 GRC. ADR Phase 1 was developed during 2013-2015 with a total expenditure of $2.29 million, as 4
shown in Table V-55. SCE will complete this project in 2017. 5
Phase 1 was implemented in first quarter 2015. The second phase is scheduled to 6
be implemented in 2017. The project is dependent on the CAISO Proxy Demand Resource (PDR) 7
implementation, which has been delayed. This, in turn, delayed the completion of SCE’s ADR project. 8
The CPUC has encouraged the growth of demand response programs to enhance 9
electric system reliability, reduce power purchases, and protect the environment. In D.05-11-009, the 10
CPUC developed a strategy and initiated programs to promote demand response. In proceeding A.05-11
05-006, the CPUC adopted programs and goals for demand response for SCE and the other California 12
energy Investor Owned Utilities. Demand response constitutes between one and two percent (in terms of 13
MW) of SCE’s portfolio. SCE anticipates that demand response will become a more significant factor in 14
determining overall power requirements going forward. Load aggregators are now allowed to create 15
demand response blocks241 from individual consumers participating in the aggregation programs. An 16
effective demand response program might be equivalent to a significant dispatchable generation 17
resource. 18
240 Refer to WP SCE-04, Vol. 2 Bk C pp. 243-248. 241 Demand response blocks (also known as load control groups or blocks) are aggregations of individual
consumers that can be dispatched (activated) as an all-or-nothing option. For example, a load aggregator may create a demand response block of 2 MW. This block can then be not dispatched (0 MW) or fully dispatched (2 MW), but it cannot be partially dispatched (e.g., 1 MW).
CIT-00-SD-PM-000182 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - 0.02 1.34 0.93 2.59 0.87 - - - 5.75 Previous GRC Request* - - - - 4.15 1.40 0.25 5.80 *The Commission adopted SCE's request for this project in D.15-11-021.
Recorded Forecast
185
In R.07-01-041, the CPUC required demand response programs be integrated into 1
the CAISO markets. Under the market rules of CAISO, the location of the demand response resources 2
will affect SCE’s power costs due to Locational Marginal Pricing, and SCE’s costs for congestion 3
management and Resource Adequacy (RA). SCE must be able to accurately track and model how 4
demand response is affecting SCE’s net power need in order to manage its energy procurement for its 5
customers. 6
The ADR project is implemented in two phases. ADR Phase 1 was implemented 7
in 2015. It enabled Demand Response bidding into the market starting June 17, 2015. 8
The scheduled implementation of ADR Phase 2 in 2017 will provide the 9
additional capabilities to facilitate the full functioning and usage of demand response resources by 10
aggregating the demand resource attributes and providing them to the CAISO Day-Ahead and Real-11
Time markets. 12
b) Need for Project 13
In D.14-12-024 the CPUC required SCE to develop a plan for integrating demand 14
response into the CAISO markets by January 1, 2018 to retain the Resource Adequacy value of demand 15
response resources. CAISO requires SCE to provide demand forecast information into its market to 16
mimic generation resources. SCE will require analytical tools to operate in the CAISO market with 17
aggregate demand response as a dispatchable resource. Demand Response resources differ from other 18
energy resources because they bridge the retail and wholesale businesses. Multiple retail accounts (with 19
corresponding meters) that have similar attributes (e.g., location or demand response program 20
participation) are aggregated into a single wholesale resource to be bid into the CAISO market. 21
The Aggregated Demand Response application will provide the following 22
functions: (1) Store the definition of wholesale demand response resources that can be bid into the 23
CAISO market based on an aggregation of retail customers; (2) Forecast the capacity of each of the 24
defined demand response resources on a daily and hourly basis; (3) Generate bids for each demand 25
response resource and submit these bids to the CAISO markets; (4) Retrieve awards from CAISO for the 26
demand response resources and translate these awards into instructions to the retail demand response 27
186
participants;242 (5) Monitor the actual performance of demand response resources by comparing their 1
actual usage during a demand response event to a baseline usage to determine the actual load reduction; 2
and (6) Perform settlements by calculating data (including resource performance data) for submission to 3
CAISO and validate CAISO settlement payments and charges. 4
(1) Benefits 5
This project will comply with R.07-01-041 and will enable SCE’s 6
customer demand response programs to participate in the wholesale CAISO energy markets. Based on 7
the attributes of the various customer demand response programs these programs will be aggregated into 8
wholesale energy resources. These resources will then be bid into to the CAISO Day-Ahead and Real-9
Time energy markets. 10
c) Scope and Cost Forecast 11
The total project costs are forecast to be $5.8 million. SCE is requesting $3.5 12
million to implement Phase 2 and complete the project by 2017.243 The capital forecast for this project 13
was developed using SCE’s internal cost estimation model. This model utilizes industry best practices 14
and SCE subject matter expertise to estimate project cost components. SCE’s forecast for this project 15
includes costs for SCE employees, supplemental workers, consultants, as well as software and vendor 16
costs. See this project’s workpaper for the cost breakdown information. 17
(1) Alternatives Considered 18
Alternative 1: SCE considered keeping the existing solution without 19
modification. We did not pursue this option because it would not meet our needs. If we choose not to 20
implement the remaining functionality, then all the DR programs would not be fully integrated into 21
CAISO. This will introduce operational risks of a partly integrated solution. Potential impacts are 22
suboptimal market participation by SCE and CAISO non-performance penalties. The partial resource 23
integration may also result in Resource Adequacy disallowances. 24
242 This function is complicated by the fact that a bid can be partially awarded by CAISO. For example, if a
demand response resource represents 1,000 retail customers participating in one of SCE’s demand response programs, the bid submitted to CAISO is for 2MW and CAISO awards 1MW to the resource in the auction, the Aggregated Demand Response system needs to determine which retail customers will participate in the program, by how much and for how long.
243 Refer to WP SCE-04, Vol. 2 Bk C p. 248.
187
Alternative 2: SCE considered replacing the existing solution by 1
procuring an existing COTS solution to meet the needs of the demand response integration function. We 2
did not pursue this option because it would not meet our needs. SCE evaluated third-party vendor 3
solutions as part of Phase 1 of this project. However, significant product flexibility (and potentially 4
costly custom development) would be required to manage changing CAISO requirements. Also, demand 5
response bidding into a wholesale market like CAISO is a relatively new field with few proven vendor 6
solutions available. SCE therefore developed the ADR system in-house. 7
8. Commodity Management Platform (CMP) 8
Table V-56 Commodity Management Platform (CMP) 244
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 9
D.15-11-021 adopted the Commodity Management Platform project (formerly 10
known as the Renewable Contract Management System) as part of SCE’s 2015 GRC. This project 11
implements a new system to manage renewable power contracts and replaces the existing Wholesale 12
Energy System (WES). WES cannot handle the volume and complexity of the new renewable contracts 13
currently online and those that are expected to come online in the next five years in growing numbers. 14
b) Need for Project 15
WES was implemented in 1998 and does not have the required functionality 16
needed going forward. Specifically, the current WES system is customized to settle legacy power 17
purchase contracts based solely on whether the power was generated during an on-peak or off-peak 18
period (i.e., using a time-of-use definition). New renewables contract terms require payment and 19
settlement functions based on interval data (hourly or sub-hourly) and are tied to CAISO operations and 20
settlements. Since the WES system cannot handle hourly data, all contracts that require hourly or sub- 21
244 Refer to WP SCE-04, Vol. 2 Bk C pp. 249-255.
Recorded ForecastCIT-00-SD-PM-000112 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast 1.25 5.63 0.68 0.70 5.49 8.59 2.85 - - - - 25.18 2015 GRC Authorized* 1.25 5.63 0.68 0.70 12.52 7.31 28.08 2015 GRC - SCE Request 1.25 5.63 0.68 5.00 15.52 - - - 28.08 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized our adjusted 2014-2015 request. The 2015 GRC did not address 2010-2012 recorded costs.
188
hourly data (which includes most of the new contracts) are handled manually using spreadsheets, Access 1
databases, and user-developed SAS tools. Data is extracted from WES and other sources, processed 2
manually using spreadsheets and other tools, and finally the results are manually entered back into WES 3
for further processing. This process is cumbersome and increases the risk of human error. 4
In addition, WES was built on technologies that are no longer supported. The 5
limitations of the current system have forced SCE to add more user-developed applications (UDAs) and 6
manual processes to handle the growing volume and complexity of new renewable, combined heat and 7
power, and Qualifying Facilities (QF) contracts. These workarounds cannot handle the growing volume 8
and complexity of contracts, which presents a growing risk of data integrity issues. 9
The Commodity Management Platform, in contrast, can handle hourly (and sub-10
hourly) data through a formula editor that will enable advanced contracts to be configured in the system 11
and automate the processing of these contracts. This will reduce the need for spreadsheet workarounds. 12
Due to the current market structure, the new contracts are far more complex than the legacy contracts 13
that SCE administers. The contracting terms now include, but are not limited to: 14
• Processing for forecast, scheduled, and metered kWh. 15
• Multi-settlement intervals per hour (5, 10, and 15-minute intervals). 16
• CAISO settlement transaction adjustments. 17
• Payment adjustments based on CAISO prices. 18
• Annual energy delivery target monitoring. 19
(1) Benefits 20
The Commodity Management Platform will accommodate contract 21
compliance, payment, and reporting processes for existing and envisioned future power purchase 22
contracts. The Commodity Management Platform will also provide enhanced controls and auditing 23
capability above what can be done using spreadsheets. The system will store a timestamp and user name 24
for all changes to data in the system, and store both the previous and new value of the changed data. 25
This enables users and auditors to trace the source and timing of all data changes. This greatly aids in 26
identifying and diagnosing the source of discrepancies in calculation results and counterparty invoices 27
and provides controls that cannot be replicated using spreadsheets. 28
c) Project Scope and Forecast 29
The total project costs are $25.2 million. The capital forecast for this project was 30
developed using SCE’s internal cost estimation model. This model utilizes industry best practices and 31
189
SCE subject matter expertise to estimate project cost components. SCE’s forecast for this project 1
includes costs for SCE employees, supplemental workers, and consultants, software and vendor costs, 2
and hardware costs. See this project’s workpaper for the cost breakdown information. 3
SCE has recorded $22.3 million through the end of 2015 and is requesting $2.85 4
million in 2016 to finish the implementation of the project.245 SCE had planned to finish this project in 5
2015, however, we needed to implement new requirements in 2016 associated with a new Formulae 6
Editor for configuring contracts, a new Energy Curtailment work flow, and extension of Integration and 7
User Acceptance testing. Even with these new requirements, SCE plans to implement this project with 8
lower total costs over the 2013 – 2016 period than what the Commission authorized in the 2015 GRC. 9
(1) Alternatives Considered 10
Alternative 1: SCE considered not pursuing this project. However, if we 11
chose not to pursue this project, then we would need to continue to rely on spreadsheets and manual 12
workarounds to settle current and future renewable contracts. We would also not be able to handle the 13
growing volume and complexity of contracts, which would increase the risk of human error and data 14
integrity issues. 15
Alternative 2: SCE considered custom development to replace the 16
existing WES system. This would be a high-complexity development due to the need to be able to define 17
different settlement calculations for different contract types. Also, a custom development would prevent 18
us from consolidating renewable and conventional power contracts into a single system in the future. 19
9. Generation Management System (GMS) Upgrade 20
Table V-57 Generation Management System (GMS) Upgrade246
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
245 Refer to WP SCE-04, Vol. 2 Bk C p. 255. 246 Refer to WP SCE-04, Vol. 2 Bk C pp. 256-262.
Recorded ForecastCIT-00-SD-PM-000149 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - 0.89 0.40 0.98 0.25 - - - - 2.52 2015 GRC Authorized* 0.89 0.80 1.69 2015 GRC - Original Request - - 1.50 0.19 - - - 1.69 * In D.15-11-021, the Commission adopted 2013 recorded costs and authorized our original 2014 request, plus the underspent amount in 2013.
190
a) Project Description 1
D.15-11-021 adopted the Generation Management System project as part of 2
SCE’s 2015 GRC. SCE is requesting $0.25 million in 2016 to implement the final phase of the project to 3
be compliant with the NERC CIP V5 requirements. 4
The Generation Management System (GMS) was implemented in 2002 and 5
enables SCE personnel to more accurately determine SCE’s net energy position throughout each 6
operating day, thereby improving the opportunity to lower energy costs to its customers. 7
GMS supports SCE’s goal to minimize costs to its customers by providing real-8
time information for Qualifying Facilities (QF), conventional and renewable power production and 9
enabling SCE to update hour-ahead schedules for generation deviations from planned schedules. GMS 10
functionality allows Power Supply operations personnel to calculate and monitor: Automated 11
Generation Control (AGC); telemetry data; schedules from the Power Costs Inc. (PCI) system; dynamic 12
schedules information; CAISO scheduling information; energy trader information; real time data from 13
various locations; net short positions; and adjustments to hour-ahead schedules based on generation 14
status. The installed GMS software version depends on obsolete versions of Microsoft Windows and 15
Windows Server, which causes maintenance issues. The system was designed and warranted by the 16
vendor for 25 connections to power plants. A software upgrade in 2005 enabled the system to handle 50 17
connections, but is insufficient to handle SCE’s current 105 connections, and projected future portfolio 18
requirements as more renewable generation resources come on-line (an average of 60 new connections 19
are expected each247 year, with 225 total connections by the end of 2018). 20
The new system that has been implemented in June 2016 resolves all of these 21
issues. 22
b) Recorded Costs and Forecast 23
SCE has recorded $2.27 million through the end of 2015 and is requesting $0.25 24
million in 2016 to finish the final phase of the project.248 The capital forecast for this project was 25
developed using SCE’s internal cost estimation model. This model utilizes industry best practices and 26
SCE subject matter expertise to estimate project cost components. SCE’s forecast for this project 27
247 Please refer to workpaper CSB-00-P1 in Power Supply (PS) Volume 2 – Energy Procurement for details. 248 Refer to WP SCE-04, Vol. 2 Bk C p. 262.
191
includes costs for SCE employees, supplemental workers, and consultants, software and vendor costs, 1
and hardware costs. See this project’s workpaper for the cost breakdown information. 2
10. Power Supply Projects less than $3 Million 3
Table V-58 Power Supply Projects less than $3M249
Work Breakdown Structure (WBS) Forecasted Capital Expenditures (Nominal $Millions)
Table V-58 above lists capitalized software projects with total project costs less than $3 4
million in capital funding and will be starting and finish within the years 2016 to 2020. Please see the 5
workpapers for these projects for detailed information on each project and its associated costs. 6
249 Refer to WP SCE-04, Vol. 2 Bk C pp. 263-281.
WBS Project Description 2016 2017 2018 2019 2020 TotalCIT-00-DM-DM-000028 Usage Measurement System (UMS) - - 1.20 - - 1.20 CIT-00-SD-PM-000148 PPD Control Systems Hardware Refresh 2.19 - - - - 2.19 CIT-00-SD-PM-000230 Gas Solar Control Systems Refresh 0.40 1.57 0.60 - - 2.57 CIT-00-DM-DM-000079 Work Mgt and Reliability-Centered Maint - - 0.50 1.00 1.00 2.50
Total 2.59 1.57 2.30 1.00 1.00 8.46
192
D. Ethics and Compliance 1
1. Enterprise Content Management 2
Table V-59 Enterprise Content Management250
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 3
The Enterprise Content Management project is focused on improving SCE’s 4
capabilities to manage a diverse and complex set of business records. SCE has been in business for more 5
than 125 years. This has created a large volume of business information that supports the Company’s 6
operations. The substantial majority of this information exists in a variety of electronic formats, such as 7
electronic spreadsheets, PDF files, and structured251 IT-supported database systems used across the 8
enterprise. This project will implement a set of eight solutions to minimize the risks of record-keeping 9
non-compliance and provide advanced content management capabilities to improve reliability, safety, 10
and electronic discovery (eDiscovery) operations. The project will also deploy tools and controls to 11
improve the accuracy of records across SCE’s operating units and improve classification of information 12
for SCE to meet its information protection needs. 13
To achieve these objectives, we will implement the following solutions: 14
1. Digital Signatures: Integrate digital signature technology and capability 15
with identified critical business processes that have signature requirements 16
as a part of their workflow. 17
250 Refer to WP SCE-04, Vol. 2 Bk C pp. 286-292. 251 Structured database systems refer to any data that resides in a fixed field within a database.
CIT-00-DM-DM-000068 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - - - - - 3.40 5.20 5.40 5.40 19.40 Previous GRC Request - - - - - - - -
Recorded Forecast
193
2. Centralization of Critical Records: Continue to migrate critical 1
records252 from file shares and individual laptops to SCE’s standard 2
content management repository. 3
3. Records Management Enhancements: Enhance records management 4
capabilities for repositories such as SharePoint to comply with records 5
management requirements; apply automation and manage lifecycle of 6
records in alignment with records retention schedule. 7
4. Management of Email Records: Apply records management controls for 8
email records in Microsoft Outlook. 9
5. Automate Records Management: Provide the ability to automate routine 10
records management responsibilities for employees. 11
6. Preserve Digital Records with Extended Retention: Deploy capabilities 12
to preserve and enable access to SCE records that have extended retention 13
timelines ranging from 25 years to 100 years and beyond. 14
7. Enterprise Search: Provide a single user interface for employees to 15
search content across multiple SCE repositories. 16
8. Manage Structured Data Lifecycle: Implement lifecycle management to 17
structured data records to properly store and disposition records according 18
to appropriate procedures. 19
b) Need for Project and Scope 20
As the volume of business data and information grows, and the pace of regulatory 21
and business changes quicken, the importance of having timely, effective, and controlled records 22
management increases. The variety of solutions and platforms where this information is processed and 23
stored is rapidly increasing. This increasingly complex information landscape, combined with the 24
growing pace of change in technology, creates challenges for SCE’s information governance and record-25
keeping capabilities. At the same time, SCE’s operations are highly dependent on this information being 26
252 Critical records include records that, if mismanaged, will affect grid reliability, employee or customer safety,
compliance, Company financials, or the business reputation of SCE.
194
accessible, accurate, and compliant with nearly 5,800 legal citations and regulations.253 Each of the eight 1
initiatives identified below are needed to support the evolving information governance needs and 2
mitigate the associated risks and challenges. 3
• Digital Signatures 4
Need: If a signature is required to confirm an electronic document as a 5
business record, the document must be printed, signed, and then scanned to create the finalized business 6
record. This paper-based signature process is inefficient and expensive, especially when considering the 7
scope of the Company’s operations. In some cases it may be hard to confirm the integrity and reliability 8
of the signature process. In a multi-page document, text and other content can be altered and inserted 9
without being noticed. If the signature process does not have integrity or reliability, the Company may 10
undergo reputational damage, significant financial penalties, and non-compliance events. 11
Scope: This project will implement a digital signature technology, which is 12
supported by standard technology products at SCE. This includes the Microsoft Office suite of 13
applications, including Word, Excel, Outlook, Adobe Acrobat-based documents, and other IT-supported 14
document formats, including specialized computer-aided design and drafting (CADD) document 15
formats. The implementation includes: 16
(1) Digital signatures for all eligible employees, with the ability to digitally 17
sign these documents. 18
(2) Electronic signatures to enhance workflow of critical business processes 19
that involve signatures. 20
• Centralization of Critical Records 21
Need: A significant portion of SCE's critical records are contained in 22
repositories that require enhanced controls such as versioning, legal hold management, metadata 23
tagging, and audit trails. These records are critical to operations, safety, and compliance. To mitigate the 24
risks of mismanagement of these critical business records, these repositories must be migrated to the 25
Company’s enterprise content management system so we can apply proper controls to the business 26
records. These controls provide for advanced security and protection, version control capabilities, 27
253 For example, these regulations and citations include the California Public Utilities Code and Code of Federal
Regulations.
195
enforcement of records retention policies, standardized metadata tagging, advanced search features, 1
legal hold management, and audit trail capabilities. 2
Scope: The centralization of critical records will include performing 3
Redundant, Obsolete, and Trivial (ROT) analysis, deleting redundant and obsolete records that meet the 4
Company’s retention timeline, and migrating records to SharePoint. As part of this migration, the project 5
will establish metadata associated with each record, develop an access-based permission design, and 6
apply retention schedules to manage the lifecycle of critical records of the Company. 7
• Records Management Enhancements 8
Need: SCE’s enterprise content management repositories, ranging from 9
SharePoint to network file shares, serve day-to-day operational needs. But they lack robust records 10
management capabilities that enable records classification, declaration, and lifecycle management in 11
accordance with SCE’s retention schedules. SCE must enhance the existing capabilities of these tools to 12
help verify routine records management operations can be performed. With this solution, SCE can 13
effectuate our records retention policy and enforce legal holds to support pending litigation or 14
investigations. 15
Scope: These records management enhancements will manage content 16
residing within the Company’s repositories, such as SharePoint. This in-place records management 17
capability will provide employees with the ability to properly manage new business information 18
repositories that may be added in the future. This will also provide capabilities to meet the management 19
capabilities found in Department of Defense (DOD) 5015.20 standard industry measure of records. 20
• Management of Email Records 21
Need: Over the past few decades, email has become pervasive across all 22
aspects of the Company’s business operations. The Company produces a significant volume of email 23
every day to support its operations. According to the Association of Records Managers and 24
Administrators (ARMA), an international consortium of leading records management experts, on 25
average 5%254 of all emails are classified as business records. These records must be retained pursuant to 26
the Company’s record retention schedule and given the same records management protections and 27
254 ARMA reference - http://www.arma.org/r1/news/newswire/2016/06/22/report-e-mail-is-greatest-compliance-
risk.
196
controls as document-based records. There is currently no effective solution implemented at SCE to 1
provide for proper records management of email-based records. 2
Scope: This project will implement an email-based records management 3
solution within the Company’s email system (MS Outlook). As part of this scope, the solution will 4
provide capabilities to meet Department of Defense (DOD) 5015.20 standard industry measure of 5
records management capabilities. 6
• Automate Records Management 7
Need: The method for determining records retention qualification (identifying, 8
declaring, and retaining the records) is currently performed manually. This requires time from all SCE 9
employees, and also makes the process vulnerable to unintentional errors and incorrect judgment. 10
Automating our recordkeeping will involve managing the lifecycle of records based on pre-defined rules 11
and automating these rules in record repositories where the records are found. This model is based on 12
pre-establishing rules and using technology to automate how we identify, qualify, and retain records. 13
This continues throughout the entire lifecycle of the records. 14
Scope: This project will implement Rules-Based Recordkeeping (RBR) to 15
enable an automated method of recordkeeping, based on pre-established rules. The solution will 16
leverage pre-established document handling rules, and automate the qualification, declaration, and 17
records retention classification of documents. This solution will collect metadata from the user and 18
apply pre-defined records lifecycle management rules across structured systems like SAP and 19
unstructured repositories such as SharePoint. The objective is to automate records retention rules for 20
about 70 critical record types. 21
• Preserve Digital Records with Extended Retention 22
Need: The Company’s Records Management Program addresses the life-cycle 23
of SCE’s records from creating the record, to retaining it, to disposing of it. Many files have a long 24
retention time and are in electronic format. Over time, the long-term preservation of these records is at 25
risk due to frequent changes in technological electronic formats and media standards, technological 26
obsolescence, and degradability of technological media. Each file format can present challenges in the 27
long-term obsolescence of past format standards. Similarly, the obsolescence and changes to previously 28
standard electronic information storage media presents reliability issues in accessibility of data. 29
The inaccessibility of information related to equipment specifications, 30
engineering drawings, or operational procedures could cause reliability issues in maintaining operations, 31
197
or trigger delays when attempting to restore or resume critical operations. Risks also arise in our ability 1
to produce information related to eDiscovery and other compliance matters if the required 2
documentation is not accessible or retrievable. This solution will mitigate risks associated with 3
managing the long-term preservation of records, which will have significant impacts on SCE's 4
operations, compliance, and ability to support eDiscovery requests. 5
Scope: The digital records preservation capability will preserve on a long-term 6
basis the Company’s records in the face of changes in technological electronic format and media 7
standards, technological obsolescence, and degradability of technological media. This solution will let 8
electronic records with extended retention times remain accessible through a standard electronic format. 9
• Enterprise Search 10
Need: The diversity of electronic information sources has increased greatly, 11
and continues to do so, as the volume of electronic business information has grown exponentially. While 12
the Company has worked to centralize its critical information in its enterprise content management 13
systems, many other business documents exist on diverse sources, such as email, files shares, and local 14
disk drives. The time consumed to retrieve documents or records to support critical operations of the 15
Company is high. Implementing Enterprise Search will significantly reduce the time and cost to retrieve 16
documents/records255 and enhance the Company’s eDiscovery operations. 17
Scope: The Enterprise Search capability will make business information 18
accessible and retrievable from multiple enterprise-type sources, such as enterprise content systems, 19
email, file shares, and cloud-services repositories. The Enterprise Search solution will search and present 20
content from multiple repositories into a single results user interface, thereby providing a consolidated 21
view of this information. This Enterprise Search capability will integrate structured and unstructured 22
information such as engineering standards, specifications and safety checklists, in its results. Users will 23
also be able to filter search results by author, repository grouping, and other pieces of metadata to aid in 24
locating the most current and most relevant content. The Enterprise Search solution will also honor the 25
in-place security protections of the information in its existing repositories. 26
255 Refer to WP SCE-04, Vol. 2C pp. 31-52.
198
• Manage Structured Data Lifecycle 1
Need: Much of the Company’s data is stored in its IT-supported application 2
systems (e.g., SAP). Due to increasing demands and business need for more information, the data in 3
these systems are increasing exponentially. This “structured” information must be properly managed 4
throughout its full data lifecycle, from creation to disposition. 5
We currently have no tool in place to give structured data lifecycle 6
management capabilities. So we manage, archive, and dispose of the data in the Company’s IT-7
supported application systems. Structured data lifecycle management will address issues triggered by the 8
exponential increase in unmanaged structured data and improper retention. This tool will help reduce 9
storage costs associated with archived data and improve performance of day-to-day operational systems. 10
Scope: Unlike the typical lifecycle management solutions based on 11
unstructured data (e.g., Word, Excel, Visio, JPEGs etc.), this effort will enable records management, 12
archiving, and legal disposition of the data residing within the Company’s IT-supported structured data 13
transactional systems, including SAP. 14
c) Cost Forecast 15
SCE is requesting $19.4 million to implement the ECM project. This forecast will 16
implement the scope described above for each of the eight ECM workstreams. We developed our capital 17
forecast for this project using SCE’s internal cost estimation model. This model utilizes industry best 18
practices, references from historical projects, and SCE subject matter expertise to estimate project cost 19
components. SCE’s forecast for this project includes costs for software licenses, vendor labor, system 20
integrator labor, and SCE labor and infrastructure hardware. See this project’s workpaper for the cost 21
breakdown information.256 A forecast for each workstream is presented in Table V-60 below. 22
256 Refer to WP SCE-04, Vol. 2 Bk C p. 292.
199
Table V-60 ECM Forecast by Workstream
(Nominal $Millions)
(1) Alternatives Considered 1
One alternative would be to manage this complex information landscape 2
manually. To manually address the diverse information management needs, we will require significant 3
dedicated staff. This staff will need to augment operational, safety and compliance teams, who will 4
continue to spend additional time and effort to carry out routine tasks associated with searching, 5
declaring, classifying, and dispositioning SCE’s records. This manual alternative could cause high costs 6
over a long period. The distributed employee teams would lack necessary governance compared to a 7
technology solution that enables consistent governance with the help of technology tools. Choosing a 8
manual solution may affect the reliability of company operations, impact regulatory or legal compliance 9
matters, cause unsafe conditions for employees and customers, or may cause financial or reputational 10
damage. 11
ECM Workstream 2017 2018 2019 2020 Total Digital Signatures 0.50 0.50 0.00 0.00 1.00 Centralization of Critical Records 0.00 0.50 0.50 0.50 1.50 Records Management Enhancements 1.50 1.50 1.70 1.10 5.80 Management of Email Records 1.40 1.30 0.30 0.00 3.00 Automate Records Management 0.00 1.40 1.00 1.50 3.90
Preserve Digital Records with Extended Retention
0.00 0.00 0.20 0.30 0.50
Enterprise Search 0.00 0.00 1.00 1.70 2.70 Manage Structured Data Lifecycle 0.00 0.00 0.70 0.30 1.00
Total 3.40 5.20 5.40 5.40 19.40
200
2. Electronic Document Management / Records Management (eDMRM) 1
Table V-61 Electronic Document Management / Records Management (eDMRM) 257
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 2
The Commission authorized the Electronic Document Management and Records 3
Management (eDMRM) project in SCE’s 2015 GRC. Since we developed the showing in SCE’s 2015 4
GRC application, several technology advances occurred in the content management industry. These 5
changes have spurred SCE to revise the scope of work for this project and evaluate the necessary 6
capabilities around content management going forward. This testimony will update the Commission as 7
to the revised scope of work for eDMRM, discuss the work completed to date, and detail the remaining 8
work to be done under this reduced scope of work. 9
As described in SCE’s 2015 GRC, eDMRM was originally scoped to serve as 10
SCE’s primary enterprise content management technology. In 2014, however, SCE began an enterprise-11
wide transition to Microsoft Office 365, which replaced our existing IBM Lotus Notes email client with 12
Microsoft Outlook, and added new enterprise content management tools such as Microsoft SharePoint 13
and OneDrive. As a result of this strategic shift, eDMRM is no longer SCE’s primary enterprise content 14
management system. Microsoft SharePoint and OneDrive are now the two main content repositories 15
deployed to SCE employees for this purpose. Due to these changes, the scope of eDMRM will no longer 16
include capabilities for centralizing all Company’s records, migration of self-service for low-risk 17
records, and Lotus Notes email records management. 18
SharePoint records management capabilities in its current state are not mature, 19
and as such, they will be enhanced over the next two to three years. In the interim, limited investment is 20
257 Refer to WP SCE-04, Vol. 2 Bk C pp. 293-299.
CIT-00-SD-PM-000217 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - - 2.10 6.89 1.42 2.60 - - - - 13.01 Previous GRC Request* - - 2.10 4.70 11.40 8.60 5.80 32.60 *The Commission adopted SCE's request for this project in D.15-11-021.
Recorded Forecast
201
required to migrate some high risk records lacking records management controls into eDMRM, manage 1
Engineering Drawings within eDMRM, and improve system performance. 2
Figure V-12 Content Management Tools
b) Need for Project 3
eDMRM houses a large volume of critical records that have been migrated from 4
fileshares over the past few years. Current system performance limitations impact adopting and using 5
eDMRM. Not implementing enhanced records retention and disposition capabilities will affect SCE’s 6
ability to comply with the records retention and disposition requirements. SCE plans to deploy a pilot of 7
the digital signature capability for select business processes across the company. This will help improve 8
the business process associated with obtaining signatures for records that already exist in eDMRM and 9
reduce costs associated with scanning and maintaining original documents with wet signatures. 10
c) Scope and Cost Forecast 11
As part of SCE’s 2015 GRC, we originally requested $32.6 million to implement 12
eDMRM over the 2013-2017 timeframe. Since that authorization, SCE has modified eDMRM and 13
expects the total project cost to be $13.01 million over the 2013 – 2016 timeframe. SCE recorded $10.41 14
million from 2013 – 2015 to establish the eDMRM technical infrastructure, migrate around 10 million 15
Public Safety and other high-risk records to eDMRM, implement engineering drawing management 16
unique to eDMRM, and create disposition and retention management capabilities. 17
SCE is requesting $2.6 million in 2016 to complete the project at the revised 18
scope by implementing capabilities previously outlined as part of a multi-phased implementation. These 19
capabilities consist of: improving eDMRM system performance; migrating additional critical records 20
repositories from file shares to eDMRM; enhancing the eDMRM-based engineering drawing 21
Before 2014 2014 and Beyond
eDMRM
MS SharePoint(Office365)
Document Management(Primary)
+Records Management
(Primary)
--Did not Exist--Document Management
(Primary)
+Records Management
(Primary-enhancements required)
Records Management (limited use)
202
management module we implemented in 2015; piloting digital signature technology to streamline 1
manual “wet signature” processes across a small set of organizations within the enterprise; and 2
completing and operationalizing records retention and disposition capabilities within eDMRM. 3
We developed the capital forecast for this project using SCE’s internal cost 4
estimation model. This model utilizes industry best practices and SCE subject matter expertise to 5
estimate project cost components. SCE’s forecast for this project includes costs for system integrator 6
labor, SCE labor, and infrastructure hardware. See this project’s workpaper for the cost breakdown 7
information.258 8
SCE’s revised scope and cost forecast for eDMRM represents a reduction of 9
$19.5 million from SCE’s original request. As discussed above, this reduced scope is primarily due to a 10
shift in SCE’s strategy to use Microsoft Office 365 as our enterprise content management platform for 11
managing enterprise content and email. 12
(1) Alternatives Considered 13
SCE considered forgoing the eDMRM enhancements planned for 2016. 14
This approach has significant compliance implications. Not implementing and operationalizing the 15
eDMRM records retrofit and disposition functionality would prevent SCE from fully effectuating its 16
Records Management Policy, which is based on over 5,800 legal and regulatory compliance 17
requirements. Not enhancing functionality in the engineering drawing module and overall system 18
performance will create a risk in terms of employees being able to efficiently access high-risk 19
documents, while performing critical reliability and business resiliency operations. 20
258 Refer to WP SCE-04, Vol. 2 Bk C p. 299.
203
E. Finance Capital Projects 1
1. Plant Ledger Upgrade and Tax Module Installation 2
Table V-62 Plant Ledger Upgrade and Tax Module Installation259
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 3
In SCE’s 2015 GRC, the Commission authorized SCE to implement the Tax 4
Department Repairs project. The tax repairs scope of the project was originally scheduled to complete in 5
2014, however during the technical design phase in late 2013, SCE identified dependencies within the 6
PowerPlan suite of capital accounting software that required SCE to align the implementation with an 7
upgrade to the latest commercially available version in order to meet the original project objectives.260 8
The PowerPlan capital accounting software was implemented in 2008 and was due for a refresh given 9
needed updates to the software. Reflecting its increased scope, SCE has renamed this project the Plant 10
Ledger Upgrade and Tax Module Installation project. 11
The upgrade to the PowerPlan capital accounting software version was needed not 12
only to support the full implementation of tax repairs functionality, but also to address maintenance and 13
efficiency impacts associated with the outdated version. SCE originally implemented PowerPlan version 14
10.2.1.2 in 2008, and at this time this version is no longer eligible for standard support from the vendor. 15
This results in more costly custom maintenance support agreements. Moreover, SCE identified over 16
259 Refer to WP SCE-04, Vol. 2 Bk C pp. 300-318 260 In addition to the Tax Repairs module, the eight modules in SCE’s current suite of PowerPlan capital
accounting software that required upgrade are the following: Project Cost Management (work order processing, CWIP, unitization), Asset Management (fixed asset accounting), Depreciation (book depreciation and reserve analysis), Depreciation Studies (actuarial analysis and analysis of book accrual rates), PowerTax (tax depreciation, book/tax reconciliation, deferred taxes), Tax Provision (current/deferred tax provisions), Property Tax (filings, analysis, payments), and Charge Repository (SAP-PowerPlan GL interfaces, allocations, validations).
204
ninety accounting business process inefficiencies that are driven by the aging PowerPlan software 1
version. The accounting business process inefficiencies were focused primarily in the following areas: 2
construction work in progress, property tax, and tax provisions related to depreciation. As a result of tax 3
repairs requirements, software obsolescence, and capital accounting process inefficiencies, SCE 4
proceeded with the PowerPlan upgrade project. From inception through year-end 2015, $5 million was 5
recorded to complete the following project activities: solution design, test installation of the upgraded 6
software, and build of integration with SCE’s SAP system. SCE requests $4.2 million for 2016 capital 7
expenditures to complete solution development, integrated system testing, data conversion, and 8
production implementation. The project will upgrade the PowerPlan application to the latest version to 9
eliminate the need for software customization, improve SCE’s capital accounting processes, and enable 10
new business capabilities to support the company’s property tax and income tax filings. 11
b) Need for Project 12
PowerPlan is the software application that manages the sub-ledger for SCE’s 13
fixed asset accounting, property tax and income taxes. As SCE’s business needs for managing capital 14
asset accounting and IRS tax regulations continue to change, PowerPlan capital accounting software 15
must be upgraded to remain technically supported and enable the processes. Since its initial 16
implementation as part of SCE’s Enterprise Resource Planning (ERP) program, however, this 17
application has not been upgraded to a more current version supported by the vendor. Moreover, when 18
PowerPlan was originally deployed, the software did not account for all business scenarios within their 19
product suite. As a result, SCE invested in customizations to enable the processes not fully supported. 20
Over the last eight years, PowerPlan has designed and improved their software to account for many of 21
the SCE customizations as part of their standard product offering. This project will update the software 22
and hardware infrastructure and add application functionality. This upgrade will accomplish the 23
following key objectives: 24
• Reduce risk by ensuring the application has the technology updates needed to 25
mitigate information security exposures. 26
• Reduce application failures resulting from outdated vendor technology. SCE 27
requires custom support maintenance with PowerPlan to resolve issues with 28
the core product and SCE specific customizations. This is costly due to the 29
significant amount of customizations made to the software to meet SCE’s 30
205
business needs. By upgrading to the newer version, SCE can leverage the 1
software’s standard functionalities and support provided by PowerPlan. 2
• Address some of ninety accounting business-process inefficiencies associated 3
with functionality gaps in the software. These gaps were primarily driven by 4
the software being several versions out of date. Since the original 5
implementation, PowerPlan has enhanced the standard product to 6
accommodate the industry’s demand for these functionalities. Through 7
implementation of the PowerPlan Tax Repairs module, the upgrade will 8
enable SCE to maintain a systematic method of accounting, by which the 9
company determines whether expenditures to maintain, replace, or improve 10
transmission and distribution linear property are immediately expensed as a 11
repair for tax purposes, or treated as capitalized assets that depreciate over 12
time for tax purposes. 13
c) Scope and Cost Forecast 14
The Plant Ledger Upgrade and Tax Module Installation project will be completed 15
in Q3 2016 with a total forecast cost of $9.2M. The capital forecast for this project was developed using 16
SCE’s internal cost estimation model. This model utilizes industry best practices, references from 17
historical projects and SCE subject matter expertise to estimate project cost components. SCE’s forecast 18
for this project includes costs for software licenses, system integrator labor, and SCE labor and 19
infrastructure hardware. See this project’s work paper for the cost breakdown information.261 The 20
revised scope of the project includes major elements as described above and summarized below: 21
• Upgrade of the PowerPlan application to the latest version - 2015.1.3.0. SCE 22
originally implemented PowerPlan version 10.2.1.2. 23
• Enablement of the Tax Repairs module, which automates the determination as 24
to whether a capital expenditure is deemed a repair for tax purposes and 25
enables SCE to continue to provide customers with the flow-through tax 26
261 Refer to WP SCE-04, Vol. 2 Bk C p. 305.
206
benefit associated with the tax-repair deduction in a systematic and auditable 1
manner adhering to IRS guidelines.262 2
• Standard integration of capital asset data across various modules within the 3
PowerPlan capital asset accounting suite, including the Asset Management’s 4
plant ledger and Tax Provision, and reducing manual reconciliations thru 5
spreadsheets. 6
• Standard integration between PowerPlan and SCE’s existing ERP Finance and 7
Enterprise Asset Management (EAM) system, reducing the timeline for 8
month-end accounting close. 9
• Elimination of significant application software customization and accounting 10
process inefficiencies. 11
(1) Alternatives Considered 12
The following alternatives were considered: 13
Alternative 1: In 2014, SCE evaluated the option not to upgrade the 14
system and lose support from the product vendor due to version non-compliance. Under this option, 15
SCE would assume responsibility for maintenance and enhancement of the application. This option 16
would introduce risk to a critical corporate financial application used for fixed asset accounting and tax 17
filings. Similar to other software products in the SCE portfolio, the application was planned to refresh in 18
the 2016 timeframe. Due to the project timeline and application refresh cycle, upgrading the application 19
as part of this project was the option selected. 20
Alternative 2: In 2014, the SCE team evaluated an option of introducing a 21
new third-party vendor product to manage the tax repairs functionality. This option was not selected 22
because the PowerPlan Tax Repairs module offered this capability and integrated more effectively with 23
SCE’s eight other PowerPlan capital asset accounting modules and existing IT applications. 24
262 Refer to WP SCE-04, Vol. 2 Bk C pp. 306-318.
207
2. Corporate Projects less than $3 Million 1
Table V-63 Corporate Projects less than $3M263
Work Breakdown Structure (WBS) Forecasted Capital Expenditures (Nominal $Millions)
Table V-63 above lists capitalized software projects with total project costs less than 2
three million dollars in capital funding and will start and finishing within the years 2016 to 2020. Please 3
see the workpapers for these projects for detailed information on each project and its associated costs. 4
F. Operational Services Capital Projects 5
1. C-CURE 9000 6
Table V-64 C-CURE 9000 2016-2020 Forecast264
Work Breakdown Structure (WBS) Authorized, Recorded, & Forecast Capital Expenditures (Nominal $Millions)
a) Project Description 7
The Commission authorized the C-CURE 9000 project in SCE’s 2015 GRC. SCE 8
is now updating the Commission on the status of this project and the additional expenditures required in 9
this GRC to complete the project. 10
263 Refer to WP SCE-04, Vol. 2 Bk C pp. 319-342. 264 Refer to WP SCE-04, Vol. 2 Bk C pp. 344-349.
WBS OU Project Description 2016 2017 2018 2019 2020 TotalCIT-00-DM-DM-000071 Legal Legal Re-platform - 2.20 - - - 2.20 CIT-00-DM-DM-000075 Regulatory Affairs Reg Affairs - TM2 Replacement - - - - 1.00 1.00 CIT-00-OP-SM-000025 Finance Integrated Budget Planning 2.80 - - - - 2.80 CIT-00-SD-PM-000216 HR Union Negotiations 2.20 - - - - 2.20
Total 5.00 2.20 - - 1.00 8.20
CIT-00-SD-PM-000139 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 TotalRecorded / Forecast - 0.09 3.02 2.03 3.71 0.70 - - - - 9.55 2015 GRC Authorized - 0.09 3.02 - - - - 3.11 2015 GRC - Original Request - 0.09 5.03 - - - - 5.13 * In D.15-11-021, the Commission adopted 2013 recorded costs.
Recorded Forecast
208
Many SCE facilities, including all major buildings, control centers, and 1
generation facilities, are protected by a centrally controlled alarm and event-management system. This 2
physical access control system (PACS) logs all access and attempted access to SCE facilities in 3
connection with our Corporate Security ID badging system. 4
The upgraded PACS system was implemented in 2014 in alignment with 5
requirements described in our 2015 GRC. In 2014, however, FERC authorized the North American 6
Electric Reliability Corporation (NERC) to make the new version of Critical Infrastructure Protection 7
(CIP) Version 5 standards enforceable by early 2016. Because of this new standard and other emerging 8
cybersecurity needs, additional modifications to the PACS application were necessary. SCE decided to 9
use the active project team to perform modification to the PACS in support of these regulatory changes. 10
SCE identified this as an efficient solution to meet the regulatory changes required for the PACS. These 11
modifications increased scope and increased recorded costs by $4.5M. The upgrade will be completed in 12
2016, meeting the NERC CIP V5 and cybersecurity standards. 13
b) Need for Project 14
The PACS upgrade project completed the foundational application upgrade 15
aligned with the business drivers described in the 2015 rate case. Due to the change in regulatory 16
requirements and cybersecurity needs, the following requirements were added, leading to increased cost: 17
• NERC CIP v5 expanded its scope to include many additional bulk system 18
substation facilities, necessitating integration with the PACS. Additionally, 19
NERC CIP v5 required enhanced physical access controls at high-impact 20
facilities, which required additional integration with new physical access 21
control devices. 22
• Application redundancy is needed to meet cybersecurity standards for SCE’s 23
critical information systems responsible for the reliable operation of the 24
electric grid. If a disaster situation occurs at one of SCE’s data centers, the 25
PACS should have a redundant technical environment at a geographically 26
separate data center to support continuity of the system, which will provide 27
assurance that physical security controls remain operational. 28
c) Scope and Cost Forecast 29
The PACS upgrade project recorded $5 million as part of the original scope 30
described in SCE’s 2015 GRC. Due to the new scope for NERC CIP V5 and cybersecurity, an additional 31
209
$4.5M was required. In 2016, the projected cost is $700,000 to fully decommission the old version of 1
the software and implement the disaster recovery environment.265 2
• NERC CIP v5 – The increased number of in-scope substation facilities and 3
increased security measures for high impact facilities necessitated that the 4
system be redesigned to support the expanded scale. This also includes the 5
integration of new access control devices to support the increased 6
requirements for high impact facilities. 7
• Disaster Recovery – Procure hardware, software, and labor to install 8
redundant application in SCE data centers. This requires configuring the 9
PACS to maintain data integrity through a failover and deploying the 10
infrastructure to perform a failover and failback between data centers. 11
(1) Alternatives Considered 12
Two other options were considered for the PACS upgrade project due to 13
the new scope introduced in 2014 by NERC: 14
Option 1: Build a second independent environment. This would create 15
operational challenges in maintaining two access control systems. Implementing this option would 16
require the IT and Corporate Security departments to increase staff to operate multiple application 17
environments. Security personnel must use two computing environments to monitor security alarms, 18
increasing operational risks and challenges. This solution was not considered viable because of the 19
operating challenges in coordination, communication, and response management. 20
Option 2: Utilize security officers to provide human monitoring of access 21
controls if the PACS became unavailable. This option was not considered because the vastness of SCE’s 22
territory would pose an adverse impact on the response time to dispatch security personnel to provide 23
access control and monitoring capability. There is significant cost associated with staging a guard force 24
in a state of readiness to respond and ultimately deploying them in an emergency situation. 25
265 Refer to WP SCE-04, Vol. 2 Bk C p. 349.
210
2. Operational Services Projects less than $3 Million 1
Table V-65 Operational Services Projects less than $3M266
Work Breakdown Structure (WBS) Forecast Capital Expenditures (Nominal $Millions)
Table V-65 above lists capitalized software projects with total project costs less than 2
three million dollars in capital funding and will start and finish within the years 2016 to 2020. Please see 3
the workpapers for these projects for detailed information on each project and its associated costs. 4
266 Refer to WP SCE-04, Vol. 2 Bk C pp. 350-392.
WBS OU Project Description 2016 2017 2018 2019 2020 TotalCIT-00-OP-SM-000026 Business Resiliency Mobile Field Response - - - 0.74 - 0.74 CIT-00-DM-DM-000065 Business Resiliency Crisis Information Management System - - - - 2.60 2.60 CIT-00-DM-DM-000066 Business Resiliency Seismic Risk Assessment - - 2.00 - - 2.00 CIT-00-DM-DM-000062 Corporate Environmental Services EHSync Environmental Clearance Phase 2 - 0.94 0.37 0.37 0.37 2.05 CIT-00-DM-DM-000059 Corporate Real Estate Facilities Management System 0.35 - - - - 0.35 CIT-00-SD-PM-000214 Corporate Safety Safety Observations 0.62 - - - - 0.62 CIT-00-DM-DM-000074 Supply Chain Management Ariba Deployment and Supplier Portal Dec 1.70 1.00 - - - 2.70
Total 2.67 1.94 2.37 1.11 2.97 11.06