Global Business Services © Copyright IBM Corporation 2007ERCOT and IBM Confidential | 23-Sep-15 | External Review of Nodal Program PMO Control System Executive

Global Business Services

© Copyright IBM Corporation 2007ERCOT and IBM Confidential | Apr 19, 2023 |

External Review of Nodal Program PMO Control System

Executive Discussion

Electric Reliability Council of Texas, Inc. (ERCOT) Board of Directors MeetingMay 16, 2007


© Copyright IBM Corporation 2007ERCOT and IBM Confidential | Apr 19, 2023 |2

Agenda

Background

Findings

Recommendations

Other Topics – Software Effort Estimation



Nodal Market Protocols and Requirements

Nodal Program People

Plans, Policies, Templates, Tools and

Techniques

Results

Actions

Nodal Program Management Control System provides order to the Actions of Nodal Program

X X =Nodal

DeliveryERCOT

Readiness

Market Participant Readiness Nodal

Success

ERCOT Nodal ProtocolsSection 2: Definitions and

AcronymsAugust 1, 2006

(Effective Upon Texas Nodal Market Implementation

ERCOT Nodal ProtocolsSection 2: Definitions and

AcronymsAugust 1, 2006


ERCOT Nodal ProtocolsSection 1: Overview

August 1, 2006(Effective Upon Texas Nodal Market

Implementation

ERCOT Nodal ProtocolsSection 5: Transmission Security Analysis and

RUCAugust 1, 2006


ERCOT Nodal ProtocolsSection 6: Adjustment Period and Real-Time

Operations(Effective Upon Texas Nodal Market

Implementation

ERCOT Nodal ProtocolsSection 4: Day Ahead

OperationsAugust 1, 2006


ERCOT Nodal ProtocolsSection 7: Congestion

Revenue Rights(Effective Upon Texas Nodal Market

Implementation

Triangle of Balance

Scope

Schedule Cost

Business Needs

Program Management Control System RUP Other ERCOT Policies and Procedures

Produce Influence Influence



The Program Management Office Control System (PMOCS) supports program direction and increases productivity

Limited Experience

Program Performance

Highly Skilled

Low HighProgram Management Office System Structure

(PMOCS) Strength and Performance Level

Individual

Project Manager’s Skills

Probable Success

Probable Success

Highly Probable Success

Success Unlikely

Program Critical Path Plan Project Schedules Program Resource Plans and

Commitments Program Budget Project Estimates and

Forecasts Progress Reporting Scope and Change Controls Risk Management Plans Issues Tracking Communications Plans Deliverables Controls Quality Management Plan Technical Support Tools –

Version Control, Configuration Management, etc.

Other Program Management Tools and Techniques

Examples of PMOCS ContentThe

PMOCS involves fourteen

categories of support

Change Control

Commun

icatio

ns

Manage

ment

People Management

Supplier Management

u

Project/Program Definition St

akeh

olde

r

Enga

gem

ent

Organization

Managem

ent

Technical EnvironmentIs

sue

Man

agem

ent

Risk

Management

Deliverables Management

Tracking &

Control

Quality

Managem

ent

Work Plan

Management

PMOCS Elements PMOCS Impact

Triangle of Balance

Scope

Schedule Cost

Business Needs



The IBM reviews consider PMOCS structure and use

This is the 3rd Review and it included: All fourteen categories of control with the PMOCS Assessment of the current state of the PMOCS Structure Evaluation of PMOCS Performance Consideration of a sub-set of the Nodal Program Projects:

• Project Management Office (PMO)• Enterprise Integration (EIP)• Energy Management System (EMS)• ERCOT Readiness and Transition (IRT)

Review of plans and other documents, interviews and other testing Provision of recommendations for improvement

Review based on PMOCS in place as of March 20, 2007

Progress Review Schedule includes five more reports:

July 2007 – Report 4 October 2007 – Report 5 January 2008 – Report 6 March 2008 – Report 7 Final 2009 – Report 8

Note: Program Management Control System (PMOCS)

Communicatio

ns

Management

People Management

Supplier Management

Change Control

u

Project/Program Definition S

take

hold

er

Eng

agem

ent

Organization

Managem

ent

Technical EnvironmentIs

sue

Man

agem

ent

Risk

Management

Deliverables Management

Tracking &

Control

Quality M

anagement

Work Plan

Management

PMOCS Areas: Fourteen Control

Categories


© Copyright IBM Corporation 2007ERCOT and IBM Confidential | 19 Apr 2023 |6

Findings – The PMOCS is Operational, but certain areas of structure can be improved

Most components meet best practices standards

A few of the components in the PMOCS do not meet structural expectations:

Work plan integration and content Contractor extension procedures Organization target and decision framework Internal stakeholder communications Integrated testing phases

Source: March 2007 ERCOT PMOCS Structure Analysis

0

2

4

6

8

10

12

14

PMOCS Structure Assessment

Areas Needing Improvement

Areas Meeting or Exceeding Design

Standards



Findings - The PMOCS is working in most areas, but certain areas of performance can be improved

Most components of the PMOCS are being used effectively

A few of the components in the PMOCS do not meet performance expectations:

Slow migration to single Risk and Issue Management tool usage and contingency planning

Undisciplined updating of Project Schedules Absence of re-estimate triggers Low compliance with Quality Management

Reviews as required by Nodal Program

Source: March 2007 ERCOT PMOCS Performance Analysis

0

2

4

6

8

10

12

14

PMOCS Performance Assessment

Areas Needing Improvement

Areas Meeting or Exceeding

Performance Expectations



The PMOCS has been improved, but there are some areas that require emphasis and focus

Report 1 Report 2 Report 3

Control Dimension 10 Oct 2006 15 Jan 2007 24 Apr 2007

Program Definition

Stakeholder Engagement

Change Management

People management

Supplier Management

Risk Management

Issue Management

Communications Mgmt

Work Plan Mgmt

Tech Environment

Deliverables Mgmt

Organization Mgmt

Tracking and Control

Quality Management

IBM Review Reports

Control A

ctivity

Meets or Exceeds

Under

Perform

Structure

Structure and Performance Ratings

Significant Improvements Needed

Effective Controls

Combined Review Ratings

Minor Improvements Needed

Not Evaluated

Moderate Improvements Needed

Unsatisfactory

Trend



Recommendation Impact Action

Integrate work plans Reduced PM administrative burden and improved information

Simply process with use of software to support project plan and task lists

Forecasts linked to productivity and deliverables in work scheduled

Ability to manage budget variance Establish and monitor Productivity Measure to trigger need for budget review

Update Communications Plan to ensure specific cascading communications

Directed communications to all organization levels

Provide more detail in existing plan to address internal communications actions

Accelerate transition process for single Risk and Issue Tool

Issues are not being tracked even if evaluated; risks are not completely evaluated and mitigation not documented

Review progress and make course adjustments

Determine ERCOT Organization target and components of employee transition

Consistent decision framework for ERCOT Transition

Leadership team define tenets of target

Simplify approval process for contractor extensions

Avoid work disruption and delays Consider alternative process for approved contractors for ongoing effort.

Accelerate implementation of and compliance with QA Policy

Avoid program delays due to absence of requirements traceability

Change behaviors by placing emphasis on the process

Provide manager responsible for all testing events including EDS

Improved integration, reduction of duplication of effort, better readiness measure

Assign ERCOT owner (s) of testing process – FAT to EDS

Recommendations for Improvement – Priority for Action




Variances to estimated effort and cost will exist throughout a project

Variance (risk) declines as more is known about the work

Actions to manage variance are critical

Forecasting based on effort drivers identify factors to be managed

Track and manage to metrics to reduce variance from authorized spending

RequirementsAnalysis

SystemDesign

SystemBuild

SystemInstall

.25x

2x

1.5x

1.25x

4x

.8x

FeasibilityStudy

.7x

.5x

Ex

pe

cte

d r

an

ge

of

es

tim

ati

on

ac

cu

rac

y

Software development life-cycle

x

N1 2 3 4

Software project cost as a function of life-cycle phase 1,2

1 Software Engineering Economics, Barry W. Boehm, Prentice Hall PTR, 19812 “Software Size Estimation of Object-Oriented Systems,” IEE Transactions on Software Engineering, Vol. 16, No.5, May 1990

Business Requirements

TargetToday

Potential variance from estimate (degree of risk)

Budget Estimate



Forecasting, tracking and quality assurance based on deliverables provides a common measurement of progress and success

Alarm Data Model Energy Management System (example deliverable) Compare effort (budget and actual) to produce deliverable Variance analysis yields insight into effort for other similar work Manage work around deliverables Provides more better variance control Supports more accurate forecasts Provides information for program direction and control focus




ED

S 4

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Alarms Processing

FAT

EMS 2A

ED

S 1

ED

S 2

ED

S 3

FAT – Rel 3

EMS 2B EDW 1

FAT – Release 4

NMMS 1 NMMS 2 EIP 1

FAT – Release 5

REG 1 MMS 1

EMS 2C

EIP 2 S&B 1MIS 1EIP 3

EDW 2

FAT – Release 6

EMS 3EDW 3

EIP 4

FAT – Release 7

EDW 4CRR 1CMM 1 EIP 5

FAT – Release 8

MMS 2S&B 2

EDW 5 EIP 6

FAT – Release 9

MMS 2S&B 2

OS

NMMS 3 EMS 4EDW 6

EIP 7EMS 5 DIS

P2P Checkout

SE Verification

ITEST

RTM Go-Live

DAM Go-Live

EDS 2 Release 3: SE VerEMS- State Estimator & Tel Perf Stds,NSA, SPS/RAP, SCADA and DynRatings for Single Model

EDW– SE Statistics

EDS 2 Release 4: NM VerNMMS- Network Model, NOMCR

EMS- EMS /NMMS Interface

EIP

EDS 3 Release 5: LMPsMMS– SCED

EMS- NSA, RLC

COMS- S&B (FIP, FOP,Ver Costs)- Registration

EIPMISEDW

EDS 3 Release 6: LFC TestingEMS- LFC

EIPEDW

EDS 3 Release 7: CRRCRRCOMS- CMM

EIPEDW

EDS 4 Release 8: RT SettlementsMMS- RT Settlements InterfaceS&BEIP, EDWEDS 4 Release 9: DAM, RUCMMS- All Markets- OSEMS – Load Forecast, Out EvalNMMSS&BEIP, EDW

ITEST

ITEST

LegendIntegration TestingProject FAT Activities

Settlements Statements

Upload Bids

Test Settlements

Outages DAM, RUC, SASMFull Nodal Functionality

24*7 Operation 168 Hr Test

NOMCR Chk out Network Model Verification

Rel 1

Rel 2

EDS 1 Release 1:EMS COTS

EDS 1 Release 2:ICCP Upgrade

Rel 3

FAT

EDS

FAT

ITEST

EDS

FAT

ITEST

EDS

SCED TestingRegUpload Offers

Rel 4

Rel 5

Rel 7

Rel 6

6 month LMP Posting

CRR Data Validation Trial Ops of CRR

LFC Comms Testing

ITEST

EDS

FAT

FAT

EDS

ITEST

ED

S 4 Go-Live

Activities

EDS 1 EDS 2 EDS 3 EDS 4

MP Interface Spec

Start of EDS activities

FAT milestones

EDS Preparatory Milestones

Rel 8

Rel 9

LFC Testing

ReadinessDeclaration

Readiness Declarations

Early Start

The Early Delivery Software (EDS) releases - involve multiple iterations of design and build activities – albeit to a small degree

Results of test iterations result in changes to design and build of the systems

Estimate risk should decrease after each EDS

Estimate Risk will continue exist until all work is completed

Today


Documents

Global Business Services © Copyright IBM Corporation 2007ERCOT and IBM Confidential | 23-Sep-15 | External Review of Nodal Program PMO Control System Executive