Loss of EMS Events 2015 System Operator Seminar. Training Objectives 2 System Operators will be able to identify the ERO Event Analysis Process Category

Loss of EMS Events2015 System Operator Seminar

2

Training Objectives

• System Operators will be able to identify the ERO Event Analysis Process Category associated with a loss of EMS event

• System Operators will be able to demonstrate through a paper exercise the steps to take in response to a loss of Energy Management System (EMS) Event

3

NERC Event Analysis of EMS Event

ERO Event Analysis Process EMS Event Categories 1f - Unplanned evacuation from a control center facility with BPS SCADA functionality for 30 minutes or more.1h - Loss of monitoring or control, at a control center, such that it significantly affects the entity’s ability to make operating decisions for 30 continuous minutes or more. 2b - Complete loss of SCADA, control or monitoring functionality for 30 minutes or more.

4

Examples of 1h Events

Examples include, but are not limited to the following: i. Loss of operator ability to remotely monitor, control Bulk Electric System (BES) elements, or both. ii. Loss of communications from SCADA RTUs iii. Unavailability of ICCP links reducing BES visibility iv. Loss of the ability to remotely monitor and control generating units via AGC v. Unacceptable State Estimator or Contingency Analysis solutions

5

Loss EMS Events

Category FRCC NERC (including FRCC)1f 1 211h 2 752b 10 110Total 13 206

Category FRCC NERC (including FRCC)1 17 4072 11 1363 154 35 2Total 28 563

Category% NERC EMS Event

to All Events1 24%2 81%All 37%

Data for all qualifying events from October 2010 through December 2014.

6

Loss of EMS Events

Are loss of EMS events impactful to System Operators?

Have loss of EMS events been impactful to the BES?

7

Loss of EMS Events

Reasons for EMS events not having BES impact• Most commonly occur during maintenance

• Planned for off peak hours and weekends with fairly flat loads • EMS staff is nearby • Heightened SO awareness

• Preparation and good SO action• Processes and Procedures • Training (Practice makes perfect)• Interconnected Systems

8

FRCC RC Conservative Operations

Purpose:• Implemented to maximize the FRCC’s ability to operate

the BES reliably during limited or complete loss of EMS monitoring capabilities

• Identifies triggers to implement Conservative Operations

• Identifies actions and communications the RCSO should take during partial or complete loss of EMS/SCADA or RTCA

Procedure title: FRCC Reliability Coordinator Conservative Operations

Procedure FRCC-MS-OPRC-029

9


• RCSO should declare Conservative Operations (Ops) if RC’s EMS, Data Acquisition, or RTCA is lost and ETR is >30 minutes or if RCSO has a reliability concern during loss of functionality

• In this presentation we will refer to this as a loss of functionality• RCSO verify on duty supervisor is notified, request ETR, and (if

applicable) ask about a plan to failover

• RCSO notify SE RC of loss of functionality and potential call for Conservative Ops

Estimated Time of Repair (ETR)

Southeastern (SE)

10


Operating Entities (OEs) as referenced in the RC Conservative Operations Procedure• Balancing Authorities (BAs)• Distribution Providers (DPs)• Generator Owners (GOs)• Generator Operators (GOPs)• Load Serving Entities (LSEs)• Transmission Operators (TOPs)• Transmission Owners (TOs)

11


• RCSO to request SE RC notify RCSO of any new contingencies and schedule discrepancies of >200 MWs until functionality is back

• RCSO to notify all OEs via hotline and FTMS of loss of functionality and Conservative Ops could be called• OEs to acknowledge message over hotline

• RCSO to instruct all BAs to maintain ACE as close to zero as possible and notify RCSO of deviations >±50MWs

12


• OEs to notify RCSO of any new contingencies

• OEs to notify RCSO if they get nominal voltages exceed or if they have any real-time overloads on their transmission facilities• For nuclear switchyards, provide notification to the RCSO if the local transmission station voltage is outside of established limits.

13


• RCSO to notify the MRCOO (or designee) of loss of functionality and Conservative Ops may be called

• RCSO to periodically check the RTCA internet sharing site to view other TOP’s CA output

• RCSO should log all pertinent data and OE reports received in a time sequenced manner

14


If the loss of functionality ETR is unknown or >30 minutes, the RCSO should declare Conservative Ops. Actions are similar but the RCSO will notify that Conservative Ops is being implemented

Additional steps for SE RC: • In addition to previous notify SE RC that Conservative Ops is

implemented• RCSO to contact SE RC for updates on actual vs. scheduled net

interchange (ANI vs. SNI) in 15 minute intervals• RCSO update SE RC on status of functionality every 60 minutes until

Conservative Ops is terminated

15


Additional steps for OEs:

• RCSO to notify OEs via hotline and FTMS of loss of functionality and that Conservative Ops is being implemented• Request acknowledgement

• OEs to provide ACE on the hotline • RCSO to review ACE data against the data provided by SE RC

• RCSO request off schedule BAs to correct ACE and instruct FPL to redispatch to make up remaining discrepancies

• RCSO to update the OEs on the status of the EMS every 60 minutes until the Conservative Operations have been terminated

16


RCSO to notify MRCOO (or designee) of loss of functionality and implementation of Conservative Ops

RCSO should:• Verify with Next-day planner that outaged lines have been

returned to service• Defer any planned outages that haven’t been taken until

Conservative Ops is terminated

17


RCSO should:• Assess the need to reduce imports/exports with OEs• Coordinate any interface reductions with interface owners• Coordinate with OEs and assess the need to return to

service any planned outage

18


When EMS/SCADA or RTCA is back:• RCSO to terminate Conservative Ops and notify SE RC,

OEs and MRCOO

19

“Conservative Operations”

All OEs need to understand what it means when the RC calls for Conservative Operations

20


FRCC RC requests:- OEs acknowledge the message on the hotline- BAs maintain ACE as close to zero as possible and notify

FRCC RC of any deviations > 50 MWs- OEs notify FRCC RC of any new contingencies- OEs notify FRCC RC if nominal voltages exceed alarm

limits, or if any real time overloads exist.- OEs notify FRCC RC if the local transmission stations on

nuclear switchyards is outside of established limits

21


• Internally each company may define Conservative Operations differently• It’s also important to understand what it means within

your company

22

Loss of EMS Exercise

• An automated planned disk management activity began to run on HPC EMS server• EMS was operating under normal conditions

• A file system error occurred on the primary EMS server which causes a critical process to stop and the server becomes hung• This causes the replication of data to the backup EMS

servers to be unavailable

• EMS-2, EMS-3, and EMS-4 become disabled

23


Physical Relocation Was not an Option!!!• Primary EMS server remained enabled

• Therefore no automatic fail-over took place• Gateway server lost it’s connection with the primary server• Memory replication services application on the primary

EMS server losses it’s connection to the three other servers that are designed to step in and take over

• The three backup EMS servers become invalid and unavailable for failover (status disabled)

24


• FRCC RC and HPC share an EMS• When HPC losses EMS functionality it usually means the RC will too

• HPC and the RC lost monitoring and control of their EMSs for over 1 hour

• HPC and the RC both called for Conservative Ops• RC worked with Southeaster RC to monitor the state

ACE

25


Assessment Instructions• Three roles Reliability Coordinator, HPC Balancing

Coordinator and HPC System Operator• Checklist for each• Sequence of Events provided• Complete your work package and make sure to get a

signature before you break

26

Questions?

Documents

Loss of EMS Events 2015 System Operator Seminar. Training Objectives 2 System Operators will be able to identify the ERO Event Analysis Process Category