ERO Enterprise and Corporate MetricsQuarter 2 Status

Mark Lauby, Senior Vice President and Chief Reliability OfficerCorporate Governance and Human Resources Committee MeetingAugust 10, 2016

2 RELIABILITY | ACCOUNTABILITY

Leading at Q3

Metric 1: Reliability Results• No Category 4 or 5 events Metric 3: Risk Mitigation Effectiveness• 3.2-Extreme Physical Events• 3.4-Protection System Misoperations• 3.6-Model Building• 3.7-Equipment PerformanceMetric 4: Program Execution Effectiveness• 4B-Standards Guidance and Training or Outreach

3 RELIABILITY | ACCOUNTABILITY

Metric 2: Assurance Effectiveness• One Category 3 event occurred; gap analysis needed

Metric 3: Risk Mitigation Effectiveness• 3.1-Changing Resource Mix: Long-term Reliability Assessment (LTRA)

collaboration with Regional Entities to be completed • 3.5-Extreme Conditions: Monitoring for intervention strategies

Metric 4: Program Execution Effectiveness• 4A-Reliability Standards and Cost-effectiveness: Standards grading to be

incorporated in risk-based input• 4C-Registration: Documented review to be completed• 4E-Operating Model: Dependent on Registration oversight, review of

Certification, and user management completion• 4F-Effectiveness Survey: Action plan approval pending• 4G-GridEx: Updating NERC Crisis Action Plan

Watching in Q3

4 RELIABILITY | ACCOUNTABILITY

Metric 3: Risk Mitigation Effectiveness• 3.3-Cybersecurity Preparedness: User Communities pushed to 2017Metric 4: Program Execution Effectiveness• 4D-CMEP: Watching mitigations and IRA completions

Emerging Activities in Q3

ERO Enterprise Effectiveness Survey Results and Action PlansKristin Iwanechko, Associate Director, Member Relations and MRC SecretaryCorporate Governance and Human Resources Committee MeetingAugust 10, 2016

RELIABILITY | ACCOUNTABILITY2

• Sub-metric F• Issued May 2016 Survey questions modified from January 2015 survey 72 survey questions across 12 topic areas

• Ratings on a 5-point scale 1 – Strongly Disagree 2 – Disagree 3 – Neither Disagree nor Agree 4 – Agree 5 – Strongly Agree

Background

RELIABILITY | ACCOUNTABILITY3

• 265 survey respondents 235 represented 368 NCR numbers 30 did not represent an NCR number

• Positive results overall Average ratings from 2.94 to 4.20

Results and Analysis

RELIABILITY | ACCOUNTABILITY4

Highest Rated Items

4.20

4.19

4.14

4.14

4.13

1 2 3 4 5

The personnel certification program is valuable inpromoting skilled, trained, and qualified BPS operators.

The audit process (including pre- and post-audit activities) for my entity’s most recent audit was well-

defined, organized, and followed established procedures.

The audit report from my most recent audit identifiedclear, definitive, and actionable items to address.

My organization trusts the E-ISAC.

The E-ISAC shares information on threats and incidents(e.g., bulletins, assessments, weekly/monthly reports,

etc.) that is reliable.

RELIABILITY | ACCOUNTABILITY5

Lowest Rated Items

3.40

3.36

3.27

3.20

2.94

1 2 3 4 5

The ERO Enterprise ensures efficiencies and minimizesduplication and activities not affecting reliability

outcomes.

There is timely and effective response during the RegistryAppeals Process.

The Reliability Standard requirements are clearly stated.

Reliability Standards are practical to implement.

Reliability Standards address risk to reliability in a cost-effective manner.

RELIABILITY | ACCOUNTABILITY6

Most Favorable Items

247

117

151

194

228

13

11

19

16

27

11

5

2

11

8

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

The audit process (including pre- and post-audit activities) for my entity’s most recent audit was well-

defined, organized, and followed established …

The personnel certification program is valuable inpromoting skilled, trained, and qualified BPS operators.

The E-ISAC shares information on threats and incidents(e.g., bulletins, assessments, weekly/monthly reports,

etc.) that is informative.

The ERO Enterprise communicates information aboutReliability Standards, their development, and

opportunities for stakeholder participation in an…

The audit report from my most recent audit identifiedclear, definitive, and actionable items to address.

Favorable Neutral Unfavorable

RELIABILITY | ACCOUNTABILITY7

Most Unfavorable Items

117

91

82

106

64

25

46

97

60

84

26

30

42

55

73

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

The organization registration process is well-definedand the definitions are clear.

The E-ISAC is my primary resource for threatinformation and analysis.

Reliability Standards are practical to implement.

The Reliability Standard requirements are clearly stated.

Reliability Standards address risk to reliability in a cost-effective manner.

Favorable Neutral Unfavorable

RELIABILITY | ACCOUNTABILITY8

• Subset of 37 questions• Statistically significant increase (3.42 to 3.77) The risk-based registration process ensures that appropriate entities are

registered or deregistered commensurate with risk to the Bulk Electric System.

• Statistically significant decrease (3.93 to 3.61) The implementation of spot checks was well-defined, organized, and

followed established procedures.

Year-Over-Year Analysis

RELIABILITY | ACCOUNTABILITY9

• Five lowest-rated items• Five most unfavorable items• Items with higher than 6% unfavorable ratings and also lower

than 60% favorable ratings• Items with statistically significant decrease in year-over-year

comparison• Themes from anonymized comments

Analysis to Determine Areas for Focus

RELIABILITY | ACCOUNTABILITY10

• Regional consistency in compliance monitoring and enforcement

• Regional consistency in organization registration• Cost-effectiveness, clarity, and practicality of Reliability

Standards

Action Plans

RELIABILITY | ACCOUNTABILITY11

• Compliance Monitoring Developing consistent templates Consistency in the multi-region registered entity (MRRE) program Regional Entities’ implementation of risk-based compliance monitoring Monitoring approaches

• Compliance Enforcement Process reviews Assessing the risk of noncompliance Settlements and penalty determinations Developed MRRE templates for enforcement activities

• 2016 ERO Enterprise and Corporate Metrics Sub-metric D: implementation of risk-based CMEP Sub-metric E: develop and implement oversight plan modifications

Regional Consistency in Compliance Monitoring and Enforcement

RELIABILITY | ACCOUNTABILITY12

• NERC-led panel• Common registration form• ERO Registration Procedure and Risk-Based Registration

Implementation Guidance posted• Risk-based approach to review registration documents used by

Regional Entities• Review JRO/CFR processes during Regional workshops• 2016 ERO Enterprise and Corporate Metrics Sub-metric C: review structure and consistency of registration program Sub-metric E: develop and implement oversight plan modifications

Regional Consistency in Organization Registration

RELIABILITY | ACCOUNTABILITY13

• Enhanced periodic reviews• Common Reliability Standard Metric• Cost-effectiveness method• 2016 ERO Enterprise and Corporate Metrics Sub-metric A: develop and pilot method for determining cost effectiveness

Cost-effectiveness, Clarity, and Practicality of Reliability Standards

RELIABILITY | ACCOUNTABILITY14

• Request CGHRC acceptance of final report and action plans no later than November 2016

• Complete actions as described in the action plans• Next survey to be issued in 2018

Next Steps

RELIABILITY | ACCOUNTABILITY15