Advantage Power Pricing Pilot · Advantage Power Pricing Pilot Confidential and Proprietary Page 6 ©2016 Navigant Consulting, Inc. Do not distribute or copy DISCLAIMER This report

Advantage Power Pricing Pilot

Confidential and Proprietary Page i ©2016 Navigant Consulting, Inc. Do not distribute or copy


Impact and Process Evaluation: Winter 2015/2016 and Summer 2016

Prepared for:

Alectra Utilities

161 Cityview Blvd.

Vaughan, ON L4H 0A9

T: 905.532.4417

F: 905.532.4447 www.alectrautilities.com

Submitted by: Navigant Consulting Ltd.

Bay Adelaide Centre,

333 Bay St, Suite 1250

416.927.1641

navigant.com

Reference No.: 183051

July 4, 2017

http://www.alectrautilities.com/


Confidential and Proprietary Page 1 ©2016 Navigant Consulting, Inc. Do not distribute or copy

TABLE OF CONTENTS

Disclaimer ...................................................................................................................... 6

Executive Summary ...................................................................................................... 7

Key Findings and Recommendations 8 Program Overview 9 Evaluation Approach 11 Impact Evaluation Findings 12 Process Evaluation Findings 15

1. Introduction ............................................................................................................. 18

1.1 Program Overview 18 1.1.1 APP Design Principles.................................................................................................... 19 1.1.2 APP Rates ...................................................................................................................... 20 1.1.3 APP Technology ............................................................................................................. 21 1.1.4 APP Participants ............................................................................................................. 23 1.1.5 APP Events and Pricing ................................................................................................. 25

1.2 Objectives of the Evaluation 28 1.2.1 Impact Evaluation Objectives ......................................................................................... 28 1.2.2 Process Evaluation Objectives ....................................................................................... 29

2. Evaluation Approach .............................................................................................. 30

2.1 Impact Evaluation 30 2.1.1 Regression with Pre-Program Processing ..................................................................... 30 2.1.2 Control Group Selection ................................................................................................. 31 2.1.3 Hourly Demand Impact Estimation ................................................................................. 33 2.1.4 Daily Energy Impact Estimation ..................................................................................... 36

2.2 Process Evaluation Approach 37

3. Impact Findings ....................................................................................................... 40

3.1 Event Demand Impacts 41 3.1.1 Overall Average Event Impacts - Winter ........................................................................ 41 3.1.2 Overall Average Event Impacts - Summer ..................................................................... 43 3.1.3 Hourly Event Impacts – Winter 2015/2016 ..................................................................... 45 3.1.4 Hourly Event Impacts – Summer 2016 ........................................................................... 48 3.1.5 Average Event Impacts by Conservation Setting ........................................................... 53

3.2 System Peak Demand Impacts 57 3.3 Energy Impacts 61 3.4 Participant Cost Impacts 67

3.4.1 Commodity Cost Impacts ............................................................................................... 67 3.4.2 Participant Rebates ........................................................................................................ 70

3.5 Conservation Settings Over Time 72

4. Process Evaluation Findings ................................................................................. 75

4.1 Participant Demographics 76 4.2 Motivations for Program Participation 79

4.2.1 Engaged and Partially Engaged Respondents .............................................................. 79



4.2.2 BYOD/Yukon and Energate Respondents ..................................................................... 80 4.2.3 Risk-Free Aspect of the APP Program ........................................................................... 81

4.3 Enrollment Process 82 4.3.1 Engaged and Partially Engaged Respondents .............................................................. 82 4.3.2 BYOD/Yukon and Energate Respondents ..................................................................... 83

4.4 Participant Technology Engagement 84 4.4.1 Frequency of Viewing Electricity Pricing Information ..................................................... 84 4.4.2 Frequency of Acting Upon Electricity Pricing Information .............................................. 87 4.4.3 Methods of Viewing Electricity Pricing Information ........................................................ 89

4.5 Customer Satisfaction 92 4.5.1 Participants’ Experience with the APP ........................................................................... 92 4.5.2 Participants’ “Most-liked” aspects of the APP program .................................................. 94 4.5.3 Participants’ “Least-liked” aspect of the APP program ................................................... 95

4.6 Participant Support of the Initiative 97 4.6.1 Likelihood to Recommend APP Program to Friends and Family ................................... 97 4.6.2 Likelihood to Recommend the Energate System ........................................................... 99

5. Conclusion & Recommendations ........................................................................ 101

Attached as separate documents: Appendix A: APP Hourly Load Plots (PDF) Appendix B: APP Hourly Impacts (XLSX)



TABLE OF FIGURES

Figure 1: APP Prices, November 2015 through October 2016 .................................................................. 21 Figure 2: RPP TOU Prices, November 2015 through October 2016 ......................................................... 21 Figure 3: Conservation Setting Offsets and APP Prices ............................................................................ 22 Figure 4: Yukon Cycling Strategies ............................................................................................................ 23 Figure 5: APP Enrollment Over Time, By Technology ............................................................................... 23 Figure 6: Participant Density Map by FSA (All Participants) ...................................................................... 24 Figure 7: Participant Density Map by FSA (All Participants) ...................................................................... 25 Figure 8: APP Price Level Days in Winter of 2015/2016 ........................................................................... 26 Figure 9: APP Price Level Days in Summer of 2016 ................................................................................. 27 Figure 10: Final Distribution of Hours by Price .......................................................................................... 27 Figure 11: Day-Type Average Daily Temperature Thresholds .................................................................. 32 Figure 12: Participant Surveys – Timing and Completion Rate ................................................................. 38 Figure 13: Process Evaluation Overview ................................................................................................... 39 Figure 14: Average Program Impacts by Price – Winter 2015/2016 ......................................................... 42 Figure 15: Average Program Impacts by Price – Summer 2016 ............................................................... 43 Figure 16: HIGH Price Summer APP Period Impacts Compared to 50% Cycling Residential DR Impacts

................................................................................................................................................................... 45 Figure 17: Average APP High Day – Winter, Energate (Standard) ........................................................... 46 Figure 18: Average APP High Day – Winter, Energate (Pioneer) ............................................................. 47 Figure 19: Winter CPP Event, Energate (Standard), January 20, 2016 .................................................... 47 Figure 20: Winter CPP Event, Energate (Standard), January 20, 2016 .................................................... 48 Figure 21: Average APP High Day – Summer, Energate (Standard) ........................................................ 49 Figure 22: Summer CPP Event, Energate (Standard), August 11, 2016 .................................................. 50 Figure 23: Average APP High Day – Summer, BYOD .............................................................................. 51 Figure 24: Summer CPP Event, BYOD, August 11, 2016 ......................................................................... 52 Figure 25: Summer CPP Event, Yukon, August 11, 2016 ......................................................................... 53 Figure 26: Average Event Impact (kW) by Conservation Setting and Technology – Winter ..................... 54 Figure 27: Average Event Impact (%) by Conservation Setting and Technology – Winter ....................... 55 Figure 28: Average Event Impact (kW) by Conservation Setting and Technology – Summer 2016 ......... 55 Figure 29: Average Event Impact (kW) by Conservation Setting and Technology – Summer 2015 ......... 56 Figure 30: Distribution of Critical Peak Period Conservation Setting, Summer 2015 vs 2016 .................. 56 Figure 31: Average Event Impact (%) by Conservation Setting and Technology – Summer 2016 ........... 57 Figure 32: Seasonal Peak Hour Schedule and Associated Prices ............................................................ 58 Figure 33: Average Participant Impacts (kW) During Winter System Peak Coincident Hours .................. 59 Figure 34: Average Participant Impacts (kW) During Winter System Peak Coincident Hours .................. 60 Figure 35: Average Participant Impacts (kW) During Winter System Peak Coincident Hours by

Technology and Conservation Setting ....................................................................................................... 61 Figure 36: Average Participant Impacts (kW) During Summer System Peak Coincident Hours by

Technology and Conservation Setting ....................................................................................................... 61 Figure 37: Winter Energy Savings (kWh) by Conservation Setting and Technology ................................ 62 Figure 39: Winter Energy Savings (%) by Conservation Setting and Technology .................................... 63 Figure 38: Summer Energy Savings (kWh) by Conservation Setting and Technology ............................. 64 Figure 40: Summer Energy Savings (%) by Conservation Setting and Technology ................................. 64 Figure 41: Hourly Impact Example Table – BYOD Summer High Price Days ........................................... 66 Figure 42: Average Total Summer Commodity Cost Savings ($) .............................................................. 68 Figure 43: Average Total Summer and Winter Commodity Cost Impact (%) ............................................ 69 Figure 44: APP Summer Rebate Incentive Summary Statistics ................................................................ 70



Figure 45: Distribution of APP Rebates by Technology ............................................................................. 72 Figure 46: Weekly Conservation Setting Distribution Over Time ............................................................... 73 Figure 47: Conservation Settings, Summer 2015 vs Winter and Summer 2015/2016 .............................. 73 Figure 48: Conservation Setting Distribution – First and Last Week of Period of Analysis ....................... 74 Figure 49: Participant Conservation Setting Changes Over Time ............................................................. 74 Figure 50. Respondent Age Bracket (All Respondents) ............................................................................ 77 Figure 51. Advertisement that Motivated Participation (Engaged vs. Partially Engaged) ......................... 78 Figure 52. Full-time Residents in Participant Households (All Respondents) ........................................... 78 Figure 53. Participants’ First Experience with Conservation Programs (All Respondents) ....................... 79 Figure 54. Motivations for Program Participation (Engaged vs. Partially Engaged) .................................. 80 Figure 55. Motivations for Program Participation (BYOD/Yukon vs. Energate) ........................................ 81 Figure 56. Importance of “risk-free” aspect of APP program, in decision to participate (2015 vs. 2016

Process Evaluation) ................................................................................................................................... 81 Figure 57. Sufficiency of Initial Information for Enrollment (Engaged vs. Partially Engaged) ................... 82 Figure 58. Ease of Signing up for APP (Engaged vs. Partially Engaged) ................................................. 83 Figure 59. Sufficiency of Initial Information for Enrollment (BYOD/Yukon vs. Energate) .......................... 83 Figure 60. Ease of Signing up for APP (BYOD/Yukon vs. Energate) ........................................................ 84 Figure 61. Frequency of Viewing Electricity Pricing Information Available Through the Program (Engaged

vs. Partially Engaged) ................................................................................................................................ 85 Figure 62. Frequency of Viewing Electricity Pricing Information Available through the Program

(BYOD/Yukon vs. Energate) ...................................................................................................................... 85 Figure 63. Frequency of Viewing Electricity Pricing Information Available through the Program (Survey 1

vs. Survey 4) .............................................................................................................................................. 86 Figure 64. Preference for Communication of APP related information (All Respondents) ........................ 87 Figure 65. Frequency of Acting Upon Electricity Pricing Information (Engaged vs. Partially Engaged) .... 88 Figure 66. Frequency of Acting Upon Electricity Pricing Information (BYOD/Yukon vs. Energate) .......... 88 Figure 67. Frequency of Acting Upon Electricity Pricing Information (Survey 1 vs. Survey 4) .................. 89 Figure 68. Methods for Viewing Electricity Pricing and Consumption Information (Engaged vs. Partially

Engaged) .................................................................................................................................................... 90 Figure 69. Methods for Viewing Electricity Pricing and Consumption Information (BYOD/Yukon vs.

Energate).................................................................................................................................................... 91 Figure 70. Change in Interaction Level with Energate Thermostat and Portal throughout Period (All

Respondents) ............................................................................................................................................. 91 Figure 71. Participants’ Experience with APP including registration, installation, customer support, etc.

(Engaged vs. Partially Engaged) ............................................................................................................... 92 Figure 72. Participants’ Experience with APP including registration, installation, customer support, etc.

(BYOD/Yukon vs. Energate) ...................................................................................................................... 93 Figure 73. Participants’ Experience with APP including registration, installation, customer support, etc.

(Survey 1 vs. Survey 4) .............................................................................................................................. 93 Figure 74. Participants’ “Most-liked” Aspects of APP (Engaged vs. Partially Engaged) ........................... 94 Figure 75. Participants’ “Most-liked” Aspects of APP (BYOD/Yukon vs. Energate) .................................. 95 Figure 76. Participants’ “Least-liked” Aspects of APP (Engaged vs. Partially Engaged) .......................... 95 Figure 77. Participants’ “Least-liked” Aspects of APP (BYOD/Yukon vs. Energate) ................................. 96 Figure 78. Effectiveness of Customer Care Centre in offering Technical Support (All Respondents) ...... 97 Figure 79. Likelihood of Participant Recommending APP to Friends and Family (Engaged vs. Partially

Engaged) .................................................................................................................................................... 98 Figure 80. Likelihood of Participant Recommending APP to Friends and Family (BYOD/Yukon vs.

Energate).................................................................................................................................................... 98 Figure 81. Likelihood of Participant Recommending APP to Friends and Family (Survey 1 vs. Survey 4)99 Figure 82. Energate Participants’ Likelihood to Recommend Energate System (Energate Respondents)99



Figure 83. Energate Participants Expect to Use Energate System one year from now in the (Engaged vs.

Partially Engaged) .................................................................................................................................... 100



DISCLAIMER

This report was prepared by Navigant Consulting, Inc. (Navigant) for PowerStream Inc. The work

presented in this report represents Navigant’s professional judgment based on the information available

at the time this report was prepared. Navigant is not responsible for the reader’s use of, or reliance upon,

the report, nor any decisions based on the report. NAVIGANT MAKES NO REPRESENTATIONS OR

WARRANTIES, EXPRESSED OR IMPLIED. Readers of the report are advised that they assume all

liabilities incurred by them, or third parties, as a result of their reliance on the report,



EXECUTIVE SUMMARY

The Advantage Power Pricing (APP) is an experimental pilot program designed to test the response of

technology-enabled residential customers to dynamic electricity prices. The central guiding principles of

the program design being tested by this pilot are to empower customers to achieve bill savings and to

contribute toward the Ontario Ministry of Energy’s long-term demand response (DR) goals through

customer price-response.

The program design was developed by Alectra and Energate in close consultation and collaboration with

staff from Ontario’s Independent Electricity System Operator (IESO), the Ontario Ministry of Energy, and

the Ontario Energy Board (OEB). The pilot design is closely modeled on the Oklahoma Gas and Electric

(OG&E) SmartHours program, modified to reflect the locally-specific needs of the Ontario market and

(beginning in the winter of 2015/2016) to explore the potential of non-Energate control technologies.

The summer1 2015 pilot was funded by the Ministry of Ontario Smart Grid Fund and through contributions

(in resources and in kind) by Energate, PowerStream, Util-Assist and Environics Analytics. Additional

funding for the installation of Energate thermostats and related load control equipment was provided by

the peaksaver PLUS® residential demand response initiative. In autumn of 2015, the pilot was extended

to cover the winter of 2015/2016 and the summer of 2016 with funding provided by the IESO

Conservation Fund.

Since its original inception, the pilot has been extended to cover the winter of 2015/2016 and the summer

of 2016 with funding provided by the IESO Conservation Fund.

This report represents Navigant’s evaluation of the APP pilot for the summer of 2016 and Winter of 2015-

2016. It is comprised of two evaluation elements:

1. Impact Evaluation. The estimation of program demand and energy impacts through the use of

participant and non-participant hourly electricity (AMI/SmartMeter) consumption data and

econometric modeling.

2. Process Evaluation. An evaluation of the effectiveness of the APP design and delivery

mechanism in meeting programmatic objectives and an assessment of the pilot’s strategic

possibilities based on an analysis of five separate participant surveys.

This executive summary is divided into five sections:

Key Findings and Recommendations. This section provides the most important impact and

process findings of the evaluation, and provides recommendations for adjustments in program

design to improve the program’s ability to meet its twin goals of customer engagement and peak

demand reductions.

Program Overview. This section briefly summarises the APP program design and

implementation since it’s inception in the summer fo 2015.

Evaluation Approach. This section briefly describes the analytic approach used by the impact

and process evaluations teams.

1 Unless stated otherwise, all references to summer in this report should be understood to refer to the Ontario Energy

Board’s Regulated Price Plan “summer” that runs from May 1st through to the end of October.



Impact Evaluation Findings. This section summarises the most important findings of the impact

evaluation.

Process Evaluation Findings. This section summarises the most important findings of the

process evaluation.

Key Findings and Recommendations

Navigant’s key finding of its evaluation of the APP for the winter of 2015/2016 and the summer of 2016

pilot are:

1. The Yukon technology group delivered the most substantial savings during the coincident

system peak hour. Former peaksaver participants equipped with the Yukon software delivered a

DR impact of 0.81 kW during the system peak hour on September 7th, 2016. During another one

of the five system coincident peak hours, on July 13th, this group delivered 1.2 kW of impacts.

2. DR savings delivered by Energate participants varied substantially depending on the

conservation setting selected. The Energate participants with the most aggressive

conservation settings delivered nearly 2 kW during the system coincident peak hour on July 13th.

By contrast, participants selecting the “Balanced” conservation setting (approximately 40% of

Energate participants with tracked conservation setting data) delivered approximately 1.3 kW of

DR, and participants selecting the “Max Comfort” conservation setting (approximately 30% of

Energate participants with tracked conservation setting data) delivered only 0.12 kW of DR. The

overall average impacts of the Energate group naturally reflects this distribution of savings.

3. On average participants reduced their winter commodity costs by between 9% and 27%

and their summer commodity costs by between 0 and 10% (depending on the technology

group). Winter savings were in large part driven by the small number of higher priced days.

4. BYOD participants deliver the most consistent inter-seasonal impacts. BYOD participants’

DR contributions were consistent in both summer and winter, and in fact delivered the second-

highest impacts (after Pioneer customers) in the winter.

5. The APP participant population skews older and is principally motivated by bill savings.

Fifty-nine percent of participants are 50 years or older, and approximately 80% indicated that their

primary motivation for participation was to save money.

6. APP has been successful in penetrating a segment of PowerStream’s customer-base

previously unengaged with utility-based energy efficiency efforts. Nearly 70% of APP

participants indicated that they had not previously participated in any PowerStream offered

Conservation and Demand Management (CDM) initiative.

7. Nearly 80% of respondents said that the risk-free aspect of the program was very

important in their decision to enroll. Maintaining the first-year risk-free “commodity cost

guarantee”2 is likely to be essential for any successful wider program roll-out.

Based these six key findings, Navigant has the following recommendations for improving the APP design

and maximizing the value of future impact evaluations in future years:

2 PowerStream provides participants with a guarantee that their electricity commodity costs will not be higher than

under standard RPP TOU rates for the first year of participation. See below for more details.



1. The IESO should be more directly involved in call Critical periods. One of the 5CP hours of

the summer occurred immediately following the end of a Critical peak period. As a result of

snapback, the Yukon group contributed much less DR than it potentially could have for this event.

2. Alectra should recommend to BYOD customers that they pre-cool their homes if they

respond to prices with thermostat adjustment. BYOD participants could potential offer

substantially more DR during regularly scheduled summer events, provided they pre-cooled to

ensure DR impacts do not decay, and comfort does not become an issue.

3. Alectra should adjust Energate conservation setting responsiveness such that even the

least aggressive conservation settings contribute some DR. Many participants selecting

these less aggressive settings are effectively free-riding on the program and contributing trivial

amounts of DR but taking advantage of the bill protection and APP schedule to reduce their

average cost of electricity without benefiting Alectra or the system.

4. Alectra should consider leveraging its link with participants to promote additional energy

efficiency initiatives. Three quarters of APP participants have not previously engaged with other

Alectra-offered CDM initiatives, indicating a substantial marketing opportunity for other, non-APP

CDM initiatives. Given the high-levels of program satisfaction indicated by participants, such

participants are likely to be receptive to messaging regarding the incremental benefits such

programs could offer them. Given the importance to survey respondents of bill savings, these

participants are likely to be most receptive to high-impact, low-cost programs (e.g., LED lighting

retrofits, timers for dehumidifiers, behavioural programs, etc.)

5. Alectra should continue to provide participants with ongoing energy management advice

to help participants optimize their APP response to the twin goals of reducing system

peak demand and achieving bill savings. Participant impressions of the initiative as well as the

extent to which they interacted with the program’s tools consistently increased throughout the

program period. Maintaining this on-going engagement will be key to maintaining and improving

program achievements over time.

Program Overview

The APP pilot design was developed by Alectra and Energate in collaboration and consultation with the

Ministry of Energy, the IESO and the OEB. As part of the design phase, all of the parties agreed that the

structure of the pilot should be determined by a number of key design principles.

The core philosophy embodied by these principles was that the pilot design should empower, and not

compel, customers, to deliver demand response by offering them an opportunity to realize meaningful bill

savings through price response in a manner consistent with the goals of Ontario’s Long Term Energy

Plan (LTEP). As such, a key imperative of the pilot was that prices to which participants were subject, and

the periods these prices were applied, should be reflective of, and responsive to, IESO-identified system

needs.

Customers enrolled in the APP remained subject to the standard Regulated Price Plan (RPP) time-of-use

(TOU) rates, were billed by Alectra under these rates and were required to pay them as usual. In addition,

participants received a monthly “shadow bill” that tracked what their bill would have been under the APP

rate. At the close of the summer, if participants’ shadow bills were lower than their actual TOU bills, they

received the difference from Alectra as a rebate. Participants whose APP shadow bill exceeded their

standard TOU bill were not penalized.



To support participant price-response, all customers enrolled in the program for the summer of 2015 were

equipped with an Energate Foundation that allowed participants to automate their thermostat’s response

to APP price fluctuations. Beginning in the winter of 2015/20163, three alternative technology groups were

added to the program:

Energate Pioneer. A small group of customers (26) were provided with the Energate Foundation,

and a set of Energate’s Pioneer thermostats. The Pioneer thermostats extends the Energate’s

automatic thermostat control to independently controlled electric baseboard heaters.

Yukon. Nearly 200 former peaksaver PLUS® participants were recruited into the APP. These

participants’ legacy peaksaver thermostats were controlled using the Cooper Power Systems

Yukon Advanced Energy Services Platform. Although approximately a quarter of this group was

recruited for the winter of 2015/2016, the automatic price response functionality was not deployed

until the summer of 2016.

BYOD. Nearly 500 customers were enrolled as “bring-your-own-device” participants. These

participants were provided with no automatic controls by Alectra.

The Energate Foundation allows for automatic thermostat response to be fixed at one of five settings,

from the “Max Comfort” (which applied no set-point offset) through to “Max Savings” (which applied the

most aggressive set-point offset).

The APP rate design consists of two main elements:

A daily “APP period” from 3pm to 9pm on non-holiday weekdays. The price in this period varies

based on system conditions and may take one of three possible values – “High”, “Medium” or

“Low” prices. System conditions used to set prices differ by season

o Summer (May through October). In summer months, APP price-levels are determined

based on the day-ahead IESO forecast demand during the APP period.

o Winter (November through April). In winter months, APP price-levels are determined

based on both the day-ahead IESO forecast demand, and the day-ahead spot price for

natural gas at the Dawn hub.4

An irregular (unscheduled) Critical peak price. Critical peak periods may be called at any point

of any day in the summer, and at any time between 10am and 6pm in the winter. Critical peak

periods are limited to lasting no more than four hours. Participants receive at least two hours’

warning prior to any Critical peak period.

All other hours are classified as “Off-Peak”. This period will be referred to in this report as the “APP Off-

Peak” to distinguish it from the RPP TOU Off-Peak period.

The APP prices for the summer of 2015, along with the approximate number of hours in the summer to

which they were anticipated to apply are shown in the figure below. The APP prices were set to be

“revenue-neutral” with standard RPP TOU rates, based on historical consumption levels and patterns. In

3 Note that three BYOD participants had previously been recruited for the summer 2015 portion of the pilot. These

were customers that had refused the Energate technology. 4 The alternative winter metric was determined in consultation with the IESO. The inclusion of the gas price in the

function for setting price-types was motivated by the IESO’s preference that winter price-periods more closely match

the fluctuations in the Hourly Ontario Energy Price (HOEP).



other words, absent any change in participants’ load as a result of the program, and assuming a set

distribution of price-type days, the total revenue recovered from participants should be approximately the

same as if they had been subject to standard RPP TOU rates.

Figure ES - 1: Summer 2015 APP Prices

Season APP

Price Period Price

(₵/kWh) % of Summer

Hours

Win

ter

201

5/2

01

6 Critical Peak 70 0.3%

High 59 0.8%

Medium 29 4%

Low 17.4 12%

APP Off-Peak 4.9 83%

Su

mm

er

201

6

Critical Peak 65 0.4%

High 52 3%

Medium 26 5%

Low 13 8%

APP Off-Peak 5.9 83% Source: Alectra interval data, IESO day-ahead historical day-ahead demands ,OEB RPP prices, and Navigant analysis

APP participants were recruited beginning in October of 2014. PowerStream advertised the program

using direct mail, bill inserts and email alerts. Only participants deemed eligible for peaksaver® PLUS

enrollment were considered for enrollment into the APP.

Enrollment in the program has grown over time. As shown in Figure ES - 1, there were, by the summer of

2016, nearly 2,000 participants enrolled in the program.

Figure ES - 1: APP Enrollment Over Time, By Technology

Technology

Number Enrolled By: Total

Enrollment May, 2015 November,

2015 May, 2016

Energate 965 207 3 1,175

Energate (Pioneer) 0 26 0 26

BYOD 3 226 235 464

Yukon 0 46 138 184

Total 968 505 376 1,849 Source: Alectra

Evaluation Approach

Impacts were estimated through the use of an econometric technique known as regression with pre-

program matching (RPPM). RPPM is a quasi-experimental technique that establishes a control group for

impact estimation through the careful matching of pre-participation participant and potential control

customer consumption patterns.



This approach minimizes the potential for model specification bias by using non-participating customers

with demonstrably similar historical consumption patterns as the basis for the counterfactual (baseline)

that drives impact estimation. Navigant used hourly interval data from participants and from over 100,000

non-participants (the base pool from which controls were selected) from the summers of 2013, 2014 and

2015, and 2016 in its analysis.

The process evaluation was driven by data provided by four different participant surveys. In addition to

allowing Navigant’s evaluation team to assess the reported behaviours, opinions and satisfaction of

participants, the longitudinal (i.e., surveying the same participants at different points in time) nature of the

survey deployment allowed for a nuanced analysis of how the reported behaviours, opinions and

satisfaction of participants changed over time. A summary of the four surveys is provided below.

Figure ES - 2: Participant Surveys: Timing and Completion Rate

Survey Group Timing Completes

Pre-pilot (Survey 1) Post Enrollment

December, 2015 175

Final Summer 275

Spring (Survey 2) BYOD/Yukon

April, 2016 82

Energate 52

Winter-exit (Survey 3) BYOD/Yukon

August, 2016 98

Energate 295

Final Survey All Participants January, 2017 590

Source: Navigant analysis

Impact Evaluation Findings

Navigant’s key impact evaluation findings include:

1. The APP continues to deliver substantial demand reductions coincident with the IESO

system peak. The average per participant demand impact during the IESO system peak hour on

September 7th was 0.76 kW for standard Energate participants, 0.81 kW for Yukon participants

and 0.26 kW for BYOD participants.

2. BYOD participants deliver consistent, material DR impacts across both seasons. BYOD

participants delivered an average of 0.18 kW of DR during High price summer days and 0.17 kW

during winter High price days.

3. BYOD participants are schedulers, not responders. These participants delivered material and

consistent DR – within the standard APP period of 3pm to 9pm. When Critical peak prices were

imposed outside this window, they delivered very little DR.

4. Yukon participants appear to rely principally on their technology to respond. Yukon

participants delivered no DR in the winter months, prior to the activation of their technology,

despite still being subject to APP prices. That said, Yukon participants’ summer impacts were

higher than any other technology group.



5. Energate Pioneer customers deliver substantial winter DR. Pioneer participants delivered an

average of 0.6 kW on Critical and High price events, substantially more in the winter than any

other technology group. These results are, however estimated based on only a very small group

of 26 participants.

6. Standard Energate participants are increasingly dividing into high-value and low-value

participants. Since the summer of 2015, participants have increasingly migrated away from the

“balanced” conservation setting to either the less aggressive or more aggressive conservation

settings, with those selecting the more aggressive conservation settings (e.g., “Savings”)

delivering more DR than participants with those settings did in the summer of 2015, and those

selecting the less aggressive conservation settings delivering less DR than customers

participants with those settings did in the summer of 2015.

7. On average participating in the APP increased customer energy consumption. As would be

expected given the price signal of the APP rate, on average participants have responded to the

program by increasing overall energy consumption by a small (and not statistically significant)

amount.

The average demand impact per participant during the APP period (3pm to 9pm, prevailing time), by price

level and during Critical peak periods (unscheduled) by technology group is presented in Figure ES -

2(winter 2015/2016) and Figure ES - 3 (summer 2016) below.

Figure ES - 2: Average Program Impacts by Price – Winter 2015/2016

Technology APP Price Mean Lower

90% CI*

Point Estimate

(kW)

Mean Upper

90% CI*

Stat. Significant?

Win

ter

BYOD

Critical Peak Event -0.03 0.21 0.45 No

High Price Day 0.08 0.17 0.27 Yes

Medium Price Day 0.06 0.10 0.14 Yes

Low Price Day -0.01 0.02 0.06 No

Energate (Standard)





Energate (Pioneer)

Critical Peak Event 0.14 0.60 1.06 Yes



Low Price Day 0.13 0.22 0.31 Yes

Yukon

Critical Peak Event -0.43 -0.02 0.38 No

High Price Day -0.22 -0.06 0.11 No

Medium Price Day -0.15 -0.08 -0.01 Yes

Low Price Day -0.15 -0.09 -0.03 Yes

* The average of the 90% confidence interval surrounding the estimated impacts in the given price period.

Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis



Figure ES - 3: Average Program Impacts by Price – Winter 2015/2016


90% CI*

Point Estimate

(kW)

Mean Upper

90% CI*

Stat. Significant?

Su

mm

er

BYOD





Energate (Standard)





Energate (Pioneer)




Low Price Day -0.10 -0.04 0.01 No

Yukon




Low Price Day 0.00 0.05 0.10 No

* The average of the 90% confidence interval surrounding the estimated impacts in the given price period. Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis

These impacts are the average across all of the relevant events and all participants.

The degree to which average participant summer impacts per event vary with outdoor temperature may

be observed in Figure ES - 3. In this plot, the estimated impact (Y-axis) on High price days for each

technology group (on days where the average event temperature was higher than 20 degrees Celsius) is

in the summer of 2016 is plotted against the average outdoor temperature. Different colour dots capture

the different technology types levels.

To compare the APP results with those of other residential DR programs, Navigant has also plotted the

reported ex-post DR impacts for individual days from four residential A/C programs that employ 50%

cycling strategies5.

5 Many of these programs also employ other cycling strategies – e.g., 65%, 100%, etc. but often much less frequently

than the 50% events. For consistency across programs, only the 50% cycling strategy impacts have been plotted.



Figure ES - 3: APP Period Impacts (Summer) Compared to 50% Cycling Residential DR Impacts

Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data,

Duke Energy Progress, Ontario Power Authority, Pacific Gas & Electric, Southern California Edison and Navigant analysis

Process Evaluation Findings

Navigant’s key process evaluation findings include:

1. Most participants would recommend APP to friends and family. The key metric of participant

satisfaction: approximately 85% of participants would recommend participation in the APP to

friends and family.

2. The primary motivation for program participation is to reduce electricity bills. 78% of all

respondents stated that reducing their electricity bill was their primary motivation for participating

in the program.

3. Prior experience with conservation amongst APP survey respondents is low. Only 25% of

respondents indicated that they had previously participated in a conservation initiative offered by



PowerStream (i.e. before PowerStream became Alectra). This indicates that the APP program is

likely attracting a new type of customer to conservation.

4. Participant engagement with the technology and the program is high, but declined over

time. Most participants viewed electricity pricing information once a day or more, however

frequency of interaction fell 8% between the initial and final survey. As well, although most

participants acted upon pricing information every day, the proportion of respondents that

undertook daily actions fell 5% from the initial to final survey.

5. Customer satisfaction with the APP is high, and improved over time. Participant feedback

demonstrates that the majority participants found the program “as good as expected” indicating a

reasonable degree of satisfaction. Responses stating that the program was “better than

expected” increased 14% between the initial and final survey. Despite this finding, a significant

number of participants indicated dissatisfaction with the program (18% stated the program was

worse than expected).

6. Most (59%) program participants are 50 or more years old. To reach younger and more

diverse demographics, Alectra may consider changing APP marketing and advertising efforts

and/or change the program’s design to attract a more varied age-range of customers.

7. Some participants were dissatisfied with their interaction with the Customer Care Centre

(CCC). 7% of respondents indicated that their interaction with the CCC was dissatisfactory. While

this is relatively low, it does provide an area that Alectra could focus on to improve customer

satisfaction.

8. Other motivations for program participation include: receiving the free Energate

thermostat, the environmental aspects of APP and the “risk-free” aspect of APP. 85% of

Energate respondents indicated that the no-cost thermostat was important in their decision to

participate, emphasizing that future participation in the program may be negatively impacted if the

thermostat is no longer offered. Also, the environmental benefits of the APP program are an

important motivator for participation, particularly for BYOD/Yukon participants (8% more

BYOD/Yukon participants selected this answer than Energate respondents). 76% of all

respondents indicated that the “risk-free” aspect of the APP program motivated their decision to

participate, emphasizing the importance of providing risk mitigation support to participants

through future program design.

9. Providing sufficient program information is important to program enrollment. The clear

majority of participants (84% to 91% depending on the group) agreed that the initial information

provided to them was sufficient to enroll. A small proportion did state that the information was

insufficient, particularly the Partially Engaged group (16%). The most common issues participants

noted regarding the initial information package include: communications were hard to understand;

a lack of communication around the specifics of the rates and eligible equipment; participants

were unclear about the risk-free aspect of the program.

10. Participants’ “most-liked” aspect of the program was saving money on electricity bills.

Emphasizing the money-saving aspect of the program may benefit customer satisfaction as well

as future recruitment efforts.



11. Participants’ “least-liked” aspect of the APP were the high peak rates. Certain respondents

found that there was a lack of communication about the peak rates when they were initially

enrolling in the program.6

6 Navigant notes that in public-facing on-line documents available to non-participants the evaluation team could only

find one explicit reference to the non-Off-Peak prices, on page 13 of the participant brochure. No mention of any price

level aside from the Off-Peak rate occurs in any of the FAQ pages, or on the main “landing page” for the program.



1. INTRODUCTION

The Energate-Alectra Advantage Power Pricing (APP) pilot is an experimental program designed to test

the response of technology-enabled residential customers to dynamic electricity prices. The central

guiding principles of the program design being tested by this pilot are to provide customers with an

opportunity to achieve significant bill savings, and to contribute toward the Ontario Ministry of Energy’s

long-term demand response (DR) goals by empowering customers with information and enabling

technologies.

The program design was developed by Alectra (formerly PowerStream) and Energate in close

consultation with staff from Ontario’s Independent Electricity System Operator (IESO), Ministry of Energy,

and the Ontario Energy Board (OEB). The pilot is closely modeled on the very successful Oklahoma Gas

and Electric (OG&E) SmartHours program, modified to reflect the locally-specific needs of the Ontario

market.

The summer7 2015 pilot was funded by the Ministry of Ontario SmartGrid Fund, and through contributions

(in resources and in kind) by Energate, PowerStream and Environics Analytics. In autumn of 2015, the

pilot was extended to cover the winter of 2015/2016 and the summer of 2016 with funding provided by the

IESO Conservation Fund. The program has now been extended through to the end of Summer 2018.

This report represents Navigant’s evaluation of the APP pilot for the winter of 2015/2016 and the summer

of 2016. It is comprised of two evaluation elements:

1. Impact Evaluation. The estimation of program demand and energy impacts through the use of

participant and non-participant hourly electricity (AMI/SmartMeter) consumption data and

econometric modeling.

2. Process Evaluation. An evaluation of the effectiveness of the APP design and delivery

mechanism in meeting programmatic objectives and an assessment of the pilot’s strategic

possibilities based on an analysis of five separate participant surveys.

This introductory chapter is divided into two sections:

Program Overview. This section provides an overview of the guiding principles of the program

design, the program design elements and the customers enrolled to participate.

Objectives of the Evaluation. This section lists the objectives establish for this evaluation by

Alectra and Energate in consultation with staff from the IESO, the Ministry of Energy and the

OEB.

1.1 Program Overview

This section of Chapter 1 provides an overview of the APP as it was deployed in the summer of 2015,

and is divided into five sub-sections:

7 Unless stated otherwise, all references to summer in this report should be understood to refer to the Regulated

Price Plan “summer” that runs from May 1st through to the end of October.



APP Design Principles. A summary of the design principles adopted by Energate and

PowerStream in consultation with the Ministry of Energy, the IESO and the OEB.

APP Rates A description of the rates imposed as part of the APP and the approach used to

generate those rates.

APP Technology. A description of the technology deployed to APP participants in the winter of

2015-2016 and summer of 2015.

APP Participants. A description of how participants were recruited, from which parts of

PowerStream’s service territory and from what demographic groups.

APP Events. A summary of the APP pricing events deployed in the winter of 2015-2016 and

summer of 2015.

1.1.1 APP Design Principles









needs.

The pilot design’s guiding principles8 were:

1. Participation is voluntary. No customers will ever be subjected to the APP rate structure without

their consent and customers will be able to withdraw from the program at any time.9

2. Participants must be empowered, not compelled, to deliver demand response. Participants

are provided with a strong price signal and encouraged to use technology (provided by Alectra or

by the participants themselves) to respond to that signal, but are not required to undertake any

response.

3. Participants will face no risk in the first year of participation. For the first year of

participation, participants will be offered “no-lose” billing to ensure that recruitment and retention

goals are met.10

4. The program pricing structure will be designed to meet long-term forecast, rather than

historical, system peak capacity needs. The goal of the APP is to test the demand response

(DR) capabilities of the program design to better serve long-term provincial capacity needs.

8 These principles have been edited and condensed from the original set of nine principles to reflect pilot evolution

and based on their relevance to the pilot evaluation. 9 Participants that withdrew prior to the close of summer 2015 were not eligible for the APP rebate that acted as the

program incentive.. 10

For administrative and regulatory reasons, bill protection is in fact so far been provided to all participants

throughout their entire participation in the pilot.



5. APP prices will be reflective of system requirements and costs. Price changes will be

correlated with system requirements and costs.

6. APP prices will be set in a manner consistent with the OEB’s principles of cost-recovery

as laid out in the RPP. The dynamic prices will be set such that, absent any change in

participants’ load as a result of the program, the total revenue recovered from participants will be

approximately the same as if they had been subject to standard RPP TOU rates.

1.1.2 APP Rates









needs.

Customers enrolled in the APP remained subject to the standard Regulated Price Plan (RPP) time-of-use

(TOU) rates, were billed by Alectra under these rates and were required to pay them as usual. In addition,

participants received a monthly “shadow bill” that tracked what their bill would have been under the APP

rate. At the close of the summer, if participants’ shadow bills were lower than their actual TOU bills, they

received the difference from Alectra as a rebate. Participants whose APP shadow bill exceeded their

standard TOU bill were not penalized.

The prices to be applied in all APP periods were determined by Navigant based on guidance from

PowerStream, Energate, the Ministry, the IESO and the OEB. Prices were determined based on historical

PowerStream customer data in a manner consistent with the OEB’s approach to setting the RPP TOU

prices: absent any change in load in response to the program, APP rate prices should be approximately

revenue neutral to the standard RPP TOU prices.

The APP prices for the summer of 2015, along with the approximate number of hours in the summer to

which they were anticipated to apply are shown in the figure below. The APP prices were set to be

“revenue-neutral” with standard RPP TOU rates, based on historical consumption levels and patterns. In

other words, absent any change in participants’ load as a result of the program, and assuming a set

distribution of price-type days, the total revenue recovered from participants should be approximately the

same as if they had been subject to standard RPP TOU rates. Figure 1, below, shows the prices for the

two periods considered in this analysis, as well as the distribution of hours assumed for the purposes of

price-setting.



Figure 1: APP Prices, November 2015 through October 2016

Season APP

Price Period Price

(₵/kWh) % of Summer

Hours

Win

ter

201

5/2

01

6 Critical Peak 70 0.6%

High 59 3%

Medium 29 5%

Low 17.4 9%

APP Off-Peak 4.9 82%

Su

mm

er

201

6

Critical Peak 65 0.5%

High 52 3%

Medium 26 5%

Low 13 9%

APP Off-Peak 5.9 83% Source: Alectra interval data, IESO day-ahead historical day-ahead demands ,OEB RPP prices, and Navigant analysis

The standard RPP TOU prices that applied in the same period, along with the percentage of hours to

which they apply is shown in Figure 2, below.

Figure 2: RPP TOU Prices, November 2015 through October 2016

Season RPP TOU

Price Period Price

(₵/kWh) % of Hours

Winter 2015/ 2016

On-Peak 17.5 18%

Mid-Peak 12.8 18%

Off-Peak 8.3 64%

Summer 2016

On-Peak 18.0 18%

Mid-Peak 13.2 18%

Off-Peak 8.7 64% Source: OEB RPP prices, and Navigant analysis

Additional details regarding the status quo RPP TOU rate may be found on the Ontario Energy Board’s

website.11

1.1.3 APP Technology

To support participant price-response, all customers enrolled in the program for the summer of 2015 were

equipped with an Energate Foundation that allowed participants to automate their thermostat’s response

to APP price fluctuations. Price-responsive DR is delivered by adjusting the thermostat set-point. In

addition, a small number (approximately 30) of load switches were installed on electric water heaters and

pool pumps. Customer take up was lower than anticipated, due in part to customers not being sure if their

water heater was electric. During DR events, the switches, controlled through the Foundation thermostat,

11

Ontario Energy Board, Electricity Prices,

http://www.ontarioenergyboard.ca/oeb/Consumers/Electricity/Electricity%20Prices

http://www.ontarioenergyboard.ca/oeb/Consumers/Electricity/Electricity%20Prices



would turn off this equipment. Beginning in the winter of 2015/201612

, three alternative technology groups

were added to the program:

Energate Pioneer. A small group of customers13

were provided with the Energate Foundation,

and a set of Energate’s Pioneer thermostats. The Pioneer thermostats extends the Energate’s

automatic thermostat control to independently controlled electric baseboard heaters.

Yukon. Nearly 200 former peaksaver PLUS® participants were recruited into APP and

volunteered to receive price signals through the Yukon platform . These participants’ legacy

peaksaver thermostats were controlled using the Cooper Power Systems Yukon Advanced

Energy Services Platform. Although approximately a quarter of this group was recruited for the

winter of 2015/2016, the automatic price response functionality was not deployed until the

summer of 2016. Unlike the Energate Foundation, the Yukon delivers price-responsive demand

response via A/C cycling. This is described in additional detail below.

BYOD. Nearly 500 customers were enrolled as “bring-your-own-device” participants. These

participants were provided with no automatic controls by Alectra.

The Energate Foundation is a bundle of enabling technologies that consists of a “smart” thermostat14

and

an online portal (“Home Energy Gateway”). The bundle automatically controls the participant’s thermostat

in response to price changes according to customer-set preferences (“conservation settings”). The five

conservation settings range from the “Max Comfort” (which applies no set-point offset) through to “Max

Savings” (which applies the most aggressive set-point offset). The degree-offsets by conservation setting

and price-type are summarized in Figure 3, below.

Figure 3: Conservation Setting Offsets and APP Prices

APP Price Period

Degree (C) Offset by Conservation Setting

Max Comfort

Comfort Balanced Savings Max

Savings

Critical Peak 0.00 1.00 2.00 3.00 4.00

High 0.00 0.75 1.50 2.25 3.00

Medium 0.00 0.50 1.00 1.50 2.00

Low 0.00 0.25 0.50 0.75 1.00

APP Off-Peak 0.00 0.00 0.00 0.00 0.00 Source: Alectra

The Yukon technology works deploying a different cycling strategy for each price level. Cycling differs

substantially from thermostat set-point as an approach to delivering A/C direct load control DR. Cycling

works not by adjusting the thermostat set-point, but by constraining the A/C compressor’s run time. So,

for example, simple 50% cycling typically works by only providing power to the A/C compressor for fifteen

minutes out of every thirty.

12

Note that three BYOD participants had previously been recruited for the summer 2015 portion of the pilot. These

were customers that had refused the Energate technology. 13

Approximately 35 customers were provided with this device. Useable data for 26 of these were included in the

analysis. 14

The Energate thermostat is not a “learning” thermostat. It is a two-way communicating thermostat with an

advanced display that can be controlled remotely via the online portal or a mobile device app.



Yukon participants could select from one of two menus of cycling strategies associated with APP prices: a

“Full” option and a “Lite” option. Fewer than fifteen participants selected the “Lite” option. The cycling

strategies associated with each of the APP prices is described below.

Figure 4: Yukon Cycling Strategies

APP Price

Period

Max. A/C Run Time

Lite Option Full Option

Critical Peak 25 min/hour (58% Cycling) 12 min/hour (80% Cycling)

High 30 min/hour (50% Cycling) 18 min/hour (70% Cycling)

Medium 36 min/hour (40% Cycling) 24 min/hour (60% Cycling)

Low 42 min/hour (30% Cycling) 30 min/hour (50% Cycling) Source: Alectra

1.1.4 APP Participants

APP participants were recruited beginning in October of 2014. PowerStream advertised the program

using direct mail, bill inserts and email alerts. Only participants deemed eligible for peaksaver® PLUS

enrollment were considered for enrollment into the APP.

Enrollment in the program has grown over time. As shown in Figure 5, there were, by the summer of

2016, nearly 2,000 participants enrolled in the program.

Figure 5: APP Enrollment Over Time, By Technology

Technology

Number Recruited by Period: Total

Enrollment May, 2015 November,

2015 May, 2016

Energate 965 207 3 1,175

Energate (Pioneer) 0 26 0 26

BYOD 3 226 235 464

Yukon 0 46 138 184

Total 968 505 376 1,849 Source: Alectra

15

Note that the time intervals shown in Figure 5 correspond to the beginning of discrete price periods. A

participant recruited in July of 2015 would therefore be counted in the “November 2015” column since

their first period of participation in the program would have been the period from November 2015 through

April 2016.

Figure 6 shows the APP participant density across the various regions within the PowerStream service

territory. Participant density (as a percentage of total participants) is mapped to individual Canada Post

15

Number of participants are the number of participants in each cohort still enrolled when enrollment data was

provided to Navigant.



Forward Sortation Areas (FSAs) within PowerStream service territory. An FSA is an area corresponding

to the first three digits of an individual’s postal code. This includes all APP participants enrolled by the

beginning of the summer 2016 period.

As is evident from this map, the vast majority of participants in the APP live in York Region, with a small,

relatively dense pocket located in Barrie. Note that some FSAs are too small geographically for the

numeric percentage to appear on the map.

Figure 6: Participant Density Map by FSA16

(All Participants)

Source: Alectra, Util-Assist tracking data, and Navigant analysis

Figure 7, below reproduces the same map, but segregates the four technology groups: Energate

(Standard), Energate (Pioneer), BYOD and Yukon participants

16

Chart sizing means that not all densities are displayed numerically as data labels.



Figure 7: Participant Density Map by FSA17

(All Participants)

Source: Alectra, Util-Assist tracking data, and Navigant analysis

1.1.5 APP Events and Pricing

The APP price to be deployed for any given APP price period was determined through the comparison of

a system-level metric with a set of threshold values. The threshold values were selected by Alectra in

consultation with Navigant based on historical values of the metric, such that, given historical values of

the metric, the desired distribution of price-levels throughout the season should be achieved. The goal

was to ensure that 50% of APP period prices would be Low, 30% would be Medium and 20% would be

High.

17

Chart sizing means that not all densities are displayed numerically as data labels.



The metrics used for price-setting were:

Summer. The day-ahead IESO forecast system demand between 3pm and 9pm.

Winter. An “APP score” developed by Navigant in consultation with Alectra and the other pilot

stakeholders (in particular the IESO). The APP score is a function the IESO day-ahead forecast

of demand in the APP period and the day-ahead forecast spot gas price at Dawn.

Throughout the course of the two seasons, thresholds were periodically adjusted in an attempt to ensure

that the number of price-types actually called were reasonably in line with those used for price-setting

purposes.

Critical peak events were called on an ad hoc basis by Alectra for testing purposes, with the goal of

calling these events during times of anticipated system peaks.

The distribution of APP prices across the winter of 2015/2016 is illustrated in Figure 8, below. The

distribution of APP prices across the summer of 2016 can be seen below that in Figure 9 further below.

Critical peak events are flagged by the purple boxes. One significant challenge in the winter of 2015/2016

was that it was a very mild winter, resulting in both relatively low day-ahead forecast IESO demands and

low gas prices. This meant that despite a number of threshold adjustments, there were considerably

fewer High price days than originally anticipated.

Figure 8: APP Price Level Days in Winter of 2015/2016

Source: PowerStream and Navigant analysis



Figure 9: APP Price Level Days in Summer of 2016

Figure 10, below, provides the final distribution of hours across all five of the different APP price periods.

This may be compared to Figure 1 which provides the anticipated distribution of hours used to set the

APP prices.

Figure 10: Final Distribution of Hours by Price

Season APP

Price Period Price

(₵/kWh)

% of Season Hours

(Actual)

# of days* (Actual)

% of Seasonal Hours (For

Price-Setting)

Win

ter

201

5/2

01

6 Critical Peak 70 0.3% 4 0.6%

High 59 0.8% 6 3%

Medium 29 4% 33 5%

Low 17.4 12% 86 9%

APP Off-Peak 4.9 83% 57 82%

Su

mm

er

201

6

Critical Peak 65 0.4% 5 0.5%

High 52 3% 26 3%

Medium 26 5% 39 5%

Low 13 8% 60 9%

APP Off-Peak 5.9 83% 57 83%

Source: PowerStream and Navigant analysis

Critical peak events were called on the following days:

Winter

o January 20, 2016: 8am – 10am



o February 11, 2016: 5pm – 9pm

o February 12, 2016: 7am – 9am

o February 24, 2016: 6pm – 9pm

Summer:

o July 6, 2016: 3pm – 7pm

o July 13, 2016: 5pm – 7pm

o July 22, 2016: 4pm – 8pm

o August 11, 2016: 1pm – 5pm

o September 7, 2016: 4pm – 6pm

1.2 Objectives of the Evaluation

This section of Chapter 1 is divided into two sub-sections. The first outlines the objectives of the impact

evaluation, the second outlines the objectives of the process evaluation.

1.2.1 Impact Evaluation Objectives

The objectives of the impact evaluation were to estimate:

1. The average seasonal energy impact of participation (kWh/participant for summer 2016)

estimated for the evaluation period by:

o Participant comfort setting group (Energate participants only)

o Technology group (base Energate system, Yukon system, BYOD)

o Hour of day

2. The average impact (kW/participant) in each hour of the APP period, in the pre-cooling period,

and in the snapback period immediately following the APP period by:

o APP price period (“Critical Peak”, “HIGH”, “MEDIUM”, “LOW”).

o The incremental impacts of Critical Peak over superseded variable peak price18

o Participant conservation setting group (Energate participants only)

o Technology group (base Energate system, Energate Pioneer, Yukon system, BYOD)

3. The average event period impact (kW/participant) by:

o APP price period (“Critical Peak”, “HIGH”, “MEDIUM”, “LOW”)

o The incremental impacts of Critical Peak over superseded variable peak price19

o Participant comfort setting group (Energate participants only)

o Technology group (base Energate system, Energate Pioneer, Yukon system, BYOD)

4. The average impact (kW/participant) on:

o The single highest Ontario demand hour of the summer of 2016 (to be defined by the

IESO for PowerStream prior to the start of evaluation)

o The peak hours of each of the five days in summer of 2016 and the winter of 2015/2016

when Ontario system demand was highest.20

These top five peak hours21

are (from

highest to lowest, by season):

18

This will be estimated using an ancillary regression developed based on the in-sample ex-post estimated impacts. 19

This will be estimated using an ancillary regression developed based on the in-sample ex-post estimated impacts.



Winter 2015/2016

6pm – 7pm, January 1, 2016

6pm – 7pm, February 11, 2016



6pm – 7pm, February 13, 2016 (a Saturday)

Summer 2016

5pm – 6pm, September 7, 2016

6pm – 7pm, August 10, 2016

5pm – 6pm, August 11, 2016

6pm – 7pm, July 13, 2016

5pm – 6pm, August 12, 2016

1.2.2 Process Evaluation Objectives

The objectives of the process evaluation were to assess:

1. Participant satisfaction with the initiative, technology, and its benefits;

2. Customer behaviour change brought on through participation in the initiative;

3. Customer motivations for participating;

4. Participant demographics and characteristics; and,

5. What enhancements could be incorporated into the current offering that could increase the

chances of program success for a wider program deployment.

20

That is the peak hour of the five days with highest system demands over the period in question. This is not

necessarily the same as the five highest demand hours of the year, as it is possible that several of the highest

demand hours of the year might be observed on the same day. 21

As per the convention of this report, all times should be considered Eastern Prevailing Time unless explicitly stated

otherwise.



2. EVALUATION APPROACH

This chapter describes the analytic approaches deployed by Navigant to complete the process and

impact evaluation.

This chapter is divided into two sections. The first describes Navigant’s approach to estimating program

impacts, and the second describes Navigant’s approach to the process evaluation.

2.1 Impact Evaluation

Impacts were estimated using hourly AMI data from participants and from over 100,000 eligible non-

participants over a four year period (2013 through 2016). This section of Chapter 2 provides a description

of the analytic approach used. It is divided into four sub-sections:

Regression with Pre-Program Processing, This sub-section describes the theoretical basis for

Navigant’s econometric approach.

Control Group Selection. This sub-section describes the approach used to select the control

group participants that were included in the analysis.

Hourly Demand Impact Estimation. This sub-section describes the set of regression equations

estimated by Navigant to deliver all of the estimated demand response impacts.

Daily Energy Impact Estimation. This sub-section describes the set of regression equations

estimated by Navigant to deliver all of the estimated energy savings impacts.

2.1.1 Regression with Pre-Program Processing

All of the estimated impacts reported in this evaluation were estimated using a technique known as

regression with pre-program matching (RPPM) 22

. RPPM is a quasi-experimental approach whereby a

control group is selected by comparing participant pre-participation consumption with a large pool of non-

participants and assigning to each participant the non-participant with the most similar pre-program period

consumption patterns. This control customer is sometimes referred to as the given participant’s match, or

matched control.

In program evaluation, the basic logic of matching is to balance the participant and non-participant

samples by matching on the exogenous covariates known to have a high correlation with the outcome

variable. Doing so increases the efficiency of the estimate and reduces the potential for model

specification bias.

Formally, the argument is that if the outcome variable Y is independently distributed conditional on X and

D (conditional independence assumption), where X is a set of exogenous variables and D is the program

22

See, for instance, Ho, Daniel E., Kosuke Imai, Gary King, and Elizabeth Stuart. 2007. Matching as nonparametric

preprocessing for reducing model dependence in parametric causal inference. Political Analysis 15(3): 199-236.

Abadie, A. and G.W. Imbens. 2011. “Bias-Corrected Matching Estimators for Average Treatment Effects”. Journal of

Business and Economic Statistics 29(1):1-11.



variable, then the analyst can gain some power in the estimate of savings and reduce potential model

specification bias by assuring that the distribution of X is the same for treatment and control observations.

Regression analysis is used to control for remaining non-program differences between participants and

their matches. In this context, the development of a matched control group is viewed as a useful “pre-

processing” step in a regression analysis to assure that the distributions of the covariates (i.e., the

explanatory variables on which the output variable depends) for the treatment group are the same as

those for the comparison group that provides the baseline measure of the output variable.

Typically the control variables with the highest correlation with a customer’s energy use during the

evaluation period –and thus, the primary variables for matching –are the customer’s energy use in a

similar period in the past.

A recent report authored at Lawrence Berkeley National Laboratory cites the RPPM (sometimes referred

to as “matched control group” or MCG method) method as a reasonable alternative to establishing

baseline conditions when the “gold standard” of program evaluations, a randomized control trial (RCT), is

not an option.23

The RPPM method is now common in the economics literature—and the energy

industry—for program evaluations conducted with observational, rather than experimental, data.24

2.1.2 Control Group Selection

In order to select the control group PowerStream provided Navigant with hourly AMI data for participants

and for over 100,000 eligible non-participants extending from May 2013 through until October 2016.

For matching controls to participants, Navigant established a set of 30 day types, and compared

participant and non-participant average hourly loads on those 30 days from the 2013 calendar year25

.

Each participant’s 2013 consumption pattern was compared with the 2013 consumption patterns of all

available non-participants from the same demographic group (see below for details), in the same period,

by season.

The non-participant whose historical patterns deviated the least (in a given season) from those of the

given participant became that participant’s match. A different match was selected for each season.

Historical consumption patterns for each individual and each season were summarized by a 360 element

vector of average levels of consumption (15 day-types in each season times 24 hours – see below).

The two seasons26

of the matching period were each divided into three two-month periods. In the winter:

January/February, March/December, and April/November. In the summer, May/October,

June/September, and July/August. Five day types were applied to each period. Four of these day-types

23

State and Local Energy Efficiency Action Network. 2012. Evaluation, Measurement and Verification (EM&V) of

Residential Behavior-Based Energy Efficiency Programs: Issues and Recommendations. Prepared by A. Todd, E.

Stuart, S. Schiller, and C. Goldman, Lawrence Berkeley National Laboratory. 24

See, for instance, Cameron, A. Colin, and P.K. Trivedi, Microeconometrics: Methods and Applications, Cambridge

University Press, 2005. 25

Controls were matched using the 2013 calendar year, rather than 2014, due to the relatively cool summer of 2014.

The temperature variation in the summer of 2015 (the first program summer) was closer to that of 2013 than 2014,

making this the more appropriate year for matching. 26

Winter and summer definitions were selected to be in line with those of the Ontario Energy Board’s (OEB)

Regulated Price Plan (RPP); summer beginning May 1 and running through to the end of October, with remaining

months considered as winter.



were assigned to non-holiday weekdays where the average daily temperature met a certain temperature

threshold (see for the criteria), and the fifth was applied to all holidays and weekends.

Figure 11: Day-Type Average Daily Temperature Thresholds

Day Type

Weekdays Weekends/

Holidays

1 2 3 4 5

Winter

Period A: Jan &

Feb >= 0.6

-3.1 to 0.6

-10.7 to

-3.1 < -10.7

Period B: Mar &

Dec >= 9.1

4 to 9.1

-2 to 4

< -2 All Days

Period C: Apr &

Nov >= 12.6

5.3 to 12.6

-0.9 to 5.3

< -0.9

Summer

Period D: May &

Oct >= 19.8

11.9 to 19.8

6.7 to 11.9

< 6.7

Period E: Jun & Sept

>= 23 18.5 to

23 14.1 to 18.5

< 14.1 All Days

Period F: Jul & Aug

>= 25.4 20 to 25.4

16.6 to 20

< 16.6

Source: Environment Canada weather data and Navigant analysis

The weather data thresholds were selected based on the observed weather in the period from May 1,

2015 through April 30, 201627

such that for each period (e.g., July/Aug) 10% of the non-holiday weekdays

were day type 1, 40% were day type 2, 40% were day type 3 and the remaining 10% were day type 4.

Effectively, the day-type thresholds were selected in such a way as to increase the weighting of more

extreme temperature days, improving the likelihood that matched controls temperature-sensitive loads

with be very close to those of the participants.

These day types thresholds were assigned to the matching period customer AMI data28

, and these

interval data were then averaged by day-type and hour. The matching period used the year of 2013 as

the matching period. This year was used (instead of 2014) based on the observation that in some cases

there existed no days in the summer of 2014 that matched the temperature thresholds; days were simply

hotter in the summers of 2013, 2015, and 2016 than in 2014.

Each participant’s average load by hour of day and day type was compared to every non-participant from

the same demographic group’s load in the same hour of day and day type within the matching period, by

season (i.e., a separate match was selected for summer and winter).

27

This period was selected for the summer 2015 analysis and was maintained for the winter 2015/2016 analysis and

the summer 2016 analysis to ensure consistency across evaluation years. 28

Note that the matching period distribution of days meeting the various day-type criteria did not necessarily match

that of the 2015/2016 period used to establish the weather thresholds – i.e., even though 10% of the days from May 1

2015 through April 30, 2016 were type 1 days, this does not mean that 10% of the 2013 days were type 1 days.



Customer demographic groupings were provided by the project’s demographic analysis vendor,

Environics. The names provided by Environics for the four demographic segments are:

Affluent Older Families

Asian Families

Middle Aged Families

Multi-Ethnic Families

Each participant’s final match was selected as the non-participant for whom the sum of squared

deviations (when the two match period data series were compared) was smallest.

2.1.3 Hourly Demand Impact Estimation

All demand impacts were estimated using a set of linear regressions. The models shown below were

estimated separately for each unique combination of season, technology, demographic segment and

conservation setting29

– in all this equation was estimated 70 times.

A few important points about this split of the data:

Non-Energate participants do not have conservation settings and so were divided only by season,

technology and demographic.

Energate Pioneer participants were such a small group – Navigant had data for only 26

participants – that they were not split up by demographic or conservation setting.

Energate participants can change their conservation settings, so a given participant’s data may

be included in a number of different sub-groups, although data for any given point in time may

appear only in a single sub-group.

The conservation data provided by Energate is only weekly in frequency, so it is possible that

participants included in (for example) a “Max Savings” regression could, at some point, have had

another conservation setting during that week.

Sub-groups with fewer than 200 observations of hourly demand in total, and sub-groups that had

both: fewer than five participants and unknown conservation settings or an unknown demographic

group were dropped from the analysis. Altogether these dropped observations account for

approximately 0.3% of the data in the estimation set.

The regression equation that delivered the estimated weekday impacts is:

29

Energate Pioneer participants were such a small group – Navigant had data for only participants – that they were

not split up by demographic or comfort setting.



360

, 1, ,

1

3 3

2, , 3, , ,

2 1

3 3

4, , , , , ,

1 1

6 6

5, 1 , , 6, , , , ,

1 1

1

2

3

i t t d i t d

d

w i t w w i t i t w

w w

pc pr i t pc t pr t i t

pr pc

h pr i t h t h pr pr i t i t h t

h pr h

y prekw Dt

APP AM APP MAweath AM

APP PCond PR MAweath

PR APP Hr PR APP MAweath Hr

3

1

5

7, 2 , ,

4

6

8, 3 , ,

3

3 6

9, , , , ,

1 1

27

10 , , ,

1

4

_ 5

_ 6

8 _ 7

8

h pr i t i t t t

h

h pr i t i t t t

h

s pr pr i t i t s t

pr s

i t c t i t

c

PR APP MAweath CPP SB Hr

PR APP MAweath CPP SB Hr

PR APP MA weath APP SB

APP CPP

Where:

,i ty = Hourly consumption of customer i in hour of sample t.

t = A time-wise fixed effect. This is equivalent to including a dummy variable for each

unique combination of hour of day and date in the estimation sample. This

controls for all effects that vary across time, but not across participants within the

sub-group included in the regression. For example: the average effect of

weekdays, weekends, and holidays, or the average impact of weather on loads,

etc.

,i tprekw = The average 2014 (pre-participation) consumption of customer i on the hour of

the appropriate day-type (see discussion in section 2.1.2 above) observed in

hour t

dDt = A set of 360 dummy variables flagging the appropriate pre-participation period to

apply in hour t. Only 288 of these dummies appear in the weekday equation

(three seasonal periods, four day types, 24 hours), with the remaining 72

appearing only in the weekend/holiday equation (see below). NB: there are 360

dummies for the winter regression and 360 different dummies for the summer

regression.

,i tAPP = A dummy variable equal to one if customer i has become subject to the program

pricing by hour t.

wAM = A set of three dummy variables capturing three different morning periods. Where

w = 1 is the period between midnight and 6am, w = 2 is the period from 6am to

9am and w = 3 is the period between 9am and noon.



,i tMAweath = The five hour moving average of (summer) cooling degree hours or (winter)

heating degree hours (both base 18 degrees C) observed for customer i in hour

of sample t.

prPR = A set of dummy variables corresponding to the three standard prices possible in

the APP period, where pr = 1 is the Low price, pr = 2 is the Medium price, and pr

= 3 is the High price. NB this variable always takes the value of zero if hour t is a

Critical peak hour.

,h tHr = A set of six dummy variables capturing each hour of the APP period, where h =1

is the first hour of the APP period (3pm – 4pm) and h = 6 is the final hour of the

APP period (8pm – 9pm)

_ tCPP SB = A dummy variable equal to one if hour t is observed within three hours of the end

of a Critical Peak period and is also in the APP period, and zero otherwise.

,_ s tAPP SB = A set of six dummy variables capturing the effects of snapback in the first six

hours immediately following the end of the APP period, where s = 1 is the first

hour following the APP period (9pm – 10pm) and s = 6 is the sixth hour following

the APP period (2am – 3am).

,8 i tMA weath = The eight hour moving average of (summer) cooling degree hours or (winter)

heating degree hours (both base 18 degrees C) observed for customer i in hour

of sample t.

,c tCPP = A set of sixteen dummy variables, one for each of the sixteen Critical peak hours

observed in the winter of 2015/2016 and the summer of 2016.

The purpose of each grouping of equation terms is summarized below:

(1) The time-wise fixed effect and the pre-participation average demands effectively create the

baseline against which program effects (for which all other model variables control) are

compared. This is equivalent to including a dummy variable for each unique combination of hour

of day and date in the estimation sample. This controls for all effects that vary across time, but

not across participants within the sub-group included in the regression. For example: the average

effect of weekdays, weekends, and holidays, or the average impact of weather on loads, etc.

(2) These terms capture program impacts in the morning before pre-cooling or the APP period

begins. The periods from 6am on are modeled with both an intercept and a temperature dummy

to capture the possibility of both A/C and behavioural price-response. The period from midnight to

6am is modeled only with an intercept based on the assumption that purely behavioural price

response is unfeasible for most participants in that time period.

(3) This term captures the effects of pre-conditioning (either pre-heating in winter or pre-cooling in

summer), assumed to vary by the day’s APP price level.

(4) This term captures the impact within the APP period of each of the different price levels. Both

intercept and slope (temperature) terms are included for Low prices, but only slope terms are

included for the Medium and High period. This is since Medium and High price days are highly

correlated with temperature which, in some initial model specifications tested, led to some

obviously spurious parameter estimates on either the intercept or the slope term.



(5) This term captures the effect of intra-APP period snapback when a Critical peak period ends

within the APP period on a Medium price day.

(6) This term captures the effect of intra-APP period snapback when a Critical peak period ends

within the APP period on a High price day.

(7) This term captures the snapback effects of the APP period by price level and as a function of the

temperature both during, and after, the APP period.

(8) This term captures the effect of a Critical peak price. The original intention had been to model the

Critical peak events as a function of temperature (to allow for stakeholders to extrapolate impacts

using other sets of temperatures), but the attendant multicollinearity stymied such attempts.30

Navigant has recommended that in future years Critical peak events be deployed in the style of a

randomized control trial (RCT) (i.e., with some portion of participants not subjected to the CPP

event and so act as a control) to avoid this issue.

The regression equation that delivered the estimated weekend impacts is:

360

, 1, ,

1

2 , 3 , , ,

1

2

i t t d i t d

d

i t i t i t i t

y prekw Dt

APP APP MAweath

Where all variables are as defined above.

2.1.4 Daily Energy Impact Estimation

Energy impacts were estimated using a very similar approach to that used for estimating demand impacts

except that all observations were aggregated to the daily level.31

As with the demand estimation, a

separate model is estimated for each combination of conservation setting and demographic segment,

although weekend and weekday differences are captured by dummy variables instead of separate

equations.

Energy impacts were estimated using this model: 15 2

, 1, , 2 , ,

1 1

i t t d i t d i t wk i t

d wk

y prekw Dt APP Wk

Where:

,i ty = Daily consumption of customer i in hour of sample t.

t = A time-wise fixed effect.

,i tprekw = The average 2014 (pre-participation) consumption of customer i on the the

appropriate day-type (see discussion in section 2.1.2 above) observed on day t

dDt = A set of 30 dummy variables flagging the appropriate pre-participation period to

apply on day t.

30

Critical peak events are highly correlated both with weather values and, most importantly, with High price days. 31

Energy impacts could equally be calculated using the results of the hourly demand model, and in fact the impacts

of the two models are statistically equivalent. Using the aggregated energy model, however, greatly simplifies the

estimation of both the impacts and the uncertainty associated with them.



,i tAPP = A dummy variable equal to one if customer i has become subject to the program

pricing by day t.

wkWk = A set of two dummy variables capturing whether a day is a non-holiday weekday

or a weekend.

2.2 Process Evaluation Approach

The process evaluation in this report represents the second set of APP related process evaluation

findings developed by Navigant. An initial set of observations were prepared for the Summer 2015

iteration of the program and are available in the complementary report entitled “Advantage Power Pricing

Pilot: Impact and Process Evaluation (July 7 2016)”. Where beneficial, findings from this previous

evaluation effort are compared with the observations made during the Summer 2016 deployment period.

Participant responses from four surveys were used to inform this process evaluation:

Survey 1, conducted in December 2015, was administered separately to the following two

participant groups:

o The “Post Enrollment” group (consisting of customers who did not participate in the APP

program during the summer of 2015); and,

o The “Final Summer Participant” group (consisting of customers who did participate in

the APP during the summer of 2015).

Surveys 2 and 3 were carried out in April, 2016 and August, 2016, respectively. These surveys

were each delivered separately to the following two participant groups:

o “The Energate” group, including participants that were provided an Energate System free

of charge as part of their enrollment in the program.

o “The Bring-Your-Own-Device/Yukon (BYOD/Yukon)” group, consisting of participants

who enrolled their own smart thermostat in the program (i.e., they did not receive the

Energate system upon enrolling in the program). This also includes participants with

thermostats operating on the Yukon platform.

Survey 4 was delivered in January of 2017 to all participants



Figure 12: Participant Surveys – Timing and Completion Rate

Survey Group Timing Completes

Pre-pilot (Survey 1) Post Enrollment

December, 2015 175

Final Summer 275

Spring (Survey 2) BYOD/Yukon

April, 2016 82

Energate 52

Winter-exit (Survey 3) BYOD/Yukon

August, 2016 98

Energate 295

Final Survey All Participants January, 2017 590

Source: Navigant analysis

Each participant survey was administered via email and strategically timed in order to gain

valuable insight from participants at critical stages of the initiative’s delivery.

The pre-pilot survey was deployed separately to new participants (Post Enrollment) and

participants of the previous trial (Final Summer). The Post Enrollment survey was designed to

gauge initial perceptions of the program, effectiveness and quality of the program’s marketing

material and PowerStream support staff, as well as participants’ key motivations for program

participation. The Final Summer participant survey was designed to assess how perceptions of

the technology and program have changed since the previous trial (summer of 2015).

The release of the spring and winter-exit surveys were timed to coincide with critical peak

events in order to, amongst other factors, assess any changes to participant comfort resulting

from the activation.

Lastly, the final survey was administered at the conclusion of the program as a means to assess

the value participants assigned to the program as well as the ability of the program’s technology

– specifically the Energate thermostat – to influence their homes’ energy use.

The feedback sought through each survey provided insight into participant attitudes and

perceptions of the APP as well as the energy efficiency actions they had taken in response to

various tools and information made available through the program. Much of the process

evaluation compares results among two sets of groups to provide as much detail into the

program as possible.

BYOD/Yukon Participants vs. Energate Participants (as defined above)

Engaged Participants vs. Partially Engaged Participants

o Engaged Participants are defined as those who have responded to all surveys

(i.e. four out of four).



o Partially Engaged Participants are those who have responded to three or fewer

of the four surveys.

Figure 13, below demonstrates the various areas explored through each survey completed:

Figure 13: Process Evaluation Overview

Process

Evaluation

Element

Evaluation Goal

Participant

satisfaction with

the initiative,

supportive

technology, and its

benefits

Select survey questions were used to gauge participant expectations

of the initiative and technology prior to engagement as well as the

perceived value to the customer of participating in the pilot.

Questions were asked in surveys during the trial in order to assess the

performance of the initiative’s technology against expectations, overall

satisfaction of the participant as well as to identify any positive or

negative experiences. Positive and negative experiences were

explored to provide Alectra with actionable intelligence for improving

the program’s design.

Behaviour change

brought on

through

participation in the

initiative.

This assessment provided insight into how the technology and/or

program impacted the way participants monitor their electricity use and

act upon this information. Findings from this assessment can be used

to adjust the future value-proposition for the initiative, future customer

engagement activities as well as provide an understanding of the

energy efficiency gains the APP influenced.

Exploration of the

motivations for

participation and

previous CDM

program

participation

This exploration was critical to understanding what aspects of the

program motivated specific customer segments to participate. This

assessment also confirmed whether or not the participants were a

first- or multiple-time participant in CDM programs.

Marketing and

advertising

effectiveness

assessment

Specifically, this assessment identified how participants first heard of

the program as well as the most influential factor(s) in their decision to

participate.

Observed bill

savings and

information

presentation.

This assessment determined the participants’ perceived financial

benefit from participating against their expectations. As well, questions

were asked regarding the format of the information presented to the

participants on their energy use in order to determine effectiveness as

well as areas for improvement.



3. IMPACT FINDINGS

This chapter presents the findings of Navigant’s evaluation of the impacts of the APP in the winter of

2015/2016 and the summer of 2016.

Navigant’s key impact evaluation findings include:

1. The APP continues to deliver substantial demand reductions coincident with the IESO

system peak. The average per participant demand impact during the IESO system peak hour on

September 7th was 0.76 kW for standard Energate participants, 0.81 kW for Yukon participants

and 0.26 kW for BYOD participants.

2. BYOD participants deliver consistent, material DR impacts across both seasons. BYOD

participants delivered an average of 0.18 kW of DR during High price summer days and 0.17 kW

during winter High price days.

3. BYOD participants are schedulers, not responders. These participants delivered material and

consistent DR – within the standard APP period of 3pm to 9pm. When Critical peak prices were

imposed outside this window, they delivered very little DR.

4. Yukon participants appear to rely principally on their technology to respond. Yukon

participants delivered no DR in the winter months, prior to the activation of their technology,

despite still being subject to APP prices. That said, Yukon participants’ summer impacts were

higher than any other technology group.

5. Energate Pioneer customers deliver substantial winter DR. Pioneer participants delivered an

average of 0.6 kW on Critical and High price events, substantially more in the winter than any

other technology group. These results are, however estimated based on only a very small group

of 26 participants.

6. Standard Energate participants are increasingly dividing into high-value and low-value

participants. Since the summer of 2015, participants have increasingly migrated away from the

“balanced” conservation setting (50% of participants in summer 2015, only 38% in summer 2016)

to either the less aggressive or more aggressive conservation settings, with those selecting the

more aggressive conservation settings (e.g., “Savings”) delivering more DR than participants with

those settings did in the summer of 2015, and those selecting the less aggressive conservation

settings delivering less DR than customers participants with those settings did in the summer of

2015.

7. On average participating in the APP increased customer energy consumption. As would be

expected given the price signal of the APP rate, on average participants have responded to the

program by increasing overall energy consumption by a small (and not statistically significant)

amount, even as they have substantially reduced their consumption during the highest-priced

periods.

The remainder of this chapter is divided into five sections:

1. Event Demand Impacts. This section provides the estimated average demand (kW) impacts by

APP price level, and for each of the critical peak events, as well as hourly plots of average actual

and counterfactual (baseline) participant demand.



2. System Peak Demand Impacts. This section provides the estimated average demand (kW)

impacts during each of the the winter of 2015/2016 and the summer of 2016 system peak hours,

as defined by the IESO.

3. Energy Impacts. This section provides the estimated average seasonal energy (kWh) impacts

for the winter of 2015/2016 and the summer of 2016.

4. Participant Cost Impacts. This section provides the estimated participant bill impacts (i.e., the

total impact on customer electricity commodity costs of participating in the APP), as well as the

distribution of rebates paid out by PowerStream.

5. Conservation Settings Over Time. This section describes how participant’s Energate

conservation settings evolved during the summer.

3.1 Event Demand Impacts

This section of Chapter 3 is itself divided into five sub-sections:

Overall Average Event Impacts - Winter. This sub-section provides and discusses the overall

average APP period impacts by price level and technology for the winter of 2015/2016.

Overall Average Event Impacts - Summer. This sub-section provides and discusses the overall

average APP period impacts by price level and technology for the summer of 2016.

Hourly Event Impacts – Winter 2015/2016. This sub-section provides and discusses a graphical

overview of hourly event impacts for the winter of 2015/2016, some numerical examples of those

impacts in text. Complete average hourly impacts may be found in Appendix B, attached as a

separate Excel spreadsheet document.

Hourly Event Impacts – Summer 2016. This sub-section provides and discusses a graphical

overview of hourly event impacts for the summer of 2016, some numerical examples of those

impacts in text. Complete average hourly impacts may be found in Appendix B, attached as a

separate Excel spreadsheet document.

Average Event Impacts by Conservation Setting. This sub-section provides and discusses the

average APP period impacts by Energate conservation setting.

3.1.1 Overall Average Event Impacts - Winter


level and technology type for the winter of 2015/2016 is presented in Figure 14, below. In addition to

providing an estimate of the average demand impact by price-level in the APP period, this table presents

the average 90% confidence interval surrounding those estimates as well as a column that flags whether

the estimated impact is statistically significant or not. An estimate with a confidence interval that straddles

zero is not statistically significant.



Figure 14: Average Program Impacts by Price – Winter 2015/2016


90% CI*

Point Estimate

(kW)

Mean Upper 90% CI*

Stat. Significant?

Win

ter

BYOD





Energate (Standard)





Energate (Pioneer)





Yukon

Critical Peak Event -0.43 -0.02 0.38 No

High Price Day -0.22 -0.06 0.11 No

Medium Price Day -0.15 -0.08 -0.01 Yes

Low Price Day -0.15 -0.09 -0.03 Yes


Navigant analysis

These impacts are the average across all the relevant events and all participants included in the

estimation set.

Two facts must be borne in mind when examining the table in Figure 14:

The updated Yukon control software was not deployed until the summer of 2016. Yukon

participants’ response is therefore entirely behavioural (no automation is in place).

The high degree of uncertainty associated with the critical peak events is largely a function of how

few critical peak hours there are. Recall that only 0.3% (11) of the nearly 4,400 hours in the winter

were critical peak hours.

The three key pieces of information provided by Figure 14 are:

1. Energate Pioneer participants deliver the highest impacts. This is as expected, since this

(very small) group of participants includes only participants whose principal heating fuel is

electricity, and whose electric heat is controlled by the Energate Pioneer devices.

2. BYOD participants deliver the second-highest impacts. Despite not having access to any

utility-provided automation for price-response, this group delivered relatively substantial demand

response impacts – nearly twice those delivered by the standard Energate customers with full



price-response automation. This suggests that BYOD participants undertook meaningful

behavioural changes in response to the prices.

3. Yukon participants delivered no demand response. Yukon participants show statistically

significant increases in demand at the lower price-levels and no statistically significant change in

demand at the higher prices..

As noted above, it should be recalled that Yukon participants had access to no automated price-response

during the winter of 2015/2016. Conclusions regarding the quality of DR delivered by technology group

must therefore be regarded preliminary until the evaluation of APP winter of 2016/2017 can provide

estimated impacts for the fully-equipped Yukon group.

3.1.2 Overall Average Event Impacts - Summer


level and technology type for the summer of 2016 is presented in, below. In addition to providing an

estimate of the average demand impact by price-level in the APP period, this table presents the average

90% confidence interval surrounding those estimates as well as a column that flags whether the

estimated impact is statistically significant or not. An estimate with a confidence interval that straddles

zero is not statistically significant.

Figure 15: Average Program Impacts by Price – Summer 2016


90% CI*

Point Estimate

(kW)

Mean Upper 90% CI*

Stat. Significant?

Su

mm

er

BYOD





Energate (Standard)





Energate (Pioneer)




Low Price Day -0.10 -0.04 0.01 No

Yukon




Low Price Day 0.00 0.05 0.10 No


Navigant analysis



The two key pieces of information provided by Figure 15 are:

1. Yukon participants deliver more DR than Energate participants during critical peak events.

Although impacts from the two groups are very similar during High and Medium events, Yukon

participants deliver materially more DR on the hottest, highest priced days.

2. Despite having no automation, BYOD customers deliver substantial impacts. BYOD

participants’ deliver about a third of the impact of Yukon participants, despite lacking any price-

responsive automation.

The degree to which average participant impacts per High price APP period vary with outdoor

temperature may be observed in Figure 16. In this plot, the estimated impact (Y-axis) of each of the four

technology groups on High-priced days is plotted against the average outdoor temperature. Different

colour dots capture the different technology groups.

To compare the APP results with those of other residential DR programs, Navigant has also plotted the

reported ex-post DR impacts for individual days from four residential A/C programs that employ 50%

cycling strategies32

.

The variation in impacts, and the approximate relationship between temperature and average per-

participant APP impacts can be clearly seen in Figure 16. Some clear trends emerge from this plot:

Energate Pioneer participants’ summer impacts are more or less unaffected by the weather.

Price-response from this (very small) sub-group may stem entirely from non-weather sensitive

loads.

Yukon participants deliver materially higher DR impacts than standard Energate participants

during the hottest days of the year.

32

Many of these programs also employ other cycling strategies – e.g., 65%, 100%, etc. but often much less

frequently than the 50% events. For consistency across programs, only the 50% cycling strategy impacts have been

plotted.



Figure 16: HIGH Price Summer APP Period Impacts Compared to 50% Cycling Residential DR

Impacts

Source: PowerStream interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather

data, Duke Energy Progress, Ontario Power Authority, Pacific Gas & Electric, Southern California Edison and Navigant analysis

The non-PowerStream data points on the plot above are drawn from:

Duke Energy Progress’ (formerly Progress Energy Carolinas) EnergyWise Home (program year

2011)

Ontario Power Authority’s peakSaver program (program years 2010, 2011, 2012 and 2013)

Pacific Gas and Electric’s SmartAC Program (program years 2012, 2013 and 2014)

Southern California Edison’s Summer Discount Program (program years 2013 and 2014)

3.1.3 Hourly Event Impacts – Winter 2015/2016

This sub-section of section 3.1 provides a graphical summary and discussion of winter hourly program

impacts on High days and during some critical peak periods. The complete set of values corresponding to

these graphs (and for other price periods, and by conservation setting) may all be found in Appendix B in

a separate Excel spreadsheet document. The complete set of plots for all technology groups and day

types may be found in Appendix A.



Figure 17, below shows the average actual and counterfactual (baseline) demand of all participants with

the standard Energate technology in the estimation sample on High price days.

The actual observed hourly consumption (or average hourly

demand) is shown by a solid black line. The dashed black

line corresponds to the counterfactual. This is the baseline –

what Navigant has estimated average participant demand

would have been on these days had there been no APP

program. The very finely dotted lines on either side of the

counterfactual dashed line trace out the 90% confidence

interval, and the thicker blue dash-dotted line (above the

rest, and read off the right axis) tracks the average hourly

outdoor drybulb temperature in Celsius.

Figure 17: Average APP High Day – Winter, Energate (Standard)

Source: Alectra interval data, Util-Assist tracking data, Environment Canada weather data, and Navigant analysis

The program effect is clearly evident, but modest. There is a small impact immediately prior to the event

(pre-heating) and after it (snapback) and a modest, reasonably consistent impact throughout the event.

The impact is considerably larger for the Energate Pioneer participants on the same set of High days, as

may be seen in Figure 18, below. The load profile of the Pioneer participants is much flatter than those of

the standard Energate participants, reflective of the fact that all of the Pioneer participants use electric

baseboards as their primary heating equipment, whereas the requirement that homes have central air

means that the vast majority of standard Energate participants heat their homes with gas (and thus have

a load profile more reflective of occupancy-related loads like lighting and cooking).

Legend for Load Profile Plots



Figure 18: Average APP High Day – Winter, Energate (Pioneer)


Over the course of the winter of 2015/2016, two critical peak events were called in the morning, to test the

degree to which participants (and their equipment) would respond to critical peak events outside of the

standard APP period. One of these, for the standard Energate participants, is shown in Figure 19, below.

As during the High periods, the standard Energate customers contributed modest DR impacts during the

morning critical peak periods. There appears to have been a high degree of variation in the critical peak

impacts, however: winter Critical peak impacts, although reasonably substantial, were not statistically

significant for any technology group except for the Pioneer Energate participants.

Figure 19: Winter CPP Event, Energate (Standard), January 20, 2016




BYOD participants contributed more significant impacts during the High priced periods, and during the

afternoon Critical peak periods than the Energate participants, but slightly less during the winter morning

Critical peak events. The average impact of the Critical peak impact amongst standard Energate

participants on the January 20th event was 0.12 kW (see Appendices A and B), but only 0.1 kW for BYOD

customers (see Appendices A and B and Figure 20, below). Further, BYOD participants contributed no

DR impacts at all during the other morning Critical peak event, but Energate participants did.

Given that the average BYOD winter Critical peak impact was more than one and a half times the

Energate impact, this indicates that the BYOD participants are much less capable of delivering DR

outside of the APP period than they are during it. This suggests that these participants are highly

dependent on scheduling, either explicit scheduling automation (e.g., via thermosat programming) or by

learned scheduled price response habits (similar to standard time-of-use price response).

Figure 20: Winter CPP Event, Energate (Standard), January 20, 2016


3.1.4 Hourly Event Impacts – Summer 2016

This sub-section of section 3.1 provides a graphical summary and discussion of summer 2016 hourly

program impacts on High days and during some critical peak periods. The complete set of values

corresponding to these graphs (and for other price periods, and by conservation setting) may all be found

in Appendix B in a separate Excel spreadsheet document. The complete set of plots for all technology

groups and day types may be found in Appendix A.

Figure 21, below, shows the average load profile of standard Energate participants on High priced days

during the summer of 2016 as the solid black line. The counterfactual – what Navigant estimates the load

profile would have been absent the program, is illustrated by the black dashed line. As in the summer of

2015, the load profile is characterized by four significant features:



A very modest shift upward in consumption in the early morning hours, likely a response to the

very low APP Off-Peak price.

A substantial period of pre-cooling in the hours immediately prior to the APP period. This pre-

cooling ensures customer comfort during the event as well as consistent demand response

across the event.

A deep trough during the event as the Energate Foundation responds to the Alectra price signal

and adjusts the thermostat set-point.

Post-event snapback as the compressor works to restore the home to its typical (i.e., non-price-

response) temperature set-point.

There has been no significant change in these features since the summer of 2015.

Figure 21: Average APP High Day – Summer, Energate (Standard)


As noted above, pre-cooling has a dual purpose: in addition to ensuring participant comfort during the

event, it is delivers a consistent DR impact across the entire APP period. The Energate Foundation

delivers DR by adjusting the thermostat set-point. The physical result of this is that the A/C compressor

that it controls will shut down completely until the home’s internal temperature has risen to the new set-

point, at which point the A/C compressor will begin to cycle as required to maintain the new (higher)

temperature. The result of this form of DR is that impacts tend to be very high in the first hour or so of an

event (since the A/C unit has effectively been switched off), but then begin to decay substantially after

that, as the home’s internal temperature rises.

This effect is partially visible in Figure 22, below. This shows the average load profile of Energate

participants on August 11th, when a Critical peak period was called beginning at 1pm. Combined with the

standard APP period (which was High that day), this mean that participants were subject to a temperature

set-back for eight hours. The decay in the DR impact as the Critical peak period transitions to the High

price period is clearly visible in Figure 22, below.



Figure 22: Summer CPP Event, Energate (Standard), August 11, 2016


As in the winter months, BYOD participants show a moderate, but consistent price-response during high

days in the summer, as shown in Figure 23, below. These participants’ load profile impacts due to the

program in non-APP period parts of the day are also much more consistent than those of the Energate

participants. BYOD participants have modestly increased consumption in the morning (as would be

expected given the price difference), but are clearly not pre-cooling.

The source of the DR contributed by the BYOD customers is unclear. Although impacts are clearly

correlated with temperature, this may be due in large part to the fact that prices are correlated with

temperature. That is, it is possible that BYOD participants may be contributing modest, but meaningful,

amounts of DR from non-A/C behavioural actions (e.g., setting timers on appliances). This hypothesis is

supported by two pieces of evidence: the very consistent impacts estimated during High days (i.e., no

late-period decay in impacts as indoor temperature hits a new set-point), and the inter-seasonal

consistency of impacts. Recall that the average High day DR contribution by BYOD customers in the

winter was 0.17 kW and 0.18 kW in summer. This may be contrasted with the average High day

contribution of a standard Energate participant of 0.1 kW in the winter and 0.53 kW in the summer.



Figure 23: Average APP High Day – Summer, BYOD


Although evidence does exist to suggest that BYOD participants do provide DR from a more varied set of

end-uses than Energate or Yukon participants, an examination of BYOD load profiles on one of the

hottest summer days does suggest that A/C set-point adjustment remains an important part of the mix.

See, for example, the BYOD load profile for the August 11, 2016 Critical peak event (Figure 24, below).

Impacts on this particular (very hot) day clearly demonstrate decay over the period of the event, for the

reasons mentioned above. This indicates that, on the very hottest days at least, A/C scheduling is an

important (but perhaps not the only – per the evidence above) component of the DR delivered by BYOD

participants. If these participants are responding via thermostat programming, it is possible that

encouraging pre-cooling and very aggressive set-point thermostat adjustment during the APP could

deliver more substantial DR impacts.

A second very important piece of information is revealed by the plot in Figure 24: although the Critical

peak event begins at 1pm (hour ending 14), demand response from the participants does not begin until

3pm – the start of the APP period. This reinforces the point discussed in the section above: the BYOD

participants are reliable contributors of DR in all seasons, but they rely on scheduling. The evidence

suggests that it is unlikely that this group of participants would contribute as much DR were they subject

only to a Critical peak (that is to say, intermittent and unscheduled) price signal.



Figure 24: Summer CPP Event, BYOD, August 11, 2016


As was noted previously, the Yukon participants’ enabling technology differs from that of the Energate or

BYOD participants in the manner in which it enables price-responsive DR. The Energate Foundation

automatically adjusts participants’ set-point in response to price signals and participant settings. BYOD

participants appear to be scheduling their thermostats to increase set-point during APP events. Yukon

participants’ enabling technology on the other hand delivers price-responsive DR through cycling.

Cycling ensures a consistent DR impact without the need to pre-cool, but it also means that the transition

from a higher to a lower price in the same curtailment period will yield not a reasonably gradual decay in

the DR impact, but rather a short-term snapback within the curtailment period. Observe Figure 25, below.

A sudden and substantial snapback is clearly evident occurring immediately after the end of the Critical

peak period, but while participants are still subject to the High price.



Figure 25: Summer CPP Event, Yukon, August 11, 2016


This effect is to be expected, given the technology being used to deliver the price-response automation,

but it has important ramifications for when a Critical peak event must be called. In the summer of 2016,

the third-highest system coincident peak demand day was this day, August 11. The peak demand hour,

however was between 5pm and 6pm, exactly the hour in which this intra-curtailment snapback period

occurred: the Critical peak period began one hour too soon and resulted in a greatly diminished DR

impact during one of the summer’s top five system coincident hours.

3.1.5 Average Event Impacts by Conservation Setting

This sub-section of section 3.1 provides the average estimated impact per participant by price level and

conservation setting, and contrasts the conservation-setting specific impacts from summer of 2016 with

those of summer of 2015. Recall that the conservation settings only apply to the standard Energate

participants. BYOD and Yukon participants do not set conservation settings, and there are an insufficient

number of Pioneer participants to support any reasonable estimated impacts at the conservation setting

level.

Note also that in the summer of 2016 there was a meaningful number of Energate participants for whom

Energate could not provide any conservation setting data. The impacts of these individuals is included

under the “Unknown” category.

Figure 26, below shows the average per participant kW impacts by conservation setting and APP price

period, and technology group for the winter of 2015/2016. Estimated impacts that are not statistically

significant at the 90% level are indicated with an “n/s”.



Figure 26: Average Event Impact (kW) by Conservation Setting and Technology – Winter

Price Period: Critical Peak

Event High Price

Day Medium

Price Day Low Price

Day

Co

nse

rva

tio

n S

ett

ing

Max Comfort 0.02 n/s 0.03 n/s 0 n/s -0.02 n/s

Comfort 0.21 n/s 0.1 n/s 0.05 -0.03 n/s

Balanced 0.15 0.09 0.04 0.01 n/s

Savings 0.25 n/s 0.18 0.13 0.06

Max Savings 0.26 0.33 0.21 0.15

Unknown 0.11 n/s 0.1 n/s 0.05 0.02 n/s

Energate Average* 0.13 n/s 0.1 0.05 0.02 n/s

Energate Pioneer 0.6 0.62 0.36 0.22

BYOD 0.21 n/s 0.17 0.1 0.02 n/s

Yukon -0.02 n/s -0.06 n/s -0.08 -0.09 Source: PowerStream interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather

data, and Navigant analysis

In nearly every case, impacts track as expected, with more aggressive conservation settings yield higher

DR impacts, and higher-priced days yielding higher impacts than lower priced days. Two notable

exceptions to this pattern are the fact that:

Participants having selected the “Comfort” setting deliver a higher point estimate of impact during

critical peak periods than the more aggressive “Balanced” conservation setting participants; and,

Participants having selected the “Max Savings” conservation setting deliver less DR during the

critical peak periods than during the High price APP period.

Both effects are likely explained by uncertainty. The point estimate of Critical peak period impacts for the

“Comfort” setting participants is higher than that of the “Balanced” participants, but this value is also not

statistically significant; the confidence bounds are very wide. So much uncertainty is associated with the

estimate that nothing conclusive can be said about this deviation from the expected pattern of savings.

Likewise, although statistically significant, the estimated Critical peak period DR impact delivered by “Max

Savings” participants is highly uncertain, with the 90% confidence interval straddling the estimate High

APP period DR impact of the same participants.

The percentage impacts (the point estimates above, divided by the estimated counterfactual) are shown

in Figure 27 below.



Figure 27: Average Event Impact (%) by Conservation Setting and Technology – Winter


Event High Price

Day Medium

Price Day Low Price

Day

Co

nse

rva

tio

n S

ett

ing

Max Comfort 2% n/s 2% n/s 0% n/s -2% n/s

Comfort 15% n/s 8% n/s 4% -3% n/s

Balanced 12% 8% 4% 1% n/s

Savings 19% n/s 15% 12% 6%

Max Savings 20% 27% 19% 14%

Unknown 7% n/s 7% n/s 4% 2% n/s

Energate Average* 10% n/s 9% 5% 2% n/s

Energate Pioneer 19% 23% 14% 12%

BYOD 10% n/s 10% 6% 2% n/s

Yukon -2% n/s -5% n/s -7% -10% Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis

Figure 28, below shows the average per participant kW impacts by conservation setting and APP price

period, and technology group for the summer of 2016. Estimated impacts that are not statistically

significant at the 90% level are indicated with an “n/s”.

Figure 28: Average Event Impact (kW) by Conservation Setting and Technology – Summer 2016


Event High Price

Day Medium

Price Day Low Price

Day

Co

nse

rva

tio

n S

ett

ing

Max Comfort 0.13 n/s 0.09 0.03 n/s 0.02 n/s

Comfort 0.77 0.46 0.24 0.06 n/s

Balanced 1.16 0.75 0.4 0.07

Savings 1.44 1.05 0.55 0.09

Max Savings 1.45 1.03 0.63 0.11

Unknown 0.21 n/s 0.1 0 n/s -0.02 n/s

Energate Average* 0.79 0.53 0.27 0.05

Energate Pioneer 0.17 n/s 0.17 0.13 -0.04 n/s

BYOD 0.28 0.18 0.08 0.01 n/s

Yukon 0.93 0.56 0.25 0.05 n/s Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis

It is striking to compare the impacts for the summer of 2016 shown in Figure 28 with those of the previous

summer of 2015, shown in Figure 29 below. What is striking about this comparison is while the summer

2016 impacts are substantially higher in the Critical peak period for the more aggressive conservation

settings, they are also lower for the less aggressive conservation setting. Likewise, overall average

impacts for the Critical peak period are lower in 2016 than in 2015.



Figure 29: Average Event Impact (kW) by Conservation Setting and Technology – Summer 2015


Event High Price

Day Medium

Price Day Low Price

Day

Co

nse

rva

tio

n

Se

ttin

g

Max Comfort 0.20 0.13 0.01 -0.02

Comfort 1.02 0.45 0.16 0.02

Balanced 1.05 0.60 0.28 0.08

Savings 1.19 0.78 0.36 0.12

Max Savings 1.15 0.75 0.38 0.14

Average 0.88 0.52 0.23 0.06 Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis

It is unclear as to why the “Comfort” and “Max Comfort” setting participants deliver less DR in summer of

2016 than in summer of 2015, but the driver of the lower overall result is due in large part to the growth in

the size of the groups that deliver lower impacts.

In particular, for the summer of 2016, Energate customers for whom no conservation setting data were

available made up approximately 10% of Energate participants during critical peak events. Recall that

from Figure 28 that this group of participants contributed approximately as much DR as did the “Max

Comfort” conservation setting participants. Figure 30 shows the distribution of conservation settings

during Critical peak periods across the two years. Although in 2016 there are .

Figure 30: Distribution of Critical Peak Period Conservation Setting, Summer 2015 vs 2016

Conservation Setting Summer

2015 Summer

2016

Max Comfort 24% 26%

Comfort 6% 9%

Balanced 47% 35%

Savings 11% 9%

Max Savings 13% 10%

Unknown NA 10% Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis

The percentage impacts (the point estimates above, divided by the estimated counterfactual) are shown

in Figure 31 below.



Figure 31: Average Event Impact (%) by Conservation Setting and Technology – Summer 2016


Event High Price

Day Medium

Price Day Low Price

Day

Co

nse

rva

tio

n S

ett

ing

Max Comfort 5% n/s 4% 2% n/s 2% n/s

Comfort 30% 21% 14% 6% n/s

Balanced 46% 35% 25% 7%

Savings 57% 48% 34% 10%

Max Savings 61% 51% 42% 11%

Unknown 9% n/s 5% 0% n/s -2% n/s

Energate Average* 32% 25% 17% 6%

Energate Pioneer 11% n/s 12% 11% -4% n/s

BYOD 12% 9% 6% 1% n/s

Yukon 39% 28% 17% 5% n/s Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis

3.2 System Peak Demand Impacts

In addition to average impacts by APP price period, Navigant has estimated the average program impact

in the peak hour of the five highest peak days in the winter of 2015/2016 and the summer of 2016. These

hours, five for the winter and five for the summer, are referred to below as “5CP” hours for concision. In

fact only some of these hours (those in the summer) are the system-coincident peak hours.33

Figure 32 shows the dates and times of the five seasonal peak hours considered for this part of the

analysis. This table also provides the price to which participants were subject during that specific hour.

33

All five of the summer hours shown in this table were the top five Ontario coincident peaks in the period from May

1, 2016 through April 30, 2017. All of the top five Ontario coincident peaks in the period from May 1 2015 through

April 30, 2016 occurred in July, August or September of 2015.



Figure 32: Seasonal Peak Hour Schedule and Associated Prices

Hour Name Date Hour Ending

(EPT): APP Price

Win

ter

5CP Hour 1 2016-01-04 19 Medium

5CP Hour 2 2016-02-11 19 Critical

5CP Hour 3 2016-01-19 19 Medium

5CP Hour 4 2016-01-18 19 High

5CP Hour 5 2016-02-13 19 Off-Peak

Sum

mer


5CP Hour 2 2016-08-10 19 High

5CP Hour 3 2016-08-11 18 High


5CP Hour 5 2016-08-12 18 High Source: Alectra, IESO

The average impacts per participant for the peak hours of the five peak winter days are shown in Figure

33, below. Key to note is that the fifth highest system peak demand day in the winter of 2015/2016 was a

Saturday (February 13). Since this was an APP Off-Peak period, it should be expected that DR impacts

during this hour were negative (i.e., demand increased).



Figure 33: Average Participant Impacts (kW) During Winter System Peak Coincident Hours

Technology Hour Name Date Hour

Ending (EPT):

Mean Lower 90% CI

Point Estimate

(kW)

Mean Upper 90% CI

Stat. Significant?

Win

ter

BYOD

5CP Hour 1 2016-01-04 19 0.11 0.18 0.24 Yes

5CP Hour 2 2016-02-11 19 0.06 0.29 0.53 Yes

5CP Hour 3 2016-01-19 19 0.08 0.13 0.17 Yes

5CP Hour 4 2016-01-18 19 0.06 0.18 0.30 Yes

5CP Hour 5 2016-02-13 19 -0.02 0.01 0.03 No

Energate

5CP Hour 1 2016-01-04 19 0.00 0.05 0.10 Yes

5CP Hour 2 2016-02-11 19 -0.04 0.13 0.30 No

5CP Hour 3 2016-01-19 19 0.01 0.04 0.07 Yes

5CP Hour 4 2016-01-18 19 -0.01 0.07 0.16 No

5CP Hour 5 2016-02-13 19 -0.03 -0.01 0.00 No

Energate (Pioneer)

5CP Hour 1 2016-01-04 19 0.67 0.80 0.93 Yes

5CP Hour 2 2016-02-11 19 0.20 0.66 1.12 Yes

5CP Hour 3 2016-01-19 19 0.36 0.46 0.55 Yes

5CP Hour 4 2016-01-18 19 0.16 0.44 0.72 Yes

5CP Hour 5 2016-02-13 19 -0.34 -0.30 -0.25 Yes

Yukon

5CP Hour 1 2016-01-04 19 -0.29 -0.18 -0.07 Yes

5CP Hour 2 2016-02-11 19 -0.42 -0.02 0.39 No

5CP Hour 3 2016-01-19 19 -0.32 -0.24 -0.16 Yes

5CP Hour 4 2016-01-18 19 -0.37 -0.17 0.03 No

5CP Hour 5 2016-02-13 19 -0.11 -0.07 -0.03 Yes Source: PowerStream interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather


Figure 34, below, provides the impacts for the five summer 2016 system peak coincident hours.



Figure 34: Average Participant Impacts (kW) During Winter System Peak Coincident Hours

Technology Hour Name Date Hour

Ending (EPT):

Mean Lower 90% CI

Point Estimate

(kW)

Mean Upper 90% CI

Stat. Significant?

Sum

mer

BYOD

5CP Hour 1 2016-09-07 18 0.06 0.26 0.45 Yes

5CP Hour 2 2016-08-10 19 0.17 0.22 0.28 Yes

5CP Hour 3 2016-08-11 18 0.22 0.43 0.63 Yes

5CP Hour 4 2016-07-13 19 0.12 0.31 0.50 Yes

5CP Hour 5 2016-08-12 18 0.23 0.29 0.34 Yes

Energate

5CP Hour 1 2016-09-07 18 0.51 0.76 1.01 Yes

5CP Hour 2 2016-08-10 19 0.58 0.65 0.72 Yes

5CP Hour 3 2016-08-11 18 0.35 0.61 0.87 Yes

5CP Hour 4 2016-07-13 19 0.67 0.92 1.16 Yes

5CP Hour 5 2016-08-12 18 0.70 0.76 0.83 Yes

Energate (Pioneer)

5CP Hour 1 2016-09-07 18 -0.07 0.27 0.60 No

5CP Hour 2 2016-08-10 19 -0.18 -0.09 0.00 Yes

5CP Hour 3 2016-08-11 18 -0.18 0.17 0.52 No

5CP Hour 4 2016-07-13 19 -0.08 0.25 0.58 No

5CP Hour 5 2016-08-12 18 -0.01 0.07 0.16 No

Yukon

5CP Hour 1 2016-09-07 18 0.52 0.81 1.10 Yes

5CP Hour 2 2016-08-10 19 0.83 0.91 0.99 Yes

5CP Hour 3 2016-08-11 18 -0.07 0.23 0.53 No

5CP Hour 4 2016-07-13 19 0.91 1.20 1.48 Yes

5CP Hour 5 2016-08-12 18 0.74 0.82 0.90 Yes

The most surprising result in Figure 34 is the estimated Yukon impact for the 5CP Hour 3. It is only 0.23

kW, and it is statistically non-significant. This is a major contrast to the impacts on the second highest and

fourth highest peaks (0.91 and 1.2 kW respectively). The reason for this low impact is that on August 11th

a Critical peak event was called from 1pm to 5pm, whereas the 5CP hour occurred between 5pm and

6pm. Following the Critical peak period there was a short period of snapback, as the Yukon device

switched from its most aggressive cycling mode (in the Critical peak period from 1pm to 5pm) to the

slightly less aggressive cycling mode for the High period than ran for the remainder of the APP period that

day (from 5pm to 9pm). This is very much evident in Figure 25, above, which clearly shows demand

briefly spiking in the period immediately follows the Critical peak period. Essentially, the Critical peak

event was ended one hour too soon.

Figure 35, below, below, shows the estimated by conservation setting and technology for the winter

system peak coincident hours.



Figure 35: Average Participant Impacts (kW) During Winter System Peak Coincident Hours by

Technology and Conservation Setting

Hour Name: 5CP Hour 1 5CP Hour 2 5CP Hour 3 5CP Hour 4 5CP Hour 5

Date: 2016-01-04 2016-02-11 2016-01-19 2016-01-18 2016-02-13

Hour Ending (EPT): 19 19 19 19 19

Co

nse

rva

tio

n S

ett

ing

Max Comfort 0 n/s 0 n/s 0.01 n/s 0 n/s -0.04

Comfort 0.09 0.17 n/s 0.11 0.15 -0.13

Balanced 0.02 n/s 0.17 0 n/s 0.05 n/s 0.01 n/s

Savings 0.11 0.18 n/s 0.10 0.06 n/s -0.01 n/s

Max Savings 0.29 0.27 0.24 0.34 0.07

Unknown 0.03 n/s 0.16 n/s 0.05 n/s 0.09 n/s 0 n/s

Energate Average* 0.05 0.13 n/s 0.04 0.07 n/s -0.01 n/s

Energate Pioneer 0.80 0.66 0.46 0.44 -0.30

BYOD 0.18 0.29 0.13 0.18 0.01 n/s

Yukon -0.18 -0.02 n/s -0.24 -0.17 n/s -0.07

*Excludes Pioneer Devices Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis

Figure 36, below, shows the estimated by conservation setting and technology for the summer system

peak coincident hours.

Figure 36: Average Participant Impacts (kW) During Summer System Peak Coincident Hours by

Technology and Conservation Setting

Hour Name: 5CP Hour 1 5CP Hour 2 5CP Hour 3 5CP Hour 4 5CP Hour 5

Date: 2016-09-07 2016-08-10 2016-08-11 2016-07-13 2016-08-12

Hour Ending (EPT): 18 19 18 19 18

Co

nse

rva

tio

n S

ett

ing

Max Comfort 0.07 n/s 0.08 0.24 0.12 n/s 0.07

Comfort 1.05 0.32 0.3 n/s 0.82 0.56

Balanced 1.18 0.98 0.73 1.32 1.13

Savings 1.45 1.40 1.30 1.85 1.54

Max Savings 1.32 1.39 1.48 1.60 1.60

Unknown 0.03 n/s 0.05 n/s 0.12 n/s 0.32 n/s 0.20

Energate Average* 0.76 0.65 0.61 0.92 0.76

Energate Pioneer 0.27 n/s -0.09 0.17 n/s 0.25 n/s 0.07 n/s

BYOD 0.26 0.22 0.43 0.31 0.29

Yukon 0.81 0.91 0.23 n/s 1.20 0.82

*Excludes Pioneer Devices Source: PowerStream interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather


3.3 Energy Impacts

As was the case for the summer 2016, energy savings (or increases) were, for the most part statistically

insignificant.



A statistically non-significant result means that the hypothesis that the program has had no effect (on

energy use) cannot be rejected at the selected level of confidence – the confidence interval that bounds

the estimate straddles zero. Put more simply, it indicates that a very high degree of uncertainty as to

whether the program has had any effect.

A statistically non-significant result may be interpreted in one of two ways:

1. The program has had no impact. This is an interpretation sometimes applied for purely

behavioural types of treatment (e.g., in-home displays, online portals, etc.)

2. An impact exists, but is highly uncertain. Sufficient external evidence exists of a program

effect (or the likelihood of a program effect) that the estimated point estimate of the impact, is

reported as the best available (although imperfect) estimate of the program effect.

Figure 37Error! Reference source not found., below shows the estimated average energy savings

during the winter of 2015/2016, by day type and in total. Savings are shown by technology, and by

conservation setting for the standard Energate participants. Note that a negative estimate indicates

an increase in consumption (i.e., negative savings). Figure 37 is immediately followed by Figure 38,

which shows the same savings (or increases, where the value is negative) as a percentage of the

participant’s baseline energy use.

Figure 37: Winter Energy Savings (kWh) by Conservation Setting and Technology

Day Type: Weekdays Weekends/

Holidays Seasonal

Total

Co

nse

rva

tio

n S

ett

ing

Max Comfort -104 -67 -172

Comfort -164 -117 -282

Balanced -93 -61 -154

Savings -42 -59 -101

Max Savings 183 55 239

Unknown -18 -14 -32

Energate Average* -64 -50 -114

Energate Pioneer 123 -48 75

BYOD -28 -30 -58

Yukon -334 -141 -475

* Does not include Pioneer participants. Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis



Figure 38: Winter Energy Savings (%) by Conservation Setting and Technology


Holidays Seasonal

Total C

onse

rva

tio

n S

ett

ing

Max Comfort -4.3% -5.5% -4.6%

Comfort -6.8% -9.7% -7.6%

Balanced -4.2% -5.3% -4.4%

Savings -1.7% -4.6% -2.6%

Max Savings 7.1% 4.3% 6%

Unknown -0.7% -1% -0.8%

Energate Average** -2.6% -4% -3%

Energate Pioneer 2.1% -1.7% 0.8%

BYOD -0.8% -1.7% -1.1%

Yukon -12.7% -11% -12%


Navigant analysis

Navigant’s interpretation of these results is that participants have increased their winter energy

consumption because of the program, but that the magnitude of that increase is highly uncertain, and that

there are some individual groups that appear to be achieving some modest (although not statistically

significant) winter energy savings.

Specifically, during the winter months it appears that Energate participants that have selected the “Max

Savings” conservation setting, and participants with the Energate Pioneer technology (i.e., those whose

primary space-heating equipment is electric baseboard) are the only ones achieving any savings.

On average, non-Pioneer Energate participants increased their winter energy consumption by

approximately 3%, Pioneer participants reduced their energy consumption by approximately 1%, BYOD

participants increased consumption by about 1%, and Yukon participants substantially increased

consumption, consumption 12% more electric energy than they otherwise would have (absent the

program) in the winter of 2015/2016.

Likely a major reason for the Yukon participants’ high level of increased consumption is that during the

winter of 2015/2016 the update to the legacy peaksaver system had not yet been made that would

automate these customers’ response. This means that there exists no automated daily demand response

to (partially) compensate for the increased consumption that results from the much lower Off-Peak period

price.

In the summer 2016 period, all the technology groups increase consumption on average (although this

increase is not statistically significant), as shown in Figure 39, below. As with winter energy impacts, the

kWh level impacts in Figure 39 are immediately followed by the energy savings (or when negative,

increases) as a percentage of the participant baseline.



Figure 39: Summer Energy Savings (kWh) by Conservation Setting and Technology


Holidays Seasonal

Total C

onse

rva

tio

n S

ett

ing

Max Comfort -136 -66 -202

Comfort -49 -82 -131

Balanced -56 -53 -109

Savings -77 -88 -164

Max Savings -24 -56 -80

Unknown -152 -110 -261

Energate Average** -87 -68 -155

Energate Pioneer -87 -59 -145

BYOD -117 -79 -196

Yukon -156 -106 -261


Navigant analysis

Figure 40: Summer Energy Savings (%) by Conservation Setting and Technology


Holidays Seasonal

Total

Co

nse

rva

tio

n S

ett

ing

Max Comfort -4.5% -4.5% -4.5%

Comfort -1.6% -5.6% -2.8%

Balanced -2% -3.8% -2.5%

Savings -2.6% -6% -3.6%

Max Savings -0.8% -3.9% -1.8%

Unknown -5.2% -7.9% -6%

Energate Average** -2.9% -4.8% -3.5%

Energate Pioneer -3% -4.1% -3.3%

BYOD -4.3% -5.9% -4.7%

Yukon -5.6% -7.7% -6.2%


Navigant analysis

The result that most participants have increased consumption is reasonable and intuitive. These

participants have, in exchange for their participation, been provided with a steeply discounted Off-Peak

rate, 5.9 cents/kWh, compared to 8.7 cents/kWh for the standard RPP TOU Off-Peak period or 18

cents/kWh for the standard RPP TOU On-Peak period.

The relative level of changes in energy consumption are also aligned with expectations for each of the

groups: those Energate participants selecting “Max Savings” are likely to be the most frugal and have

also increased energy use in the summer the least, and have achieved some (very small and imprecise)

savings in the winter. Pioneer participants achieve savings in the winter (when their space-heat is

controlled), but not in the summer, and their winter savings are achieved on weekdays, when the APP

prices will result in their heat being controlled, but not on the weekends when no automation exists.



All customer groupings have slightly increased their seasonal energy consumption by 3 – 6%. As would

be expected (as in winter) percentage increases in consumption are higher on weekends and holidays

(when there are no APP periods to incent energy reductions) than on weekdays.

The tables above are all aggregated and therefore “net” values – that is, they show the net result of

demand reduction during higher-priced periods and demand increases during lower priced periods. What

is less clear from the tables above is the pattern of demand changes that lead to this net increase in

consumption.

Some sense of these hourly changes may be obtained by examining the plots shown above (e.g., Figure

21, above, and all those provided in Appendix A, showing event load profiles). For readers interested in

diving into these patterns in greater detail, Appendix A provides a set of user-generated dynamic tables

that allow the user to examine the hourly impacts of the program by technology, day-type, demographic,

and (where applicable) conservation setting. An example of such a table is presented in Figure 41 below,

showing the average BYOD participants’ response on High-priced days.



Figure 41: Hourly Impact Example Table – BYOD Summer High Price Days

Hour Ending

(Eastern Prevailing

Time)

Load Absent

Program

Load With Program

Load Impact (Positive = Demand

Reduction)

Drybulb Temperature

(Celsius)

APP Period

APP Price Level

CPP Period

1 1.15 1.28 -0.13 22 0 High 0

2 0.96 1.08 -0.12 22 0 High 0

3 0.84 0.93 -0.10 21 0 High 0

4 0.76 0.83 -0.07 21 0 High 0

5 0.72 0.78 -0.07 20 0 High 0

6 0.74 0.80 -0.06 20 0 High 0

7 0.78 0.87 -0.08 20 0 High 0

8 0.89 0.97 -0.08 20 0 High 0

9 0.98 1.06 -0.09 20 0 High 0

10 1.05 1.14 -0.09 22 0 High 0

11 1.17 1.27 -0.10 24 0 High 0

12 1.25 1.37 -0.11 25 0 High 0

13 1.40 1.51 -0.11 27 0 High 0

14 1.49 1.61 -0.12 28 0 High 0

15 1.58 1.65 -0.07 29 0 High 0

16 1.65 1.50 0.15 29 1 High 0

17 1.80 1.62 0.18 29 1 High 0

18 1.99 1.77 0.22 29 1 High 0

19 2.03 1.83 0.20 29 1 High 0

20 2.00 1.80 0.21 28 1 High 0

21 1.91 1.76 0.15 27 1 High 0

22 1.87 1.97 -0.10 26 0 High 0

23 1.72 1.88 -0.15 24 0 High 0

24 1.40 1.61 -0.21 23 0 High 0 Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis

Generally speaking, across technologies and seasons, the pattern of change is very similar, and is in line

with the goal of any time-differentiated rates: participants are reducing their demand during the higher-

priced hours, and increasing their demand in the lower-priced hours. The overall effect is to improve

customers’ load factor: that is to reduce the differential between their peak demand and their average

demand. Increasing customer load factors is a generally desirable system goal: the more consistent

customer loads are, the easier it is to plan for supplying those loads.

Although the general pattern of shifting is common across technology groups, there are some important

differences in the specifics:



BYOD participants’ load increases tend to be reasonably consistent across Off-Peak hours.

These participants’ response suggests that changes in usage patterns may be attributed to

scheduled thermostat (or other device34

) programming, or some consistent behavioural shift.

Energate participants’ load increases are much more pronounced in the period immediately

preceding the APP period or Critical peak events. This is by construction: the Energate device is

programmed to apply pre-cooling (or pre-heating, in the winter) so that event-impacts remain

consistent over the course of the event.

Yukon participants’ load increases fall somewhere between those of the Energate participants

and the BYOD participants. They are certainly not as consistent and predictable as the BYOD

participants’, but neither is the pre-cooling as pronounced as that of the Energate participants.

The general pattern that emerges is that participants appear to rely on the automation available to them:

dynamic response automation (Energate) leads to dynamic shifts in demand, whereas scheduling

automation (BYOD) leads to more consistent shifts in demand. These technology characteristics should

be carefully considered for future program implementations and suggest that some technologies are

better suited for some rate structures than others.

3.4 Participant Cost Impacts

This section of Chapter 3 provides the estimated program impacts on participant costs and is divided into

two sub-sections:

Commodity Cost Impacts. This sub-section provides the average estimated impact on

participant electricity commodity costs, compared to what they would have been had they not

participated in the program at all.

Participant Rebates. This sub-section summarizes the rebates issued to APP participants.

Rebates represent the difference between what participants actually paid in standard TOU rates

for their consumption in the period from November 1, 2015 through October 31, 2016, and the

commodity costs imposed by the APP and tracked by shadow bills.

3.4.1 Commodity Cost Impacts

This sub-section provides the average estimated impact of the APP on participant commodity costs

compared to what they would have paid had they not participated in the APP at all. These average

commodity cost impacts are calculated by:

Applying the APP prices to the actual consumption observed for the participants in the sample.

Applying the standard TOU rate prices to counterfactual (baseline) estimated consumption for all

participants in the sample.

Averaging these two values within the customer groups of interest (i.e., technology group, and for

Energate customers, conservation setting) by hour.

Taking the difference of the seasonal total of the average values calculated above.

34

For example, dehumidifier, pump, or other device timers.



This analysis represents a truer estimate of participant commodity cost benefits than an examination of

the rebates (incentives) provided by PowerStream, since it is effectively a comparison of what those costs

would have been without the program with the commodity costs imposed by the APP rate. That is, it

accounts for the way participant loads have changed. By contrast the rebates only track the difference in

participant commodity costs given the program-induced behaviour change.

Figure 42, below shows the average commodity cost impact of program participation by conservation

setting,35

participant technology and season.

Figure 42: Average Total Summer Commodity Cost Savings ($)

Price Period: Winter

2015/206 Summer

2016

Co

nse

rva

tio

n S

ett

ing

Max Comfort $60 -$20

Comfort $54 $25

Balanced $66 $64

Savings $80 $83

Max Savings $111 $112

Unknown $81 -$16

Energate Average* $70 $35

Energate Pioneer $246 $50

BYOD $119 $1

Yukon $38 $29


Navigant analysis

As may be seen in Figure 42, bill savings in the winter of 2015/2016 were considerable, despite relatively

modest DR response (see Figure 26), and material increases in consumption by the participants (Error!

Reference source not found.). This is due to the relatively small number of higher priced APP periods

called in that winter. Recall that APP prices are set to be revenue neutral under the assumption that a

certain proportion of each price period is called. The winter of 2015/2016 was very mild, and had much

lower gas prices than in many recent years, resulting in many fewer High or Medium price days than

anticipated.

A comparison of Figure 1, showing the distribution of hours used for price-setting purposes, with Figure

10, showing the actual distribution of hours by price period, shows that the winter had fewer than a third

as many High price periods as assumed for the purpose of rate-setting and half as many Critical peak

hours assumed for price-setting. This disconnection between rate-setting assumptions (based on

historical averages) and actual pilot system conditions resulted in a structural benefit for all participants.

This disconnection is to be expected: system conditions used for rate-setting are based on representative

historical test-years and in any given year actual system conditions will result in participants either

achieving a structural benefit (if, e.g., the weather is milder than in the test year) or a structural loss (if,

e.g., the weather is more extreme). In a full program deployment, provided the test year truly is

35

Conservation settings (unlike demographic groups) may change over the course of the summer. The estimated

commodity cost impact shown here is the estimated seasonal impact assuming that the given participant uses the

same conservation setting for the entire summer.



representative (and deviations in price-frequency from the assumed rate-setting distribution are random),

the structural benefits in any one year would be compensated for by a structural loss in another. .

Average summer commodity cost savings were generally quite small, and Energate customers’ savings

were about half of what they were in the summer of 2015. Energate customers for all conservation

settings (except “Max Savings”) reduced their commodity costs more in the summer of 2015 than in the

summer of 2016. This consistency across different types of participants indicates that the principal cause

of reduced commodity cost savings may have been the increase in the cost of APP Off-Peak electricity

from 4.9 to 5.9 cents per kWh (and the attendant effect on the other prices as a result of the revenue

neutrality constraint). The increase in the Off-Peak price resulted in a smaller differential between Off-

Peak and non-Off-Peak prices in the summer of 2016 compared to the summer of 2017. This then also

reduces the effective arbitrage potential of shifting energy consumption to the Off-Peak period (or even

increasing it in that period). Economic theory would suggest that this reduced arbitrage opportunity is a

contributing factor to reduced commodity cost savings. This suggests that more savings could be

achieved with more aggressive price differentials, although more aggressive such differentials might

discourage participant recruitment, particularly when bill protection is removed.

Figure 43, below, expresses these commodity cost savings as a percent of total RPP commodity costs

that would have been incurred, on average, by participants had they not participated in the program.

Figure 43: Average Total Summer and Winter Commodity Cost Impact (%)36

Price Period: Winter

2015/206 Summer

2016

Co

nse

rva

tio

n S

ett

ing

Max Comfort 15% -4%

Comfort 14% 5%

Balanced 17% 12%

Savings 20% 16%

Max Savings 27% 22%

Unknown 18% -3%

Energate Average* 17% 7%

Energate Pioneer 27% 10%

BYOD 20% 0%

Yukon 9% 6% Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, Environment Canada weather data, and

Navigant analysis

Since the econometric approach used to estimate impacts relies on estimating average relationships

(e.g., between demand and the APP price), the variation around individual participant impacts would

make it inappropriate (and potentially misleading) to report the distribution of individual participant

commodity cost savings.

36

In this figure the values represent savings, so a negative number indicates an average increase in participant

commodity cost.



3.4.2 Participant Rebates

This sub-section provides some summary statistics relating to the rebates (incentives) issued to APP

participants for their participation in the program.

Recall that program participants, while enrolled in the program, were required to continue to pay their

standard monthly bill, calculated based on the existing RPP TOU rate. PowerStream also provided a

shadow bill that tracked participants’ costs under the alternative APP rate. At the end of each seasonal

period (corresponding to the RPP season), if the total billing amount under the APP rate was less than

the actuals paid by a participant, that participant received a rebate for the difference in the form of a

cheque. When the APP rate would have resulted in a participant paying more than under the conventional

RPP TOU rate, no rebate was issued and no penalty applied.

Figure 44, below, reports the maximum, minimum and mean rebates calculated by PowerStream for all

APP participants that were included in the econometric estimation sample. Negative values for rebates

are shown to provide context only – participants with a calculated negative rebate would have received no

rebate from PowerStream, nor been charged any incremental fee. Values over $500 are shown for

reference, though the maximum rebate was capped at $500 per season.

Figure 44 reports summary statistics of actual rebates by conservation setting and technology type, but it

must be understood that many of the participants whose rebates contribute to the distributions shown

below may have used more than one conservation setting over the summer. In this case, the

conservation setting to which each participant is assigned is that which they selected for the most days in

the seasonal period in question.

Figure 44: APP Summer Rebate Incentive Summary Statistics

Price Period:

Winter 2015/206 Summer 2016

Max

Rebate Mean

Rebate Min

Rebate Max

Rebate Mean

Rebate Min

Rebate

Co

nse

rva

tio

n S

ett

ing

Max Comfort $256 $80 $8 $537 $40 -$252

Comfort $391 $85 $10 $581 $108 -$106

Balanced $456 $83 $20 $743 $137 -$81

Savings $583 $92 $27 $528 $165 -$7

Max Savings $252 $87 $26 $613 $181 -$51

Unknown $400 $84 $6 $441 $57 -$160

All Energate* $583 $83 $6 $743 $105 -$252

Energate Pioneer $568 $243 $72 $205 $73 $13

BYOD $1,000 $127 $17 $611 $17 -$160

Yukon $237 $86 -$5 $298 $39 -$79


Navigant analysis

When comparing summary statistics regarding customer rebates to the customer commodity cost savings

it is important to bear in mind what, precisely the rebate metric is tracking.



A participant with a high rebate is one for whom the differential in commodity costs under the APP and

TOU rate is very high – given their behavioural response to the pilot. Put another way, a participant could

achieve a substantial rebate by increasing their consumption considerably in the APP Off-Peak periods.

This would substantially increase their commodity costs over what they would have been absent the

program, while still being much lower than they would have been under the APP rate, thus delivering a

substantial rebate.

Consider the following (entirely hypothetical) example:

APP participant X enrolls in program for no other reason than to reduce his electricity commodity

costs between 11am and 3pm on non-holiday weekdays (APP Off-Peak period, but RPP TOU

On-Peak period)

Participant X increases his consumption from 11am to 3pm on non-holiday summer weekdays by

500 kWh, and otherwise makes no change to his consumption profile.

The cost impacts for this participant are:

o Commodity cost impact: 5.9 cents37

x 500 kWh = $29.50 increase in participant

commodity cost.

o APP Rebate: 18 cents38

x 500 kWh - 5.9 cents x 500 kWh = $60.50 rebate from Alectra

Thus rebates are achieved both by motivated participants working hard to reduce their demand during

times of system need, but also by participants that may (in effect) be gaming the pilot in order to avoid

status quo RPP TOU charges. In all likelihood most participants fall on the spectrum between these two

extremes.

Figure 45, below, provides distributions of APP rebates by participant technology and seasonal period.

Some of the key features revealed by this set of plots are:

Energate customers’ rebates are much more varied in the summer, when there is much more

potential for response (because the Energate device controls space-cooling).

Energate Pioneer customers’ rebates are much more varied in the winter, when there is much

more potential for response (because the Energate Pioneer devices control space-heating)

The more symmetrical shape of the rebate distribution for both BYOD and Yukon customers is

the result of the fact that there was greater price variation in the summer, meaning that rebates

are more reflective of the variation in participant price-response.

37

APP Off-Peak rate, summer of 2016. 38

TOU On-Peak, summer of 2016



Figure 45: Distribution of APP Rebates by Technology


3.5 Conservation Settings Over Time

This section of Chapter 3 provides a summary of how standard Energate participants’ conservation

settings evolved over the course of the period from November 1, 2015 through to the end of October

2016. Recall that participants using the Energate Foundation could select one of five conservation

settings that determined the automatic thermostat response for each price-event. “Max Comfort”

participants’ thermostats do not automatically adjust.

One limitation of this analysis, as noted previously above, is that conservation setting data were available

only as weekly snapshots. This means that although it is possible to track the trends in conservation

settings across the summer, it is not possible to track any high frequency conservation setting changes.

For example the data are insufficiently granular to capture the effects of participants that generally keep

their conservation setting at “Balanced” but switch it to “Max Savings” on a particular day (in anticipation

of a Critical or High price period, for example). Further, no conservation settings data were available for

approximately 10% of the standard Energate participants (listed as “Unknown” in the plot below). These

participants also contributed very little in the way of demand response impacts (see section 3.1).



In general, however, weekly conservation settings were very stable over the period of analysis, with the

relative shares of participants by conservation setting changing very little over time, as may be seen in

Figure 46, below.

Figure 46: Weekly Conservation Setting Distribution Over Time

Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, and Navigant analysis

.

As may be seen in in Figure 46, not only are conservation settings mostly static, but the vast majority of

participants use the default one “Balanced”. Although approximately a fifth of participants use the

“Savings” or “Max Savings” settings, nearly a third of participants are using the “Max Comfort” setting.

When the “Unknown” conservation setting is removed and the distribution is compared with the

distribution of conservation settings from the summer of 2015, it is clear that a higher proportion of

Energate participants are now using the less aggressive conservations settings.

Figure 47: Conservation Settings, Summer 2015 vs Winter and Summer 2015/2016

Conservation Setting

Distribution in Last Week of October

2015 2016

Max Savings 12% 13%

Savings 9% 9%

Balanced 50% 38%

Comfort 6% 9%

Max Comfort 22% 31% Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, and Navigant analysis

Given the very modest contribution that participants selecting the “Comfort” and “Max Comfort” settings

make to overall program impacts, future iterations of the APP should consider either eliminating the “Max

Comfort” option from the device, or re-calibrating it such that even “Max Comfort” delivers some impacts.



Despite the very stable distribution of conservation setting selections over the course of the period of

analysis, there does appear to exist a trend for migration away from the “Balanced” conservation setting

over time, as illustrated in Figure 48, below..

Figure 48: Conservation Setting Distribution – First and Last Week of Period of Analysis

Conservation Setting

Distribution in Week of: Difference*

2015-11-01 2016-10-24

Max Savings 9% 12% 2%

Savings 7% 8% 1%

Balanced 39% 34% -6%

Comfort 5% 8% 2%

Max Comfort 27% 27% 1%

Unknown 12% 12% 0% *Rounding may affect values shown

Source: Alectra interval data, Util-Assist tracking data, Energate conservation setting data, and Navigant analysis

Figure 49, below, shows the distribution of how individual participants changed their conservation settings

between the first setting selected and the final one. A negative number on the X-axis indicates a group of

participants that move to a less aggressive setting (e.g., “Balanced” to “Comfort”). A positive number

indicates a move to a more aggressive setting. Thus, given there are only five conservation settings, this

figure indicates (see far right column) that 1% of participants started with a “Max Comfort” setting and had

moved to a “Max Savings” setting by the end of the summer.

Figure 49: Participant Conservation Setting Changes Over Time

Source: PowerStream interval data, Util-Assist tracking data, Energate conservation setting data, and Navigant analysis

More than three quarters of participants did not change their conservation setting. Of those that did

change their conservation settings between the beginning and the ending of the pilot were as likely to

make them less aggressive as they were to make them more aggressive.



4. PROCESS EVALUATION FINDINGS

This chapter presents the findings of Navigant’s process evaluation of the APP pilot in the winter of

2015/2016 and the summer of 2016.

The key findings from the process evaluation are as follows:

12. Most participants would recommend APP to friends and family. The key metric of participant

satisfaction: approximately 85% of participants would recommend participation in the APP to

friends and family.

13. The primary motivation for program participation is to reduce electricity bills. 78% of all

respondents stated that reducing their electricity bill was their primary motivation for participating

in the program.

14. Prior experience with conservation amongst APP survey respondents is low. Only 25% of

respondents indicated that they had previously participated in a conservation initiative offered by

PowerStream (i.e. before PowerStream became Alectra). This indicates that the APP program is

likely attracting a new type of customer to conservation.

15. Participant engagement with the technology and the program is high, but declined over

time. Most participants viewed electricity pricing information once a day or more, however

frequency of interaction fell 8% between the initial and final survey. As well, although most

participants acted upon pricing information every day, the proportion of respondents that

undertook daily actions fell 5% from the initial to final survey.

16. Customer satisfaction with the APP is high, and improved over time. Participant feedback

demonstrates that the majority participants found the program “as good as expected” indicating a

reasonable degree of satisfaction. Responses stating that the program was “better than

expected” increased 14% between the initial and final survey. Despite this finding, a significant

number of participants indicated dissatisfaction with the program (18% stated the program was

worse than expected).

17. Most (59%) program participants are 50 or more years old and live with only one other

person. To reach younger and more diverse demographics, Alectra may consider changing APP

marketing and advertising efforts and/or change the program’s design to attract a more varied

age-range of customers.

18. Some participants were dissatisfied with their interaction with the Customer Care Centre

(CCC). 7% of respondents indicated that their interaction with the CCC was dissatisfactory. While

this is relatively low, it does provide an area that Alectra could focus on to improve customer

satisfaction.

19. Other motivations for program participation include: receiving the free Energate

thermostat, the environmental aspects of APP and the “risk-free” aspect of APP. 85% of

Energate respondents indicated that the no-cost thermostat was important in their decision to

participate, emphasizing that future participation in the program may be negatively impacted if the

thermostat is no longer offered. Also, the environmental benefits of the APP program are an

important motivator for participation, particularly for BYOD/Yukon participants (8% more

BYOD/Yukon participants selected this answer than Energate respondents). 76% of all

respondents indicated that the “risk-free” aspect of the APP program motivated their decision to



participate, emphasizing the importance of providing risk mitigation support to participants

through future program design.

20. Providing sufficient program information is important to program enrollment. The clear

majority of participants (84% to 91% depending on the group) agreed that the initial information

provided to them was sufficient to enroll. A small proportion did state that the information was

insufficient, particularly the Partially Engaged group (16%). The most common issues participants

noted regarding the initial information package include: communications were hard to understand;

a lack of communication around the specifics of the rates and eligible equipment; participants

were unclear about the risk-free aspect of the program.

21. Participants’ “most-liked” aspect of the program was saving money on electricity bills.

Emphasizing the money-saving aspect of the program may benefit customer satisfaction as well

as future recruitment efforts.

22. Participants’ “least-liked” aspect of the APP were the high peak rates. Certain respondents

found that there was a lack of communication about the peak rates when they were initially

enrolling in the program.39

The remainder of this chapter provides the results of the process evaluation and is divided into the

following sections:

Participant Demographics. This section provides demographic detail on the surveyed

participants.

Motivations for Program Participation. This section explores reported participant motivations

for enrollment.

Enrollment Process. This section summarises the participants’ experience enrolling in the APP

program.

Participant Technology Engagement. This section discusses participants’ engagement with the

APP technology throughout the pilot period.

Customer Satisfaction. This section provides a summary of participant-reported program

satisfaction.

Participant Support of the Initiative. This section summarises participant support for the

continuation of the APP, including their likeliness to recommend the program.

4.1 Participant Demographics

The figures below highlight program participant demographic features collected by the surveys, and

provide a snapshot of some of the key characteristics of APP participants. This information may be

helpful in refining program messaging and recruitment.

Figure 50 illustrates the age range of participants as identified through survey responses.40

Most

respondents were evenly distributed across the following three age categories: 40-49, 50-59 and 60-69.

39

Navigant notes that in public-facing on-line documents available to non-participants the evaluation team could only

find one explicit reference to the non-Off-Peak prices, on page 13 of the participant brochure. No mention of any price

level aside from the Off-Peak rate occurs in any of the FAQ pages, or on the main “landing page” for the program. 40

Only Survey 1 and 3 included questions about participant age.



Age of respondents was relatively consistent across the four participant groups (i.e., Engaged, Partially

Engaged, Energate and BYOD/Yukon). Since the summer of 2015, there has been modest growth in the

group of participants under the age of 40 (from 17% in summer 2015, to 26% in this round of surveys),

and a quite substantial growth in the oldest demographic. Only 7% of respondents to the summer 2015

survey were over the age of 60. Whereas 37% - over a third – in this round of surveys report being over

60.

Figure 50. Respondent Age Bracket (All Respondents)

Source: Survey 1 and 3

These results are important to consider when developing effective strategies to educate future customers

on the initiative’s value-proposition.

Figure 51 documents the type of advertisement that most motivated program enrollment amongst survey

respondents. As shown in this plot, half of the survey respondents indicate that the communication

channel most responsible for their enrollment was email. Conversely, only 2% of survey respondents

indicated that social media motivated them to participate. If Alectra wishes to increase participation

amongst younger age brackets, additional emphasis on its social media messaging may be helpful.



Figure 51. Advertisement that Motivated Participation (Engaged vs. Partially Engaged)

Source: Survey 1

Engaged respondents were 7% more likely than Partially Engaged respondents to state that they were

motivated to participate in the APP after seeing an advertisement in the mail (Figure 51). Conversely,

10% of Partially Engaged respondents indicated that word of mouth motivated their participation vs. 0%

for Engaged participants.

The effectiveness of the various advertising methods for recruitment was relatively consistent across all

participant groups. If the survey population is representative of Alectra’s target participant group,

messaging through emails and the Alectra webpage will likely be effective during future program

recruitment efforts.

Figure 52 documents the distribution of the number of full-time residents in survey respondents’

households. Most (55%) respondents’ households contained three or more residents, likely indicating that

the program is popular amongst families, although a material number of respondents’ (36%) households

have only two full-time residents.

Figure 52. Full-time Residents in Participant Households (All Respondents)

Source: Survey 1 and 3



When asked if the APP program was their first experience with a conservation initiative, 74% of

respondents said “yes” (see Figure 53, below). As in the summer of 2015, The APP continues to attract a

customers with no previous experience with CDM programs, potentially expanding the market for CDM in

Alectra’s service territory.

Figure 53. Participants’ First Experience with Conservation Programs (All Respondents)

Source: Survey 1

4.2 Motivations for Program Participation

The motivations for program participation across each participant group (Engaged vs. Partially Engaged,

BYOD/Yukon vs. Energate) are discussed in the following section.

4.2.1 Engaged and Partially Engaged Respondents

Post enrollment participants (i.e. those that did not participate in the summer 2015 APP) were asked

during Survey 1 to identify their main motivation for participating in the APP program (Figure 54). The

most common response for both Engaged and Partially Engaged respondents was “to save money on my

electricity bills” with 81% and 77% of responses, respectively. This is consistent with the responses seen

during previous year evaluation efforts.

In addition, over two thirds of respondents indicated that they expected to save over $50 on their

electricity bills over the course of six months. These two findings offer insight into the messaging that

could be used in advertising campaigns for future recruitment efforts (i.e. emphasize the potential

electricity-bill savings aspect of the program).



Figure 54. Motivations for Program Participation (Engaged vs. Partially Engaged)

Source: Survey 1

A relatively small number of respondents indicated that the primary motivator for their participation was “to

receive an Energate thermostat or in-home display,” (4% for Engaged and 2% for Partially Engaged).

Another question from Survey 1 asked respondents how important receiving a thermostat or in-home

display was in motivating the respondent to participate in the program. Nearly all (85%) of respondents

that did receive a device stated that the device was an important factor in motivating them to participate.

The main motivation may to be save money, but free equipment is clearly an important secondary factor

for many participants.

4.2.2 BYOD/Yukon and Energate Respondents

Like Engaged and Partially Engaged respondents, the most common reason for participation amongst

both BYOD/Yukon and Energate respondents was “to save money on my electricity bills,” with 78% and

79% of responses respectively (Figure 55). This emphasizes the finding that reducing the electricity bill is

the most important aspect for program participation for all respondents.

One major area of difference between the BYOD/Yukon and the Energate respondents is that the former

were more likely to respond that the main motivating factor for participation was “for environmental

reasons,” with 12% of respondents vs. 4% indicating that this was a meaningful factor. Marketing

materials and program information that emphasize the environmental aspects may be more effective with

individuals that already have a smart thermostat and those that have previously participated in a

provincial DR or CDM program than those that do not.



Figure 55. Motivations for Program Participation (BYOD/Yukon vs. Energate)

Source: Survey 1

4.2.3 Risk-Free Aspect of the APP Program

In addition to the importance of reduced electricity bills, free devices and environmental attributes, the

risk-free aspect of APP was “very important” to 76% of all respondents in their decision to enroll (Figure

56, 2016). Further, 95% of all respondents stated that the risk-free aspect was at least somewhat

important. This response was consistent across all respondents, and is consistent with the results from

the 2015 APP evaluation (Figure 56, 2015). This highlights the importance of providing some level of risk

mitigation support to participants through future program design.

Figure 56. Importance of “risk-free” aspect of APP program, in decision to participate (2015 vs.

2016 Process Evaluation)

Source: PowerStream APP Summer 2015 Evaluation and Survey 1



4.3 Enrollment Process

To assess the program enrollment process for participants, the sufficiency of initial program information

and the ease of sign-up were investigated. The results of this investigation are summarized in the

following section.

4.3.1 Engaged and Partially Engaged Respondents

When asked if initial information was sufficient for enrollment, 91% of Engaged and 84% of Partially

Engaged respondents stated that it was (Figure 57). This is a positive result for both the Engaged and

Partially engaged group, but it may be beneficial to investigate what further information was needed,

particularly by the Partially Engaged group. The survey allowed respondents to provide open-ended

responses as to why the information was not sufficient, the most common reasons are listed below, and

acting on these may increase the number of APP participants in the future.

Program communications were hard to understand (e.g. lack of visual representations)

Communications did not provide enough details on rates (e.g. what the peaks will be)

Communications did not provide enough information on eligible equipment (e.g. thermostats,

heating types)

Clarity surrounding the risk-free aspect of the program

The second complaint, in particular, should likely be addressed by Alectra. Navigant found that that in

public-facing on-line documents available to non-participants the evaluation team could only find one

explicit reference to the non-Off-Peak prices, on page 13 of the participant brochure. No mention of any

price level aside from the Off-Peak rate occurs in any of the FAQ pages, or on the main “landing page” for

the program.

Figure 57. Sufficiency of Initial Information for Enrollment (Engaged vs. Partially Engaged)

Source: Survey 1

Figure 58, below, documents how easy the survey respondents reported enrollment to be. A substantial

number of Engaged participants (45%) indicated that they “strongly agreed that it was easy to sign up for

the program, 22% than Partially Engaged participants. However, most respondents, 88% of Engaged

and 85% of Partially Engaged, agreed that it was easy to sign up to the program.



Figure 58. Ease of Signing up for APP (Engaged vs. Partially Engaged)

Source: Survey 1

4.3.2 BYOD/Yukon and Energate Respondents

Most of the BYOD/Yukon and Energate respondents stated that the initial program information was

sufficient for purposes of understanding how the APP would function in their homes (84% and 89%,

respectively - Figure 59). This closely mirrors the results of Engaged vs. Partially engaged, leading to the

overall conclusion that the program information is sufficient for most participants.

Figure 59. Sufficiency of Initial Information for Enrollment (BYOD/Yukon vs. Energate)

Source: Survey 1

There was little variation between BYOD/Yukon and Energate responses regarding the ease of signing

up for APP (Figure 60). The results confirm the previous conclusion that, overall, it is easy for participants

to sign up to APP – 89% and 84% of BYOD/Yukon and Energate respondents agreed that it was easy to

sign up.



Figure 60. Ease of Signing up for APP (BYOD/Yukon vs. Energate)

Source: Survey 1

4.4 Participant Technology Engagement

To understand the level of participant engagement with the APP technology, the frequency that

respondents viewed and acted upon electricity pricing information was analyzed. Additionally,

participants’ preferred methods for viewing electricity pricing information was also assessed. A discussion

of the results and insight of this analysis are provided in the following section.

4.4.1 Frequency of Viewing Electricity Pricing Information

This sub-section provides a summary of how often participants examined the electricity price changes to

which they were subject under APP.

4.4.1.1 Engaged and Partially Engaged Respondents

The frequency that Engaged and Partially Engaged respondents viewed electricity pricing information

through the program is shown in Figure 61. The most common response amongst both groups of

respondents was “once a day or more.” This response was 10% more common amongst Engaged

participants than Partially Engaged participants, supporting the idea that engagement with technology

may be correlated with survey participation.



Figure 61. Frequency of Viewing Electricity Pricing Information Available Through the Program

(Engaged vs. Partially Engaged)

Sources: Survey 1, Survey 2, Survey 3 and Survey 4

4.4.1.2 BYOD/Yukon and Energate Respondents

The questions related to the frequency of APP interactions was also asked to both BYOD/Yukon and

Energate participants, and “once a day or more” was the most common response (66% and 65%,

respectively – Figure 62). This shows that viewing APP information has been incorporated into the daily

routines of most participants, suggesting a high-level of engagement with the technology.

Variation in responses between BYOD/Yukon and Energate respondents was very low.

Figure 62. Frequency of Viewing Electricity Pricing Information Available through the Program

(BYOD/Yukon vs. Energate)

Sources: Surveys 2 and Survey 3



4.4.1.3 Variation in Response throughout the Trial

Viewing pricing information once a day was the most common response amongst all four surveys.

However, comparing the results of all respondents to Survey 1 and Survey 4 shows responses of “once a

day or more” fell from 63% in survey 1 to 55% in survey 4. Conversely, respondents that reported viewing

the information “less than once per week” rose from 5% in Survey 1 to 13% in Survey 4. These results

imply that engagement with technology, specifically viewing pricing information, may have declined

slightly throughout the duration of the program.

Figure 63. Frequency of Viewing Electricity Pricing Information Available through the Program

(Survey 1 vs. Survey 4)

Sources: Survey 1 and Survey 4

One explanation for this decline could be that participants engaged with the program in different ways and

no longer needed to view pricing information every day (e.g. they set schedules). Another potential

reason for the decline is that it is natural consumer behaviour to lose interest in a program/initiative over

time.

To increase the frequency that participants view pricing information, it is beneficial to understand what

methods of viewing pricing information that are preferred by participants. Figure 64 depicts responses to

the question, “which communication methods [about pricing information] would you chose to receive,”

asked in Survey 4.

The most preferred communication method reported by respondents was monthly mailed information

reports, with 17% of responses. This preference may have contributed to the high proportion of “less than

once per week” responses found in Survey 4; some participants would rather view pricing information

infrequently. The second most commonly preferred price communication channel (13%), is daily email

alerts. Emphasizing this method of communicating pricing information may be the most effective way of

encouraging participants to view their electricity pricing information once a day or more - if this is in fact a

goal - since it is a more frequent form of communication.



Figure 64. Preference for Communication of APP related information (All Respondents)

Source: Survey 4

4.4.2 Frequency of Acting Upon Electricity Pricing Information

This sub-section outlines how frequently survey respondents reported that they took action in response to

pricing information.


In addition to viewing electricity pricing information, the frequency that a participant will act upon this

information is a significant metric of engagement. Figure 65 documents how frequently Engaged and

Partially Engaged participants reported taking action in response to prices, across the four surveys. The

first significant factor to note is that only a small minority of respondents (6% of Engaged and 9% of

Partially Engaged) indicated that no action was taken in response to electricity pricing.

Respondents that indicate that they take actions daily represent highly engaged participants who have

adopted the technology, and the program, into their daily routine. Engaged respondents were 10% more

likely to respond that they took actions every day depending on the peak price, indicating a relatively

higher engagement than Partially Engaged respondents.



Figure 65. Frequency of Acting Upon Electricity Pricing Information (Engaged vs. Partially

Engaged)

Sources: Survey 1, Survey 2 and Survey 3


BYOD/Yukon and Energate respondents mostly responded that they took some action in response to

pricing information. Also, only 11% and 8% of BYOD/Yukon and Energate respondents, respectively,

indicated that they took no action in response to pricing information (Figure 66). This supports the

statement that engagement with the program is high among all participants.

There is less variation in the frequency of acting upon pricing information between the BYOD/Yukon vs.

Energate group than the Engaged vs. Partially Engaged group. This mirrors the results of the frequency

of viewing pricing information from the two pairs.

Figure 66. Frequency of Acting Upon Electricity Pricing Information (BYOD/Yukon vs. Energate)

Sources: Surveys 2 and Survey 3



4.4.2.3 Variation in Response through the Trial

The frequency of responses from all participants for “yes, I took various actions everyday depending on

peak price,” declined from 51% in survey 1 to 46% in Survey 4. This finding shows that daily engagement

with the technology may have declined slightly throughout the program. This finding is corroborated by

Figure 63 (viewing electricity pricing information daily declined between Survey 1 and 4).

The reasons for this decline may be due to natural factors (decreased interest in the program),

technological factors (participants interacted with the technology in different ways e.g. setting automatic

schedules) or program factors (pricing information is not communicated frequently enough or using the

preferred method).

Figure 67. Frequency of Acting Upon Electricity Pricing Information (Survey 1 vs. Survey 4)


4.4.3 Methods of Viewing Electricity Pricing Information

This sub-section documents the manner in which survey respondents reported that they viewed pricing

information.


As discussed previously, the method of viewing electricity pricing information is an important input into

how participants engage with the APP program technology. The most common method across all groups

is email, which is an easily accessible and a regular method of viewing the information (Figure 68). This

method is also available to all customers, unlike the Energate thermostat, Energate portal and Energate

App, which are only available to those with an Energate system.

Very few respondents indicated that they used text messaging to view pricing information, so it may not

be a beneficial communication method in future program efforts (unless a different demographic is



targeted for participation). This is also supported by the responses from survey 4, in which the least

preferred method of communication is daily text messages (Figure 64).

Figure 68. Methods for Viewing Electricity Pricing and Consumption Information (Engaged vs.

Partially Engaged)

Sources: Survey 1, Survey 2, and Survey 3


Email was also the most common methods used to view electricity pricing information for BYOD/Yukon

and Energate participants. Energate participants reported less of a need for monthly information reports

than BYOD/Yukon participants did, 49% vs. 29% (Figure 69). This may be due to Energate respondents

having more methods for viewing pricing information than BYOD/Yukon participants (APP and Energate

portals, Energate app and the Energate thermostat itself).



Figure 69. Methods for Viewing Electricity Pricing and Consumption Information (BYOD/Yukon vs.

Energate)


4.4.3.3 Energate Interaction Throughout the Trial

Figure 70 highlights how Energate respondents changed their level of interaction with their thermostat

over the trial period. The majority (53%) of respondents consistently engaged with their thermostat and

portal, however, 10% stated that they reduced their interaction over time. As with all programs similar in

nature to APP, over time, participant interactions with a provided technology decline.

Figure 70. Change in Interaction Level with Energate Thermostat and Portal throughout Period (All

Respondents)

Source: Survey 3



4.5 Customer Satisfaction

To assess satisfaction with the APP program, results of survey questions that asked about participants’

overall experience with the program, as well as their “most-liked” and “least-liked” aspects of the initiative

were evaluated. The results of this assessment are summarized in the section below.

4.5.1 Participants’ Experience with the APP

This subsection documents survey respondents’ overall experience of the APP.


Most (83% of Engaged and 86% of Partially Engaged) participants stated that the program was “about

what I expected” or, “better than expected.” Engaged participants were 6% more likely than Partially

Engaged participants to state that the program was better than expected. A small number of respondents

stated that the program was worse than expected (15% for Engaged and 14 % for Partially Engaged),

indicating that there are some aspects of APP that cause dissatisfaction for a material number of

participants.

Figure 71. Participants’ Experience with APP including registration, installation, customer

support, etc. (Engaged vs. Partially Engaged)



Most (87% and 90% of BYOD/Yukon and Energate, respectively) respondents stated that their

experience with APP was “about what I expected,” (Figure 72). This is a positive result given that

participants most likely signed up for the program with high expectations, and only a relatively small

number of respondents stated that the program was much worse than expected (2% for BYOD/Yukon

and 1% for Energate).




support, etc. (BYOD/Yukon vs. Energate)


4.5.1.3 Variation in response through the trial

“Better than expected” responses were 14% more common in Survey 4 than in Survey 1, indicating that

participants’ experience with the program improved throughout the program (Figure 73). Survey 1

respondents include those that recently signed up for the program, and in some cases, recently went

through the installation process. As a result, it is important to determine the effect that the program

enrollment process had on satisfaction of Survey 1 participants.


support, etc. (Survey 1 vs. Survey 4)


In Survey 1, 50% of survey respondents indicated that they had to wait three or more days to be

contacted by an Alectra (then PowerStream) representative to schedule their installation, and 18% of

respondents indicated that assistance from a Alectra representative after installation was not effective.

Both factors may be improved upon to increase initial satisfaction levels as well as to ensure customers

are able to begin engaging their device immediately upon installation.



4.5.2 Participants’ “Most-liked” aspects of the APP program

This sub-section documents the aspects of the program that APP participants most liked.


Figure 74 highlights the responses for Engaged and Partially Engaged participants’ in terms of their

“most-liked” aspects of APP. “Saving money on my electricity bills,” was the highest for both groups,

which mirrors the responses for the major motivators for participating in the program (Figure 54). Insight

into electricity usage is also a highly-liked aspect of the program, so it may be beneficial if Alectra

emphasizes this and the money saving aspect in regular communications with participants (or during

program recruitment efforts).

For Engaged participants, the environmental benefits had the lowest instance of “liked” responses (4%).

(Figure 74).

Figure 74. Participants’ “Most-liked” Aspects of APP (Engaged vs. Partially Engaged)



Saving money on electricity bills was also the “most-liked” aspect of APP for BYOD/Yukon and Energate

respondents, making this the overall “most-liked” aspect of the program for all participants (Figure 75).

The rest of the BYOD/Yukon vs. Energate responses closely resemble the responses of Engaged vs.

Partially Engaged groups, except that the former pair shows less variation than the latter.



Figure 75. Participants’ “Most-liked” Aspects of APP (BYOD/Yukon vs. Energate)

Sources: Survey 3

4.5.3 Participants’ “Least-liked” aspect of the APP program

This sub-section documents the aspects of the program that APP participants least liked.


To determine why participants’ experience with APP did not meet expectations, respondents were asked

what they “liked least” about APP (Figure 76). The most common responses across Engaged and

Partially Engaged participants was that peak rates were too high (41% and 48% respectively), and that

peak times were inconvenient (27% and 24%). Very few respondents indicated that more

education/communication concerning actions to take during the program is needed (i.e. “Difficulty

knowing what action to take to save money”). There was a significant amount of “other” responses. Of

these, many indicated that it would be more convenient to end the peak time earlier. However, the most

common “other” response was that the participant had no issues with the program.

Figure 76. Participants’ “Least-liked” Aspects of APP (Engaged vs. Partially Engaged)





Like Engaged vs. Partially Engaged respondents, high peak rates were the “least-liked” aspect of APP for

both BYOD/Yukon and Energate respondents (48% and 50% respectively), making this the “least-liked”

aspect of the program overall. It may be beneficial for Alectra to analyze this result and adjust future

program communications accordingly. E.g. provide more information about peak rates, and strategies to

avoid consumption during them, through ongoing communications.

Figure 77. Participants’ “Least-liked” Aspects of APP (BYOD/Yukon vs. Energate)

Sources: Survey 3

4.5.3.3 Other Reasons for Dissatisfaction with APP Program

Another reason why respondents may have indicated that the program was worse than expected is their interaction with the Customer Care Centre (CCC). 7% of participants indicated that the support offered by the CCC was dissatisfactory (Figure 78). Although this is a relatively low amount, it provides an area that can be improved in further iterations of the program. Some respondents provided an explanation of why their experience the with the CCC was dissatisfactory. The most common groups of responses are listed below:

It took a long time to connect with the CCC, or the CCC was unavailable (e.g. weekend)

The CCC failed to follow up on a request (e.g. return a phone call)

The CCC did not have answers to the participants’ questions



Figure 78. Effectiveness of Customer Care Centre in offering Technical Support (All Respondents)


4.6 Participant Support of the Initiative

To assess participants’ support of the initiative, responses to questions that determined the likelihood that

participants would recommend the APP program and the Energate System to their friends and family

were investigated. Results of this evaluation are summarized below.

4.6.1 Likelihood to Recommend APP Program to Friends and Family


Most Engaged and Partially Engaged respondents were “very likely” to recommend the APP program to

friends or family (69% and 75%, respectively). Engaged respondents reported 6% less that they were

“very likely” to recommend APP than Partially Engaged respondents. To improve upon the likelihood to

recommend rating, the satisfaction criteria specific to Engaged participants should be considered and

acted upon if possible.



Figure 79. Likelihood of Participant Recommending APP to Friends and Family (Engaged vs.

Partially Engaged)



Like above, BYOD/Yukon and Energate respondents are very likely to recommend the APP program to

friends and family (70% and 78% respectively), showing strong support for the future of the program

overall. BYOD/Yukon respondents, were 8% less likely to recommend APP than Energate respondents.

To improve upon the likelihood to recommend rating, the satisfaction criteria specific to BYOD/Yukon

respondents may be considered and acted upon if possible. For example, the BYOD/Yukon group were

not interested in receiving the Energate thermostat (Figure 75), but other incentives may improve their

likelihood to recommend the program.

Figure 80. Likelihood of Participant Recommending APP to Friends and Family (BYOD/Yukon vs.

Energate)




4.6.1.3 Variation in response through the trial

“Very likely” responses demonstrate high support for the future of the APP program. It is the highest

response for all participant groups and this metric improved by 6% between survey 1 and survey 4

(Figure 81).

Figure 81. Likelihood of Participant Recommending APP to Friends and Family (Survey 1 vs.

Survey 4)


4.6.2 Likelihood to Recommend the Energate System

80% of Energate participants were “very likely” to recommend their Energate system (Figure 82), which is

higher than any group’s recommendation of the APP program. This provides evidence supporting the use

of the Energate system, or some form of incentivized device, in future iterations of the program.

Figure 82. Energate Participants’ Likelihood to Recommend Energate System (Energate

Respondents)

Surveys: Survey 3

More evidence of the impact the Energate system had on participants’ support for the program can be

seen in Figure 83, as the majority (88% and 84%) of Energate users, both Engaged and Partially

Engaged respectively, believe they will be using their Energate system one year from now in the same



manner as they are today. Note that a chart has not been shown for BYOD/Yukon vs. Energate, as few

BYOD and no Yukon participants had an Energate system,

Figure 83. Energate Participants Expect to Use Energate System one year from now in the

(Engaged vs. Partially Engaged)

Surveys: Survey 3



5. CONCLUSION & RECOMMENDATIONS

The first winter and second summer of the APP pilot have been successful at meeting its design goals.

The pilot design has been shown to deliver meaningful peak demand response capability at the time of

system peak, it has done so while also delivering cost saving to participants, and it has done so without

any substantial detrimental impacts on customer satisfaction. Further, it has delivered valuable,

actionable intelligence about the trade-offs inherent to four different groups of technologies that may be

used help participants respond to dynamic rates.

Navigant’s key finding of its evaluation of the APP for the winter of 2015/2016 and the summer of 2016

pilot are:

1. The Yukon technology group delivered the most substantial savings during the coincident

system peak hour. Former peaksaver participants equipped with the Yukon software delivered a

DR impact of 0.81 kW during the system peak hour on September 7th, 2016. During another one

of the five system coincident peak hours, on July 13th, this group delivered 1.2 kW of impacts.

2. DR savings delivered by Energate participants varied substantially depending on the

conservation setting selected. The Energate participants with the most aggressive

conservation settings delivered nearly 2 kW during the system coincident peak hour on July 13th.

By contrast, participants selecting the “Balanced” conservation setting (approximately 40% of

Energate participants with tracked conservation setting data) delivered approximately 1.3 kW of

DR, and participants selecting the “Max Comfort” conservation setting (approximately 30% of

Energate participants with tracked conservation setting data) delivered only 0.12 kW of DR. The

overall average impacts of the Energate group naturally reflects this distribution of savings.

3. On average participants reduced their winter commodity costs by between 9% and 27%

and their summer commodity costs by between 0 and 10% (depending on the technology

group). Winter savings were in large part driven by the small number of higher priced days.

4. BYOD participants deliver the most consistent inter-seasonal impacts. BYOD participants’

DR contributions were consistent in both summer and winter, and in fact delivered the second-

highest impacts (after Pioneer customers) in the winter.

5. The APP participant population skews older and is principally motivated by bill savings.

Fifty-nine percent of participants are 50 years or older, and approximately 80% indicated that their

primary motivation for participation was to save money.

6. APP has been successful in penetrating a segment of PowerStream’s customer-base

previously unengaged with utility-based energy efficiency efforts. Nearly 70% of APP

participants indicated that they had not previously participated in any PowerStream offered

Conservation and Demand Management (CDM) initiative.

7. Nearly 80% of respondents said that the risk-free aspect of the program was very

important in their decision to enroll. Maintaining the first-year risk-free “commodity cost

guarantee”41

is likely to be essential for any successful wider program roll-out.

41

PowerStream provides participants with a guarantee that their electricity commodity costs will not be higher than

under standard RPP TOU rates for the first year of participation. See below for more details.



Based these six key findings, Navigant has the following recommendations for improving the APP design

and maximizing the value of future impact evaluations in future years:

1. The IESO should be more directly involved in call Critical periods. One of the 5CP hours of

the summer occurred immediately following the end of a Critical peak period. As a result of

snapback, the Yukon group contributed much less DR than it potentially could have for this event.

2. Alectra should recommend to BYOD customers that they pre-cool their homes if they

respond to prices with thermostat adjustment. BYOD participants could potential offer

substantially more DR during regularly scheduled summer events, provided they pre-cooled to

ensure DR impacts do not decay, and comfort does not become an issue.

3. Alectra should adjust Energate conservation setting responsiveness such that even the

least aggressive conservation settings contribute some DR. Many participants selecting

these less aggressive settings are effectively free-riding on the program and contributing trivial

amounts of DR but taking advantage of the bill protection and APP schedule to reduce their

average cost of electricity without benefiting Alectra or the system.

4. Alectra should consider leveraging its link with participants to promote additional energy

efficiency initiatives. Three quarters of APP participants have not previously engaged with other

Alectra-offered CDM initiatives, indicating a substantial marketing opportunity for other, non-APP

CDM initiatives. Given the high-levels of program satisfaction indicated by participants, such

participants are likely to be receptive to messaging regarding the incremental benefits such

programs could offer them. Given the importance to survey respondents of bill savings, these

participants are likely to be most receptive to high-impact, low-cost programs (e.g., LED lighting

retrofits, timers for dehumidifiers, behavioural programs, etc.)

5. Alectra should continue to provide participants with ongoing energy management advice

to help participants optimize their APP response to the twin goals of reducing system

peak demand and achieving bill savings. Participant impressions of the initiative as well as the

extent to which they interacted with the program’s tools consistently increased throughout the

program period. Maintaining this on-going engagement will be key to maintaining and improving

program achievements over time.

Documents

Advantage Power Pricing Pilot · Advantage Power Pricing Pilot Confidential and Proprietary Page 6 ©2016 Navigant Consulting, Inc. Do not distribute or copy DISCLAIMER This report