Residential Behavior Programs

Ryan FirestoneOctober 20, 2015

2

Presentation Outline

• RTF history• 7th Power Plan• Where do savings come from?• Modeling principals for the RTF• Where does this measure fit within the

Guidelines?• Measure cost• Next steps for staff and subcommittee(s)

3

OverviewTwo fundamental issues:

• Established impact evaluation methods do not get us the level of detail necessary to count savings appropriately– We’re unlikely to prescriptively measure our way out of this issue– RTF judgement here may be of great value for regional standardizing

• We expect behavior programs (and their implementation) to vary significantly enough that a prescriptive Standard Protocol is not practical– We’d like to provide guidance for impact evaluations– Guidelines identify Program Impact Evaluation as a savings estimation

methodology, but don’t describe measure-specific RTF guidance products

4

RTF History• Standard Protocol approved in March, 2010

– Experimental design (treatment and control/comparison group)– Billing analysis– Normalization for weather and other factors, difference-in-difference measure

of savings– More impact evaluation guidance than prescriptive methodology

– “The RTF didn’t take up the issue of whether behavior-based programs meet the definition of conservation.” meeting minutes from that meeting

• Status set to “Under review for compliance w/ RTF Guidelines” after the Guidelines were developed.

• Staff/CAT proposal: Deactivate this Standard Protocol

5 Power Plan• Residential behavior is being modeled as a conservation resource for the first

time in the 7th Plan (draft).– Focus on long term savings

• Savings that aren’t expected to persist aren’t counted– Curtailment ≠ Conservation

• Savings from reductions in utility (e.g., thermostat adjustment) aren’t considered conservation under the Power Act.

– Not double counting• Potential from Plan-modeled equipment/appliances not re-counted in behavior analysis

– Council staff estimated actions that would satisfy Plan requirements, and saving potential• Reduce water heater setpoint – (BULK OF SAVINGS)• Reduce lighting HOU when not in room• Reduce HVAC usage when not at home

– Up to 70% of homes (need control group) – 172 kWh/home– 48 aMW potential

• Analysis files (see “Residential Behavior” section)• 7th Plan Conservation Resource Advisory Committee Presentations

– December 17, 2014– January 16, 2015

6

RTF Residential Behavior Subcommittee – Sept. 17 Meeting

Meeting to prepare for October RTF discussion • RTF history• 7th Power Plan• Where do savings come from?• Modeling principals for the RTF• Measure categorization (Standard vs.

Custom Protocol)• Next steps for staff and subcommittee

(Meeting presentation and notes)

[discussion from meeting covered later in presentation]

AttendeesJennifer Anziano (RTF Manager)Andie Baker (Abacus Research)Rebecca Blanton (PSE)David Bopp (Flathead Electric)Leona Doege (Avista)Ryan Firestone (RTF Contract Analyst)Lauren Gage (Bonneville)Mark Jerome (CLEAResult)Don Jones Jr. (PacifiCorp)Jim Perich-Anderson (PSE)Josh Rushton (RTF Contract Analyst)Mohit Singh-Chhabra (RTF Contract

Analyst)Bonnie Watson (Bonneville)Gary Wood (Seattle)

7

Uniform Methods Project (UMP)• DOE protocols for determining savings from energy efficiency measures and programs• Chapter 17: Residential Behavior Programs• Applicable to residential behavior programs with large (1,000’s-10,000’s) number of

participants– Each with individual billing data (e.g., by house)

• Experimental Design: – Randomized Control Trial – subjects randomly assigned to group that gets or does not get messaging– Randomized Encouragement Design – all subjects can opt in, subjects randomly assigned to group that

gets or does not get encouragement to participate.• Analysis:

– Difference (kWhcontrol – kWhtreatment)

– Difference-in-Difference ( (kWhpre - kWhpost)treatment – (kWhpre-kWhpost)control

– Simple average, panel regression w/ or w/out fixed-effects– Avoid double counting of trackable program savings – analyze participation data– Avoid double counting of untrackable (upstream) program savings (e.g., lighting) – use surveys

• Similar methods in State and Local Energy Efficiency Action (SEE Action) “Issues and Recommendations” report, evaluations, etc.

8

Where Do Savings Come From?

Purchase Behavior Usage Behavior

Traditional difference-in-difference billing analyses measure total savings relative to control group

Savings can be from changes in equipment and/or changes in the use of existing equipment

Even more categorization is necessary to determine “claimable” savings and align them with other claimed savings

Avoid double counting with non-behavior program

claimed savings

Conservation CurtailmentUpstream program

participation

Non-program purchase

Non-program removal

Trackable program

participation

Avoid double counting with non-behavior program

claimed savings

Avoid double counting with

Momentum savings

Need to consider

persistence

Not considered conservation

under standing interpretation of the Power Act.

Not to scale

9

Modeling Principles for an RTF Behavior Measure

1) Don’t count savings from curtailment (e.g., thermostat adjustment)

2) Avoid double counting– program-incented equipment-based savings – momentum savings

3) Don’t apply risk mitigation and capacity credits to things we don’t expect to persist

• Subcommittee feedback:– Current evaluation methods are a good starting point– Disaggregating savings to address double counting issues and to

exclude curtailment seems appropriate, but is it feasible?

10

Modeling Principles for an RTF Behavior Measure

1) Don’t count savings from curtailment (e.g., thermostat adjustment)2) Avoid double counting

– program-incented equipment-based savings – momentum savings

3) Don’t apply risk mitigation and capacity credits to things we don’t expect to persist

• Staff/CAT recommendation: Use persistent savings as a proxy for conservation (i.e., not curtailment)– Savings that do not persist are likely to reflect a compromise of

utility– Addresses issues 1) and 3) above

• Double Counting issue would still need to be addressed

11

Persistence as a Proxy for Conservation

Avoid double counting with non-behavior program

claimed savings

Conservation CurtailmentUpstream program

participation

Non-program purchase

Non-program removal

Trackable program

participation

Avoid double counting with non-behavior program

claimed savings

Avoid double counting with

Momentum savings

Need to consider

persistence

Not considered conservation

under standing interpretation of the Power Act.

Persistent savings

12

What Kind of Measure is This?

• Standard Protocol?– Requires prescriptive best practice method for estimating savings

• No best practice yet for disaggregating savings: Evaluations to date have not had much success

– Method may be dependent on:• Targeted behaviors• Control vs. comparison group• Granularity of billing data• Normalization requirements• Program size

• Subcommittee agreed that evaluation guidance was more appropriate than a prescriptive protocol

13

What Kind of Measure is This?• RTF Staff/CAT recommend developing this as measure-specific Program Impact

Evaluation guidance– Weigh-in on areas we have something to say, but leave details to

programs/evaluators• Experimental design: sample size, control and treatment group selection• Normalization for weather, rate changes, etc.• What actions to count/not count• Estimating total savings• Estimating disaggregated savings• How to handle persistence

– Would this be worthwhile to the region?

– The Guidelines may need to be modified to accommodate this• Program Impact Evaluation is identified as a savings estimation method, but

Guidelines do not describe measure-specific guidance• Custom Protocol is identified as a less-prescriptive protocol for site-specific savings

estimation methods

14

Measure Cost• For Standard Protocol and Custom Guidance, RTF

does not estimate costs or cost effectiveness

• However, Guidelines say, “costs and benefits should be estimated and documented as described in these Guidelines, as appropriate.”

• Caution: Regional costs include not only program costs, but also customer costs to acquire new equipment

15

Next Steps• Today

– Deactivate the current Standard Protocol• It does not meet the criteria of a Standard Protocol in our Guidelines• It’s probably not as thorough as the guidance we would develop – it’s a good starting point, though

– RTF gives Staff/CAT guidance on key issues:• Should this be developed as Impact Evaluation Guidance (it’s not currently spelled out in the

Guidelines)? – Would this be worthwhile to the region?

• How much disaggregation of savings is necessary?• Is persistent savings a reasonable proxy for conservation (i.e., not curtailment)?• Other recommendations?

• Staff/CAT to review recent evaluations and other material– Summarize methods, outcomes, and challenges

• Work with Residential Behavior Subcommittee to begin developing this measure

16

Motion“Deactivate the Standard Protocol measure:‘Residential: Methods for Evaluating Behavior-Based Energy Conservation Programs in the PNW’.”

and

“Instruct Staff to develop a Residential-Behavior-Program Impact Evaluation Guidance document.”

17

Additional Material

Slides from January 16, 2015 CRAC Presentation

18

19

20

21

22

23

Additional Material

Slide on Curtailment from March 2, 2010 RTF Meeting

Slide 24

B O N N E V I L L E P O W E R A D M I N I S T R A T I O N

The Power Act and RBBP Historically, the Council has excluded measures and practices that reduced the level of

service or utility (in economic terms) provided to consumers by the current measure or practice. – 839a(3). "Conservation" means any reduction in electric power consumption as a result of

increases in the efficiency of energy use, production, or distribution. [Northwest Power Act, §3(3), 94 Stat. 2698.]

– Savings from lowering thermostats for space heating have not been considered conservation under the Act

– Savings from lowering the thermostat on water heaters from 140 F or 130 F to 120 F have been.

Rationale for behavioral change programs as “utility neutral”, or non-sacrificial?– Persistent savings are unlikely to be sacrifices– Programs are asking customers to reduce behavior when it does not change utility (e.g.,

turning off lights or thermostat down when not in the room)– Recent interest in sustainability has created utility for reducing energy consumption,

particularly enabled through technology or information