Upload
danghanh
View
219
Download
0
Embed Size (px)
Citation preview
Connexus Energy Ramsey, MN
Measurement and Verification Report of
OPower Energy Efficiency Pilot Program
Madison, WI
. Minneapolis, MN
. Marietta, OH
. Indianapolis, IN
. Sioux Falls, SD
Contact: Chris Ivanov 1532 W. Broadway Madison, WI 53713
Direct: 608-268-3516 Fax: 608-222-9378
Email: [email protected] Web Site: www.powersystem.org
July 28, 2010
Confidential, Copyrighted, and Proprietary
This document contains information confidential to Connexus Energy (Connexus or
Cooperative) and Power System Engineering, Inc. (PSE). Unauthorized reproduction or
dissemination of this confidential information is strictly prohibited.
Copyright 2010 Power System Engineering, Inc.
This document includes methods, designs, and specifications that are proprietary to Power
System Engineering, Inc. Reproduction or use of any proprietary methods, designs, or
specifications in whole or in part is strictly prohibited without the prior written approval of
Power System Engineering, Inc.
NEITHER POWER SYSTEM ENGINEERING, INC. NOR CONNEXUS ENERGY
SHALL BE RESPONSIBLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, OR
CONSEQUENTIAL DAMAGES (INCLUDING LEGAL FEES AND COURT COSTS)
ARISING OUT OF OR CONNECTED IN ANY WAY TO THE UNAUTHORIZED USE,
MODIFICATION, OR APPLICATION OF THIS DOCUMENT OR THE
PROPRIETARY INFORMATION, METHODS, AND SPECIFICATIONS SET FORTH
IN THIS DOCUMENT, WHETHER IN WHOLE OR IN PART.
TABLE OF CONTENTS
1.0 Executive Summary ............................................................................................................... 1
2.0 Background of the Program ................................................................................................. 4
3.0 Impact Evaluation Approaches and Results ....................................................................... 6
3.1 True Impact Test Approach and Results ............................................................................ 6
3.2 OLS Model Approach and Results .................................................................................... 8
3.3 Fixed Effects Model Approach and Results .................................................................... 10
4.0 Conclusion ............................................................................................................................ 13
5.0 Appendix ............................................................................................................................... 14
5.1 Thoughts on Further Research ......................................................................................... 14
5.2 Model Results .................................................................................................................. 15
5.3 PSE Background .............................................................................................................. 16
5.4 Authors ............................................................................................................................. 16
1.0 Executive Summary
1.0
Connexus OPower Energy Measurement-Verification Report 1 MN0480806 8/9/10
1.0 Executive Summary
In response to Minnesota’s new state-wide conservation goals, Connexus Energy (Connexus)
partnered with OPower, formerly Positive Energy, to launch a new energy efficiency program.
Power System Engineering, Inc. (PSE) was hired as an independent third-party evaluator for this
pilot program. PSE’s role was to validate the methods and estimates of energy savings attributed
to the program in the first year.
Data was provided to PSE by OPower staff; in particular, Tyler Curtis. This included monthly
billing data, program-specific characteristics, customer demographics, and climatic data. The
time span of the data stretched from January 2007 to January 2010. This time period
encompassed over two years of data before the pilot began (January 2007 through February
2009) and 11 months of data during the pilot (March 2009 through January 2010). The dataset
included over 2.5 million observations encompassing nearly 80,000 member accounts.
PSE examined the sample selection, program design, and data outputs. In evaluating the energy
savings of the program, three measurements were calculated. These included the True Impact
Test, an Ordinary Least Squares econometric model, and a Fixed Effects econometric model.
These evaluation techniques are standard methods of measuring energy efficiency impacts.
The True Impact Test is a non-parametric analysis which examines the change in the differences
of the control and treatment groups from pre-pilot to post-pilot time periods. The advantage of
this method is that it is relatively simple to understand and is a powerful tool in evaluating
program impacts. In algebraic terms, the True Impact Test can be described by the following
equation:
The other two evaluation methods use econometric analysis to estimate a specified model. Each
model calculates a parameter estimate of the impacts of the pilot program on energy usage. The
first model uses an Ordinary Least Squares (OLS) estimation procedure. The second model
1.0
Connexus OPower Energy Measurement-Verification Report 2 MN0480806 8/9/10
employs a Fixed Effect estimation procedure, which is the same approach that OPower uses in its
models.
The advantage of the OLS model is that it allows the researcher to include customer-specific
information (e.g., income, housing structure, age) into the regression. This enables estimation of
demographic impacts on electricity usage. The advantage of the Fixed Effects model is that it
automatically adjusts for all participant differences in estimating the program impacts. The
downside of the Fixed Effect model is that it cannot include time invariant customer information.
For example, many demographic variables such as sex or race do not vary over time and cannot
be included in the Fixed Effects Model.
The three estimates are presented in the table below. There is stability in results across the three
estimation procedures. The True Impact Test estimates a daily kilowatt hour (kWh) reduction of
0.621 for each participating member of the pilot program during the post-pilot time period.
Similarly, reductions of 0.622 and 0.637 are seen for the OLS and Fixed Effects models,
respectively. Reduction percentages range from 2.05 to 2.10 percent.
Given the stability of the results derived from separate estimation procedures, the large sample
size, and the randomness of sample selection, PSE concludes there are tangible energy savings
resulting from this program. Results are robust with a high degree of confidence attached to
them.
Percentage Reduction
Daily Annual
True Impact Test 0.621 227 2.05%
OLS Model 0.622 227 2.05%
Fixed Effects Model 0.637 232 2.10%
Average: 0.626 229 2.07%
Annual KWH Savings per customer
Estimated Per Customer Savings of OPower Program
1.0
Connexus OPower Energy Measurement-Verification Report 3 MN0480806 8/9/10
OPower’s fixed effects model estimated a program impact of 2.27%. This is slightly higher than
PSE’s Fixed-Effect model which estimated a reduction of 2.10%. The small disparity in results is
most likely due to the OPower process of continually updating their results as new data becomes
available and using a rolling program start date commencing after the second Home Energy
Report mailing was received by a particular household.
Further research is required to better quantify the long-run impacts of this program. Research
items, such as estimating the impacts of multiple years of treatment and the possible legacy
impacts of treatment after the program is discontinued, would have important ramifications on
cost benefit tests and on optimal program design and cycling. Research on customer
demographics and their impact on program energy savings would also add value. Additionally,
interaction effects with other energy efficiency programs would be of interest. Along these same
lines, determining the ways the pilot participants lowered energy usage would be important
information (e.g., conservation efforts, appliance replacement).
2.0 Background of the Program
2.0
Connexus OPower Energy Measurement-Verification Report 4 MN0480806 8/9/10
2.0 Background of the Program
In response to Minnesota’s new state-wide conservation goals, Connexus Energy (Connexus)
partnered with OPower to launch a new energy efficiency program. Power System Engineering,
Inc. (PSE) was hired as an independent third-party evaluator for this pilot program. PSE’s role
was to validate the methods and estimates of energy savings attributed to this program.
The program provides residential participants with a Home Energy Report designed to motivate
and educate recipients to improve the energy efficiency of their homes. The program is designed
as a large scale behavioral experiment. Control and treatment groups were randomly selected
out of a total of 80,000 households. Each group consisted of approximately 40,000 households.
Households had to meet the following criteria to be included in one of the groups:
1. Exactly one active electric account per household.
2. Account history dating to January 2007.
3. Valid meter read cycle.
4. Not on a medical rate plan.
5. At least 50 “neighbors.”
The Home Energy Report relies on providing residences with a comparison of energy usage to
their “neighbors,” thus the definition of what constitutes a neighbor is important. OPower
defines the neighbor benchmark by the following characteristics.
Size of home: Neighbors are selected based on similarity in home size. Homes
greater or less than 25 percent of the participant’s home are excluded from the
comparison.
Distance: A radius of 0.5 miles of the targeted household is searched in order to find
100 similar households. At least 50 households must be available within this radius
for it to be included in the pilot.
Heating Fuel: Households are compared with other homes that have the same
heating fuel.
3.0 Impact Evaluation Approaches and
Results
3.0
Connexus OPower Energy Measurement-Verification Report 6 MN0480806 8/9/10
3.0 Impact Evaluation Approaches and Results
Section 3.0 provides a more detailed look at the three approaches used to quantify electrical
savings resulting from the pilot program. Research results and findings are provided for each
approach. These three approaches do not depend on each other. Thus, they are independent
estimations with only the data as a common element.
The three approaches all reveal similar findings. The pilot program did induce consumers to
lower energy use, provoking annual savings of about 229 kWh per participant. This amounts to
around 9,160 MWh annually saved by the 40,000 participants. The True Impact Test estimated
savings of about 2.05 percent, the OLS Model estimated savings at 2.05 percent, and the Fixed
Effects Model estimated savings of 2.10 percent. Results for the OLS and Fixed Effects Models
were statistically significant at a 99 percent confidence threshold.1
3.1 True Impact Test Approach and Results
The True Impact Test is the most straightforward method of evaluating energy efficiency
programs. The True Impact Test is a non-parametric analysis which examines the change in the
differences of the control and treatment groups from pre-pilot to post-pilot time periods. The
advantage of this method is that it is relatively simple to understand and examine the results and
is a powerful tool in evaluating program impacts. In algebraic terms, the True Impact Test can
be described by the following equation:
1 The True Impact Test is a non-parametric approach to estimating savings and thus is not able
to provide a confidence level based on statistical tests.
Percentage Reduction
Daily Annual
True Impact Test 0.621 227 2.05%
OLS Model 0.622 227 2.05%
Fixed Effects Model 0.637 232 2.10%
Average: 0.626 229 2.07%
Annual KWH Savings per customer
Estimated Per Customer Savings of OPower Program
3.0
Connexus OPower Energy Measurement-Verification Report 7 MN0480806 8/9/10
To calculate the True Impact Test, PSE took the difference in the average daily use of the
treatment group during the pilot ( ) to the average daily use of the control group during the
pilot ( ) and then subtracted this difference from the difference in the average daily use of
the treatment group before the pilot started ( ) to the average daily use of the control group
before the pilot began ( ). This provides an estimate of the change in the post-pilot treatment
energy use that can be attributed to the pilot itself. Given the large sample size and design of the
program, this estimate is convincing. The graph below further illustrates the finding. The
difference between the treatment and control group significantly widens after the start of the
energy efficiency program.
In an attempt to separate the seasonal differences in the impacts, the seasons were divided into
three categories.
1. Winter (December, January, February, March, April).
2. Summer (July, August, September).
3. Shoulder (May, June, October, November).
30.288
28.622
30.222
27.935
26.500
27.000
27.500
28.000
28.500
29.000
29.500
30.000
30.500
Daily KWH Use per Customer of Test Groups
Control Group Treatment Group
Treatment PeriodPre-Treatment Period
3.0
Connexus OPower Energy Measurement-Verification Report 8 MN0480806 8/9/10
The table below shows the average daily use per group on an annual, winter, summer, and
shoulder basis. Treatment energy savings in the summer are estimated at 0.647 kWh per day.
This is a reduction of 1.82 percent over the control group’s usage. The winter months show an
impact of 0.677 average kWh reductions per day. The shoulder months have an estimated
reduction in average daily use of 0.517 kWh.
3.2 OLS Model Approach and Results
To further substantiate the True Impact Test results, an Ordinary Least Squares regression was
estimated. This regression incorporated pre-pilot data (January 2007 to February 2009) and the
post-pilot data March 2009 to January 2010) and estimated variables to determine their impact
on average daily electricity use.
Six explanatory variables were included in the OLS Model.
1. Intercept Term (a): This variable measures the expected electricity usage with zero
values of the other variables considered. This would be the expected usage if there was
no pilot, zero heating degree days, and zero cooling degree days.
2. Test Indicator Binary Variable (T): This variable equals “1” if the customer was
selected to be in the treatment group and “0” if the customer was placed in the control
Annual Winter Summer Shoulder
Pre-Treatment Period (Jan. 2007-Feb. 2009)
Control Group 30.288 30.779 35.639 25.473
Treatment Group 30.222 30.760 35.497 25.392
Difference (T-C) -0.066 -0.019 -0.142 -0.081
Treatment Period (March 2009 - January 2010)
Control Group 28.622 29.557 31.184 25.697
Treatment Group 27.935 28.861 30.395 25.100
Difference (T-C) -0.687 -0.696 -0.789 -0.598
Treatment KWH Impact: -0.621 -0.677 -0.647 -0.517
Percent Reduction: -2.05% -2.20% -1.82% -2.03%
True Impact Test of OPower Program
(Daily KWH Use per Customer)
3.0
Connexus OPower Energy Measurement-Verification Report 9 MN0480806 8/9/10
group. These values are constant across the pre-pilot and post-pilot time frames. This
variable allows the researcher to estimate the inherent differences between the two
groups.
3. Post-Pilot Binary Variable (P): This variable equals “1” if the time period is after the
start of the pilot program (March 2009). It equals “0” if the time period is before the
start of the pilot program (prior to March 2009). This variable allows an estimation of
the impact on average daily use of the different time periods.
4. Test Indicator multiplied by the Post-Pilot (P*T): This variable equals “1” if the
observation is in the treatment group and the month is after the start of the pilot. It
equals “0” if the observation is either in the control group or is in the treatment group
but before the pilot start date. This variable estimates the impact the pilot program
has on average daily use. PSE’s estimate of program savings is taken from the
estimated coefficient on this term.
5. Heating Degree Days (HDD): This variable measures the heating degree days for the
billing month. The parameter estimates the impact that heating degree days have on
average daily usage.
6. Cooling Degree Days (CDD): This variable measures the cooling degree days for the
billing month. The parameter estimates the impact that cooling degree days have on
average daily usage.
The estimated equation takes the following functional form:
Econometric analysis is used to estimate the values of a, b, c, d, e, f in order to minimize the
squared sum of the error term ( ). The parameter estimates (b,c,d,e,f) are interpreted as the
marginal energy use of the variable they are multiplied by. For example, parameter “f” is a
measure of how the average daily use will increase when cooling degree days increase by one.
In the context of this report, the most interesting parameter is “d.” This measures the marginal
impact on average daily use of being in the treatment group during the pilot period. The
parameter estimate for “d” is -0.622, meaning that average daily use is estimated to be reduced
by over half a kWh if the participant is currently being sent Home Energy Reports.
3.0
Connexus OPower Energy Measurement-Verification Report 10 MN0480806 8/9/10
Estimated Impacts
OLS Model
Daily kWh -0.622
Percent Reduction -2.05%
While the econometric method is more complicated and less easy to comprehend relative to the
True Impact Test, it does have the advantage of being able to test how much confidence policy
makers can have on the estimated impact. One way to do this is to calculate the t-statistic on the
parameter estimate “d.” PSE estimates a t-statistic of 12.40. For context, a t-statistic of 1.645
indicates a 90 percent level of confidence and a t-statistic of 2.326 indicates a 99 percent degree
of confidence. A t-statistic of 12.40 is, therefore, highly significant and offers a great deal of
confidence in the result.
3.3 Fixed Effects Model Approach and Results
To further substantiate the results of the pilot, PSE estimated a Fixed Effects Model. A Fixed
Effects Model is an econometric method used to capture all household-specific effects on energy
consumption. It is quite unlikely that household characteristics would significantly influence the
estimated impact of the program, given the large sample size and random selection of the
treatment and control groups. However, a Fixed Effect Model can alleviate any potential
concerns.
The logic behind the approach is to estimate separate intercepts for each customer. These
intercepts incorporate the household-specific characteristics that influence energy consumption.
It is implausible to actually estimate 80,000 separate intercepts, as is the case for this dataset.
However, econometricians have discovered a computational trick which leads them to the same
estimation results. By subtracting all included variables by the average of the variable for each
individual household, the same parameter estimates are attained.
The Fixed Effects Model has three main disadvantages.
1. Degrees of freedom loss. By implicitly including 80,000 intercept terms, the
estimation loses 79,999 degrees of freedom. The loss of degrees of freedom makes the
3.0
Connexus OPower Energy Measurement-Verification Report 11 MN0480806 8/9/10
estimation less precise. However, given that the dataset is so large and contains over
2.5 million observations, this loss is negligible.
2. Variables that do not vary with time. The transformation involved in the Fixed
Effects Model wipes out the possibility of including variables that do not vary with
time. This limits the researcher’s ability to identify variables which influence program
impacts to only those variables where data is available and varies over time for each
household. For example, an income variable that does not vary could not be included
in the Fixed Effects Model.
3. Ease at which the results can be interpreted by a non-econometrician. For this
reason, PSE included the True Impact Test and the OLS Model in this report. All three
tests show similar results.
PSE included all of the same variables in the Fixed Effects Model that were included in the OLS
model. Given the nature of the Fixed Effects Model, however, the intercept term and the
treatment indicator term do not vary over time by household and were thus unable to be
estimated. The equation used in the estimation is as follows:
The parameters have the same interpretations as they did in the OLS Model. Of interest to this
report is the parameter estimate of “d.” Again, this is the estimated impact on average daily use
for those households in the treatment group currently receiving Home Energy Reports.
The parameter estimate for “d” is equal to -0.637, revealing that those households who are being
treated are, on average, reducing average daily usage by 0.637 kWh. As was the case for the
OLS Model, the parameter estimate was highly significant with a t-statistic of 23.46.
Estimated Impacts
OLS Model
Daily kWh -0.637
Percent Reduction -2.10%
4.0 Conclusion
4.0
Connexus OPower Energy Measurement-Verification Report 13 MN0480806 8/9/10
4.0 Conclusion
Given the stability of the results derived from three independent estimation procedures, the large
sample size, and the randomness of sample selection, PSE concludes there are tangible energy
savings resulting from this program. Results are robust with a high degree of confidence
attached to them.
The estimated savings range from 0.621 to 0.637 regarding kWh average daily use. In
percentage terms, this is a range of 2.05 to 2.10 percent reductions in electricity use for those
receiving the Home Energy Reports relative to those that are not. PSE also concludes there are
higher savings resulting from receiving monthly reports relative to quarterly reports.
Using a rolling start date OPower claims energy savings of 2.27 percent over the life of the pilot
program. PSE used a start date to March 2009 for the fixed effect model and it produced a
savings of 2.10 percent. PSE can verify a savings between 2.05 and 2.10 percent for the first
year.
5.0 Appendix
5.0
Connexus OPower Energy Measurement-Verification Report 14 MN0480806 8/9/10
5.0 Appendix
The Appendix begins with a list of further research options that PSE believes would be
worthwhile investments in uncovering the fullest potential of this program. The second part
presents the models for the OLS Model and Fixed Effects Model discussed in this report.
5.1 Thoughts on Further Research
Listed below are research items which could significantly enhance the understanding and optimal
structuring of the program.
1. Diminishing returns and legacy impacts: Determining the incremental gains for each
additional year of treatment would be of great interest. At what point do energy
savings level off? Do savings actually decrease as customers become desensitized to
the reports over time? Another important question is: Do savings continue after a
household is discontinued from the program and how long do these legacy savings last?
Would cycling households on and off the program be a cost effective strategy?
2. Interactions with energy efficiency programs: How does this program interact with
other energy efficiency programs? Does it enhance or detract from them? Separating
out the interaction impacts would very useful in determining accurate estimates of
energy efficiency program savings.
3. Behavioral changes behind energy savings: An issue of this program is what is
causing the decline in energy use for residences involved in the program. Is it a
conservation effect, whereby households are more alert to turn off lights and adjust
thermostats? Or are they more inclined to purchase energy efficient appliances and
light bulbs?
4. Demographic relationships with energy savings: What types of households are more
or less inclined to change energy use behavior? Is program targeting a valuable option?
5. Differences in energy savings due to neighbor comparisons “Boomerang Effect”:
How does the comparison between the households use and the neighborhood energy
use impact energy savings? Do those households who are classified as energy efficient
relative to their neighbors actually increase energy use? What comparison percentages
drive the largest energy reductions?
5.0
Connexus OPower Energy Measurement-Verification Report 15 MN0480806 8/9/10
5.2 Model Results
OLS Model
Dependent Variable: kwhdur
Independent Estimated Standard t-
Variable Coefficient Error Statistic
constant 20.92733 3.44746e-002 6.07036e+002
testind -6.55090e-002 2.79682e-002 -2.34227
post 0.40916 3.58424e-002 11.41544
posttst -0.62150 5.01023e-002 -12.40453
hdddur 0.23123 8.39772e-004 2.75350e+002
cdddur 2.60318 7.17852e-003 3.62635e+002
Number of Observations 2585514
R-squared 5.11016e-002
Corrected R-squared 5.10997e-002
Sum of Squared Residuals 8.99848e+008
Standard Error of the Regression 18.65570
Durbin-Watson Statistic 1.76353
Mean of Dependent Variable 29.64004
Fixed Effects Model
Dependent Variable: kwhdur3
Independent Estimated Standard t-
Variable Coefficient Error Statistic
post3 -2.73721e-002 1.94173e-002 -1.40968
posttst3 -0.63681 2.71456e-002 -23.45898
hdddur3 0.22990 4.52831e-004 5.07703e+002
cdddur3 2.62359 3.87046e-003 6.77849e+002
Number of Observations 2585514
R-squared 0.16062
Corrected R-squared 0.16062
Sum of Squared Residuals 2.61489e+008
Standard Error of the Regression 10.05665
Durbin-Watson Statistic 1.85488
Mean of Dependent Variable -2.69004e-009
5.0
Connexus OPower Energy Measurement-Verification Report 16 MN0480806 8/9/10
5.3 PSE Background
Founded in 1974, PSE is a full-service consulting firm serving the utility industry with offices in
Wisconsin, Indiana, Minnesota, Ohio, and South Dakota. PSE has expertise in the areas of
demand response, energy efficiency, revenue decoupling, merger valuations, load forecasting,
cost and reliability performance benchmarking, T&D system planning and design, resource
planning, communication technologies, smart grid investments, rate design, alternative
regulation, and cost of service studies.
5.4 Authors
Chris Ivanov, Economist
Mr. Ivanov began his career at Wisconsin Public Power Inc. and is now a utility consultant.
While at WPPI, he prepared, evaluated, and managed electric load forecasts, weather
normalization, and small area forecasts for its 49 members. As a consultant, Mr. Ivanov
prepares, evaluates, and manages electric load forecasts, surveys, and economic analyses for a
wide range of clients including distribution and G&T cooperatives. His current focus is on
assisting utilities with DSM studies and load forecasts. He has a Masters in Applied Economics
from Marquette University and is currently finishing his MBA.
Steve Fenrick, Economist
Mr. Fenrick has nearly a decade of consulting experience in the utility industry. His work has
focused on cost and reliability performance benchmarking, incentive regulation, demand-side
management program designs and evaluation, revenue decoupling, and load forecasting. Mr.
Fenrick has worked with cooperatives, investor-owned utilities, regulatory commissions, and
international utilities. He earned a Bachelor of Science in Economics from the University of
Wisconsin-Madison and is finishing a Masters degree in Agricultural and Applied Economics
from the same university.