1

TOPICS

2013 Custom Impact Overview

2013 Custom Impact Elements

Evaluation Results

• Gross Impact Findings

• Net Impact Findings

• Project Practices Assessment (PPA) Findings

Selected IALC 2013 Findings and Recommendations

Q&A/Discussion

Emphasis on new sampling and evaluation approaches, tailored scope allowing for measure- and program-level results, and new recommendations

2013-14 INDUSTRIAL, AGRICULTURAL AND LARGE COMMERCIAL (IALC) ROADMAP

DRAFT REPORT WEBINARPY2013 CUSTOM IMPACT EVALUATION

April 20, 2015

IALC CUSTOM IMPACT EVALUATION AGENDA

» IALC Evaluation Overview and Objectives» Data Collection and Approach» Evaluation Sample by IOU» Aggregation of Sample Results» Gross Impact Results

• Discrepancy Factor Impacts• Discrepancy Factor Assessment

» Net Impact Results» Project Practices Assessment (PPA) Results» IALC 2013 Recommendations

2

3

2013 IALC CUSTOM IMPACT OVERVIEW » Evaluation of Custom (non-deemed measures) across multiple

programs and the four major California IOUs • Core Calculated Programs• Third Party Programs• Local Government Partnerships and Universities / Colleges • And More!! • Over 100 programs within the IALC 2013 scope

» Why is Custom Impact Evaluation Important?• Custom measures account for 19% of electric [499 GWh] and

56% of statewide natural gas savings claims [36 Million therms]• History of low realization rates for savings claims

- How do we address the disconnect between ex-ante and ex-post project treatment?

IALC EVALUATION OVERVIEW

» IALC Roadmap is composed of nonresidential, non-deemed claims:• Non-deemed lighting measures moved to Commercial RM

» PY2013 IALC savings claims:

4

CUSTOM EVALUATION OBJECTIVES

» Custom Priorities and Researchable Issues• Estimate achieved gross impacts identify factors affecting

gross realization rates (GRRs) identify opportunities for improvement

• Estimate free ridership identify factors affecting free ridership identify opportunities for improvement

• Evaluate PA conformance with CPUC policies, decisions, and best practices

• Collect data and information to assist with other research or study areas

5

DATA COLLECTION AND APPROACH

» The gross impact evaluation developed gross realization rates (GRRs) for statistically representative samples of custom projects• The gross impact evaluation utilized project-specific

measurement and verification (M&V) to estimate GRRs• Greater M&V rigor was used for strata 1 and 2 projects

- Analysis involved on-site metering and extensive engineering analysis to calculate the ex-post impact of sampled strata 1 and 2 projects

- Smaller projects received less rigor• 190 PY2013 M&V points were targeted and 189 were achieved

6

DATA COLLECTION AND APPROACH

7

DATA COLLECTION AND APPROACH

» The NTG evaluation used an interview-based approach for a representative sample of selected projects, yielding NTG ratios

» Similar to the gross impact approach, a mix of rigorous professional interviews and basic rigor CATI surveys were conducted

» 153 PY2013 points were targeted and 146 were achieved» Gross and net impact sampling by PA

• Electric and gas savings were collapsed• Sampling and analysis used source energy equivalents*

Number of Evaluation Completes (n)

IOU Gross Impact (M&V)

NTG Surveys

PG&E 55 51

SCE 53 46

SDG&E 43 28

SCG 38 21

All IOUs 189 146

* Conversion rates obtained from “2001 Energy Efficiency Standards for Residential and Non-residential Buildings, California Energy Commission,” June 2001: 1 kWh = 10,239 Btu source energy; 1 Therm = 100,000 Btu source energy. 1 MMBtu =1,000,000 Btu

8

AGGREGATION OF SAMPLE RESULTS» Sample-based gross and net results were aggregated by IOU to

represent the population of custom claims• Five strata were defined for each IOU, by MMBtu project size• Sample-weighted GRR and NTGR means were derived for each

IOU-stratum

For example: • Stratum-level means for each IOU were then aggregated to IOU

level using population-level strata weights.

Strata N MMBtu1 7 879,8502 18 952,4613 47 906,0274 126 928,7445 941 917,149

PG&E Illustration of Population-Level Weights:

9

GROSS IMPACT RESULTSLifecycle Gross Realization Rates – All IOUs

10

GROSS IMPACT RESULTSMean Lifecycle Gross Realization Rates by IOU and Energy Metric (MMBtu and kW)

Energy Metric Sample Size (n)

LC Gross Realization

Rate

Population (N)

Error Ratio**

90% Confidence Interval

FY Gross Realization

Rate

PG&E

MMBtu* 55 0.63 1,126 0.47 0.57 to 0.70 0.74

kW 37 0.44 854 1.41 0.28 to 0.61 0.53

SCE

MMBtu* 53 0.44 932 1.05 0.34 to 0.54 0.54

kW 51 0.51 836 0.79 0.42 to 0.60 0.64

SDGE

MMBtu* 43 0.51 264 0.86 0.41 to 0.61 0.75

kW 28 0.79 143 0.89 0.59 to 0.98 1.03

SCG

MMBtu* 38 0.60 159 0.88 0.48 to 0.73 0.69

* The primary sample was designed and selected at this level. The kW sample sizes are lower due to the fact that kW impacts were not claimed by IOUs in every case.

** A measure of the statistical variation in the gross realization rates.

11

GROSS IMPACT RESULTSMean Lifecycle Gross Realization Rates by IOU and Energy Metric (MMBtu and kW)

» Mean lifecycle RR by PA and energy metric all less than 0.70 (except GRR for SDG&E kW = 0.79)

» All weighted lifecycle RRs lower than first year RRs» Primarily due to adjustments in EUL (overstated for 71 of 189

sampled projects)» All PAs had projects with negative or zero GRRs (30 of 189)

12

GROSS IMPACT RESULTSDiscrepancy Factors – Downward and Upward Adjustments

13

All Downward Effect Upward Effect No Adjustment0

0.5

1

1.5

2

2.5

3

3.5

4

Ex-Ante Gross Impact Claim Ex-Post Gross Impact Result

Impa

ct E

stim

ate

(Mill

ion

MM

Btu)

N=240

N=157

N=41 N=42

GRR=0.69

GRR=0.51

GRR=1.29GRR=1.00

GROSS IMPACT RESULTSMost Influential Discrepancy Factors

» Largest downward impacts on FY MMBtu (by degree of savings impact)» Differences in operating conditions» Calculation methods » Inappropriate baselines or baseline conditions » Ineligible measures » Others include: inoperable measure, measure count, tracking

discrepancies» For zero-savers, most frequent discrepancies are:

» Inappropriate baselines» Ineligible measures

14

GROSS IMPACT RESULTSMost Influential Discrepancy Factors

15

-450,000

-400,000

-350,000

-300,000

-250,000

-200,000

-150,000

-100,000

-50,000

00.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

Inappropriate Baseline Operating Conditions Calculation Method Ineligible Measure Other

Redu

ction

in E

x-Po

st M

MBt

u

Impa

ct E

stim

ate

(Mill

ion

MM

Btu)

Ex-Ante Gross Impact Claim Ex-Post Gross Impact Result Ex-Post MMBtu Change

N=30

N=48

N=72

N=14N=12GRR=0.21

GRR=0.71

GRR=0.50

GRR=0.06 GRR=0.19

-388,936 MMBtu -370,596 MMBtu

-223,662 MMBtu

-171,455 MMBtu

GROSS IMPACT RESULTSMost Influential Discrepancy Factors by IOU – Example

16

NET IMPACT RESULTS

Net-to-Gross Ratios by IOU

Mean Net-to-Gross Ratios

Results PGE SCE SDG&E SCG

Weighted NTGR 0.55 0.57 0.59 0.66

90 Percent Confidence Interval 0.52 to 0.59 0.52 to 0.61 0.55 to 0.64 0.59 to 0.73

Relative Precision 0.07 0.09 0.08 0.10

n NTGR Completes 51 46 28 21

N Sampling Units 1,126 932 264 159

Error ratio (ER) 0.30 0.36 0.28 0.31

17

NET IMPACT RESULTS

» Corporate standard practice and corporate environmental policies

» Factors contributing to low program influence

• Project included in capital or operating budget

• Measure installed elsewhere by same company without rebate

• Measure automates existing manual practice

• Measure associated with environmental compliance» Factors contributing to higher program influence

• Timing of decision relative to discussions with program staff• First-time installation of measure• Energy efficiency is the sole or primary motivator• Program rebate is a high percentage of project cost (> 25%)

Most Influential Factors across IOUs

18

PPA Overview

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS

» Builds upon Lower Rigor Assessment (LRA) in ‘10/12 WO33

» Systematically examines PA ex-ante savings development practices

» More targeted than LRA with goal of targeting areas for

improvement

» Provides quantitative feedback based on qualitative information

» Analysis segregated for pre-2013 and 2013+ project agreement

date

» Capture effects of 2012 EAR policy guidance

» Clear improvement between periods not observed for any PA

except SDG&E (only 9 2013+ sample points)

19

20

PPA Overview

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS

» Includes examination of:

» Project eligibility

» Baseline selection

» Project EUL, RUL, costs and incentives

» Calculation methods, inputs and assumptions

» Rating-based examination of:

» Comprehensiveness of documentation

» Quality and appropriateness of ex-ante results and conclusions

» General rating scale:

» 1 – Does not meet basic expectations

» 2 – Minimal effort to meet expectations

» 3 – Meets expectations

» 4 – Exceeds expectations

» 5 – Consistently exceeds expectations

21

Project Eligibility

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS

» No clear improvement in appropriateness of project eligibility or

quality of documentation

» Primary reasons for eligibility failure:

» CPUC decisions / guidance

» Requirement that measures exceed code / ISP baseline

» Previous EAR guidance

22

Project Types – All PAs

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS

» Generally good agreement ER and ROB most frequently overturned project types

23

Project Baselines – All PAs

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS

» Generally good agreement Existing equipment most commonly overturned project

baseline (to ISP or code)

24

Project Documentation

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS

» Overturned ER, NR, ROB project types due to lack of

documentation of:

» Age and condition of existing equipment

» EULs/RULs

» Capabilities of existing / baseline equipment to perform service

» Available efficiency levels

» Overturned existing equipment baseline often due to:

» Lack of PA review of ISP and applicable codes and standards

» Instances of regressive baseline and subsequent ineligibility due to:

» Lack of PA review of normal facility practices

25

Documentation Assessment for Selected Parameters

PROJECT PRACTICES ASSESSMENT (PPA) FINDINGS

» EUL documentation is maintained in less than 50 percent of custom application files.

Evaluation EUL 2 years shorter, on average downward effect on LC GRR

» Where directly relevant, such as for ER claims, RUL is generally not documented in

custom application files.

» Where indirectly relevant, incremental cost is only documented in custom application

files about 40 percent of the time. Incremental cost values for applicable measures

were especially deficient and were populated only for 45 of 104 measures (43

percent).

26

OVERARCHING FINDINGS AND RECOMMENDATIONS

» Lifecycle MMBtu GRRs remain low (0.44 to 0.63)

» Relative to 2010-2012 LC GRRs in 2013 were:

» PG&E – higher by 5%

» SCE – higher by 3%

» SCE – lower by 28%

» SDG&E – lower by 20%

» For all PAs, NTG ratios showed moderate improvement over 2010-

2012

» Range from 0.55 to 0.66 in 2013, approximately 20%

improvement

27

OVERARCHING FINDINGS AND RECOMMENDATIONS

» To more accurately estimate ex-ante savings, the PAs should:• Improve documentation and reporting of project EUL, including a review

of evaluation EUL conclusions/rationale, • Improve quality control of determining project operating conditions, ex-

ante baseline determinations, savings calculations, and eligibility rules • Ensure adjustments to project savings based on post-installation

inspections and M&V.

» To achieve sufficient quality control, PA project documentation needs to significantly improve. SDG&E appears to be leading the other PAs in this area, given the level of improvement noted in 2013+ applications.

» To reduce continued moderate free ridership, PAs should test changes to program features designed to increase program-induced savings.

28

KEY RECOMMENDATIONS BY TOPIC AREA

» Increase focus on:• Accuracy of operating conditions, • Use of pre- and post-installation data and information, and • Keeping project documentation and tracking claims up to date

with field information » The evaluation team recommends that the PAs ensure that savings

calculations are based on actual equipment-use schedules and reflect any changes to the post-installation operating parameters

Operating Conditions

29

KEY RECOMMENDATIONS BY TOPIC AREA

» Increase efforts to ensure conformance with CPUC baseline policies and make a greater effort to examine existing equipment RUL

» Clearly identify project events in terms of natural replacement, replace on burnout, early replacement, new construction, and add-on equipment, and set the appropriate baseline accordingly

» Disseminate information on baseline selection to ensure best practices across program staff, implementers and customers

» Ensure that baseline equipment specifications are capable of meeting post-installation operating requirements, that the baseline selected is consistent with the project type, and that regressive baseline considerations are examined

» The requirement that measures exceed the ISP / code baseline is a first order consideration for project eligibility

Baseline Conditions

30

KEY RECOMMENDATIONS BY TOPIC AREA

» Continue to review and improve impact methods and models through review of evaluation results, industry best practices, and the CPUC’s ex-ante review process

» Ex-ante savings estimates and calculation methods should be more thoroughly reviewed and approved by PA technical staff prior to finalization of incentives and savings claims

» Conduct periodic due diligence to ensure programs adhere to PA and CPUC impact estimation policies, guidelines, and best practices

» Continue to work closely and collaboratively with the CPUC’s ex-ante review process

Calculation Methods

31

KEY RECOMMENDATIONS BY TOPIC AREA

» Adopt procedures to screen for and increase efficiency levels for high likelihood free ridership projects. Advise customers to • move to a higher level of efficiency• undertake a bundled retrofit to ensure deeper savings• move to another EE project under consideration

» Carefully review the list of qualifying measures; eliminate eligibility for standard practice measures

» Make changes to the incentive design. Tier incentives by technology class. Consider adopting a payback floor.

» Under development for 2014: modified NTGR framework to address projects involving replacement of existing equipment• Early replacement/dual baseline projects• Normal replacement/ROB projects

Program Influence/NTG

32

KEY RECOMMENDATIONS BY TOPIC AREA

» Improve PA program requirements, manuals, training, and quality control procedures in order to screen out ineligible projects

» It is recommended that a statewide document, similar to the PPA form, be developed for use by all PAs for custom claims; this form has been provided to the PAs for their use

» Increase communication and coordination efforts with entities responsible for disseminating and implementing CPUC guidance

» Better ex-ante documentation is needed supporting key project parameters identified in the PPA analysis

Cross-Cutting and Other Gross Impact-Related

THANK YOU

www.itron.com

CONSULTING & ANALYSIS GROUP

1111 Broadway

Oakland, CA

GROSS IMPACT RESULTSMost Influential Discrepancy Factors by IOU - SCE

15

GROSS IMPACT RESULTSMost Influential Discrepancy Factors by IOU – SDG&E

16

GROSS IMPACT RESULTSMost Influential Discrepancy Factors by IOU – SCG

17