1

Understanding the Elements of Success: Findings from the National Energy Efficiency Programmatic Best

Practices Study

CALMAC MeetingDecember 15, 2004

Kenneth James, PG&E, Study ManagerMike Rufo, Quantum Consulting, Prime Contractor

2

Project Advisory CommitteeKenneth James – PG&E

Pierre Landry – SCE

Rob Rubin – SDG&E

Jay Luboff – CPUC

Eli Kollman – CPUC

Sylvia Bender - CEC

Project TeamMike Rufo – Quantum Marissa Meyers - Quantum

Phil Willems – Quantum

Jane Peters – Research Into Action

Bruce Mast – Frontier Associates

Megdal & Associated

Shel Feldman Mgt Cons.

3

Presentation Overview

• Project Background

• Example Best Practices Study Findings

• Best Practices Website and Products

• Next Steps

4

“Benchmarking is the process of identifying, sharing, and using best practices to improve business processes.” Source: American Productivity and Quality Center

"Benchmarking is simply about making comparisons with other organizations and then learning the lessons that those comparisons reveal". Source: The European Benchmarking Code of Conduct

Benchmarking almost always occurs as a collaborative process in which participants share information. Typically the shared information is about business processes with the intention of identifying excellence and developing an understanding of how it is achieved.

Benchmarking addresses the question, “How one can improve by learning from others?” Benchmarking is inexorably tied to concept of best practices, also called business excellence or exemplary practices.

Benchmarking

5

The term “Best Practice” refers to the business practice that, when compared to other business practices that are used to address a similar business process, produces superior results.

Best practices are documented strategies and tactics employed by successful organizations and programs. Rarely is an organization or program "best-in-class" in every area. Our focus is not on identifying best programs or best organizations but, rather, best practices that exist within and across programs.

Best Practices are identified from in-depth interviews with program managers, thorough review of program documents, analysis of secondary sources, and comparison of program features and outcomes.

The focus of this Study is on best practices that can be generalized and have a high likelihood of transferability to other programs within or across program categories.

Best Practices

6

California energy crisis

CA knowledge management database -

program guidance

Adapting Best Practices for Performance

Improvement

Nationally sustained database - driver of high

program practices

Best PracticesDatabase& Website

New entrants into energy efficiency

Energy Action Plan resource “loading

order”

Background

7

A Few Key Questions Addressed by this Best Practices Project:

• What EE program design, implementation and management practices are entities currently using?

• How effective are they? • Where is there room for performance improvement? • How will this knowledge assist in meeting the challenges

and opportunities in CA new EE environment?

8

Program Design

Theory, Linkages & Partnerships

Structure, Policies & Procedures

Program Management

Project Management

Reporting & Tracking

Quality Control & Verification

Program Evaluation

Evaluation & Adaptability

Program Implementation

Outreach, Marketing & Advertising

Participation Process

Installation & Delivery

Program Design

Theory, Linkages & Partnerships

Structure, Policies & Procedures

Program Management

Project Management

Reporting & Tracking

Quality Control & Verification

Program Evaluation

Evaluation & Adaptability

Program Implementation

Outreach, Marketing & Advertising

Participation Process

Installation & Delivery

Program Components

9

Program Areas

On Beta Website:• Residential

– Lighting– HVAC– Single-Family Comp– Multi-Family Comp– Audits– New Construction

• Nonresidential– Lighting/Turnkey– Large Comprehensive

In Review:• Nonresidential

– HVAC– Trade Ally Training– New Construction

• Other– Mass Market Advertising

Still in Preparation– Res Appliances

Program Area Reports (13)

10

Study Products: Program Area White Paper Reports

• Summary of Findings• Overview of Programs (5-10 each, 80+ total)• Policy/Historic Context & Issues• Feature Benchmarking and Best Practices• Comparison of Outcomes

Report Content

11

Example of Range of Programs Covered - Large Nonresidential Comprehensive Incentives

Program Area

• CA’s Nonresidential Standard Performance Contract • NYSERDA’s Energy $martTM C/I Performance • United Illuminating’s Energy Opportunities • BC Hydro’s Power Smart • Xcel Energy’s Custom Efficiency (Colorado)• Northeast Utilities’ Custom Efficiency • Massachusetts Electric’s Energy Initiative• Alliant Energy’s Energy Shared Savings• Efficiency Vermont’s Business Energy Services • SMUD’s Commercial & Industrial Custom Retrofit

12

Study Products: Program Area White Paper Reports

• Program Synopsis• Program Focus• Program Context• Program Components• Key Sources• Contact Information

Program Profiles

13

Generally Cross-Cutting• Articulate plan/theory that states expectations, timing, approach• Link strategic approach to policy objectives and constraints• Build feedback loops into program design and logic• Do not over-promise results• Understand local market conditions• Conduct sufficient market research• Maintain program design flexibility to respond to changes in market & other factors• Put process plan (including program management) in writing• Tailor program to the unique needs of market sector targeted • Define & locate hard-to-reach customers & target programs accordingly, as

appropriate

Sample Program Design BPs

14

Generally Cross-Cutting• Clearly define program management responsibilities to avoid

confusion as to roles and responsibilities • Develop and maintain clear lines of communication• Use well-qualified engineering staff (for technical programs)• Motivate field staff and service providers• Maintain consistency in personnel over time• Delegate responsibility based on risk versus reward• Make sure at least some of the institutional memory resides in-

house, not with subcontractors• Reward high performing staff and link performance evaluations to

tangible measures which are known in advance and developed together jointly by the manager and employee

Sample Program Management BPs

15

Generally Both Program-Specific and Cross-Cutting• Define & identify key information early in program process• Clearly articulate data requirements for measuring program success• Balance level of tracking planned against resource availability• Design system to support requirements of evaluators• Use Internet to facilitate data entry & reporting; build in real time

data validation systems that perform routine data quality functions• Automate, as much as is practical, routine functions• Develop electronic application processes • Develop accurate algorithms & assumptions for savings estimates• Conduct regular checks of reports to assess program performance• Document tracking system & provide manuals for all users

Sample Reporting & Tracking BPs

16

Mostly Program-Specific• Develop inspection & verification procedures during program design• Base quality control on program’s relationship with vendors, #

involved, types of measures, volume, variability of project size• Use measure product specification in requirements & guidelines• Require pre-inspections for large or uncertain impact projects• Require builder/representative to be on site during inspection• Verify accuracy of rebates, coupons, invoices to ensure the

reporting system is recording actual product installations • Assure quality of product through independent testing procedures • Build in statistical features to the sampling protocol • Treat inspection visits as partnership-building & learning events • Ensure inspectors have plenty of hands-on-construction practice• Assess customer satisfaction with the product through evaluation

Sample QC & Verification BPs

17

Mostly Program Specific• Keep participation simple, Offer a single point of contact for customers • Develop participation strategies that are multi-pronged & inclusive• Review & understand product availability before establishing eligibility• Make program participation part of an existing, routine transaction• Use Internet/electronic means to facilitate participation• Avoid over committing to projects before design parameters are known • Set incentive levels to maximize net not gross program impacts• Use incremental costs to benchmark and limit payments• Tie rebates for popular measures to those less likely to be considered• Limit or exclude incentive payments to known free riders• Tie incentives to building performance• Offer low interest loans or financing as leverage on incentives• Use disincentives for savings inflation for performance-based options

Program Implementation

18

Mostly Program Specific• Use Energy Star logo to instill consumer confidence/utility credibility

– Leverage with cities/community-based organizations & other programs

• Include adequate retail outreach & support to ensure product is stocked & advertised & that point-of-purchase materials are accurate & clear

• Develop & disseminate case studies to showcase program projects• Use target marketing strategies to ensure that hard-to-reach populations are

informed about available programs and options• Use face-to-face marketing, where possible, especially for small biz• Give builders the opportunity to participate in development of message• Market to multiple departments with volume building organizations• Provide trade allies with training & resources to enhance marketing• Sell the customer benefits first, then energy efficiency • Keep benefits quantifiable in economic terms, Promote life-cycle cost• Take advantage of external factors (i.e., heat waves, etc.)

Program Marketing

19

Generally Both Program-Specific and Cross-Cutting• Engage the implementation team in the evaluation process• Create a culture whereby evaluation findings are valued and integrated

into program management • Present actionable findings in real time and at the end of study • Stagger the timing of process and ex post impact tasks so that process

results are communicated on a relatively real-time basis• Conduct impact evaluations routinely, but not necessarily annually• Include periodic estimation or free-ridership and spillover• Use regular process evaluation to provide timely and fresh results• Periodically review & update market information on construction

practices, EE market share, measure adoption, trends• Perform detailed market assessments particularly for MT programs• Support program review & assessment at the most comprehensive

level possible

Program Evaluation

20

The Best Practices

Benchmarking

Website

21

Top-line BP List

BP Rationales, Whitepapers, &

Gap Analysis

Comprehensive Program Profiles,

All Performance Benchmarks

Complete Documentation of Methods, Data Collection

Processes, Results

Communication of Results Model

22

Home Page – Best Practice Website

23

Find A Study

24

Links to Program Areas

25

Search Results

26

Website Products

– Program Area Chapter Reports

• Description of Report

• Full Chapter Report (PDF)

• Executive Summary (PDF)

• Best Practices Table (PDF)

– Program Profiles

• Description of Program Profile

• Program Profile Report (PDF)

27

Next Steps• Need feedback from Beta Review Group and

Project Advisory Committee (PAC)• Post review consideration, revisions made on

January • Load final chapters onto website in January to

be used for 2005-’08 Portfolio planning• Further analysis of outcomes ($/kWh saved)