Greenhouse Gas Emissions Forecasting: Learning from International Best Practices

Greenhouse Gas Emissions Forecasting:

Learning fromInternational

Best Practices

Disclaimer: The views expressed in this document do not necessarily represent those of the organizationswith which individual Round Table members are associated or otherwise employed.

Printed in Canada on recycled paper

© National Round Table on the Environment and the Economy, 2008

All rights reserved. No part of this work covered by the copyright herein may be reproduced or used inany form or by any means — graphic, electronic or mechanical, including photocopying, recording,taping or information retrieval systems — without the prior written permission of the publisher.

Library and Archives Canada Cataloguing in Publication

Greenhouse gas emissions forecasting: learning from international best practices.

Text in English and French on inverted pages.Title on added t.p.: Prévisions des émissions de gaz à effet de serre : leçons des pratiques exemplairesinternationales.Available also on the Internet.

ISBN 978-0-662-05879-3

Cat. no.: En134-41/2008

1. Greenhouse gases—Canada—Forecasting. 2. Greenhouse gas mitigation—Canada. 3. Greenhousegases—Forecasting. 4. Greenhouse gases—United States—Forecasting. 5. Greenhouse gases—GreatBritain—Forecasting. I. National Round Table on the Environment and the Economy (Canada) II. Title: Prévisions des émissions de gaz à effet de serre : leçons des pratiques exemplairesinternationales.

TD885.5.G73G73 2008 363.738’746 C2008-980260-8E

This book is printed on Environmental Choice paper containing 20 percent post-consumer fiber, usingvegetable inks.

National Round Table on the Environment and the Economy344 Slater Street, Suite 200Ottawa, OntarioCanada K1R 7Y3Tel.: (613) 992-7189Fax: (613) 992-7385E-mail: [email protected]: www.nrtee-trnee.ca

GHG Emissions Forecasting – Learning from International “Best Practices” – July 2008 i

National Round Table on the Environment and the Economy

About UsThe National Round Table on the Environment and the Economy (NRTEE) is dedicated to exploring newopportunities to integrate environmental conservation and economic development, in order to sustainCanada’s prosperity and secure its future.

Drawing on the wealth of insight and experience represented by our diverse membership, our mission is togenerate and promote innovative ways to advance Canada’s environmental and economic interests incombination, rather than in isolation. In this capacity, it examines the environmental and economicimplications of priority issues and offers advice on how best to reconcile the sometimes competing interestsof economic prosperity and environmental conservation.

The NRTEE was created by the government in October 1988. Its independent role and mandate wereenshrined in the National Round Table on the Environment and the Economy Act, which was passed by theHouse of Commons in May 1993. Appointed by Governor in Council, our members are distinguishedleaders in business and labour, universities, environmental organizations, Aboriginal communities andmunicipalities.

How We WorkThe NRTEE is structured as a round table in order to facilitate the unfettered exchange of ideas. By offeringour members a safe haven for discussion, the NRTEE helps reconcile positions that have traditionally been atodds.

The NRTEE is also a coalition builder, reaching out to organizations that share our vision for sustainabledevelopment. We believe that affiliation with like-minded partners will spark creativity and generate themomentum needed for success.

And finally, the NRTEE acts as an advocate for positive change, raising awareness among Canadians andtheir governments about the challenges of sustainable development and promoting viable solutions.

We also maintain a secretariat, which commissions and analyses the research required by our members intheir work. The secretariat furnishes administrative, promotional and communications support to theNRTEE.

Our Current ProjectsThe members of the NRTEE meet four times a year to review their research and conduct their deliberations.Our current projects focus on:

• Energy Efficiency in the Commercial Building Sector

• Climate Change Adaptation Policy for Northern Infrastructure

• Carbon Emissions Pricing Policies

For more details on past and current programs, visit our website at http://www.nrtee-trnee.ca.

GHG Emissions Forecasting – Learning from International “Best Practices” – July 2008 iii

ChairRobert Page TransAlta Professor of Environmental Managementand Sustainability Institute for Sustainable Energy, Environment andEconomy University of Calgary Calgary, Alberta

Vice-ChairFrancine Dorion St-Bruno-de-Montarville, Quebec

Janet Benjamin President, Vireo Technologies Inc. andPresident, Association of Professional Engineers andGeoscientists of BCNorth Vancouver, British Columbia

Pauline Browes DirectorWaterfront Regeneration Trust Toronto, Ontario

Elizabeth Brubaker Executive Director Environment Probe Toronto, Ontario

Angus Bruneau Corporate Director St. John’s, Newfoundland and Labrador

David Chernushenko PresidentGreen & Gold Inc. Ottawa, Ontario

Anthony Dale Vice President, Policy and Public Affairs Ontario Hospital Association Toronto, Ontario

Robert A. Dubé PresidentAtout PersonnelMontreal, Quebec

Timothy R. Haig President & CEO, BIOX Corporation Oakville, Ontario

Christopher Hilkene President Clean Water Foundation Toronto, Ontario

Mark Jaccard ProfessorSchool of Resource and Environmental ManagementSimon Fraser University Vancouver, British Columbia

Don MacKinnon President Power Workers’ Union Toronto, Ontario

Ken McKinnon ChairYukon Environmental and Socio-Economic Assessment Board Whitehorse, Yukon

Richard W. ProkopankoDirector of Corporate Affairs and Sustainability Rio Tinto Alcan Inc. Vancouver, British Columbia

Wishart RobsonClimate Change Advisor Nexen Inc.Calgary, Alberta

Robert Slater Adjunct Professor Environmental Policy Carleton University Ottawa, Ontario

Robert Sopuck Vice-President of Policy (Western Canada) Delta Waterfowl Foundation Winnipeg, Manitoba

David McLaughlin NRTEE President & CEO

Members of the National Round Table on the Environment and the Economy (NRTEE)

TABLE OF CONTENTS

1. Executive Summary....................................................................................................................1

2. Introduction ..............................................................................................................................32.1 Background and Purpose ..............................................................................................32.2 Analytical Approach ......................................................................................................4

3. The Challenge of Forecasting ..................................................................................................53.1 What is Forecasting? ....................................................................................................53.2 Modelling Approaches ..................................................................................................5

4. What are the Best Practices of GHG Emissions Reductions Forecasting? ..........................64.1 Methodology ................................................................................................................64.2 Governance....................................................................................................................74.3 Canada’s Approach ........................................................................................................8

4.3.1 Forecasting Model ..............................................................................................84.3.2 Accuracy of Canada’s Forecasts ..........................................................................9

4.4 Case Studies ................................................................................................................104.4.1 Introduction ......................................................................................................104.4.2 United States ....................................................................................................104.4.3 United Kingdom ................................................................................................13

5. Discussion ..............................................................................................................................155.1 Lessons for Canada ....................................................................................................16

6. Conclusion ..............................................................................................................................19

Appendix A: List of Abbreviations and Acronyms ..................................................................21

Appendix B: Description of Additionality, Free-ridership, Rebound Effect, and Policy Interaction Effects ............................................................................22

Appendix C: Discussion of Top-down, Bottom-up, and Hybrid Approaches to Energy-Economy Modelling ............................................................................23

Appendix D: Glossary of Useful Modelling Terms ..................................................................24

Notice to Reader ..........................................................................................................................25

1. Executive SummaryThis report responds to key concerns highlighted bythe National Round Table on the Environment andthe Economy (NRTEE) in its 2007 Response to itsobligations under the Kyoto Protocol ImplementationAct (2007 KPIA Response). Chief among thoseconcerns were differing and inconsistent forecastingmethods used among various federal departments todescribe the greenhouse gas (GHG) emissionsreductions accruing from a particular policy measureor initiative, leading to issues of additionality, freeridership, rebound effect, and policy interaction effects.The NRTEE emphasized the importance oftransparency and clarity with respect to keyassumptions and methods, and the consideration ofimportant sensitivities and uncertainties whenproducing GHG emissions forecasts. It alsoemphasized the importance of consistency inapproaches across different departments, programs,and measures, and the need to integrate the findingsin a holistic framework. In light of these concerns,the NRTEE felt it could be useful for the federalgovernment if international best practices could beidentified and highlighted in the forecasting ofemissions reductions resulting from governmentpolicies, from both a methodological and agovernance perspective.

In its 2008 Climate Change Plan for the Purposesof the Kyoto Protocol Implementation Act (2008Government KPIA Plan), the government respondeddirectly to the above concerns: “the advice of the(NRTEE) is a key factor in the government’smethods for estimating reductions.”1 In its 2008KPIA Response, the NRTEE noted the significantimprovements made by the federal government inaddressing the issues mentioned above in the 2008KPIA Plan, in particular the issues of transparency,additionality, and policy interaction effects. Movingforward, the NRTEE believes there is continualbenefit in examining and highlighting how othercountries approach similar challenges to those facedby the federal government in emissions forecasting.

Recognizing that forecasting GHG emissionsreductions policies2 is difficult and inherentlyuncertain, the challenges of forecasting, includingmodelling approaches, are probed. In this report,forecasting is defined as a depiction—an energy-economy model—of how a system will evolve infuture both with and without policy intervention.The core of the report defines emissions forecastingbest practices from methodological and governanceperspectives. After a review of Canada’s approach toemissions forecasting, the report presents case studiesto illustrate how two other industrialized countries—the United States and the United Kingdom—approach forecasting.

Considering the significant role the provinces andterritories play in the 2008 KPIA Plan, an importantelement of this report is considering their role inemissions forecasting. Initial research, however,indicated that few provinces have developedcomprehensive emissions forecasts (while the reportnotes that some provinces are in the process ofdeveloping forecasts, they are not required to by lawas is the case with the federal government). This hasbeen highlighted in the report as a key area ofconcern given that substantial emissions reductionsare attributed to provincial policies and measures infederal forecasts despite the lack of detailed provincialforecasts. These issues are discussed in sections 4.3(Canada’s Approach) and 5.1 (Lessons for Canada) ofthe report.

The report concludes with a broad discussion onfindings from the case studies, along with detailedlessons for Canada, and specific conclusions onincorporating best practices in GHG emissionsforecasting. It is important to note that the report’skey findings, listed below, do not imply that Canadadoes not currently follow some of these practices—these are broad, standard best practices that takentogether, should result in improved forecastingmethods and approaches in Canada.

Key findings and recommendations from theanalysis contained in the report from a methodologicalperspective include the following:

GHG Emissions Forecasting: Learning from International Best Practices – July 2008 1

1 Environment Canada (2008), A Climate Change Plan for the Purposes of the Kyoto Protocol Implementation Act, p. 30.

2 Business-as-usual (BAU), or “without policy” forecasting, is generally straightforward. However, the production ofGHG emissions forecasts based on policies (e.g., Regulatory Framework for Large Emitters, EcoEnergy for Buildingsand Houses) is particularly challenging.

• Hybrid energy-economy models are more effectivein producing accurate GHG emissions forecasts asthey integrate the strengths of both the traditionalbottom-up and top-down approaches tomodelling emissions forecasts;

• The use of a consistent baseline from year-to-year(including baseline data), assumptions, andconditions across the board is fundamental toensure emissions forecasts can be accuratelycompared from year to year;

• The use of consistent and agreed definitions ofterms and concepts, such as for free ridership andadditionality, across government departmentsinvolved in forecasting would ensure greatertransparency of emission forecasts and facilitateassessment of the forecasts’ accuracy.

• There is need for an international perspective inthe model so that it can respond appropriately toworld events (since in most cases, Canada is aprice taker for both commodities and energy, anda primary trader of goods and energy). Canada isacting in concert with other countries on climatepolicy and its forecasting approaches need toreflect this reality.

From a governance perspective:

• Use of an independent forecasting agency ispreferable to provide more accurate andtransparent emission forecasts for consideration by

government policy makers, external analysts, andParliamentarians and to facilitate ongoing auditand evaluation.

• Multi-source emissions forecasting from a groupof individual government departments can beaccurate, but works best both when centrallycoordinated and with independent authority bythe central coordinating department or agency toquestion other departmental forecasts.

• Regular independent reviews, audits andevaluations of government forecasts andforecasting methods by a third-party agency orprocess helps ensure accuracy of forecasts and thatforecasting methodologies are up-to-date androbust.

• Forecasting must be sufficiently resourced andfinanced by governments to ensure data is up todate and most recent improvements in forecastingmethodologies are incorporated for the benefit ofpolicy makers taking decisions based on theseforecasts.

• Regular, ongoing evaluation of past forecasts foraccuracy and effectiveness is necessary to ensurecontinuous improvement of governmentforecasting methodologies and approaches.

• Ensure transparency and clarity with respect tokey assumptions and methods.

GHG Emissions Forecasting: Learning from International Best Practices – July 20082

2. IntroductionSuccessful climate policies are those that achieveforecasted greenhouse gas (GHG) emissionsreductions. But, forecasting expected GHG emissionsreductions from specific policies and measures isdifficult, as noted by the National Round Table onthe Environment and the Economy (NRTEE) in itsResponse to its Obligations under the Kyoto ProtocolImplementation Act (2007 KPIA Response).3 It hasnot only been a challenge for Canada, but for allcountries to generate accurate projections of expectedGHG emissions reductions from policies over a giventime period. However, Canada has consistentlyproduced inaccurate forecasts over the past 10 years,in each instance underestimating the growth ofdomestic GHG emissions under business-as-usual(BAU) conditions.4

The importance of accurate forecasting cannot beoverstated. It is the forecast upon which climatepolicies and programs are developed and measured.Policy makers can be driven to differing policychoices depending upon the forecasted emissionsreductions expected to be realized. Inaccurateestimates make it difficult for the federal governmentto design policies that will result in expected forecastemissions reductions. Some countries, particularlythose with similar resource-based economies toCanada’s, have had more success in forecasting GHGemissions reductions in some policy areas. While nocountry produces forecasts that are 100 per centaccurate, some countries utilize approaches thatappear to be more reflective of actual emissionsevents. Therefore, it is well worth examining andunderstanding how these countries approach theirGHG emissions projections, both frommethodological and governance perspectives, andthen assessing whether and how these might bebeneficially applied to Canada.

2.1 Background and Purpose

In its review of the government’s climate change planunder its KPIA obligations in summer 2007, theNRTEE found that differing and inconsistentforecasting methods were used among various federaldepartments to describe the emissions reductionsaccruing from a particular initiative, leading to issuesof additionality, free ridership, rebound effect, andpolicy interaction effects.5 It is important to note thatin its just-released 2008 KPIA Plan, the governmenthas taken significant steps to address these issues.6

The areas of concern identified by the NRTEE in itsevaluation of the government’s 2007 KPIA climatechange plan are primarily those of methodology andgovernance. In its 2007 KPIA Response, from whichthis report originated, the NRTEE emphasized theimportance of transparency and clarity with respectto key assumptions and methods, and theconsideration of important sensitivities anduncertainties. It also emphasized the importance ofconsistency in approaches across differentdepartments, programs, and measures, and the needto integrate the findings in a holistic framework. Inlight of these conclusions, the NRTEE felt it could beuseful for the federal government if international bestpractices could be identified and highlighted in theforecasting of emissions reductions resulting fromgovernment policies, from both a methodological anda governance perspective.

At its November 2007 plenary meeting, theRound Table accordingly approved a proposal todevelop a best practices guide or backgrounder thatwould be submitted to the government along withthe NRTEE’s next response under its KPIAobligations in 2008.

From a methodological perspective, the NRTEEidentified certain forecasting methods for estimatingGHG emissions reductions that did not result in theexpected realized emissions, as the difference between


3 For further information, see http://www.nrtee-trnee.ca/eng/publications/c288-response-2007/index-c288-response-2007-eng.htm.

4 Simpson, J., Jaccard, M. and Rivers, N. (2007) Hot Air: Meeting Canada’s Climate Change Challenge. Toronto:McLelland and Stewart, p.165.

5 Please see Appendix B for a description of these effects.

6 Environment Canada (2008), A Climate Change Plan for the Purposes of the Kyoto Protocol Implementation Act, p. 30.

stated reductions and reference case (or BAU)emissions. The sum of stated reductions for a givenyear should correspond to the expected differencebetween the reference case and forecasts of realizedemissions. As noted above, deviating from the basiccomputation of measuring reductions against thereference case can lead to issues of additionality, freeridership, rebound effect, and policy interactioneffects.

From a governance perspective, the NRTEEidentified inconsistent forecasting methods acrossdepartments for estimating GHG emissionsreductions and gaps in the on-going evaluation offorecasting approaches. The NRTEE also found thata consistent definition for “reductions” for policyimpacts was not applied in all instances in the plan—some policy impacts were stated in different terms(e.g., in terms of their cumulative impact). Using aconsistent model and definitions for forecasting theemissions reductions resulting from specific programsand initiatives is necessary to address this.

For both the methodological and governance areasof concern listed above, it is important to note that inits 2008 KPIA Plan, the government has taken stepsto address these issues. Nevertheless, the NRTEEbelieves the application of certain best practices, asoutlined in this report, will ensure more accurateforecasting moving forward and, by extension, informeffective policy choices that will achieve GHGemissions reductions.

2.2 Analytical Approach

In order to ensure optimal climate policy outcomes,accurate and realistic forecasting is vital. The NRTEEbelieves this report could be useful in assisting thegovernment to develop effective policy by providingexamples of how other governments forecast GHGemissions reductions from policies and measures.

Flowing from the key findings of the RoundTable’s 2007 KPIA Response, the approach to this

report has been to not only define and provideexamples of international best practices from amethodological perspective, but also to also highlightbest practices from a governance perspective.

Specifically, we sought answers to two keyquestions:

• Methodology: What is the most effectiveforecasting methodology that countries shoulduse?

• Governance: What are the optimal forms ofgovernance to ensure that best practices inforecasting are followed?

A case study approach was chosen as mostappropriate. To the extent possible, countries witheconomic or national situations most applicable toCanada have been examined. It is important to notethat the selection of case studies to provide examplesof international best practices in GHG emissionsforecasting is not as straightforward as it seems. Theoriginal intention was to select countries with similarchallenges to those facing Canada in forecastingGHG policies and that followed best practices inmethodology and governance as outlined in sections4.1 and 4.2. What these case studies highlight is thatsome countries utilize best practices from amethodological perspective and some from agovernance perspective, but not all utilize bestpractices in both areas.7 Another initial purpose ofthis report was to highlight best practices in sub-national jurisdictions, particularly some Canadianprovinces and U.S. states. No province, however, hasreleased detailed plans with emission policyforecasts10, raising questions about themethodological basis behind their plans.

In order to ensure rigour in our research, analysis,and findings, this paper was then peer reviewed byseveral well-respected economic experts.11 Beyondthe peer review, the final report was reviewed andconsidered by NRTEE members themselves.


7 This issue is further discussed in section 4.4.1.

8 This issue is elaborated upon in section 4.1.

9 Please see Notice to Reader for details.

3. The Challenge of Forecasting The challenges associated with estimating emissionsreductions from climate policy measures and actionsare well documented.10 On the methodological side,these include the development of baselineassumptions11 and the impact of specific policymeasures, assumptions about financial costs andconsumers’ preferences, assumptions about thedirection and rate of technological change, costingindividual actions versus integrated actions,assumptions about macroeconomic costs, the effect ofpolicy measures on cost incidence and total costs, andthe full range of costs and benefits of GHG emissionreduction policies. There is no simple method toaccount for all these actions.

Perhaps less technically complex, but just aschallenging, are governance issues related to effectiveemissions forecasting. The NRTEE’s 2007 KPIAResponse emphasized the importance of transparencyand clarity with respect to key assumptions andmethods, and the consideration of importantsensitivities and uncertainties. It also emphasized theimportance of consistency in approach across variousdepartments, programs, and measures, and the need tointegrate findings in a holistic framework. This reportwill explore how two other countries address both setsof these challenges and determine, to the extentpossible, if it is appropriate to apply their methodologyand governance approaches to the Canadian context.

3.1 What is Forecasting?

In this report, forecasting is defined as a depiction—aneconomic model—of how a system will evolve infuture both with and without policy intervention.Since the future is of course unknown and thusuncertain, we cannot say that one forecast made today

is better than another. What we can do, however, isassess alternative forecasting methods in terms of thefollowing criteria:

1. past accuracy, which may or may not bode wellfor future forecasts;

2. sound representation of current and emergingsystem dynamics,12 which should increase theprobability of a better forecast;

3. greater transparency, which increases the abilityfor outsiders to examine and critique all keyassumptions, and perhaps test alternatives; and

4. the ability to conduct and record sensitivityanalyses, which should improve the understandingof the critical forecast model assumptions andrelated key uncertainties.

3.2 Modelling Approaches Emissions forecasts are calculated using energy-economy models. These models are designed toforecast the effects of policies on energy-related GHGemissions. The structure of most energy-economymodelling approaches ranges from detailed bottom-upmodels reflecting engineering-economic details of awide menu of technologies in each sector to top-downmodels of the whole economy calibrated on historicdata from up to hundreds of sectors. Hybrid models—those that combine the strengths of the bottom-up andtop-down approaches—are considered by manymodelling authorities as optimal approaches toforecasting.13 Canada’s E3MC model14 is an exampleof a hybrid model, along with the U.S. EnergyInformation Administration’s (EIA) model (describedin section 4.4.2).15 In the later discussion of specificcountries’ best practices approaches to emissionsforecasting, we will assess what kinds of conclusionscan be drawn as to the effectiveness of forecasts fromthe various modelling approaches used.


10 Examples include Jaccard, M., J. Nyboer and B. Sadownik, The Cost of Climate Policy (2002); Bernstein, S.,“International institutions and the framing of domestic policies: The Kyoto Protocol and Canada’s response toclimate change,” Policy Sciences, 35 (2002); and Nordhaus, R. and K. Danish, Designing a mandatory greenhouse gasreduction program for the U.S. (2003).

11 At the very least, baseline assumptions should include supporting data and data consistency with other agencies.

12 In this case, system dynamics refers to how an energy-economy model captures changes in technologies, costs, andbehaviours over time.

13 Dowlatabadi, H., D.R. Boyd, J. MacDonald (2004). Model, Model on the Screen, What’s the Cost of Going Green?Washington, D.C.: Resources for the Future, p. 10.

14 E3MC stands for Energy-Economy-Environment Model for Canada. A description of it is found in section 4.3.1.

15 Please refer to Appendix A for a discussion of modelling approaches.

4. What are the Best Practices ofGHG Emissions ReductionsForecasting?In reviewing best practices in GHG emissionsforecasting, a useful starting point is the main areas ofconcern addressed by the NRTEE in its 2007 KPIAResponse:

• Clear and transparent definitions of the modellingapproaches used are absolutely essential.Particularly important are the assumptions aboutthe time frame to which the models are beingapplied. It is also important to provide sufficientdetail on the methods and data employed.

• For any specific modelling framework, it isimportant to provide prioritized informationabout the baseline forecast assumptions aboutbehaviour and the cost of technology.

• Also critical is a clear and transparent definition ofthe particular policies being studied. In general,market-based approaches are commonly modelledwithin an economic framework. In contrast, non-market based policies are more difficult to assess,with analyses often based on ad hoc assessments.It is in that context that issues of consistency andintegration of results are most critical.

Beyond these key concerns are a range of issuespertaining to methodology and governance inemissions forecasting, which we set out below.

4.1. MethodologyFrom a methodological perspective, BAU forecast hasto be explicit about key drivers and assumptions (s.3.1, criteria 3) and about key response dynamics(s.3.1, criteria 2). Even under a BAU forecast, thislatter criterion is important. For example, what willbe the full response over the next 40 years to a long-run trend to higher real oil prices? Then, the issue ofdynamics continues to be extremely important as we

try to forecast the effect of policies. At this point,sensitivity analysis is also key (s.3.1, criteria 4).

While there are many approaches to emissionsforecasting, the two approaches consistentlydetermined as best practices from a methodologicalperspective from academic, government, andinstitutional peer review are the International EnergyAgency’s (IEA) World Energy Outlook and the U.S.Energy Information Administration’s (EIA)International Energy Outlook.16

Through its annual World Energy Outlook, the IEAproduces a BAU forecast as well as a policy forecast(World Alternative Policy Scenario). It can bedetermined that the IEA utilizes “best” forecastingpractices for the following reasons: first, it provides asystematic analysis of both aggregate measures andintensities: GDP, energy supply, and energy supply perunit GDP. Contrast this with Canada’s 2006 referencecase17 where everything is in rates and average shares,which is inherently less easy to verify or adapt. TheIEA forecasts are quite flexible, and allow one to easilysee where errors have occurred (e.g., growth wasunderestimated, so emissions will be as well). InCanada’s case, there were no hard values for GDP,population, etc., only hard values for energy use andemissions. If the Canadian reference case were to haveunderlying trends on important aggregates, it might beeasier to update emissions forecasts, which would makethem more credible. Second, and related to the first,the IEA periodically has a section in its new editionsthat examines why its previous forecasts may haveerred: Was it an error in GDP or in emissions perGDP forecasts that led to an error in emissionsforecasts?18 Third, the IEA also discusses up front thekey assumptions and sources of uncertainty. Mainuncertainties include macroeconomic conditions (e.g.,slower GDP growth than assumed in both scenarioswould cause demand to grow less rapidly), uncertaineffects of resource availability and supply on energyprices, and changes in government and energyenvironmental policies.19 It also expands on one of


16 The EIA’s domestic approach to emissions forecasting is described in Section 3.2.2

17 Natural Resources Canada (2006), Canada’s Energy Outlook: The Reference Case 2006. As noted in this report andthe NRTEE’s 2008 KPIA Response, a new baseline (March 2008) has been developed that has significantlyimproved from the 2006 baseline.

18 For example, see International Energy Agency (2004), World Energy Outlook 2004, Annex B – Energy Projections:Assessment and Comparison, p. 519.

19 IEA (2006), World Energy Outlook 2006, p. 53.

the key sources of uncertainty, the adoption of policy(as the NRTEE assessed through its 2007 KPIAResponse). The IEA responds to this uncertainty byproposing an “Alternative Policy Scenario,” in whichit examines what would be likely to happen given theunderlying forecast assumptions if certain policiesbeing discussed were adopted.

The EIA provides worldwide forecasts of energydemand and supply through 2030 in its annualassessment of global energy markets that is publishedin the International Energy Outlook (IEO). Similar tothe IEA’s long-term outlooks, the IEO employsmeasures of economic growth (GDP), population,and energy intensity to derive its forecasts. All ofthese statistics are reported either in the body of thetext or in the appendices that accompany the report.The IEO includes regional projections of carbondioxide emissions by fossil fuel. The report alsoincludes an examination of the forecast relative toprior-year releases and, in the 2006 edition of thereport, a comparison of past IEO projections with theactual historical data. This assessment can also befound in the appendices of the various IEO editions,all of which are archived on the EIA website.

In addition to a reference case projection, the IEOtypically includes a number of side cases. These casesestimate the impact on energy markets of high andlow macroeconomic growth assumptions and highand low energy prices. These results help to quantifythe uncertainty in the IEO projections. While theEIA does not routinely include climate change policyside cases in its outlook, in 2006 it did include ananalysis of the impact of the Kyoto Protocol on thecountries and regions that ratified the treaty(including, of course, Canada).

As described above, the EIA’s International EnergyModule is similar to the IEA’s forecasting approach.For the purposes of this report, it is important tounderstand that the EIA and IEA forecasts are BAU,and that it is policy forecasting that is especiallychallenging. However, both the EIA and IEA buildon their BAU forecasts with complementary analysesand alternative policy scenarios that depict differentfutures (e.g., the adoption of more aggressive GHGtargets or faster deployment of clean technologies)from the reference case or BAU forecast, whichprojects what will happen if current actions and

policies remain in place. This is particularlyimportant for Canada, as discussed in section 5.1.Therefore, for the context of this report, theapproaches described above can serve as best-practicebenchmarks from a methodological perspective foremissions forecasting.

4.2 Governance

As mentioned above, there are some basic bestpractices in regard to governance in emissionsforecasting. These include clear and transparentdefinitions of the modelling approaches used, theprovision of sufficient detail on the methods and dataemployed, and clear and transparent definitions ofthe particular policies being studied. This allows for acommon understanding of the government’semissions forecasts and permits a straightforwardevaluation of emissions reductions estimates andforecasts by not only government officials, but bythird parties as well.

Beyond this, there is the issue of on-goingevaluation. The importance of independent peerreview cannot be overemphasized as a governancebest practice. This ensures that the government’sapproach to emissions forecasting, includingmodelling, is rigorous and reflects key analytical,economic, regional, and sectoral considerations.While different types of review are possible, great careshould be taken to make sure that the review is trulyindependent of the sponsoring agency and themodelling effort. Sometimes it is even useful toshowcase the results of a report of this sort in a publicmeeting, as in the case with the EIA’s annualforecasts, involving a wide range of stakeholders. Suchmeetings are able to examine the results and methods,as well as future plans to improve the analysis—all inthe context of international best practices.

Broadly speaking, as will be discussed in section 5below, countries with integrated, centralized,independent emissions forecasting agencies are morelikely to produce accurate forecasts than thosecountries with responsibilities for data gathering andmodelling diffused across various departments. Somecountries are recognizing this, as shown by the newlycreated U.K. Committee on Climate Change and theAustralian Department of Climate Change.


4.3 Canada’s Approach

The Canadian government has set a national goal ofreducing GHG emissions, relative to 2006 levels, by20 per cent by 2020, and by 60 to 70 per cent by2050. As illustrated by the government’s most recentclimate change plan Turning the Corner,20 Canadauses forecasts to determine future GHG emissionsreductions. In the plan, federal measures alone willreduce emissions in 2020 by approximately 230megatonnes below forecasted levels, with 165megatonnes of that reduction being attributable tothe federal Regulatory Framework for Industrial GHGEmissions.

Various departments collect, analyze, and modeldata (in some cases modelling emissions forecasts forspecific policies and measures) and provide thatinformation to Environment Canada, the departmentresponsible for coordinating climate policy (includingforecasting) within the federal government. Whilethis practice does not necessarily lead to inaccurateforecasts, issues arise when the central agency isneither independent nor has the authority to overridethe data and analysis it receives from otherdepartments. The NRTEE drew attention to thisissue in its 2007 KPIA Response when it emphasizedthe importance of consistency in approaches acrossdifferent departments and programs, and the need tointegrate the findings in a holistic framework. Thisissue is clearly illustrated in the U.S. case study setout in section 4.4.2 below.

4.3.1 Forecasting Model21

The emissions and economic forecasts presented inthe government’s plan were estimated throughEnvironment Canada’s economic model (the Energy-Economy-Environment Model for Canada, orE3MC). The E3MC model is a combination of theEnergy 2020 model and the Informetrica Model(TIM). According to Environment Canada, E3MCpermits “integrated energy-economy policysimulations in a manner that fully addresses the

challenges of additionality, free-riders, reboundeffects, and policy interaction effects that commonlyarise in this type of complex analysis”22—the veryeffects the NRTEE highlighted in its KPIA Response.This most recent plan was released following theNRTEE’s 2007 KPIA Response.

Energy 2020 (E2020) is a “bottom-up” technologymodel used to forecast the effect of policies onemissions.23 The model forecasts the adoption ofenergy-using and energy-producing technologiesthroughout the Canadian economy. E2020 accountsfor the technologies in use and forecasts consumers’choices of future technologies. Consumer choices arebased on both financial factors (the operating andcapital costs of technologies) as well as non-financialpreferences, as drawn from historical data. Technologyoptions (with different efficiencies or powered bydifferent fuels) are associated with different energy useand different emissions. The model forecasts futureemissions first under a reference scenario, whichmodels emissions under current conditions, andsecond under a policy scenario, which modelsemissions under the proposed policy package. Thedifference between the two trends illustrates theforecasted effects of the proposed policies.

The Informetrica Model (TIM) is amacroeconomic model used to assess the effect of theTurning the Corner policy package on the economy.Using the investments in new technology and resultingsavings forecast by E2020, TIM models the effects ofthese factors on consumption, investment, production,and trade decisions in the rest of the economy. Theseeffects are modelled by balancing inputs and outputsof commodities and capital. TIM includes bothindustry-specific and regional breakdowns.

The E3MC model is a “hybrid” model as iteffectively iterates between the E2020 and TIMmodels by re-running one model with the outputs ofthe other. This iteration continues for each year inthe simulation until model outputs no longer change(this stability suggests the models have found anequilibrium solution).


20 For details, refer to Government of Canada (2008), Turning the Corner – Detailed Emissions and Economic Modelling.

21 In two recent reports on long-term issues related to energy and climate change, the NRTEE used the Energy 2020(bottom-up) model and the CIMS (hybrid) model.

22 Government of Canada (2008), p. iii–iv.

23 Please see Appendix C for a discussion on different modelling approaches.

4.3.2 Accuracy of Canada’s Forecasts

As the NRTEE has conducted an evaluation of thegovernment’s policies and measures in reducing GHGemissions in the accompanying 2008 KPIA Response,this report will not contain an analysis of thegovernment’s approach and accuracy of its forecasts.However, as noted earlier in this report, thegovernment has taken significant steps in its 2008KPIA Plan to address the different and inconsistentforecasting methods that various federal departmentsused to describe the emissions reductions accruingfrom a particular initiative, which the NRTEEidentified as problematic in its 2007 KPIA Response.

As observed by some economic experts, Canadahas found it a challenge to produce consistent andaccurate forecasts, “…underestimating growth of

GHGs under BAU conditions.”24 These forecastshave presumably been evaluated on the assumptionthat BAU conditions are what Canada has had. It isimportant to note there has not been a recentcomprehensive, technical analysis and review of thegovernment’s forecasting methodology andgovernance, by either an independent agency orthird-party peer review.25 In regard to the E3MCmodel, while there was a review of the Energy 2020model in 2000 through the Analysis and ModellingGroup (AMG) process, the model has been updatedconsiderably since that time. There has not been,therefore, a peer review or independent analysis of theE3MC. Canada has, in the past, subjected itsenvironment-economy models to peer review. A goodexample and potential approach for an independentreview of the current E3MC is the 2001 Royal


24 Simpson, J., M. Jaccard, and N. Rivers (2007). Hot Air: Meeting Canada’s Climate Change Challenge. Toronto:McLelland and Stewart, p.165.

25 The Commissioner of the Environment and Sustainable Development has conducted an audit of specific federalclimate change measures but not a full, comprehensive review. For details, see http://www.oag-bvg.gc.ca/internet/English/aud_parl_cesd_200609_e_936.html.

Government Forecasts of Canadian GHG Emissions

550

600

650

700

750

800

850

1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010

Gre

en

ho

use

gas e

mis

sio

ns

(meg

ato

nn

es o

f C

O2 e

qu

ivale

nt)

1997 Forecast

1999 Forecast

2002 Forecast

2006 Forecast

Actual

Source: Simpson, Jaccard and Rivers (2007), p. 166.

Society of Canada Report of an Expert Panel to Reviewthe Socio-Economic Models and Related ComponentsSupporting the Development of Canada-Wide Standardsfor Particulate Matter and Ozone.26 The Panel wasformed in response to a request from a committee ofindustry stakeholders and government regulators foran objective and independent review of methods usedto estimate and compare the costs and benefits ofparticulate matter (PM) and ozone reduction.

4.4 Case Studies

4.4.1 Introduction

The selection of case studies to provide examples ofinternational best practices in GHG emissionsforecasting is not as straightforward as may seemapparent. The original intention was to selectcountries with similar challenges to those of Canadain forecasting GHG policies, and that followed bestpractices in methodology and governance as outlinedin sections 4.1 and 4.2. Countries selected as havingsimilar economic traits to those of Canada (e.g.,resource-based, export-led) were the U.S., Norway,and Australia; with jurisdictional similarities, the U.S.and Australia; and for well-acknowledged bestpractices, the U.K. and U.S. However, this study hasrevealed that very few countries follow all bestpractices together from both methodological andgovernance perspectives in their emissions forecasting.Therefore, the U.S. and the U.K. are the solecountries highlighted in this report as having overallbest practices in emissions forecasting.

To be fair, the forecasting of emissions reductionsfrom GHG policies is an evolving practice. Somecountries have consistently attempted to adopt bestpractices from the early days of reporting emissionsprojections; others are only more recently makingconcerted efforts to improve their methodology andgovernance vis-à-vis forecasting. There is a discussionof this issue following the case studies.

Another original intention of this report was tohighlight best practices in jurisdictions other thannational-level governments, particularly someCanadian provinces and U.S. states. Given theunique jurisdictional issues of climate policy inCanada, and the fact that 16 megatonnes of annual

emissions reductions in Turning the Corner areattributed to provincial and territorial initiatives, it isimportant to determine if provinces are conductingaccurate forecasts of their climate policies andmeasures. Few provinces, however, have releaseddetailed plans with emission policy forecasts.Nevertheless, they have set GHG emissionsreductions targets and announced policies andmeasures to achieve these targets, but have notaccompanied these with detailed emissions forecastswhich would attribute specific reductions andmeasures to forecasts, with accompanyingmethodology to allow for independent validation.

This is not to say that provinces are not takingaction—British Columbia is supposed to release aclimate plan with detailed forecasts by summer 2008and Alberta is also developing its own forecasts.However, it is important that all provinces produceemissions policy forecasts utilizing best practices toensure an accurate understanding of the scale of theemissions reduction challenge in Canada andultimately to result in coordinated or complementarypolicy approaches across Canada.

Even in the case of California, often touted as aNorth American leader in climate policy and filling afederal policy vacuum on climate change, forecasts arebased on the voluntary Climate Action Registry. Nosystematic, independent forecasting is conducted.Considering the large scale of regional initiatives takingplace (including proposed emissions tradinginitiatives), and the expected need for emissionstrading systems to link, the collection of data andforecasting of emissions at both the federal andprovincial levels is vital to ensure coherent, coordinatedclimate policy making for Canada as a whole.

4.4.2 United States

Approach

In the U.S., the Energy Information Administration(EIA) performs the majority of energy-basedemissions forecasting. The EIA, created by Congressin 1977, is a statistical agency of the U.S.Department of Energy. Its mission is to provide“policy-neutral data, forecasts, and analyses topromote sound policy making, efficient markets, and


26 For details, see http://www.rsc.ca//files/publications/expert_panels/ozone_&_pm/Ozreport.pdf.

public understanding regarding energy and itsinteraction with the economy and theenvironment.”27 The Department of EnergyOrganization Act (Public Law 95-91) allows EIA’sprocesses and products to be independent fromreview by Executive Branch officials.28 Its domesticenergy projections are based on the National EnergyModeling System (NEMS), a detailed model thatutilizes various modelling methodologies to representindividual domestic energy markets (electricity,refining, industrial, etc.) Confidence in the modelmay be reflected by the number of think tanks,academic institutions, private entities, andlaboratories that use it. The EIA’s Annual EnergyOutlook (AEO) contains a reference case and oftenmore than 30 alternative scenarios. In addition, theEIA publishes an assumptions document, modeldocumentation, and an assessment of its forecasts.

In its publications, the EIA makes reference to theimportance of sustained and significant levels ofresource commitment to the production of nationalGHG emissions projections. Producing detailed

forecasts and alternative policy scenarios on an annualbasis requires substantial funds. Significant resourcesalso ensure capacity in producing the forecasts andensure that the model is based on the most recent data.

The forecasts produced by the EIA are based oncurrent laws and regulations but the EIA doesproduce forecasts on future policies when directed byCongress. This issue is particularly important forCanada, as one of the issues noted in section 4.3.2 isthat the Turning the Corner modelling analysisassumed that “provincial mitigation policies improveover time and become more consistent betweenprovinces.” Since provincial GHG reduction policiesare emerging as an important issue in Canada, thisdistinction takes on added significance. Oneimplication here is that future Canadian analysesmight consider starting with projections of GHGemissions based on existing laws and policies andthen explicitly add scenarios to reflect assumptionsabout improvements in provincial policies. Thiswould give policy makers more precise information asto the challenges and opportunities surrounding


Table 1: Comparison of Governance and Methodology in Emissions Forecasting

Governance Methodology

United States Independent statistical agency Single hybrid model (NEMS); responsible for forecasting (EIA) independent peer review of forecasting

model

United Kingdom Central department responsible for Single hybrid model (M-M model); forecasting with input from other two independent audits of government departments (DEFRA); central agency forecasts; one non-governmental review coordinating climate action across of forecasts.departments (OCC)

Norway* Independent statistical agency General equilibrium model (MSG); responsible for forecasting (Statistics commitment by government for future Norway); climate policy coordinated by independent peer review of model and Ministry of the Environment forecasts

Australia* Newly-created Department of Climate Use of multiple models across sectors.Change responsible for all climate policy, including forecasting

*Studied for the purposes of this report but not included as a best-practice case study.

27 http://www.eia.doe.gov/neic/aboutEIA/quickfacts.html

28 Further information on the EIA can be found at http://www.eia.doe.gov.

specific policies. Such an approach would, at aminimum, increase transparency and facilitateevaluation of future forecasts.

Governance

As noted, the EIA is an independent statisticalagency. The only person appointed to a position inthe EIA is the Administrator. The Administrator isalways an energy professional or statistician. The EIAresponds to requests from Congress from all politicalparties. From its inception, the EIA organizinglegislation was constructed in such a way as to makeit as independent and free from politicalmanoeuvring as possible. Its products are notreviewed or approved by the Administration. They goout under the signature of the Administrator.

Because it is an independent statistical agency, theEIA is often considered “unbiased.” Beyond itsannual projections, the EIA performs policy analysesat the request of Congress or the Administration thatspecifies the “policy.” EIA analyses usually illustratethe impact of the policy compared to its referencecase, the nature of the critical assumptions, and theuncertainty of key variables.

A corollary to the methodological issuementioned earlier concerns the treatment by theresponsible forecasting agency of analyses performedby other government departments or agencies. In theU.S., as in other countries, individual departments oragencies routinely prepare analyses of specific GHGreduction programs—typically of programs theythemselves administer. For example, the U.S. EPAroutinely projects emissions reductions resulting fromits voluntary programs. While these agency analysesare reviewed by the EIA, they are not necessarilyadopted by the EIA. This is an important point.Unless the agency in charge of forecasting GHGemissions has the authority/independence to make itsown professional judgments about the effectiveness ofindividual programs, there is a potential for theforecasts to be driven by more narrow program-driven considerations rather than a more independentand integrated analysis. A perceived strength of theEIA is that it can, and does, have the authority andindependence to disagree with other agencies on theforecasts they are provided with.

Audit/Review Function of Forecast Accuracyand Methodology

Audits and reviews of emissions forecasts are animportant evaluation and accountability tool. Internalto the U.S. Department of Energy is an InspectorGeneral with authority to review the EIA’s productsand processes. There is also the GeneralAccountability Office that has authority to reviewgovernment programs. According to communicationswith EIA officials for the purposes of this report,neither has expended much effort to date inreviewing EIA data or projections. Internal to theEIA, however, is an office of statistical methods thatreviews documentation and hires independent expertsto review model results and accuracy. Prior topublication, all of the offices of the EIA review theAEO through an established clearance process. Thereare also working groups composed of experts fromthroughout the department and industry that meetbiannually and review model methodology andprojections. The EIA also publishes a retrospectivethat evaluates its previous reference caseprojections.29 Along with these processes, the EIAholds an annual conference that highlights the AEOprojections and models. Its projections and modelshave been reviewed frequently in academic journals;it has a memorandum of understanding with theAmerican Statistical Association; it conducts biannualmeetings that often address model methodology andstatistical methods; and it participates in energymodelling fora that compare and contrast modellingmethodologies and results.

Beyond academic peer reviews in journals andstakeholder meetings, the only significant externalevaluation of U.S. emissions forecasts have been theUNFCCC through its Report on the in-depth review ofthe third national communication of the United Statesof America. The report is highly complimentary ofNEMS, stating “the rich representation of technologyin NEMS allows analysis of the impact of mitigationpolicies thanks to the model’s explicit representationof vintaged (time-dependent) capital (energyequipment and structures such as power plants), andtracking of its turnover rates.”30


29 See http://www.eia.doe.gov/oiaf/analysispaper/retrospective/pdf/table18.pdf.

30 UNFCCC (2004), Report on the in-depth review of the third national communication of the United States of America,p.11.

4.4.3 United Kingdom

Approach

While there are a number of inputs into forecastingemissions reductions in the U.K., the Department forEnvironment, Food and Rural Affairs (DEFRA)coordinates and releases the emissions forecasts on anannual basis. The U.K. MARKAL-Macro EnergyModel (M-M model) has been the main model toexplore the technological and macroeconomicimplications of reducing U.K. domestic carbonemissions by its declared target of 60% by 2050. TheM-M model is an integrated energy-macro model,covering the entire energy system in considerabletechnological detail, including electricity generation,heat, and transport. The model explores how, underdifferent assumptions about future fossil fuel pricesand the pace of technological innovation, the energysystem will evolve under a carbon constraint, andwhat the macroeconomic implications to the U.K.economy might be, including the costs to GDP. Themain driver for increased emissions is economicgrowth, as described by the macro module of themodel apparatus. Because the M-M model describesthe economy in equilibrium, it is unable to capturetransition costs that might occur as the economyadjusts to changes in energy policy or prices. As aU.K- only model, it also does not capture theimplications for U.K. trade and competitiveness as aresult of policies aimed at reducing carbon emissions.Therefore, in addition to the M-M analysis, the U.K.government has also used the Oxford Energy-Industry Model (OEIM) and global macroeconomicmodel (GMM) to explore the potential short- tomedium-term adjustment costs associated withmoving to a low carbon economy. As a purelydomestic model, the M-M model also cannot explorethe implications of international carbon trading.

The way in which activity data are broken downto estimate emissions closely resembles the basis onwhich the government monitors economic activity.As a result, much of the economic activity datagathered by government is already classified in aformat that can facilitate estimates of emissions. Oneof the major sources of activity data, covering around85 per cent of emissions, is the Digest of U.K.Energy Statistics (DUKES) produced annually by the

Department of Business, Enterprise and RegulatoryReform. It is the most authoritative source of annualdata on energy use in the U.K. The rest of the dataactivity comes from a variety of sources, including theTransport and Environment Departments. Many ofthe data providers are government departments butsome of the data used to estimate emissions comefrom trade associations and directly from industry.

Governance

While DEFRA coordinates and releases emissionsforecasts in the U.K., the newly-created Office ofClimate Change (OCC) will work across the U.K.government to support analytical work on climatechange and the development of climate change policyand strategy. Many government departments areinvolved in climate-related activities or in helping theU.K. and other countries adapt to its possible futureimpacts. All departments utilize the OCC.

The Climate Change Bill, which is currentlybeing subjected to a full public consultation alongsidepre-legislative scrutiny in Parliament, sets out afundamentally new and more structured approach tothe setting of emissions reduction targets and themonitoring of performance against them. It willtranslate into U.K. legislation, the goal of which thegovernment announced in its 2003 White Paper—namely, a 60 per cent reduction in carbon dioxide by2050 measured against a 1990 baseline. It will createa system of five-year carbon budgets to place theU.K. on a trajectory to meet its long-term goal, andrequire the Secretary of State for the Environment toset and meet carbon budgets for up to 15 years inadvance. A newly created Committee on ClimateChange will be responsible for advising thegovernment on the level of carbon budgets to be set,and for monitoring emissions through annual reportsto Parliament that will include a more comprehensiveassessment of performance after the end of each five-year budgetary period.

Audit/Review Function of Forecast Accuracyand Methodology

The forecasting of GHG emissions reductions hasbeen the subject of at least two audits31 by the U.K.National Audit Office. Of the countries researched


31 National Audit Office (2006), Emissions Projections in the 2006 Climate Change Programme Review; National AuditOffice (2008), U.K. greenhouse gas emissions: measurement and reporting.

for this report, the U.K. is the only one to have hadits entire emissions forecasting measurement andreporting audited by an independent third party. Theaudit had a number of key findings:

• The U.K. will meet or exceed its Kyoto target onall but the most pessimistic assumptions, and willfall short of its 2010 domestic target in all but themost optimistic assumptions.

• Forecasts made in 2000 have been revised toreflect a reduction in the expected savings fromindividual policy measures, changes in fossil fuelprice assumptions, and gradual refinements to theformer Department of Technology andInnovation’s (DTI) energy demand model andadjustments to the 1990 baseline. A degree ofchange in projections is to be expected; the U.K.government recognized that the 2000 estimateswere subject to considerable uncertainty.

• The forecasts are based on sophisticated modellingapproaches. The models are subject to expertreview and other quality assurance processes.

• U.K. Government has taken steps to make the2006 forecasts more robust than those in 2000.The review of projected policy impacts that tookplace in 2006 involved a more skeptical scrutinyof the emission reductions to be expected frompolicy measures. There was also more detailedanalysis of uncertainty.

• Forecasts against the 2020 and 2050 domestictargets to reduce CO2 are less well developed andnecessarily more speculative. As the 2010 targetapproaches, it is important to switch attention tothe realism and delivery of these future targets.

• International reporting requirements specify thebasis on which emissions should be estimated. TheU.K.’s estimates follow best practices and have beenreviewed favourably by international experts inGHG measurement appointed by the UN.

• The Climate Change Bill provides a newframework for setting U.K. targets. Its provisionswill introduce concepts such as the “net U.K.carbon account” and requirements to account for

the contribution made by carbon credits anddebits from emissions trading schemes. Suchprovisions could complicate the reportingframework further, or else provide an opportunityto develop a more comprehensive and transparentbasis for presenting climate change statistics.32

The forecasting of emissions in the U.K. was also thesubject of an audit undertaken by University CollegeLondon’s Environment Institute. In its report entitledU.K. Greenhouse Gas Emissions: Are We on Target? theEnvironment Institute found that while the U.K.GHG emission target of a 12.5% cut to the baselinelevels required by the Kyoto Protocol will be met, theemissions reductions forecast for the 2020 target willbe difficult to meet because of continued significanteconomic growth that will cause emissions to riseafter 2010. The audit suggests current policies wouldachieve a GHG emission reduction between -12 and-17 per cent by 2020 as opposed to the government’spolicy aim of -30 per cent. The audit states that theoverriding reasons for the possible failure of currentgovernment policies to achieve their stated targets isthat nearly all the policies are voluntary.

In its Report of the centralized in-depth review of thefourth national communication, the UNFCCCcommends the U.K. for its emissions projections.33

The U.K. was also commended not only for itsconsistency with earlier reports, but the transparentand concise nature of its report. The U.K. is furthercommended for not only including a “with measures”scenario, but a baseline scenario and two “withadditional measures” scenarios (including the effect ofplanned measures). From a governance perspective,the U.K. received praise for the establishment of theClimate Change Projects Office (CCPO), theappointment of a designated national authority forthe Clean Development Mechanism (CDM), and theappointment of a designated focal point for jointimplementation (JI) projects. The UNFCC furthercommended the U.K. for its role in the developmentof the registry software and finally, “lauds the U.K.for its solid and coherent program of action.”34


32 National Audit Office (2008), U.K. greenhouse gas emissions: measurement and reporting, p.5.

33 The NRTEE expressed similar concerns to the Canadian government in its 2007 KPIA Response.

34 UNFCCC (2007), Report of the centralized in-depth review of the fourth national communication of the UnitedKingdom of Great Britain and Northern Ireland, p.18.

5. DiscussionWhile the case studies yield many interesting findingsin relation to best practices in GHG emissionsforecasting, it is important to determine to whatextent the case studies follow the best practicescriteria discussed in sections 4.1 and 4.2. It is alsoimportant to understand how forecastingmethodology and governance relate to one another. Isone dependent on the other? Does strongmethodology alone lead to strong forecasts? Is itpossible for countries to have accurate forecastswithout strong governance in climate policy? Isstrong governance a key factor in determiningaccurate forecasts?

As both the U.S. and U.K. approaches are subjectto extensive peer review and rigorous analysis andconstant updating, it is likely from a methodologicalperspective that they produce accurate emissionsforecasts.

From a governance perspective, it is preferable tohave one central agency or group undertaking orcoordinating emissions forecasting. It does not,however, necessarily detract to have several sources offorecasts, arising from competition among forecastersand also from the characterization of modeluncertainty facilitated by multiple models. This isonly preferable, however, if the oft-cited case ofmultiple forecasters who do not communicate witheach other, do not compare results, and do not learnfrom each others’ errors, is avoided. But multipleforecast groups who talk, compare, and learn mayproduce better forecasts than just one group. Only inthis circumstance, but still with the coordination of acentral agency, could a multi-source approach toemissions forecasting be useful.

It is also important to bear in mind that differentagencies within government obviously have differentproblems to solve and different policy and programobjectives, so that different models may help get theappropriate answers they individually need.35 Thisdoes not mean, however, that this should bereplicated for the complex, integrated challenges oflarge-scale GHG emissions forecasting, which is the

Canadian government’s current approach. However,instead of individual departments submitting theirestimates to Environment Canada, a preferablesolution within this approach would be to require thedifferent departments that produce the variousforecasts to meet regularly to compare results andattempt to arrive at consensus on common questionsof interest, or at least learn from others’ attempts.Even if no consensus is possible, the variety of resultsmight provide an indication of uncertaintyattributable to a lack of knowledge of the primarymodel.

In the area of governance, both the U.S. and U.K.received praise from the UNFCCC for theconsistency in their approaches to forecasting. Asidefrom the U.S. and U.K., it is important to note therecent commitments by the other governmentsstudied for the purposes of this report but notincluded as best-practice countries; the Norwegianand Australian governments have made commitmentsto independent peer review, centralized forecasts, anddeveloped ambitious climate policy (includingaggressive reduction targets) in general. This isespecially of interest to Canada as these governmentsface similar economic and jurisdictional challenges.

With the exception of the U.K., the other casestudies (U.S., along with Norway and Australia)reveal the lack of independent reviews by governmentauditors on the accuracy of emissions forecasts. In thecase of the U.K., audits by an independent federalauthority and an academic research institute can besaid to improve understanding and credibility of itsforecasts.

An important issue is the criticism levelled at allgovernments by the UNFCCC (and in the case ofthe U.K., its own independent auditors) on thesignificant forecasted emissions reductions attributedto voluntary measures. There is extensive literature onthe limited success of attributing a level of specificemissions reductions as a result of voluntarymeasures. For example, studies highlight the paucityof credible evidence on the performance of voluntaryprograms compared to a realistic baseline.36 In its2007 KPIA Response, the NRTEE noted that with


35 For example, a specific program or measure (i.e., fuel efficiency vehicle programs) may require a different forecastingmodel than one designed for large-scale, integrated modelling.

36 Morgenstern, D. and W.A. Pizer (eds) (2007), Reality Check: The Nature and Performance of Voluntary EnvironmentalPrograms in the United States, Europe and Japan. Washington, D.C.: RFF Press.

“few exceptions, little evidence exists through whichone can evaluate the incremental effect ofinformation-provision programs for emissions controlor energy conservation.”37 This reliance on voluntaryprograms can therefore raise questions about thelikelihood that the emissions forecasts from thesemeasures will be accurate. For example, pastemissions forecasts in Canada have relied onvoluntary measures and programs to deliversignificant reductions and in all cases haveoverestimated the forecast emissions reductions.Attributing significant emissions reductions tovoluntary measures will by their very nature continueto result in inaccurate emissions forecasts. In this way,the governance and methodological issues raised inour analysis point to the need for policy makers toconsider alternative climate policy measures toachieve desired GHG emissions reductions.

How strong is the link between good forecastingmethodology and good forecasting governance inproducing accurate emissions forecasts? From agovernance perspective, all countries studied for thepurposes of this report reveal that strong, centraldepartments or agencies with independence shouldbe responsible for emissions forecasting. In the case ofthe U.S., an independent statistical agency isresponsible for producing its country’s GHGemissions forecasts.38 Where there is not anindependent statistical agency responsible forforecasting (e.g. the U.K.), there is strong politicalcommitment and centralized governance structures inplace. Therefore, a significant finding (as noted insection 4.4.2) is that unless the agency in charge offorecasting GHG emissions has the authority orindependence to make its own professionalconclusions about the effectiveness of individualprograms, there is a potential for the forecasts to bedriven by more narrow program-drivenconsiderations rather than a more independent andintegrated analysis aimed at overall GHG emissionsforecasting.

5.1 Lessons for Canada

While the above discussion, and key findingssummarized in the conclusion below, provideimportant insight for Canada on potential areas toexplore in its approach to emissions forecasting, it isimportant to determine key lessons from the casestudies that can be applied to the Canadian context.

Independence of Agency/DepartmentResponsible for Forecasting

In the U.S., as in other countries, individualdepartments or agencies routinely prepare analyses ofspecific GHG reduction programs – typically ofprograms they themselves administer. For example,the U.S. EPA routinely projects emission reductionsresulting from its voluntary programs. While theseagency analyses are reviewed by the EIA, they are notnecessarily adopted by the EIA. This is an importantpoint. Unless the agency in charge of forecastingGHG emissions has the authority/independence tomake its own professional judgments about theeffectiveness of individual programs, there is apotential for the forecasts to be driven byprogrammatic rather than analytic considerations. Aperceived strength of the EIA is that it can and doeshave the authority and independence to disagree withother agencies on issues of this sort.

At the federal level in Canada, the closest existingbody in terms of governance to the EIA would beStatistics Canada. While it does not currentlyundertake emissions forecasting, it collects emissionsdata for Environment Canada’s GHG emissionsinventory. Through legislation, Statistics Canadacould be given the responsibility and sole authority toproduce Canada’s emissions forecasts. Theindependence of Statistics Canada would ensure thatemissions forecasting would be free from interferenceand that it would have the authority to question andif necessary reject emissions reductions estimates fromgovernment departments. Alternatively, the federalgovernment could also follow the example of


37 NRTEE (2007), Response of the National Round Table on the Environment and the Economy to its Obligations underthe Kyoto Protocol Implementation Act, p. 32.

38 Statistics Canada – Canada’s independent, federal statistics agency – collects emissions data for the purpose ofreporting, not forecasting.

Australia and create a new federal department oragency responsible for all climate change policy,including forecasting.

Clearly Articulated Legislated Roles,Responsibilities, and Milestones forClimate Policy

The U.K. Climate Change Bill, which is currentlybeing subjected to a full public consultation alongsidepre-legislative scrutiny in Parliament, sets out afundamentally new and more structured approach tothe setting of emissions reduction targets and themonitoring of performance against them. It willtranslate the goal the government announced in the2003 White Paper into legislation—namely, a 60 percent reduction in carbon dioxide by 2050 against a1990 baseline. It will create a system of five-yearcarbon budgets to place the U.K. on a trajectory tomeet its long-term goal and require the Secretary ofState to set and meet carbon budgets for up to 15 years in advance. A newly created, externalCommittee on Climate Change will be responsiblefor advising the government on the level of carbonbudgets to be set, and for monitoring emissionsthrough annual reports to Parliament that willinclude a more comprehensive assessment ofperformance after the end of each five-year budgetaryperiod.

Canada should consider a truly integrated long-term climate policy with similar characteristics.Given the comprehensive nature of the climatechallenge, which crosses a range of governmentdepartments (not just environment), a central policydevelopment and coordination body that links policyand program choices with integrated emissionsforecasting would likely result in more confidence inthe efficacy of proposed emissions reductionmeasures. Legislating its role and mandate toincorporate independent forecasting and evaluationswould further strengthen its effectiveness andconfidence in the policy approaches being pursued.Providing an external review or audit of emissionforecasts would complete this new approach.

The Role of Provinces and Territories

An original intention of this report was to highlightbest practices in jurisdictions other than national-levelgovernments, particularly some Canadian provincesand U.S. states. Given the unique jurisdictional issuesof climate policy in Canada, and the fact that 16megatonnes of annual emissions reductions inTurning the Corner are attributed to provincial andterritorial initiatives, it is important to determine ifprovinces are conducting accurate forecasts of theirclimate policies and measures.

With its past two reports on long-term issuesrelated to climate change and energy,39 the NRTEEhas consistently called for a coordinated, nationalapproach to climate change issues in Canada. If thefederal government is to produce accurate nationalemissions forecasts and ensure adopted policymeasures achieve forecast reductions, there needs tobe better coordination and understanding ofprojected emissions reductions from provinces’climate policies and measures. There also needs to bea better understanding among governments inCanada as to various approaches to forecasting andhow consistent methods can be applied to ensureaccurate forecasts of GHG emissions reductions.Provinces should be encouraged to develop andrelease detailed emissions forecasts to inform theirown policy choices necessary to meet the reductiontargets they have set for themselves.

Scenario-Based Emissions PolicyForecasting

The U.S. EIA’s reference case projections are based oncurrent laws and regulations with additional forecastsincorporating future policies. This issue is particularlyimportant for Canada, as one of the concerns notedin section 4.3.2 is that the Turning the Cornermodelling analysis assumed that “provincialmitigation policies improve over time and becomemore consistent between provinces.” As noted in theNRTEE’s 2008 KPIA Response, potential futureemissions reductions from provincial actions arecounted as realized emissions reductions.


39 NRTEE (2006), Advice on a Long-term Strategy on Energy and Climate Change; NRTEE (2008), Getting to 2050:Canada’s Transmission to a Low-Emission Future.

Scenario-based emissions policy forecasting canhelp address this concern. Future Canadian analysesmight consider starting with projections of GHGemissions based on existing laws and policies andthen explicitly add scenarios to reflect assumptionsabout improvements in provincial policies. This same,scenario-based approach can be extended to policy-level analysis, presenting a forecast of economy-wideemissions with and without the policy in place, eachin a scenario where all other policies are in place.

This provides an estimate of the marginal orincremental effect of the policy including adjustmentsfor free-ridership and policy interaction effects, anddepending on the structure of the model, reboundeffects. These estimates could be included along withthe current policy-level analysis or as replacement forthem. Such an approach would, at a minimum,increase transparency and facilitate evaluation offuture forecasts.


6. ConclusionsThe NRTEE’s research and analysis of internationalbest practices in GHG emissions forecasting leads itto conclude the following:

From a methodological perspective:

• Hybrid energy-economy models are more effectivein producing accurate GHG emissions forecasts asthey integrate the strengths of both the traditionalbottom-up and top-down approaches tomodelling emissions forecasts;

• The use of a consistent baseline from year-to-year(including baseline data), assumptions, andconditions across the board is fundamental toensure emissions forecasts can be accuratelycompared from year to year;

• The use of consistent and agreed definitions ofterms and concepts, such as for free ridership andadditionality, across government departmentsinvolved in forecasting would ensure greatertransparency of emission forecasts and facilitateassessment of the forecasts’ accuracy.

• There is need for an international perspective inthe model so that it can respond appropriately toworld events (since in most cases, Canada is aprice taker for both commodities and energy, anda primary trader of goods and energy). Canada isacting in concert with other countries on climatepolicy and its forecasting approaches need toreflect this reality.

From a governance perspective:

• Use of an independent forecasting agency ispreferable to provide more accurate andtransparent emission forecasts for consideration bygovernment policy makers, external analysts, andParliamentarians and to facilitate ongoing auditand evaluation.

• Multi-source emissions forecasting from a groupof individual government departments can beaccurate, but works best both when centrallycoordinated and with independent authority bythe central coordinating department or agency toquestion other departmental forecasts.

• Regular independent reviews, audits andevaluations of government forecasts andforecasting methods by a third-party agency or

process helps ensure accuracy of forecasts and thatforecasting methodologies are up-to-date androbust.

• Forecasting must be sufficiently resourced andfinanced by governments to ensure data is up todate and most recent improvements in forecastingmethodologies are incorporated for the benefit ofpolicy makers taking decisions based on theseforecasts.

• Regular, ongoing evaluation of past forecasts foraccuracy and effectiveness is necessary to ensurecontinuous improvement of governmentforecasting methodologies and approaches.

• Ensure transparency and clarity with respect tokey assumptions and methods.

Good climate change policy stems from accurateemissions forecasting. It is the first building block.The public must have confidence that theirgovernment’s GHG policies and measures will resultin the emissions reductions promised by politicalleaders. The government has taken a significant stepin the right direction with its 2008 KPIA Plan.

Most countries face similar challenges inproducing accurate forecasts for emissions reductionsfrom policies and measures. As highlighted in theintroduction to this report, the NRTEE, recognizingthat forecasting is difficult and challenging not onlyin Canada but in all countries, wanted to provide thefederal government with examples of how othergovernments approach and produce emissionsforecasts. The case studies show there are fewcountries that truly utilize all best practices, fromboth methodological and governance perspectives, intheir approaches to emissions forecasting. However,there are best practices in other countries that couldbe applied to the Canadian context.

The NRTEE hopes that these international bestpractices can provide insight to the federalgovernment on how best to address some of Canada’sforecasting challenges as identified in the 2007 KPIAResponse—those of transparency and clarity withrespect to key assumptions and methods; theconsideration of important sensitivities anduncertainties; the importance of consistency inapproach across various departments, programs, andmeasures; and the need to integrate findings in aholistic framework.


While the NRTEE appreciates the extent towhich the federal government addressed thesechallenges in its 2008 KPIA Plan, it encourages thegovernment to continue improving its methodologyand governance in its approach to emissionsforecasting.


AEO Annual Energy Outlook (annual report on domestic energy-based emissions—U.S.)

AMG Analysis and Modelling Group

BAU Business-as-usual

CCPO Climate Change Projects Office

CDM Clean Development Mechanism

DEFRA Department of the Environment, Food and Rural Affairs (U.K.)

DTI Department of Technology and Innovation (U.K.)

DUKES Digest of U.K. Energy Statistics

EIA Energy Information Administration (U.S.)

E3MC Energy-Economy-Environment Model for Canada

GDP Gross domestic product

GHG Greenhouse Gas

GMM Global macroeconomic model (U.K.)

IEA International Energy Agency

IEO International Energy Outlook (annual report on international energy-based emissions—U.S.)

IPCC Intergovernmental Panel on Climate Change

JI Joint Implementation

KPIA Kyoto Protocol Implementation Act

M-M Model MARKAL-Macro Energy Model (U.K.)

MSG Norwegian equilibrium model

Mt megatonne

NEMS National Energy Modeling System (U.S.)

NRTEE National Round Table on the Environment and the Economy

OCC Office of Climate Change (U.K.)

OEIM Oxford Energy-Industry Model (U.K.)

PM particulate matter

UNFCCC United Nations Framework Convention on Climate Change

WEO World Energy Outlook (annual report of the IEA)


Appendix A: List of Abbreviations and Acronyms

There are four key reasons the emissions reductionswere overestimated in the Government of Canada’s2007 KPIA Plan. First, the estimates of thereductions generated by the various initiativessuffered from biases related to additionality (includingadditionality concerns resulting from a lack ofaccounting for free-ridership). Second, the emissions-reduction factors used in the calculations were, insome cases, not consistent with recent scientificevidence. Third, rebound effects were not always takeninto account in the estimates. Finally, policies weretreated independently, so policy-interaction effects areignored.

Problems of additionality arise when the statedemissions reductions do not reflect the difference inemissions between equivalent scenarios with andwithout the initiative in question. If emissionsreductions from an initiative have already beenincluded in the reference case, these emissionsreductions will be double-counted.

A key source of additionality issues that arisesfrequently and so was treated separately in theNRTEE’s 2007 KPIA analysis is the failure toaccount for free-ridership. Free-ridership is notproperly accounted for when stated reductionsinclude the results of behaviour that is rewarded butnot influenced by the policies. This can occur whensubsidies are paid to all purchasers of an item,regardless of whether they purchased the item becauseof the subsidy. Those who would have purchased theproduct regardless are termed free-riders, and their

behaviour (since it would have happened regardless ofthe policies) has already been accounted for in thereference case. Not correcting for this implies thatinduced emissions reductions will be overestimatedby the proportion of free-riders, which has beenestimated to be between 40% and 80% (NRTEE,2005).

The rebound effect describes the increased use of amore efficient product resulting from the implieddecrease in the price of use: for example, a moreefficient car is cheaper to drive and so people maydrive more. While estimates vary, emissionsreductions will generally be overestimated by between5% and 20% if estimates do not account forincreased consumption due to the rebound effect.

The relative successes of emissions-control policieswill be interdependent, and an evaluation frameworkthat takes this into account is important for properinterpretation of stated results. The 2007Government KPIA Plan provides results fromseparate evaluations of individual policies, while theseare slated to be imposed simultaneously. Thisapproach omits any policy interaction effects and willonly be accurate when the sum of all individualpolicy effects is equal to the total effect of all policies,which is not likely to be the case. A general finding ofthe NRTEE’s 2007 KPIA Response, which isconsistent with the statement above, is that in orderto deliver a statement of total expected emissionsreductions, all policies should be imposedsimultaneously in a modelled economy.


Appendix B:Description of Additionality, Free-ridership, Rebound Effect,

and Policy Interaction Effects

Appendix C:Discussion of Top-down, Bottom-up, and Hybrid Approaches to

Energy-Economy Modelling


Energy-economy models generally range fromdetailed bottom-up models reflecting engineeringeconomic details of a wide menu of technologies ineach sector, to top-down models of the wholeeconomy calibrated on historic data from a few tohundreds of sectors. Hybrid models—those thatcombine the strengths of the bottom-up and top-down approaches—are considered by manymodelling authorities as optimal approaches toforecasting.40

The essential element in a bottom-up model isthat it is a model of individual units in a system, inwhich aggregate properties are then deduced from thebehaviour of the individual units. For example, in abottom-up model of soft drink consumption in agroup of teenagers, we would look at the soft drinkconsumption of each teenager, and then add them upto get the group’s consumption. By contrast, a top-down model begins with a model of the aggregate; anattempt may be made to deduce properties of sub-units from the aggregate. In a top-down model of thegroup of teenagers, total soft drink consumption bythe group is modelled, and then we try to allocate itamong individuals.

Weaknesses in traditional top-down modelsinclude the fact that they do not explicitly represent

the technologies in the energy system, so policiesdesigned to influence technology evolution directlycan only be crudely simulated at best. Bottom-upmodels are based on theoretical assumptions abouthuman behaviour, with the result that theirpredictions do not represent the economic system.Because of their different structures and definitions ofcosts, the two types of models tend to predict verydifferent economic structures. Top-down modelsusually predict high costs of GHG emissionreduction policies while bottom-up models usuallypredict low costs. In order to address theseweaknesses, modellers have attempted to createhybrid models that integrate the strengths of both thebottom-up and top-down approaches.41 Such hybridenergy-economy models have attempted to bridge themethodological schism between top-down andbottom-up approaches by meshing the description ofthe economy in terms of specific technologies (as inbottom-up models) into an integrated energy-economy model.

In Canada, both the E3MC and CIMS modelsare hybrid models used by a variety of actors. Hybridmodels used in the U.S. and U.K. are NEMS and theU.K. MARKAL-Macro Energy Model.

40 Rivers, N. and M. Jaccard (2005). “Combining Top-Down and Bottom-up Approaches to Energy-EconomyModeling Using Discrete Choice Methods,” The Energy Journal, Vol.26, No.1; Grubb, M (1993) “Policy Modelingfor Climate Change: The Missing Models” Energy Policy, 21(3); Intergovernmental Panel on Climate Change(IPCC) (1996), IPCC Second Assessment Report: Climate Change 1995.

41 Grubb (1993), Hoffman and Jorgenson (1977), Jacobsen (1998), Bohringer (1998), and Koopmans and te Velde(2001) have all designed hybrid models.

Appendix D: Glossary of Useful Modelling Terms


TERM DEFINITION

Behavioural parameters Parameters in CIMS used to more realistically represent consumer preferences for technologies. They include discount rate, representing time-value preferences; heterogeneity, representing the extent to which different consumers have different preferences for technologies and thus perceive costs and benefits differently; and intangible costs, representing non-financial costs associated with specific technologies, such as the greater risk associated with new technologies or technologies with longer payback periods.

Behavioural Realism A criteria for assessing model usefulness; the extent to which consumer choices—and their response to price signals—is incorporated into a model and supported through empirical means.

Bottom-up models Techno-economic models that have extensive technological detail, but provide overly optimistic emissions forecasts, since they assume new technologies are perfect substitutes for older ones, and that consumers always will choose the least-cost technology option. Bottom-up models also typically do not include feedbacks with the economy as a whole.

CIMS A technology vintage model that forecasts emissions via the turnover of energy-using and energy-supplying technology stocks. CIMS models consumers’ choices of new technologies.

Computable General A specific type of top-down model that models multiple markets in equilibrium Equilibrium (CGE) with each other in an economy.Models

E3MC The Energy-Economy-Environment Model for Canada, which is Environment Canada’s emission forecasting model. Essentially the model consists of the Energy 2020 and TIM models linked together.

Energy 2020 A technology vintage model that forecasts emissions via the turnover of energy-using and energy-supplying technology stocks. Energy 2020 models consumers’ choices of new technologies.

Hybrid Models Models that combine the strengths of top-down and bottom-up models. CIMS and Energy 2020 are examples of hybrid energy-economy models.

Macroeconomic A criterion for assessing model usefulness; the extent to which interactions completeness between the energy sector and the economy as a whole are represented in a

model. This criterion particularly affects how useful the model is in assessing the costs of policy options to the economy.

Technological One criterion for assessing model usefulness; the extent of detail in which a Explicitness model includes specific energy-using and supplying technologies.

TIM The Informetrica Model—a macroeconomic model that examines the economy as a whole. TIM would be useful for examining issues such as trade and economic impact of policies. TIM can be linked to microeconomic models focusing specifically on the Canadian energy system.

Top-down models Models that represent the energy-use, technological change, and the economy as whole at an aggregate level. These models typically do include macroeconomic effects and realistically represent consumer behaviour. They usually lack any detailed representation of technologies.

Notice to Reader

This report was reviewed by the following well-respected economic experts:

• Knut H. Alfsen, Head Research Director, Centre for International Climate and Environmental Research,Oslo

• John Galbraith, Professor, Department of Economics, McGill University, Montréal

• Richard Morgenstern, Senior Fellow, Resources for the Future, Washington, D.C.

• John Nyboer, University Research Associate, School of Resource and Environmental Management, SimonFraser University, Vancouver


Documents

Greenhouse Gas Emissions Forecasting: Learning from International Best Practices