8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
1/16
McCulloughResearch
ReviewoftheERCOT
December18,2008Nodal
CostBenefitStudy
January7,2009
RobertMcCullough
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
2/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
2|P a g e
OverviewOnDecember18,2008,CRA InternationalandReseroConsultingreleasedastudythatconcludedthat
the benefits of continuing the shift to nodal markets from the existing zonal system significantly
outweighedthecosts. Theyreportedbenefitsintwodifferentforms. First,theareaofTexasservedby
ERCOTwouldexperiencea$520millioncostreduction,whilerequiringonly$222million inadditional
computer programming costs. Second, the new market design would reduce costs to ERCOTs
consumersby$5.6billion.
Acarefulreviewrevealsseriousquestionsconcerningitsaccuracyandcraftsmanship. Thescopeofthe
new study is 2011 to 2020, although only the first two years are actually modeled. The period from
2013through2020issimplyarepeatedcutandpastefromtheresultsfor2011and2012. Moreover,
thebenefitsduetothesitingofnewgenerationaresimplyassumedfromtheresultsofthe2004cost
benefitstudy. Overall,only17%ofthe$520millionsavingsactuallyresultsfromthecomputermodel
described in the report. The remaining 83% of the benefits are the product of various ad hoc
adjustments.
Similarly, the $5.6 billion in consumer benefits are also based on modeling that stops in 2012 even
thoughresultsarereportedthrough2020. Thissurprisingresultthatashift incomputeralgorithms
will dramatically reduce revenues for electric generators reproduces the conclusions of the 2004
study. In both cases, the result relies on the assumption that generator bids are fixed at short term
marginal costs. Unfortunately, this is not the case inTexasandother areas thathave adoptednodal
pricing.
Strategicbidding
(bids
vastly
higher
than
marginal
costs)
is
acentral
feature
in
ERCOT.
In
any
market where bids can be easily changed to recapture the assumed revenue loss, such gains for the
consumerarebothspeculativeandeasilycorrectedbyadjustingthebids. The$5.6billionestimatealso
containsasignificanterroracriticalratioused inthecalculation isbasedonvaluestakenfromyears
beforenodalmarketsareactually implemented. Overall,the$5.6billiontracesonly17%tocomputer
modelingandwiththeresidual83%representingpoorlydocumentedadhocadjustments.
Noattemptwasmadetocalibratethestudytoactualconditions (knownasbackcasting). Nordidthe
authors estimate the sensitivity of their results to their assumptions. Since some assumptions were
unusual,the lackofsensitivitytesting isparticularlyproblematic. Thecostreduction inthetwoyears
actuallymodeled
is
only
.04%
of
annual
production
costs,
aslender
margin
in
aworld
where
the
cost
of
oilincreased100%in2008beforefalling75%bytheendoftheyear.
Theestimateof$222millioninadditionalcomputerprogrammingcostsisalsoproblematic. Inactuality,
theDecember18,2008studyassumes$429millioninadditionalcomputerprogrammingcoststhatare
offset by $207 million of additional computer programming costs to refresh the existing market
design. WhiletheDecemberstudyestimatesmaintenancecostsforthenewcomputerprogramas$14
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
3/16
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
4/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
Atfirstglance,thistableappearstorepresentaneditorialdecisiontoomitthefullsetofresultsinthe
interestofbrevity. Acloserreadingmakesclearthattheseyearsaretheonlyyearsusedintheupdated
study:
TheNPVfrom2011to2020 isestimatedtobe$339million,assumingthatproduction
costsandresultingbenefitsobservedforthefirsttwoyearsofoperationremainatthe
samelevelonaveragethrough2020.2
ThecompletetablecanbereverseengineeredfromtheprojectedNPVonthefinallinesofTable5:
Table5:AnnualProductionCostbyScenario(inreal2008dollars)
ZonalCase NodalCase Benefit(Zonal Nodal)
($Million) ($Million) ($Million)
2009 12,928.6 12,892.0 36.6
2010 12,319.2 12,277.1 42.1
2011 12,211.2 12,164.4 46.8
2012 12,212.0 12,164.0 48.0
2013 12,211.2 12,164.4 46.82014 12,212.0 12,164.0 48.02015 12,211.2 12,164.4 46.82016 12,212.0 12,164.0 48.02017 12,211.2 12,164.4 46.82018 12,212.0 12,164.0 48.02019 12,211.2 12,164.4 46.8
2FinalReport,December18,2008,page9.
4|P a g e
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
5/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
5|P a g e
2020 12,212.0 12,164.0 48.0AverageAnnual(20112012) 12,211.6 12,164.2 47.4
ProjectedNPV
(2011
2020)
86,378
86,039
339
Therowsadded in italicswerenotmodeled intheDecemberstudy,which insteadmodeled2011and
2012,andthenthesameresultsweresimplycutandpastedfortheremaining80%ofthestudyhorizon.
This unusual methodology is surprising given the problems that arose in the 2004 study in modeling
ERCOTsproductioncostsafter2012:
TCA believes that specific predictive conclusions should not be based on TCA results
obtained for 2013, and especially 2014. This is because the massive addition of new
generatingresourcesmodeledforoutyearsisnotsupportedbytransmissionupgrades.3
Thissame
conclusion
also
occurs
later
in
the
2004
report:
Themidtermtrendcontinuesforanotheryear(2012),butisreversedin2013and2014
duetodifficultiesassociatedwiththemodelingoffurthercapacityexpansiondecisions
intheabsenceoftransmissionupgrades.4
Table34intheNovember2004reportclearlyshowstheproblem:
3FinalReport,November24,2004,page316.
4FinalReport,November24,2004,page322.
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
6/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
In2013,thebenefitsofmovingtoanodalsystemarereducedbyover50%. Inthefollowingyearthe
resultsactuallyindicatethatthezonalsystemissuperiortothenodalsystem.Althoughitmaymerelybe
acoincidence,itisinterestingthattheDecember2008studystopsmodelingintheyearwhereproblems
aroseintheNovember2004study.
The second componentof nodalbenefits is the potentialadditional precisionofdata forpowerplant
siting.
The
December
2008
study
assumes
the
value
of
improved
generation
siting
by
a
particularly
arbitraryprocedure:
Assuming70% inadditionalbenefitsattributable to improvedgenerationsitingovera
periodof2013 through 2020 (when generation capacitywillbe needed inaddition to
6|P a g e
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
7/16
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
8/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
8|P a g e
Only a small fraction of the reported results come from modeling using current loads, fuel prices,
generation, and transmission. The vast majority of the results come from ad hoc adjustments the
reuse of earlier years results in lieu of completing the modeling and proration of results from the
November2004study.ConsumerPaymentSavingsThe most significant numbers in the December 2008 study concern consumer benefits. The authors
indicate thatbetween2009and2012consumerswillseeanannualaverage$576.6million insavings
duetothereimbursementofcongestionrentsandotherbenefits. Thisvalueisthenratioedupby
the fraction that the average benefits had between 2005 and 2008 to total benefits estimated in
November2004.
This approach is extremely speculative. As noted above, the new study represents different years,
different resources, different fuel prices, and different transmission capabilities. To compound the
problems,theratiousedisbasedontwoyearswhenthenodalmarketwillnotbeoperating,sotheratio
itself, regardless of its legitimacy, is in error. Finally, the assumption addressed below, that ERCOT
generatorswillrestricttheirbidstoshorttermmarginalcost, isbothunrealisticand likelyto leadtoa
massiveoverestimateoftheconsumerpaymentsavings.
Thecalculationdescribedonpage34oftheDecember2008reportis:
1. Identifytheconsumerpaymentsavingsfor2005through2014fromtheNovember2004report$7.3billion.
2. Averagethecongestionrentsfor2009through2012fromtherevisedmodelusedintheDecember2008study.
3. Averagethecongestionrestsfor2005through2008fromtheoriginalmodelusedintheNovember2004report.
4. Dividethe2004modelresultbythe2008modelresultgettingafactor1.29.5. Dividethe2004valueforconsumerpaymentsavingsby1.29toget$5.6million.
Thiscalculation issounusualthat it isdifficulttoseeanyeconomic logic in itsderivation. There isno
reason
to
believe
that
there
is
any
logical
comparison
between
market
conditions
in
2005
through
2008
asestimated fouryearsagoandthecurrent forecast for2009 through2012. Even ifthemarkets
were similar, despite the changes in generation, fuel prices, and transmission, the appropriate
comparisonwoulduseyearsinwhichthenodalmarketwasactuallyoperating,not2009and2010when
ithasnotyetstarted.
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
9/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
LocalCongestRentfor
2011and2012
17%
Ratioto2004Study's
Values
for
Consumer
Benefits
83%
AssumedValuesConstitutes83%ofTotalConsumerBenefitsfromShiftingtoNodalMarkets
Aside from the illogic of the calculation, the economics of this argument are also suspect. ERCOT is
characterized by a severely oligopolistic market structure of many buyers and few sellers. Strategic
biddingisacontinuingfeatureofERCOTsmarketswithbidsandpricesoftenatmultiplesofshortterm
marginal cost. If, as the authors assume, local congestion rents reduced the revenues of producers,
economic theory would lead one to expect that strategic bidding may increase to recover the
generatorsrevenues.
ThemajorgeneratorinERCOT,Luminant,routinelysubmitsbidsat$300/MWh. Wheninvestigatedby
theTexasPublicUtilitiesCommission,Luminantdefendeditsstrategicbiddingpracticesbyarguingthat
suchhighbidswererequiredtocoveritstotalcostsofgeneration. Whileitsargumentmighthavehada
poorgrounding ineconomics, it was a perfectly logical argument fora price leader in an oligopolistic
market. IfLuminantsactualcosts increasedafternodalmarketswereadoptedbyERCOT, it is logical
thatitwouldaddthecoststoitsstrategicbids,notpassivelyacceptareductioninincome.
9|P a g e
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
10/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
10|P a g e
BackcastingAn important featureoftheNovember2004reportwasanattempttotest itsaccuracyagainstactual
pricesandoperationsfor2003. Thisisastandardstepforanycomplexcomputermodel. Backcasting
results for the November 2004 model were poor: overall ERCOT forecasts were 25.4 $/MWh against
actual prices of $29.1/MWh. This represents an error on theorderof15%,a fairly poorshowing for
forecastingthepreviousyear.
The December study dispenses with backcasting altogether. This omission is puzzling since the
December2008modelforecastedtwoyears(2009and2010)ofnospecialinteresttothependingpolicy
decision. These two years are not used in the calculations (with the exception of their erroneous
inclusionintheratiousedtoassumeconsumerbenefits)sincethenodalmarketisnotforecasteduntil
2011. Belowisanexplanationbyyear:
2007 Notmodeledasabackcast
2008 Notmodeledasabackcast
2009 Nodal market not in operation modeled but not used in the present value of
productioncostsavings
2010 Nodal market not in operation modeled but not used in the present value of
productioncostsavings
2011 Nodal market in operation modeled and used in present value of production cost
savings
2012 Nodal market in operation modeled and used in present value of production cost
savings
2013 Nodalmarketinoperationnotmodeledandresultsassumedtobeequalto20112012
2014 Nodalmarketinoperationnotmodeledandresultsassumedtobeequalto20112012
2015 Nodalmarketinoperationnotmodeledandresultsassumedtobeequalto20112012
2016 Nodalmarketinoperationnotmodeledandresultsassumedtobeequalto20112012
2017 Nodalmarketinoperationnotmodeledandresultsassumedtobeequalto20112012
2018 Nodalmarketinoperationnotmodeledandresultsassumedtobeequalto20112012
2019 Nodalmarketinoperationnotmodeledandresultsassumedtobeequalto20112012
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
11/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
11|P a g e
2020 Nodalmarketinoperationnotmodeledandresultsassumedtobeequalto20112012
Intheabsenceofabackcast,thereisnoevidencethattheDecemberstudyisabletoreproduceprices
andoperationsinERCOTwithanyaccuracywhatsoever.
AssumptionsTheauthorsstateonpage68:
A. Marginal Cost Bidding: All generation units are assumed to bid marginal cost(opportunity cost of fuel plus nonfuel VOM plus opportunity cost of tradable
permits).
To
the
extent
that
real
markets
are
not
perfectly
competitive,
the
model
tendstounderestimateprices.7
This decision to make this assumption is not surprising. The original November 2004 report
madeasimilarassumption:
The assumption of shortrun marginal cost bidding can be overridden, implementing
strategicbiddingbehavior,buttheeffortrequiredtodothisisconsiderable;priortothe
contractingprocess,theCBCGchosenottopursuethisapproach.Notethatthroughout
this report the use of the term marginal cost means refers to shortrun marginal
costs.8
Unfortunately,this isapoorassumptionforamodeldesignedtosimulatemarketprices. Ina
condition of perfect competition, bids will approximate marginal cost. The critical condition
precedentisperfectcompetition. ERCOTmeetsfewoftheconditionsofperfectcompetition,
andstrategicbiddingisacontinuingprobleminERCOTadministeredmarkets.
Recently,LuminantsettledacomplaintbyTexasregulatorsforstrategicbidding.9 Attheheart
of the investigation was the appropriateness of TXUs exercise of market power. In practice,
TXUtendstobidthevastmajorityofitsresourcesat$300/MWHregardlessofthemarginalcost
oftheunits.10
7FinalReport,December18,2008,page68.
8FinalReport,November24,2004,page33.
9NoticeofViolation,Docket34061,March28,2007.
10Ibid.,page26. Thetermsbidcurveandmarginalcostwereadded.
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
12/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
TXUsstrategicbiddingwasobviously importantwitnessthecomplaintfiledbyPUCstaff. Moreover,
the deviation between the marginal cost of coal units and $300/MWh bids dwarfs any possible
efficiencyadvantageslikelytoresultfrommoreprecisepricinginthenodalmodel.
Modelingofstrategicbidding isachallenge. Oligopolisticpricingdecisionsarecomplexandvarywith
circumstances. As noted above, when the largest single market participant makes strategic bids for
thousands of megawatts, it is imprudent to ignore this information when modeling. In the example
takenfromthePUCcomplaint,thesignificantdifferenceinpricesisadifferenceof$200/MWh. Thebias
thisassumptionwillcreateinthemodelingdwarfstherelativelysmallproductioncostdifferencesinthe
December
2008
report.
The most important operational assumption after bidding behavior is the cost of natural gas. The
authorsdonotincludedetailedinformationonnaturalgasandcoalpricesusedintheirmodel.Thereis,
however,ashortdiscussionofthesourcesofthenaturalgasforecastsonpages68and69. Preparedin
a year of unprecedented fuel price volatility, the authors make the decision to base forecasts on the
2008 EIAs natural gas forecast published a year ago. The Energy Information Administration makes
forecastsonbothanannualandamonthlybasis. EIAsannualforecastsarereleasedinmidDecember,
12|P a g e
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
13/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
thus making the 2008 forecast a full year out of date. While using forecasts from a year ago would
normally be defensible, 2008 was hardly a normal year. Within 2008 prices for natural gas used for
electricgeneration
increased
over
50%
by
July
2008
and
in
the
latter
portion
of
the
year,
they
fell
even
moreprecipitously.
$
$2.00
$4.00
$6.00
$8.00
$10.00
$12.00
$14.00
$/MWh
EIAShortTermandAnnualForecastsforNaturalGas
2008AEO December2008STEO
Forecasting fossil fuel prices has become increasingly difficult over the past few years and the use of
superannuatedforecastsisanextremelybadstepinanymodelingexercise. Notwithstanding,thedated
naturalgasforecast,theuseofasinglerelativelyoptimisticpointforecast isnolongerappropriate
regardless
of
its
source
or
age.
The
standard
approach
to
this
problem
is
to
prepare
sensitivities
for
high, medium, and low fossil fuel prices so that decisionmakers can see the impactof unpredictable
fossilfuelpricesonthemodelresults.
Additionalassumptionsandcalculationsarepoorlydocumentedormissing,including:
1. NPVcalculationsforsystemcosts2. NPVcalculationsforcustomersavings
13|P a g e
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
14/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
3. UPLANresults4. GEMAPSresults5. SOxandNOxrequirementsandforecastedcosts6. Plantassumptions.
Theauthorsalsoneveraddresstherealdiscountrate,avaluethatoftendrivesresultsinmodelswhere
costsandbenefitsoccuratverydifferenttimesinthefuture. ThisistrueoftheDecemberstudywhere
computer programming costs largely occur before the onset of the nodal market. The estimated
benefits, of course, result only after nodal market implementation. It should be noted also that the
authorsadopt thediscount rate from theNovember2004study inspiteof theturmoil in theworlds
financialmarketsandtheglobalrecession.
ComputerProgrammingCostsPerhaps most mysterious is the expected cost to complete the computer programming for the nodal
market. Althoughnotexplicitlystated,thetotalexpectedbill isontheorderof$750million,ofwhich
$322.1 has already been spent.11 Table 13 identifies a present value of future expenditures of $430
million.12 Whilebeyondthescopeofthisreview,thepriceforthisrelativelystraightforwardcomputer
program is quite high by industry standards, especially for a single state with limited interties to the
surroundingmarkets. Table15showsthefullcostofthecomputerprogrammingexercise:
14|P a g e
11FinalReport,December18,2008,page11.
12Ibid.,page40.
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
15/16
8/14/2019 Review of the ERCOT December 18, 2008 Nodal Cost Benefit Study
16/16
ReviewoftheDecember18,2008NodalCostBenefitStudy
January7,2009
RobertMcCullough
16|P a g e
programasarelativebargainat$14.1millionperyearoverthe firstdecadeof itsoperation,which is
approximately onethird of the maintenance costs for the existing program. Assuming that the zonal
programsmaintenance
costs
have
been
overstated
by
afactor
of
three,
this
would
increase
the
net
cost
to continue the nodal program by $106 million, yielding a net going ahead cost of $326 million, as
opposed to $222 million. Alternatively, if maintenance costs of the nodal program are assumed to
matchthecostsofthezonalprogram,thetotalgoaheadcostswould increaseto$452milliondollars,
insteadofthereported$222million.
Ineithercase,theconclusioninTable15appearsvastlyunderstated.
ConclusionsThe December 18, 2008 study has a number of problems. These appear significant enough that the
conclusions of thestudy areeffectivelyspeculative. Inaddition, thestudy is poorlydocumentedand
dependsonanumberofunusualassumptions.
1. Thestudyhasonlymodeledtwoyearsofthetenyearperiodunderanalysis. Thisfailingaffectsboth the estimate of production costs and consumer benefits. The current modeling ends
beforetheyearwhenthe2004modelappearedtohavesignificantproblems.
2. Thestudyhasnotverifiedtheirmethodologybydoingabackcast. Intheabsenceofabackcast,theresultscannotbedependedupontoevenremotelyreproduceconditionsinERCOTmarkets.
3. Several assumptions are highly questionable. The study assumes marginal cost bidding bymarket participants even though this is not present in ERCOT. In addition, the study uses anatural gas forecast from a year ago a curious choice in such volatile times. Other
assumptions are undocumented or simply copied from a study four years out of date the
discountrate,forexample.
4. The costs of the additional computer programming appear high very high and theassumptions appear inconsistent. The high maintenance costs assumed for the existing
operationalzonalmodelappeartobejustafractionoftheassumedmaintenancecostsofthe
new,unfinishedandvastlymorecomplex,nodalmodel.