Benchmark of forecasting models

Reviewing and improving the state of the art

Daniel Cabezón

Head of Meteorological Models and Special Tasks (Energy Assessment Department)

Santiago Rubín

Energy Forecasting Manager (Energy Assessment Department)

Ignacio Lainez

Director of Energy Assessment

1. Introduction

2. Benchmark of forecasting models

3. Data feed to modelers

4. Error metrics

5. Open benchmark processes

OVERVIEW

2/14

1. Introduction

Objective: To have access on time to the best power forecast available

(expected mean and uncertainty)

1. Day Ahead Market (DAM): Hourly forecast for day D+1 to be delivered before [8-10] am (localtime) at day D

• Symmetric market (deterministic) / Asymmetric market (probabilistic)

• Individual / Aggregated portfolio

2. Intra-Day Market (IDM): refresh of new updated forecasts several times inside a day

Forecast of potential power = wind farm power without energy losses (100% availability and no curtailment)

3/14

4/14

• Best method to scan and identify on real time state-of-the-art forecasting models

• Initial participation necessarily as free trial

• Every participant can get in or quit as desired

• Monthly refresh of wind farm potential power to all modelers

• Milestone every quarter

• EDPR feedback to each individual participant (keeping confidentiality)

• After each Quarter (Jan-Apr-July-Oct)

• Possibility of 1 year contract in case of excellent ratio accuracy-pricing

Static datasupply

Model calibration(M1-M3)

ForecastStart

Real time hourly data feed + Error Monitoring

Q1 Q2 Q3 Q4

QUIT

New forecaster

Previous forecaster

2. Benchmark of forecasting models

• Creation of user in EDPR FTP server: retrieve data + send forecast

• Static Data

• Initial supply of data base for calibration (layout, WT model, historical potential power, etc.)

• Monthly update of wind farm potential power

• Dynamic data

• Real time supply from EDPR Data server to EDPR FTP server

• Submission 1: Active Power + Availability + Curtailment signal

• Submission 2: Preliminary Potential Power

Data server

(SCADA)

Static (Monthly basis)

Dynamic (real time)

FTP server

5/14

3. Data feed to modelers

• Off-line curve characterization: wind farm potential power VS nacelle wind speed

• Real time estimation and monitoring of potential power at all wind farms

• On-line retrieval of:

Nacelle wind speed Potential Power Substation Power

• Applications:

1. Track forecast error in real time

2. Analyze and detect in advance non-planned energy losses (icing, etc.)

0

200

400

600

800

1000

1200

1400

1600

1800

0 5 10 15 20 25 30

Wind speed at nacelle1 [m/s]

6/14

3. Data feed to modelers. Real time potential power

0

5000

10000

15000

20000

25000

30000

1 8

15

22

29

36

43

50

57

64

71

78

85

92

99

10

6

11

3

12

0

12

7

13

4

14

1

14

8

15

5

16

2

16

9

17

6

18

3

19

0

19

7

20

4

21

1

21

8

22

5

23

2

23

9

24

6

25

3

26

0

26

7

27

4

28

1

28

8

29

5

30

2

30

9

31

6

32

3

33

0

33

7

34

4

35

1

35

8

36

5

37

2

37

9

38

6

39

3

40

0

40

7

41

4

42

1

42

8

43

5

44

2

44

9

45

6

46

3

47

0

47

7

Po

ten

tial

Po

we

r [k

Wh

]

Time [h]

Consolidated

Real Time

Real time vs Consolidated potential power

*Failure rate = 3%

7/14

3. Data feed to modelers. Real time potential power

# KPI Parameter (%) Monitored (Weeklyand Monthly)

Description

1 NMAENormalized Mean Absolute ErrorAvg [Absolute Deviated MWh] / Nominal Power [MW]

2 WMAEWeighted Mean Absolute ErrorSum(Absolute Deviated MWh) / Sum(Production [MWh])

3 ImbalancesExcess and Deficit of P50 forecast (normalized to overall production) Excess -> Sum(Deviated MWh > 0) / Sum(Production)Deficit -> Sum(Deviated MWh < 0) / Sum(Production)

4 Percentiles AccuracySignificance of percentiles = % measurements below each percentile% Frequency when Production < P10% Frequency when Production < P90

5 Uncertainty BandDifference between P90 and P10 Avg (P90-P10) / Nominal Power [MW]

6 Time Series Time evolution of P10-P50-P90 forecast against measured potential

power (red line) during the previous week / month

# ForecastDeliverable(Hourly Base)

Description

1 Power P50Expected production(@ 100 % availabilityassumption)

2 Power P10, P90 Uncertainty Range

8/14

4. Error metrics

Targets

Balance

Make accurate

Minimize

Minimize

9/14

4. Error metrics

Month

# KPI Parameter (%) Monitored

Description

1 NMAENormalized Mean Absolute Error

Avg(Absolute Deviated MWh) / Nominal Power [MW]

NMAE (%)

10/14

4. Error metrics

Month

WMAE (%)

KPI Parameter (%) Monitored

Description

2 WMAEWeighted Mean Absolute Error

Sum(Absolute Deviated MWh) / Sum(Production [MWh])

11/14

4. Error metrics

Month

# KPI Parameter (%) Monitored

Description

3 ImbalancesExcess and Deficit of P50 forecast (normalized to overall production)

Excess = Sum(Deviated MWh > 0) / Sum(Production)Deficit = Sum(Deviated MWh < 0) / Sum(Production)

12/14

4. Error metrics

Month

# KPI Parameter (%) Monitored

Description

4 Percentiles AccuracySignificance of percentiles = % measurements below each percentile

% Frequency when Production < P10% Frequency when Production < P90

13/14

4. Error metrics

Month

# KPI Parameter(%) Monitored

Description

5Uncertainty

BandDifference between P90 and P10

Avg (P90-P10) / Nominal Power [MW]

US

Brazil

Canada

Poland

Romania

Italy

Portugal

France

Spain

BelgiumUK

14/14

5. Open benchmark processes

5 Models

12 Models (Wind)

8 Models (PV)

8 Models

45%

26%13%