To what extent can we rely on ecological monitoring and ...cost-fp0903.ipp.cnr.it/Downloads/Rome_Conference/Ferretti.pdf · Marco Ferretti TerraData environmetrics ... 88 M € €10.000

To what extent can we rely on ecological monitoring

and research data?

Research, monitoring and modellingin the study of climate change and air pollution impacts on forest ecosystems

5-7 October 2010, Rome, Italy

Marco Ferretti TerraData environmetrics

[email protected]

www.terradata.itUniversità di Siena

Partly based on an the activity carried out within the Life+ project

FutMon, Activity C1-QAC-15(IT)


Talk outline


A widespread concern“What we measure affects what we do; and if our measurements are flawed, decision may be distorted”

“We are almost blind when the metrics on which action is based are ill-designed or when they are not well understood.”

J E Stiglitz, Columbia University, Nobel 2001;Amartya Sen, Harward University, Nobel 1998Paul Fitoussi, Institut d'études politiques de Paris

• Data reliability: who cares (besides Nobel prizes)?

• What, and at what extent, does affect the reliability

of M&R data in Europe (and elsewhere)?

• What can we do?


Talk outline


Some questions




• What can we do?


Talk outline


Some questions


How good we are?


Relying on data: who cares?

“We believe that many current monitoring programmes suffer from deficiencies associated with inadequate attention during programme design to the why, what and how of monitoring.”(Yoccoz NG et al., 2001, TRENDS in Ecology & Evolution, 16, 8: 446-453)

“Many of the main conservation organisations are doing or commissioning monitoring work – but will the data that are being collected ever be of much use?”(Legg CJ and Nagy L, 2006. Journal of Environmental Management, 78: 194-199).


How good we are?


Relying on data: who cares? guardian.co.uk, 1 February 2010

“Strange case of moving weather posts and a scientist under siege

It is difficult to imagine a more bizarre academic dispute. Where exactly are 42 weather monitoring stations in remote parts of rural China?….But the argument over the weather stations, …. may yet result in a significant revision of a scientific paper (1) that is still cited by the UN's top climate science body.”

(1) Jones et al., Nature 347, 169 - 172

Nature 464, 141 (11 March 2010)

Climate of fear

The integrity of climate research has taken a very public battering in recent months.…Scientists must now emphasize the science, while acknowledging that they are in a street fight.…Scientists must not be so naive as to assume that the data speak for themselves


How good we are?


Relying on data: who cares?

Cumulative, distributed investments (>100yrs):

• Infrastructure: 480 M€• „Data and information

value“: 1200 M€• Annual operational

costs: 88 M€

€10.000

€100.000

€1.000.000

€10.000.000

1 11 21 31 41 51 61 71 81 91

Infrastructure value of LTER sites

Mirtl, M. et al., 2009. LTER-Europe Design and Implementation Report Federal Environment Agency Austria. Vienna. 220 pages.

ISBN 978-3-99004-031-7.




• What can we do?


Talk outline


Some questions

ΔHow important is the problem, and how it develops?

SEtXSEtX )()( ),2(

^

),2( υαυα μ +<<−

(UN

/EC

E d

ocum

ent 1

986)


Understanding data requirements

Is there a problem?


Talk outline


Understanding error sources

• (Ambiguous objectives)

• Sampling errors

• Measurement errors

• Non-statistical errors

• Errors in models

(Parr TW et al., 2002Environmental Monitoring and Assessment, 78: 253–290


Talk outline


Forest monitoring in Europe• Large-scale (Level I, ca. 6000 plots)• Intensive (Level II, ca. 800 plots)• Many investigations:

– Tree condition– Soil– Foliar chemistry– Tree growth– Vegetation– Phenology– Ozone injury– Soil solution– Deposition– Air quality– Meteorology– Litterfall

(after UN/ECE, 2008)(approx. 263 variables measured)


Level I - Tree condition


Tree condition

• Target population– Frame attributes inconsistent

between Countries– Population changes over time

• Network design (Level I)– “Assumed” on a random basis

• Plot design (Level I and II)– Fixed area vs. fixed number of

trees – Various solutions adopted

• Consequence: design-based inference is problematic (if ever possible).

N

25 m

E

S

W 1 23

4

5

6

N

25 m

E

S

W 1 23

4

5

6

N

25 m

E

S

W 1 23

4

5

6

N

25 m

E

S

W 1 23

4

5

6

N

E

S

W



Exercise Species Crews Pairs

n n n %Czech Republic Picea abies 14 91 57 62.6

Fagus sylvatica 14 91 54 59.3Finland Picea abies 11 55 17 30.9

Pinus sylvestris 11 55 23 41.8Silver birch 11 55 31 56.4

France Pinus pinaster 5 10 4 40.0Quercus ilex 5 10 7 70.0

Significant differences

(Müller and Stierlin, 1990)

Tree defoliation (based on Mues, 2005)

Level I - Tree condition


Level II – Species richness

01020304050607080

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16Settore

N° s

peci

e

G 81906S 81906F 81906Q.A. 81906

(Bacaro et al., 2009)

QA team



Assessment method

No of countries

Plots per country, range (n)

Crews per country, range (n)

Subplots per plot, range (n)

Size of subplot,

range (m2)

Coverage 13 1-53 1-17 1-200 0.25-2500Braun-Blanquet 9 7-15 1-17 1-24 75-400Contact point 3 3-8 1-3 8-24 0.5-1

Level II – Species richness

(Ferretti et al., 2009)


Level II – Deposition chemistry


Level II – Deposition chemistryOpen field(1) Throughfall(2)

N of systems compared 20 20Sample type, n

Wet-only 1 0Funnel 18 16Gutter 0 4

Samplers used by the system, nWet-only 1 -

Funnel 1-10 8-16Gutter - 3-10

Height of collector above ground, cmWet-only 150 -

Funnel 40-300 28-230Gutter - 98-112

Total collecting area measured, cm2

Wet-only 379 -Funnel 1256 1112-5972Gutter - 1584-25771

Sampling scheme (3)straight line, fixed distances - 3

straight line cross fixed distances - 4 (4)straight line cross random distances - 1

using sampling grid - 6at random - 5

gutters in more or less a quarter circle - 2 (4)(1) After Erisman et al., 2003(2) After Bleeker et al., 2003(3) After Drajiers et al., 2001(4) One country reported more than one scheme


Comparability


Talk outline


Non statistical errors

(Durrant Houston, 2008)

(Nimis, 2001)

• Does everybody understands the same thing when they define or measure it?

– Definitions

– “Taxonomic inflation”


Talk outline


Non-statistical errors

ID var1 var2 var3 var4 1A 42 -1 1B 43 3 5.4 2A 34 3 5.7 2B 35 2 5.4 3A 32 4 9.9 (above l imit

for instrument)

3B 34 2-3 7.3 4A/B 32 3 6.9 5A 45 >4 3.1 5B 43 2 4.0

var54

4.2

7.35.13.8

6.44.92.2n/a5.5

Missing variable – important or irrelevant?

Text in a numeric field – cannot be used in calculations without encoding

Extra information will be lost if not allowed for in the database design

Extra variable – is this important?(If yes, it should have been foreseen)

-1: real result or missing value?



Talk outline


Consequences - evaluation and interpretation

Errors at plot level (P<0.05)

0 20 40 60 80 100

Crown density

Soil attributes

Foliar attributes

DBH

LAI

Species, n

Species, cover

Throughfall, mm

Error (% of mean), Min-Max

M u lt ip le r e g re s s io n m o d e ls

lo g y = α 0 + α 1 x 1 + α 2 x 2 … ..+ α n x n

lo g y = e x p e c ta t io n v a lu e re s p o n s e v a r ia b lex 1 … ..x n = p re d ic to r v a r ia b le sα 0 … .α n = r e g r e s s io n c o e f f i c ie n ts

“…inappropriate sampling may bias the outcome of multivariate and other types of data analysis, weakening the conclusions of the program.”(Ferretti & Chiarucci, 2003, Sci. Tot. Env., 310: 171–178)

?


(De Vries and Dobbertin, 2009, after Sutton et. al 2008)

Biased results?

FactorsVariance explained

(%)Country 33-39Age 2-14Environmental factors 1.3-2.4

(De Vries and Dobbertin, 2009; after Klap et al., 1998, 2000)

Tree health C sequestration




• What can we do?


Talk outline


Some questions

• Built upon past ICP Forests QA/QC activity

• Set-up of two dedicated groups• Complete revision of Standard Operating

Procedures, which leaded to:1. Re-address sampling issues at

network and plot level2. Adopt a general QA perspective3. Formal definition of Data Quality

Requirements4. A new series of data quality exercises


Talk outline


Steps undertaken within ICP Forests and Life+ FutMon



1. Sampling issues• Level I and Level II• Moving to a p-based

perspective• Preservation of time series



2. QA perspective• Harmonization of SOPs

structure and contents• Data Quality

Requirements• Training and inter-

comparison exercises• Counter-actions


Data Quality requirements

• Measurement Quality Objectives (MQOs): the desired level of precision for a given measurement.

• Data Quality Limits (DQLs): the desired minumum level of achievement of MQOs.

3. Formal definition of DQRs


Talk outline


DQRsBefore FutMon and the

revision of the ICP Forests Manual

After FutMon and the revision of the ICP

Forests Manual

0

20

40

60

80

100

Varia

bles

, %

Without DQR

With DQR

0

20

40

60

80

100

Varia

bles

, %

Without DQR

With DQR

32.9% of variables covered by DQR

66.4% of variables covered by DQR


4. Calibration exercises (2009-2010)

0

10

20

30

40

50

Varia

bles

test

ed, n

0

20

40

60

80

100

Ach

ieve

men

t of M

QO

s, %

A. Variables tested B. DQL achievement - DQL

• Data reliability: who cares?

• What, and to what extent, does affect the reliability


• What can we do?


Talk outline


Some questions

“…the rationale underlying the development of many programmes seems to be the simple idea that additional information about any system will be inherently useful”.(Yoccoz NG et al., 2001, TRENDS in Ecology & Evolution, 16, 8: 446-453)


Stop“collect (data) now-think-later”(Lindenmayer and Likens, 2009, TRENDS in Ecology and Evolution, 24, 9: 482-486)

“If enough data is collected, anything may be proven by statistical methods”.Bloch A, 2002. Murphy’s law complete, Arrow books: 282 p.

Sample designMeasurement

Equipment

-5

0

5

10

15

20

25

30

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51

0

2

4

6

8

10

12

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51

Analysis

Metadata

Metadata

Metadata

Metadata

Metadata

Metadata

Metadata

Metadata

Storage


Quality Assurance at every stage


Università di Siena

www.terradata.it

“It is EPA policy that all work funded by EPA in which environmental data will be collected, evaluated, used, or reported …… have approved QA Project Plans”(EPA QA/G-5, 2002, p. 2) (www.epa.gov/quality)

Promote the use of QA

• “it is “ugly” that initial enthusiasm for monitoring programmes wanes and programmes are abandoned;

• and it is “bad” when monitoring programmes change protocols in midstream, leaving collections of incompatible data in their wake”.


But: do not waste data series!

Stout BB, 1993, Environmental Monitoring and Assessment 26:91-98

Is it a real risk for Europe?

Università di Siena

www.terradata.it

2000 2007

UN/ECE

2009


Conclusions

• Hundreds of variables are measured on thousand of sites in Europe at the costs of millions of euros

• The risk of biased conclusions because ill-designed set-up and poor, often unknown, data quality cannot be denied

• Recent activity within ICP Forests and FutMon promoted QA as unifying framework for monitoring design and implementation, to document data quality, and to make results defensible.

• However, substantial progress will occur only if formal QA procedures will be required by funding agencies before a grant is assigned...

• ...and if long-term monitoring will be maintained.


Acknowledgments

• Colleagues of ICP Forests and C1 actions of the Life+ FutMon project, in particular:

– E. Beukert, V. Calatayud, R. Canullo , N. Clarke, N. Cools, K. Derome, B. de Vos, A. Fuerst, N. Koenig, A. Kowalska, A. Marchetto, P. Rautio, S. Raspe, M. Schaub, D. Zlindra.

• Colleagues at the vTI, Hamburg for the support in the Manual revisions

• Colleagues at the MIPAAF, CFS, Rome Italy.

Documents

To what extent can we rely on ecological monitoring and ...cost-fp0903.ipp.cnr.it/Downloads/Rome_Conference/Ferretti.pdf · Marco Ferretti TerraData environmetrics ... 88 M € €10.000