View
230
Download
0
Embed Size (px)
Citation preview
8/14/2019 EEF Data Cred
1/17
Prof. Andrew S. Pullin
Centre for Evidence-Based Conservation
Bangor University
Data credibility
8/14/2019 EEF Data Cred
2/17
Focus on outcome evaluation
Emphasis on quantitative data
Emphasis on natural sciences Emphasis on effectiveness
Emphasis on strength of evidence
Emphasis on data quality Perspective of systematic reviewer
8/14/2019 EEF Data Cred
3/17
Common Issues
The confidence with which we caninterpret data in the context of ourquestions depends upon dataquality and the strength of theevidence that they provide.
Is the measured effect real and can
we attribute the effect to theinterventions we have put in place?
8/14/2019 EEF Data Cred
4/17
What does quality mean?
The extent to which the studydesign limits the influence of errorand bias.
Inversely proportional to thelikelihood of misinterpretation.
The extent to which data sets can
be combined in a meta-analysis
8/14/2019 EEF Data Cred
5/17
Methodological development:
Stages of a systematic review
Formulate a question (stakeholder engagement)
Generate a protocol (peer reviewed)
Systematic search
Study selection Data quality assessment (critical appraisal)
Data extraction
Synthesis of data (meta-analysis)
Report on evidence base and implications
Active dissemination and information sharing
Guidelines now published as Pullin & Stewart 2006. Conserv. Biol.
8/14/2019 EEF Data Cred
6/17
Appraising methodology?
There is no such thing as a perfect study,all studies have weaknesses, limitations,biases
Interpretation of the findings of a studydepends on design, conduct and analysis
A third of ecological papers arepseudoreplicated!
About 80% of research findings are false!
Ioannidis JPA (2005) Why most published researchfindings are false. PLoS Med 2(8): e124.
8/14/2019 EEF Data Cred
7/17
Susceptibility to Bias
Selection Bias
Performance Bias
Detection Bias Attrition Bias
8/14/2019 EEF Data Cred
8/17
Dealing with Effect Modifiers
Key problem for attribution
Poor quality studies will suffer from
confounding variables
Synthesis of good quality studies canexamine influence of effect modifiersunder different conditions.
Differences in methodological quality can
be explored as an explanation forheterogeneity in study results
8/14/2019 EEF Data Cred
9/17
Are bracken control methods effective?
Lesson variable data
availability may prevent
meaningful comparison of
effectiveness.
Stewart, G.B. Pullin, A.S. & Tyler, C.(2007) The effectiveness of asulam for bracken
(Pteridium aquilinum) control in the united kingdom: A meta-analysis. Environmental
Management40, 747-760
8/14/2019 EEF Data Cred
10/17
Variable outcome measures
Key problem for synthesis ofmultiple studies
Rarely consensus on what is themost valid measure
8/14/2019 EEF Data Cred
11/17
Do in-stream devices increase
salmonid populations?
8/14/2019 EEF Data Cred
12/17
Pseudoreplication
Big issues for site-based ecology
Provided problem is transparent it
can be dealt with
8/14/2019 EEF Data Cred
13/17
Do Marine Protected Areas work?
8/14/2019 EEF Data Cred
14/17
Internal v External validity
Does eliminating variables make thedata more or less credible?
Internally valid experiments shouldbe of higher quality but may be lessfit for purpose.
8/14/2019 EEF Data Cred
15/17
Are Rhododendron control methodseffective?
Tyler, C., Pullin, A.S. & Stewart, G.B. (2006) Effectiveness of management interventions to
control invasion byRhododendron ponticum. Environmental Management37, 513- 522.
8/14/2019 EEF Data Cred
16/17
Improving data credibility
Controlled randomised - replicated
Multiple stakeholder involvement in
design Transparency of method
Accessibility of data
l d
8/14/2019 EEF Data Cred
17/17
www.environmentalevidence.org