Critical appraisal in the health professions:

a session for UBC-Maastricht MPHE programme, 2012

Dean Giustini, UBC Biomed Librarian | MHPE Teleconference, 13 July 2012

Learning goals

• To provide overview of critical appraisal & key principles

• Map out the literature in medical education

• Critically appraise relevance of published literature in medical education

• To define BEME: Best evidence in medical education

Revisit evidence-based practice

Educational research starts with scenario

• Formulate a focused question• Search for best evidence• Appraise evidence• Apply evidence in practice• Critically evaluate whole

“…research, including qualitative studies, form the basis of evidence-based practice for health professionals. …however, many practicing health educators

do not feel fully confident in their ability to critically appraise qualitative research… Jeanfreau, 2011

Mapping medical education literature

A widely-disseminated literature

Masses of information

Rapid expansion & change of literature

Essential to engage with scholarly contributions of others in teaching & learning

Reflection on one’s own teaching & best practices in general

BEME: basing your decisions about teaching practices on the available

evidence …more later!

Other reflections on educational research

Quantitative methods• Gold standard is RCT, systematic review• Numbers used to describe & analyze• Precise answers are found for defined questions

Qualitative methods• No real gold standard • Words are used to describe & analyze • Seek to understand how people make sense of & interpret • Focus on perspectives & attitudes of individual• Enquiry is based in a culture or natural setting of individual(s)

Dr. Kevin Eva says “…we need to look for fatal flaws in research methodologies that might invalidate the conclusions being drawn in any given study and, yes we want to

ensure that research in the field maintains a focus on helping educators solve the practical dilemmas they face on a day-to-day basis. ….still, the extent to which we

define our efforts purely based on any particular methodological criteria is the extent to which we miss a great opportunity to truly stake claims to progression in our

thinking and our educational innovations…”

Eva KW. Broadening the debate about quality in medical education research. Med Educ. 2009 Apr;43(4):294-6

Types of qualitative methodologies used

• No ‘gold standard’ or hierarchy• researchers match method to research question (or should)• no evidence-based pyramid• triangulation of methods/’mixed’ is popular

• Popular methodologies• Ethnography• Grounded theory• Phenomenology• other?

Daly J, et al. A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol. 2007 Jan;60(1):43-9.

Qualitative Quantitative • A focus on text, depth

• Answers ‘how / why’

• Flexible, open-ended, seeks to explore phenomena

• Richness & context  • Develops emerging theories or

models (grounded theory) • Smaller # of subjects

• Larger amount of information collected from each subject

• A focus on numbers, breadth

• Answers ‘what / when / how much’

• Structured, seeks to confirm hypotheses about phenomena

• Precision & focus • Tests existing theories or models   

• Large # of subjects

• Smaller amount of information collected from each subject

More differences between Qualitative/Quantitative Research

Cost? ….which methodologies are likely to cost more?

Making sense of qualitative research / 10 key questions

1. Was there a clear statement of the aims of the research?2. Is a qualitative methodology appropriate? 3. Was the research design appropriate to address the aims of the research? 4. Was the recruitment strategy appropriate to the aims of the research? 5. Were the data collected in a way that addressed the research issue? 6. Has the relationship between researcher and participants been adequately considered? 7. Have ethical issues been taken into consideration? 8. Was the data analysis sufficiently rigorous? 9. Is there a clear statement of findings?10. How valuable is the research?

Many of these questions recur across checklists:• Cochrane Qualitative Research Methods Group. Chapter 6 – Critical appraisal of qualitative research.

http://cqrmg.cochrane.org/supplemental-handbook-guidance• Critical Appraisal Skills Programme (CASP): making sense of evidence website: http://www.casp-uk.net/

Why does evidence fail to get into practice?

…because 75% of researchers cannot understand basic statistics….and, 70% cannot critically appraise & apply research

~ Dunn et al. Using research for practice: a UK experience of the BARRIERS scale

Critical appraisal is iterative, heuristic & perhaps even social

Qualitative methods in the literature

• Used when ‘context of teaching’ is needed• To identify & describe contexts• To draw out experiences, feelings, attitudes, beliefs, perceptions

• Diagnostic • To explain something ‘why’• To examine reasons for certain behaviours

• Evaluative • To evaluate the effectiveness of something• To measure satisfaction, needs, barriers, improvements

Data collection, qualitative research

• In-depth & unstructured interviews • Some see as the ‘gold standard’ • open-ended, respondent-led• very detailed, loose structure,• emergent issues (‘inductive’)• recorded and transcribed for analysis

• Focus groups • Around 6-10 people; discuss common issues• Facilitated; non-directive questioning, observation• Range of opinions sought • Remember group dynamics, depth & validity

Other methods used in educational research

• Case studies• Descriptive survey• Ethnography • Phenomenological research• Narrative review• Historical research• Randomized controlled study• Systematic review• Other types?

Dr. Michael Wesch, Educator, digital ethnographer

Recruitment & sampling

• Purposive sampling• Purposeful, strategic, active, systematic• Chosen for potential to obtain information to inform research

• Theoretical sampling• Is guided by emerging theories

• Sampling strategies • Revised & modified during research• Not a statistical sample• Does not need to represent group• Saturation point

Best Evidence in Medical Education

BEME is defined as: “…the implementation, by teachers in their practice, of methods and approaches to education based on the best evidence available…”

Harden RM, Grant J, et al BEME guide no. 1: best evidence medical education. Med Teach. 1999;21:553–562.

The six steps of BEME

1. Frame the question2. Develop a search strategy3. Produce the raw data4. Evaluate the evidence5. Implement change6. Evaluate the change

BEME encourages teachers to:

• Comprehensively & critically appraise medical literature in education

• Categorize the power of that evidence

• Identify the gaps & flaws in the existing literature

• Suggest appropriately planned studies to optimize the evidence

BEME aims to “…assist education practitioners to become increasingly skilled at using evidence to

optimise student learning, shape their teaching, and in designing curricula and education policies…”

Comprehensively, critically appraise literature• systematic• transparent

Categorizing power of evidence• realist• epistemological openness

Identify gaps & flaws in existing literature• published• grey literature• hand searching

Use appropriately-planned studies • optimize the evidence • education intervention more evidence-based

BEME appraisal checklist

http://www2.warwick.ac.uk/fac/med/beme/

BEME Coding Sheet

Summary

• Qualitative/quantitative/’mixed’ methods are common• Experiences, attitudes, beliefs, perceptions, behaviours

interrogated…• No gold standard methodology• Ethnography, grounded theory, phenomenology most common• Purposive sampling used; not a statistically valid one• Inductive & iterative process – also true of critical appraisal• Explicit, systematic, justified & reproducible – all apply

• BEME: Best evidence in medical education website• Bligh J, Anderson M. Medical teachers and evidence. Med Ed. 2000;34:162-163.• Cochrane Qualitative Research Methods Group. Chapter 6 – Critical appraisal of qualitative research.

http://cqrmg.cochrane.org/supplemental-handbook-guidance• Cook TD. Qualitative and quantitative methods in evaluation research. Beverly Hills, CA: Sage.• Côté L. Appraising qualitative research articles in medical education. Med Teach. 2005 Jan;27(1):71-5.• Critical Appraisal Skills Programme (CASP): making sense of evidence website: http://www.casp-uk.net/• Daly J. A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol. 2007

Jan;60(1):43-9. • Denzin N, Lincoln LS. Handbook of qualitative research. Sage Publications, 1994.• Eva KW. Broadening the debate about quality in medical education research. Med Educ. 2009 Apr;43(4):

294-6.• Eva KW. On the limits of systematicity. Med Educ. 2008 Sep;42(9):852-3.• Gosall N, Gosall G. The doctors guide to critical appraisal. PasTest Ltd, 2009.• Greenhalgh T. How to read a paper. BMJ books. London: Blackwell, 1995.• Guidelines for evaluating papers on educational interventions. BMJ. 1999 May 8;318(7193):1265-7.• Gunderman RB. Achieving excellence in medical education. Springer, 2008.• Haig A, Dozier M. Systematic searching for evidence in medical education. BEME Guide No. 3. Med Teach

2003;25(4):352–363 and 25(5):463–484.• Harden RM, Grant J. BEME guide no. 1: best evidence medical education. Med Teach. 1999;21:553–562.• Harden RM, Crosby JR. An introduction to outcome based education. ‐ Med Teach. 1999;22(1)7–14.• International Centre for Allied Health Evidence Critical Appraisal Tools

http://www.unisa.edu.au/cahe/Resources/CAT/default.asp• Johnson B, Johnson L. Educational research: quantitative, qualitative and mixed approaches. Sage, 2007.

References