Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
EDUCATIONAL INTERVENTIONS TO PROMOTE CLINICAL
REASONING: A BEME SYSTEMATIC REVIEW
SYSTEMATIC REVIEW PROTOCOL
Nanos gigantum humeris insidentes1.
1. REVIEW GROUP MEMBERS AND AFFILIATIONS
Professor Judy Mckimm. Dean of Medical Education at College of Medicine, Swansea University
Dr. Ana L. Da Silva (lead reviewer). Lecturer in Medical Education Swansea University
Dr. Paul Garrud Assistant Director of Medical Education, Division of Medical Sciences & Graduate Entry
Medicine, School of Medicine, University of Nottingham
Dr. Daniel Mclaughlin, Director of Undergraduate Studies, Division of Medical Sciences & Graduate Entry
Medicine, School of Medicine, University of Nottingham
Dr. Jean McKendree, Senior Lecturer in Medical Education, Hull York Medical School, University of York
Dr. Janine Henderson, Programme Director MB BS, Hull York Medical School, University of York
Dr Anna Hammond, Director of Communication Skills Teaching, Hull York Medical School, University of York
Dr. Simon Gay, Clinical Lecturer in Medical Education, School of Medicine, Keele University
Dr. Maggie Bartlett, Clinical Lecturer in Medical Education, School of Medicine, Keele University
Dr. Laurence Atkinson, Assistant Director of General Practice Studies at Cambridge School of Clinical Medicine
Dr. Claire Vogan, Director of Student Support at College of Medicine, Swansea University
Dr. Steve Durning, Professor of Medicine and Pathology Director, Uniformed Services University of the Health
Sciences F. Edward Hebert School of Medicine (reviewer and lead of Clinical reasoning assessment BEME
review)
All members of the group will undertake an active role in the current review. Tasks performed by each
individual member (e.g. review of RoB, search, synthesis) will be allocated based upon the members’ interest
and expertise.
CONTACT DETAILS OF LEAD REVIEWER
College of Medicine, Grove Building, Swansea University, Singleton Park Swansea SA2 8PP
Tel. 01792 606547
Twitter:AnalindaS Email: [email protected]
1.1 WIDER ADVISORY GROUP
Experts in the field will be approached to take part in this review as an wider advisory group upon protocol
approval by BEME reviewers.
1.2 SOURCES OF FUNDING
Currently this review has no external sources of funding. Funding from available grants will be sought to
support particular activities such as training sessions, preparation of outputs, group meetings and
dissemination of findings to a wider audience (e.g. stakeholders, educators).
1 John of Salisbury, Metalogicon (1159 pp. 167). Meaning: One who discovers by building on previous discoveries.
1
2. BACKGROUND AND CONTEXT
‘Our job as educators is to continue to evolve our teaching methods in the hope that our students become
more efficient and more accurate problem solvers and make fewer cognitive errors’ (Kassirer, 2010, p.1124).
The importance of clinical reasoning for the competency of healthcare professionals (Epstein, 2007; Higgs et
al., 2008; Higgs, 1992) and the quality of care they provide to patients has long been established (Stiegler et
al., 2011; Graber et al., 2005; Norman & Eva, 2010). The fascination with revealing and understanding the
mental processes by which experts can successfully diagnose and treat patients, are embedded in four
decades of prolific research (Norman, 2005). However, only recently more attention has started to be devoted
to how educational interventions could contribute to promote this thought process. This is of great
importance, as is our role as educators to ensure students and trainees are given the learning opportunities
necessary to develop into successful and safe clinicians (Kassirer, 2010, p.1124). That can only be effectively
done if based upon the best available evidence.
The past research agenda across healthcare disciplines in has focused largely on understanding how expert
clinicians solve clinical cases, mainly in comparison with novices (Elstein, 2009; Rikers & Verkoeijen, 2007),.
During the last forty years many competing explanations and conflicting hypotheses have been suggested.
Many of these hypotheses still endure today however, some consensus has emerged. Research identified a
strong link between clinical knowledge and clinical reasoning (Patel et al., 1990; Woods et al., 2007; Boshuizen
et al., 1992) and the impact of experience in clinical environments (Schmidt et al., 1990; Norman & Eva, 2005;
Norman, 2006). It seems that experts use less knowledge, but possibly of different nature, in a more
expeditious way, based on the construction of elaborate mental representations (schemas or scripts) of the
association between signs, symptoms, diagnoses and prognosis derived from their clinical experience
(Norman, 2005; Elstein et al., 1978; Schmidt et al., 1990; Charlin et al., 2000; Norman & Eva, 2005; Charlin et
al., 2007; Eva et al., 2002). Such mental representations allow experts to rapidly and accurately recognise
similarities (patterns) between old and new cases, and act accordingly. This process was designated as ‘pattern
recognition’ or ‘non-analytic reasoning’ (NAR) (Norman et al., 2007). Novices, lacking such representations,
have to rely on a more analytic, slower and more conscious system (Gobet, 2007; Croskerry, 2009; Reiter et al.,
2002) which is possibly more error prone (Sherbino et al., 2012). This explanation, based on the Dual-Process
Theory (DPT) (Evans, 1984), is one of the most widely used in the field (Norman et al., 2009).
Several authors have shown that patient characteristics, the environment and individual characteristics all
have an impact on clinical reasoning (Durning et al., 2012; Eva et al., 2010; Higgs & Jones, 2000; Smith et al.,
1991; Ajjawi & Higgs, 2012). Some recent views suggest that, as a highly context-specific type of reasoning,
clinical reasoning maybe better understood in the guise of a non-positivist paradigm. Ajjawi & Higgs, for
example, adopted a hermeneutic phenomenological approach to research professionals’ journeys and
trainees’ perceptions of their development of clinical reasoning (Ajjawi & Higgs, 2012, 2007, 2008). More
recently, Durning & Artino suggested ‘Situativity Theory’ as a/the conceptual framework to understand clinical
reasoning (Durning et al., 2012; Durning & Artino, 2011). Situativity theory is defined as ‘theoretical
frameworks which argue that knowledge, thinking, and learning are situated (or located) in experience’
(Durning & Artino, 2011, p.188) encompassing elements from situated cognition (Brown et al., 1989), situated
learning (Lave & Wenger, 1991) and other theories, that share the view that knowledge cannot be separated
of its context and agents (Durning & Artino, 2011). It is still ‘early days’ in terms of fully undestanding how such
viewpoints will help to answer the many questions about clinical reasoning, particularly how its propositions
will endure further research and practice. But expectations are certainly high, especially as these authors are
also conducting a BEME review (Durning, et al. 2013 – BEME protocol), likely to render useful and information
to all of those interested in this area. Two decades have passed since the Custers et al(1996) review into
clinical reasoning and almost one since the review by Norman(2005). Both are landmark reviews in the field,
and during this time many changes occurred calling for the need to re-examine this research area.
2
More so, because in past decades educational interventions to promote clinical reasoning were frequently
neglected or limited to some vague suggestions but recently there appears to be a growing interest in this area
(for example: Ark et al., 2007, 2006; Bowen, 2006; Struyf et al., 2005: Irby & Bowen, 2004). This shift in the
research area, allied to an growing interest in this topic by educators, shown, for example, in the high number
of citations rendered by Eva (2005) paper entitled “what every clinical teacher needs to know about clinical
reasoning” highlights the need for the present BEME review.
Today, the literature includes a vast range of suggestions regarding educational interventions to promote
clinical reasoning. Some examples are: individualized feedback; small group discussions of cases;
demonstration sessions; structure clinical placements; virtual learning patients and online cases; simulation
sessions; reflective practice/reasoning; cognitive forcing and cognitive debasing strategies (Harris et al., 2011;
Dequeker & Jaspaert, 1998; McMillan, 2010; Windish et al., 2005; Durak et al., 2007; Struyf et al., 2005;
Kurzenhäuser & Hoffrage, 2002; Elizondo-Omaña & López, 2008; Delany et al., 2013; Kuiper et al., 2008;
Rencic, 2011; Croskerry et al., 2013). This recent proliferation makes the task of choosing between
interventions complex if no comprehensive summary exists. BEME reviews, as high quality systematic reviews
in medical education, address this problem by providing medical schools and educators with a synthesis of the
best available evidence, allowing for informed choices to be made (Patrício & Vaz Carneiro, 2012).
Additionally, there seems to be also a growing number of thorough research studies designed to formally
evaluate the effect of such suggestions (Cook et al., 2010; Cook & Triola, 2009; Stieger et al., 2009; Borleffs et
al., 2003; Radomski & Russell, 2010; Ark et al., 2007; Bowen, 2006; Graber, 2003; Croskerry, 2003; Croskerry et
al., 2013; Mamede et al., 2008; Sherbino et al., 2012; Trowbridge et al., 2013; Thompson & Stapley, 2011).
These studies are an encouraging sign and provide confidence that there is sufficient high quality research on
educational interventions to make possible the present systematic review.
Finally, although clinical reasoning captured the interest and research endeavors of researchers across many
different research fields (e.g. computer sciences, cognitive psychology, artificial intelligence, sociology) and
different health professions, this body of knowledge is fractured by disciplinary boundaries. Such disciplinary
and professional silos make research findings less accessible to healthcare institutions and teachers faced with
the difficult decision of choosing what educational activities to implement.
In summary, the interest in educational interventions used to promote of clinical reasoning both from
practice and research supports the need and timely nature of the current review, maximizing the potential for
both its usefulness for practice and possible impact on shaping the future research in this field.
KEY DEFINITIONS:
The present review shares the vision of BEME reviews as an “up-to-date summary of evidence that will
enable teachers to make practical decisions and assist policy makers in their choices about the funding of
education” while simultaneously contributing to the body of knowledge in healthcare education (Hammick,
2005, p.33).
The key definitions below will guide the present review. Those definitions were carefully chosen to ensure
the present review aligns with other reviews in the field, primarily the work being carried out by the BEME
group led by Professor Steve Durning (BEME protocol- Durning et al. 2013).
Clinical Reasoning:
Definitions of clinical reasoning are varied, often lacking necessary consensus, nevertheless some common
components can be identified: i) clinical reasoning is a type of thinking process2; ii)it is present at all stages of
the clinical cycle (from selection of information to treatment/prognosis); iii) it leads to actions and decisions
about patient care (context specificity); iv)it involves interplay of factors associated with the patient, situation
2 Thinking is defined here according to Holyoak & Morrison (2012) as “the systematic transformation of mental representations of
knowledge to characterize actual or possible states of the world, often in service of goals.” (Holyoak & Morrison, 2012, p.15)
3
and the individual; v) it is susceptible to heuristics and bias; vi) it is a fundamental part of healthcare
competency (Higgs et al., 2008; Durning et al., 2011; Eva et al., 2007). Such components seem to be common
across different healthcare professions (Jensen et al., 2008). Echoing these identified communalities and the
definition used the contemporary BEME review cited above, for the purposes of the present review clinical
reasoning will be defined as the thinking process by which healthcare professionals select, interpret/analyze
and combine information in order to make decisions and take actions about a patient in each clinical situation.
Additionally, as a thinking process, clinical reasoning will be susceptible to cognitive bias and heuristics.
Therefore any educational intervention, aimed at reducing the risk of such bias or error at any stage of the
clinical cycle (diagnosis, clinical investigations, treatment, management, review) can be understood as
promoting clinical reasoning (Graber et al., 2012; Norman, 2009).
Educational interventions:
For the purposes of the current review, educational interventions will be defined as planned activities with
clearly defined outcomes that are designed to lead to a positive change in a person’s behaviour by creating
structured opportunities for knowledge, skills and attitudes/behaviours to be developed (Gordon & Findley,
2011). These can take place face-to-face either in classroom or clinical environments (real or simulated),
online or using a combination of multiple environments. Equally, these can be aimed at large, small groups or
take place as individualised sessions (Dent & Harden, 2005).
Clinical reasoning is a desirable outcome of any healthcare degree, and therefore is a tacit outcome of many
of the learning opportunities provided within those programmes. For the purposes of the present review all
educational interventions considered must explicitly state clinical reasoning promotion as a desired outcome
(see inclusion/exclusion criteria section).
Additionally, studies using clinical reasoning as one of the outcome measures to investigate overall
differences between curricula types will not be included. Such studies were reviewed elsewhere (e.g. Newman
et al., 2003, Colliver, 2000) with no ubiquitous conclusion regarding the link between clinical reasoning and a
particular curriculum type (Newman et al., 2003). Furthermore it is now understood that variability exists
within the way different schools implement the same type of curriculum. Such variability is often unreported
in the research conducted, which compromises the usefulness of such studies for the aims of the current
review.
It is, however, expected that studies retrieved by our searches will report information about the type of
curriculum as part of the description of their educational context. That information will be included in the
coding sheets to inform discussion of findings. This will narrow the focus of the review without compromising
useful information increasing the feasibility and usefulness of the recommendations (Hammick, 2005).
4
3. REVIEW QUESTION(S), OBJECTIVES AND KEY WORDS
Review Question(S): The current review will address the following question: Which educational interventions contribute towards
promoting clinical reasoning?
Within this main questions, the following sub-questions will be considered:
Which educational interventions are suggested to promote clinical reasoning?
What are their characteristics (environment, participants, content, educational context(e.g.
curriculum type), research context resources required) ?
What are the conceptual frameworks they are based upon?
Do educational intervention vary across professions? How?
What other elements, external to the interventions, can impact the success of such activities?
Review Objectives: In line with the purposes of BEME reviews, the present review main objective are:
1. Provide educators and researchers with a synthesis of the available evidence regarding educational
intervention used to promote clinical reasoning, reporting on quality of the research/findings, type of
studies conducted and type of outcome evaluated (adapted from Barr et al., 2000)
2. Make recommendations for educators and institutions, based upon the best available evidence,
regarding which educational interventions contribute towards the development of clinical reasoning.
These recommendations will included an identification of key characteristics of such interventions
(e.g. type of intervention, profession, participants level, educational context, research methods,
conceptual frameworks underpinning clinical reasoning teaching, resources required for
implementation) in order to allow for informed choices to be made.
3. Identify research trends and gaps, and make recommendations for future research in the field.
Review Key Words: clinical reasoning; diagnostic reasoning; health professions education; inter-professional education, medical
education; educational interventions; cognitive interventions; cognitive error; clinical judgment; clinical
problem solving; clinical decision making.
5
4. SEARCH SOURCES AND STRATEGIES
Search Strategy: Database searches will be performed accordingly to BEME guidelines (Haig & Dozier, 2003). Databases will
be searched using the search terms identified below, using both keyword, MESH terms, and full article search
when available. Conference Proceedings Citation Index- Science (CPCI-S) and Conference Proceedings Citation
Index Social Science & Humanities (CPCI-SSH) and websites of international medical education conferences
(mainly AMEE, RIME Conference-AACM, ASME, APMEC) will also be searched. This search will be limited to a
two year period (2012-2014), as it is reasonable to assume that, older studies, if of enough quality, would
probably have been published elsewhere and therefore will be identified by the journal search. Backward and
forward citation searches may be performed to test the quality of the search strategy, and exceptionally, to
identify relevant references not retrieved in the initial search. Hand searches of particular relevant articles
identified during the article retrieval and extraction process may also be performed (Academic Medicine,
Nursing Education, Medical Education, Medical Teacher, BMJ Patient Safety, and others). Only primary
research will be included in this review. The authors and editors of journals may be approached to obtain data
not available on the article or the supplementary materials.
Databases:
ERIC, Medline, PubMed, Scopus, Psychinfo, Pubmed, Google Scholar, EMBASE, CINAHL.
Time-span of the literature reviewed:
A pilot search conducted in SCOPUS database yielded 1477 articles (search 1 and 2 including duplicates and
all types of articles) from those only 5 were published before 1994, supporting the argument that the interest
in educational interventions is relatively recent compared with research on other aspects of clinical reasoning
(e.g. assessment) (Appendix 5). Therefore search will focus on the last two decades of research, between 1994
and 2014. Backward citation searching may identify studies outside this period in such cases, decisions about
inclusion/exclusion will be made upon their quality and only high quality studies will be exceptionally included.
Language and geography:
Keyword searches will be performed in English but no further language restrictions will be applied. In the
unlikely event of articles in other language that not being retrieved in our searches, drawing on the members
of the group’s language skills, articles in French, Spanish, Portuguese will be reviewed in original language,
other may be translated (using Google translator) if required. If necessary specialist translation service may be
used if translation performed by Google translate does not retrieve the necessary quality required to review of
the study (e.g. translation from Chinese).
No geographical restrictions will be applied. During phase 1, with the support of the librarians in the lead
institution (Swansea University), the search strategy presented here will be iteratively refined and improved,
however, based upon the pilot results it can be anticipated that in phase 1 between 500 to a maximum of
1000 records of potential relevant articles will be handled.
Table 1: Search Terms*
Search terms 1(S1)
clinical reasoning OR clinical decision making OR clinical problems solving OR diagnostic reasoning OR competency OR Diagnostic thinking OR Expert reasoning OR Critical thinking OR Critical analysis OR Situation awareness OR
AND
Promotion OR Teaching OR Pedagogy OR development OR improvement OR learning; OR teaching OR education OR educational OR interventions OR
LIMIT TO (database filters)
Population: Healthcare
professionals (full list below) Time
Period:
6
Schema OR Illness Scripts OR System 1/Pattern recognition OR System 2/hypothetic deductive reasoning OR Forward reasoning OR Backward reasoning OR Bayesian reasoning OR Probabilistic reasoning
educational activities OR educational strategies OR educational methods OR Expertise OR Competence OR
1994 to 2014 Language:
all
Search terms 2(S2)
cognitive bias OR cognitive errors OR heuristics
AND
Reduction OR educational OR interventions educational
OR activities OR educational strategies
7
5. STUDY SELECTION CRITERIA
Population: Initially all domains listed below and levels of healthcare education will be included in the searches. Based on
the quality (criteria below) of articles retrieved, discussion section may focus mainly on domains yielding a
higher number of results to be able to provide a consistent picture of the evidence available. All retrieved
studies will be reported in the findings section.
Evidence of educational interventions at all training levels, undergraduate/initial training to postgraduate
and continuum professional development will be included. Evidence will be classified accordingly to the levels
of training and a ’multiple levels’ category will be included to account for those studies presenting evidence of
educational activities that extend across multiple levels.
List of healthcare professions included: (alphabetical order)
1. Dentistry
2. Medicine
3. Nursing
4. Occupational Therapy
5. Osteopathy
6. Paramedic science
7. Pharmacology
8. Veterinary
Interventions:
Any educational intervention in accordance to the definition used for the purposes of this review with the
explicit aim of promoting clinical reasoning will be included in accordance to the inclusion/exclusion criteria
below will be included.
Study types:
No study will be excluded based on their methodology. All primary research studies reporting educational
interventions at all levels of Cook, Bordage, & Schmidt (2008) framework (description, justification and
clarification).
Studies research design will be classified accordingly to the classification developed by Barr et al.(2000)
(Appendix 3). No grey literature will be included.
Retrieved studies will be rated against the quality criteria in appendix 2.
Outcomes:
Outcomes will be classified according to the Kirkpatrick model adapted by Barr et al. (2000) to use for
systematic review in healthcare education. In order to achieve a holistic view of the research in this field,
identify potential gaps and areas for future research, no study will be excluded on the grounds of outcome
type reported. Such conclusions will be reported in our initial characterization of the field.
The strength of the conclusions identified in each study will be rated on a numerical scale in line with BEME
guidance and a previous systematic review of educational interventions (Gordon & Findley, 2011).
In order to identify messages regarding the effectiveness of educational interventions, studies reporting
highest quality and strength of conclusions will be analysed separately, with more detail and greater emphases
will be placed in their findings. In this analysis outcomes will be also discussed in the light of the findings of
Durning et al.2013 BEME review on assessment of diagnostic reasoning, to ensure consistency between both
BEME reviews. It is expected that this review will be mainly of a narrative nature. However if enough high
quality quantitative studies using with comparable measures are retrieved analysis of weighted effect sizes (at
8
95% confidence interval) possibly using a random-effects model will be calculated and considered as the
primary measure of effect has previously done by other systematic review in health profession education
(Cook et al., 2010).
Inclusion/exclusion criteria:
All of the below Inclusion conditions need to be met for article inclusion:
Clearly defined educational activities/strategies AND
Explicitly aim to develop clinical reasoning OR reduce cognitive bias OR expertise from cognitive
perspective AND
Involve healthcare professionals listed above AND
Provide a detailed description of the intervention to allow for it to be reproduced by others
OR/AND
Be primary research AND Report evaluation data (at any level of Kirkpatrick model adapted by Barr
et al., 2000.)
Exclusion conditions driven from the above inclusion criteria (excluded if)
NOT reporting on clearly identified educational intervention explicitly aimed at promoting clinical
reasoning;
Other reviews (if of high quality these maybe used for backward citation searching);
Articles reporting assessment of clinical reasoning (as that is covered in other BEME review) without
explicit mentioning of educational intervention to teach/develop it;
Any research reporting on design, validation or psychometric characteristics of assessments clinical
reasoning (as that is covered in other BEME review);
Articles using clinical reasoning as one of the outcome measure in curriculum comparisons:
Any research reporting on educational activities to develop reasoning and not CLINICAL reasoning;
Studies describing interventions to reduce cognitive bias or support decision making that are not
educational nature, such as changes in the workplace, introduction of computerized decision
making systems, use of electronic healthcare records or development of mathematic algorithms
to improve test interpretation.
9
6. PROCEDURE FOR EXTRACTING DATA
Data extraction will take place in two phases: first an iterative identification and screening of records in the
databases will take place. Title and abstracts will be scrutinized and a decision made about inclusion/exclusion
(phase 1); the second phase will involve assessment of eligibility and the final decision on inclusion. Full
articles and supplementary materials will be retrieved from databases, their eligibility for review assessed, and
those included coded and rated based on their methodological quality and strength of their findings (phase 2).
Distinction will be made between those to be included in the overall synthesis and those high-quality papers
which will be analysed and reported in greater detail. If necessary, during phase 2, backward citation
searching and hand searching may be used to identify and retrieve articles if not identified during initial
searches. Coding sheets are presented in Appendix 4 and 5.
The review lead, with the support of the librarians and a research assistant at the host institution (Swansea
University) will be responsible for the initial searches.
Reference manager software will be used to store and manage the retrieved references. Two independent
reviewers will independently, blindly review each record and decide on its inclusion or exclusion, providing
reasons for their choice. In cases of disagreement, consensus will be reach by discussion between reviewers.
Cohen's kappa coefficient (k) will be calculated. Full article extraction will be performed.
Phase 2 will follow a similar process. Reviewers will be paired and each pair will make an independent
blinded assessment of the papers eligibility for the study. As previously, consensus will be reached by
discussion between reviewers. The full papers to be included will then be redistributed for coding and quality
assessment based upon the members’ interests and methodological expertise.
The lead reviewer will be responsible for collating the information and maintaining the databases during all
steps of the review.
If the number of studies reporting quantitative measures of outcomes justifies, Rev.Man5 and GradePro may
be used to analyzed and report data of such nature.
TRAINING
Training needs will be identified on an individual basis and reviewers will be directed towards the
appropriate online training available from Campbell Collaboration (http://www.campbellcollabo
ration.org/resources/training.php),COCHRANE LEARNING (www.cochranelearning.com) and other specific
learning resources by The Cochrane Collaboration, and other sources of online training on particular aspects of
the review (e.g. assessment of rigor quality of qualitative studies). Funding may be sought for training sessions,
or development of online training packs, in such case these will be made freely available to others conducting
other BEME review s training.
10
7. SYNTHESIS OF EXTRACTED EVIDENCE – PRISMA
Reporting and presentation of findings will follow the adapted the PRISMA (Preferred Reporting Items for
Systematic reviews and Meta-Analyses) guidance (Liberati et al., 2009; Moher et al., 2009) and research
questions posed. The final summary of the findings will be structured according to the three objectives of the
review stated earlier.
All studies will be classified according to Cook, Bordage, & Schmidt (2008) framework into description,
justification and clarification studies.
All studies will be analyzed using content analysis (thematic analysis) (Denzin & Lincoln, 2000) in order to
provide a summary of themes associated with definitions of clinical reasoning use by different healthcare
professions3, description of the interventions, the theoretical frameworks underpinning such activities. The
aim here will not be to look for effectiveness but rather provide details about the nature and salient features
and context of such interventions.
Justification and clarification studies (quantitative and qualitative) will be categorised by the type of
educational activity reported, methodological choices and outcome reported using the Kirkpatrick model as
adapted by Barr et al.(2000). Special attention been paid to the description of the heterogeneity between
studies and possible different between different healthcare professions, such aspects will be discussed
separately within the discussion section . Qualitative studies will be synthesised using thematic analysis and
used to inform discussion of quantitative data (Bearman & Dawson, 2013).
It is expected that this review will be mainly of a narrative nature. However if enough high quality quantitative
studies are found with comparable measures (allowing standardization of the mean to be performed) are
retrieved these will be analysed separately. In this analysis, weighted effect sizes (at 95% confidence interval)
possibly using a random-effects model will be calculated and considered as the primary measure of effect has
previously done by other systematic reviews in healthcare education (Cook at al. 2009). In these studies
heterogeneity will be analysed using I2 statistic as described by (Liberati et al., 2009).
High quality studies will be discussed in more detail in order to answer the main research questions as it is
appropriate to place greater confidence in their findings. If possible, depending on retrieved data, for this
group of interventions the review will also consider the resources required to implement such interventions.
The PRISMA checklist (Appendix 1) will be used to monitor changes and developments carried out during the
review phases and a detailed record of potential sources of bias and limitations of the review will be kept and
presented along with its findings.
8. PROJECT TIMETABLE
The project timetable is presented below in Table 2. The timeline reflects the processes described in previous sections and, importantly, allows for conclusions from the Durning et al. 2013 BEME on assessment of diagnostic reasoning (due to be published by January 2015) to be taken in consideration and inform the assessment of the outcomes and data analysis and synthesis.
9. CONFLICT OF INTEREST STATEMENT
No conflicts of interest to report.
10. PLANS FOR UPDATING THE REVIEW
Updating the review will be undertaken in accordance with BEME guidelines.
11. CHANGES TO THE PROTOCOL
Any changes to the protocol will be recorded on the adapted PRISMA checklist (appendix 1) along with the
rationale for their need.
3 This will be done in collaboration with the Dr.Durning review group. One of the aims of this group is to investigate
definitions of clinical reasoning in medicine.
11
Table 2: Project timetable
2015
Jul
Aug
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sear
ch a
nd
dat
a ex
trac
tio
n
Phase 1:
Testing of data extraction form (coding
sheet) already ongoing
Search optimisation
Identification
Screening: title + abstract
Testing of data extraction process
Full article retrieval
Phase 2 (full article):
Assessment of eligibility and data
extraction
Quality and RoB assessment
Dat
a
anal
ysis
Discussion of findings meeting
Data analysis and synthesis
Rep
ort
an
d
pu
blic
atio
n
Preparation of report and manuscripts
PRISMA sections
1 -17
PRISMA
sections 17-
29
Manuscript and report
Presentation of findings at AMEE
12
12. REFERENCES
Ajjawi, R. & Higgs, J. (2012) Core components of communication of clinical reasoning: a qualitative study with experienced Australian physiotherapists. Advances in health sciences education : theory and practice. [Online] 17 (1), 107–119. Available from: doi:10.1007/s10459-011-9302-7 [Accessed: 2 December 2012].
Ajjawi, R. & Higgs, J. (2008) Learning to reason: a journey of professional socialisation. Advances in health sciences education : theory and practice. [Online] 13 (2), 133–150. Available from: doi:10.1007/s10459-006-9032-4 [Accessed: 13 February 2013].
Ajjawi, R. & Higgs, J. (2007) Using Hermeneutic Phenomenology to Investigate How Experienced Practitioners Learn to Communicate Clinical Reasoning. 12 (4), 612–638.
Ark, T.K., Brooks, L.R. & Eva, K.W. (2006) Giving learners the best of both worlds: do clinical teachers need to guard against teaching pattern recognition to novices? Academic medicine : journal of the Association of American Medical Colleges. [Online] 81 (4), 405–409. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16565197.
Ark, T.K., Brooks, L.R. & Eva, K.W. (2007) The benefits of flexibility: the pedagogical value of instructions to adopt multifaceted diagnostic reasoning strategies. Medical education. [Online] 41 (3), 281–287. Available from: doi:10.1111/j.1365-2929.2007.02688.x [Accessed: 2 September 2010].
Barr, H., Freeth, D., Hammick, M., Koppel, I., et al. (2000) Evaluations of interprofessional education: a United Kingdom review for health and social care. [Online] (August). Available from: http://www.opengrey.eu/item/display/10068/551729 [Accessed: 13 March 2014].
Bearman, M. & Dawson, P. (2013) Qualitative synthesis and systematic review in health professions education. Medical education. [Online] 47 (3), 252–260. Available from: doi:10.1111/medu.12092 [Accessed: 13 May 2014].
Borleffs, J.C.C., Custers, E.J.F.M., van Gijn, J. & ten Cate, O.T.J. (2003) “Clinical reasoning theater”: a new approach to clinical reasoning education. Academic medicine : journal of the Association of American Medical Colleges. [Online] 78 (3), 322–325. Available from: http://www.ncbi.nlm.nih.gov/pubmed/12634216.
Boshuizen, H.P.A.P., Schmidt, H.G.G., Hp, B. & On, S.H.G. (1992) On the role of biomedical knowledge in clinical reasoning by experts, intermediates and novices. Cognitive Science: A Multidisciplinary Journal. [Online] 16 (2), 153–184. Available from: doi:10.1016/0364-0213(92)90022-M [Accessed: 6 September 2011].
Bowen, J.L. (2006) Educational strategies to promote clinical diagnostic reasoning. The New England journal of medicine. [Online] 355 (21), 2217–2225. Available from: doi:10.1056/NEJMra054782 [Accessed: 28 September 2011].
Brown, J.S., Collins, A., Duguid, P. & Seely, J. (1989) Situated Cognition and the Culture of Learning. Educational Researcher. [Online] 1832–42. Available from: doi:10.2307/1176008.
Buckley, S., Coleman, J., Davison, I., Khan, K.S., et al. (2009) The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Medical teacher. [Online] 31282–298. Available from: doi:10.1080/01421590902889897.
13
Charlin, B., Boshuizen, H.P.A., Custers, E.J. & Feltovich, P.J. (2007) Scripts and clinical reasoning. Medical education. [Online] 41 (12), 1178–1184. Available from: doi:10.1111/j.1365-2923.2007.02924.x [Accessed: 6 July 2011].
Charlin, B., Tardif, J. & Boshuizen, H.P. (2000) Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research. Academic medicine : journal of the Association of American Medical Colleges. [Online] 75 (2), 182–190. Available from: http://www.ncbi.nlm.nih.gov/pubmed/10693854 [Accessed: 4 November 2010].
Cook, D. a, Erwin, P.J. & Triola, M.M. (2010) Computerized virtual patients in health professions education: a systematic review and meta-analysis. Academic medicine : journal of the Association of American Medical Colleges. [Online] 85 (10), 1589–1602. Available from: doi:10.1097/ACM.0b013e3181edfe13 [Accessed: 21 February 2014].
Cook, D.A., Bordage, G. & Schmidt, H.G. (2008) Description, justification and clarification: a framework for classifying the purposes of research in medical education. Medical Education. [Online] 42 (2), 128–133. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18194162.
Cook, D.A. & Triola, M.M. (2009) Virtual patients: a critical literature review and proposed next steps. Medical education. [Online] 43 (4), 303–311. Available from: doi:10.1111/j.1365-2923.2008.03286.x.
Croskerry, P. (2009) A universal model of diagnostic reasoning. Academic medicine : journal of the Association of American Medical Colleges. [Online] 84 (8), 1022–1028. Available from: doi:10.1097/ACM.0b013e3181ace703.
Croskerry, P. (2003) The Importance of Cognitive Errors in Diagnosis and Strategies to Minimize Them. Academic Medicine. [Online] 78 (8), 775–780. Available from: doi:10.1097/00001888-200308000-00003.
Croskerry, P., Singhal, G. & Mamede, S. (2013) Cognitive debiasing 2: impediments to and strategies for change. BMJ quality & safety. [Online] 22 Suppl 2ii65–ii72. Available from: doi:10.1136/bmjqs-2012-001713 [Accessed: 3 March 2014].
Custers, E.J.F.M., Regehr, G. & Norman, G.R. (1996) Mental Representations of Medical Diagnostic Knowledge: A Review. Acad Med. 71 (10), S55–S61.
Delany, C., Golding, C. & Bialocerkowski, A. (2013) Teaching for thinking in clinical education: Making explicit the thinking involved in allied health clinical reasoning. Focus on Health Professional Education A Multidisciplinary Journal. [Online] 14 (2), 44. Available from: http://search.informit.com.au/documentSummary;dn=344079003268797;res=IELHEA.
Dent, J.A. & Harden, R.M. (2005) A practical guide for medical teachers. University of Michigan, Elsevier Churchill Livingstone.
Denzin, N.K. & Lincoln, Y.S. (2000) The Sage handbook of qualitative research. 2nd edition. Norman K. Denzin & Yvonna S. Lincoln (eds.). [Online]. Thousand Oaks, California, Sage Publications, Inc.
Dequeker, J. & Jaspaert, R. (1998) Teaching problem-solving and clinical reasoning: 20 years experience with video-supported small-group learning. Medical education. [Online] 32 (4), 384–389. Available from: http://www.ncbi.nlm.nih.gov/pubmed/9743800.
Durak, H.I., Caliskan, S.A., Bor, S. & Van der Vleuten, C. (2007) Use of case-based exams as an instructional teaching tool to teach clinical reasoning. Medical teacher. [Online] 29 (6), e170–4. Available from: doi:10.1080/01421590701506866 [Accessed: 23 August 2010].
14
Durning, S., Artino, A.R., Pangaro, L., van der Vleuten, C.P., et al. (2011) Context and clinical reasoning: understanding the perspective of the expert’s voice. Medical education. [Online] 45 (9), 927–938. Available from: doi:10.1111/j.1365-2923.2011.04053.x [Accessed: 24 August 2011].
Durning, S.J. & Artino, A.R. (2011) Situativity theory: a perspective on how participants and the environment can interact: AMEE Guide no. 52. Medical teacher. [Online] 33 (3), 188–199. Available from: doi:10.3109/0142159X.2011.550965 [Accessed: 19 February 2014].
Durning, S.J., Artino, A.R., Boulet, J.R., Dorrance, K., et al. (2012) The impact of selected contextual factors on experts’ clinical reasoning performance (does context impact clinical reasoning performance in experts?). Advances in health sciences education : theory and practice. [Online] 17 (1), 65–79. Available from: doi:10.1007/s10459-011-9294-3 [Accessed: 31 July 2013].
Elizondo-Omaña, R.E. & López, S.G. (2008) The development of clinical reasoning skills: a major objective of the anatomy course. Anatomical sciences education. [Online] 1 (6), 267–268. Available from: doi:10.1002/ase.57.
Elstein, A., Shulman, L.S. & Sprafka, S.A. (1978) Medical Problem Solving: an Analysis of Clinical Reasoning. Cambridge, MA, Harvard Un. Press.
Elstein, A.S. (2009) Thinking about diagnostic thinking: a 30-year perspective. [Online] 14 Suppl 17–18. Available from: doi:10.1007/s10459-009-9184-0.
Epstein, R.M. (2007) Assessment in medical education. The New England journal of medicine. [Online] 356 (20), 2109; author reply 2110. Available from: doi:10.1056/NEJMra054784.
Eva, K.W. (2005) What every teacher needs to know about clinical reasoning. Medical education. [Online] 39 (1), 98–106. Available from: doi:MED1972 [pii] 10.1111/j.1365-2929.2004.01972.x.
Eva, K.W., Hatala, R.M., Leblanc, V.R. & Brooks, L.R. (2007) Teaching from the clinical reasoning literature: combined reasoning strategies help novice diagnosticians overcome misleading information. Medical education. [Online] 41 (12), 1152–1158. Available from: doi:10.1111/j.1365-2923.2007.02923.x.
Eva, K.W., Wood, T.J., Riddle, J., Touchie, C., et al. (2010) How clinical features are presented matters to weaker diagnosticians. Medical education. [Online] 44 (8), 775–785. Available from: doi:10.1111/j.1365-2923.2010.03705.x [Accessed: 19 July 2011].
Eva, K.W.W., Norman, G.R.R., Neville, A.J.J., Wood, T.J.J., et al. (2002) Expert-Novice Differences in Memory: A Reformulation. Teaching and Learning in Medicine. [Online] 14 (4), 257–263. Available from: http://www.tandfonline.com/doi/abs/10.1207/S15328015TLM1404_10 [Accessed: 8 December 2011].
Evans, J.S. (1984) Heuristic and analytic processes in reasoning. British Journal of Psychology. [Online] 75451–468. Available from: doi:10.1111/j.2044-8295.1984.tb01915.x.
Gobet, F. (2007) A pattern-recognition theory of search in expert problem solving. Thinking Reasoning. [Online] 3 (4), 291–313. Available from: http://dx.doi.org/10.1080/135467897394301.
Gordon, M. & Findley, R. (2011) Educational interventions to improve handover in health care: a systematic review. Medical education. [Online] 451081–1089. Available from: doi:10.1111/j.1365-2923.2011.04049.x.
Graber, M. (2003) Metacognitive training to reduce diagnostic errors: ready for prime time? Academic medicine : journal of the Association of American Medical Colleges. [Online] 78 (8), 781. Available from: http://www.ncbi.nlm.nih.gov/pubmed/12915364.
15
Graber, M.L., Franklin, N. & Gordon, R. (2005) Diagnostic error in internal medicine. Archives of internal medicine. [Online] 165 (13), 1493–1499. Available from: doi:10.1001/archinte.165.13.1493.
Graber, M.L., Kissam, S., Payne, V.L., Meyer, A.N.D., et al. (2012) Cognitive interventions to reduce diagnostic error: a narrative review. BMJ quality & safety. [Online] 21 (7), 535–557. Available from: doi:10.1136/bmjqs-2011-000149.
Haig, A. & Dozier, M. (2003) BEME guide no. 3: systematic searching for evidence in medical education--part 2: constructing searches. [Online] 25 (5), 463–484. Available from: doi:10.1080/01421590310001608667 Q8AP0TL08X7J33LY [pii].
Hammick, M.R. (2005) Evidence-informed education in the health care science professions. Journal of veterinary medical education. [Online] 32399–403. Available from: doi:10.3138/jvme.32.4.399.
Harris, A., Boyce, P. & Ajjawi, R. (2011) Clinical reasoning sessions: back to the patient. The clinical teacher. [Online] 8 (1), 13–16. Available from: doi:10.1111/j.1743-498X.2010.00407.x.
Higgs, J. (1992) Developing Clinical Reasoning Competencies. Physiotherapy. [Online] 78 (8), 575–581. Available from: doi:10.1016/s0031-9406(10)61202-3.
Higgs, J. & Jones, M.A. (2000) Clinical Reasoning in Health Professions. 2nd edition. Joy Higgs & Mark A. Jones (eds.). [Online]. Elsevier Health Sciences. Available from: http://books.google.com/books?id=YfJmzMbX9jIC&pgis=1 [Accessed: 29 November 2010].
Higgs, J., Jones, M.A. & Christensen, N. (2008) Clinical Reasoning in Health Professions. 3rd edition. Elsevier.
Holyoak, K.J. & Morrison, R.G. (2012) Thinking and reasoning: A reader’s guide. In: K J Holyoak & Robert G Morrison (eds.). Oxford Handbook of Thinking and Reasoning. [Online]. New York, U.S.A., Oxford University Press.
Irby, D.M. & Bowen, J.L. (2004) Time-efficient strategies for learning and performance. The Clinical Teacher. [Online] 1 (1), 23–28. Available from: doi:10.1111/j.1743-498X.2004.00013.x.
Jensen, G., Resnik, L. & Haddad, A. (2008) Expertise in Clinical Reasoning. In: Joy Higgs, Mark A. Jones, S Loftus, & N Christensen (eds.). Clinical Reasoning in the Healthcare Professions. 3rd edition. Philadelphia/US, Elsevier.
Kassirer, J.P. (2010) Teaching Clinical Reasoning : Case-Based and Coached. Academic Medicine. 85 (7), 1118–1124.
Kim, S.Y., Park, J.E., Lee, Y.J., Seo, H.-J., et al. (2013) Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity. Journal of clinical epidemiology. [Online] 66 (4), 408–414. Available from: doi:10.1016/j.jclinepi.2012.09.016 [Accessed: 12 March 2014].
Kuiper, R., Heinrich, C., Matthias, A., Graham, M.J., et al. (2008) Debriefing with the OPT model of clinical reasoning during high fidelity patient simulation. [Online] 5Article17. Available from: doi:10.2202/1548-923X.1466.
Kurzenhäuser, S. & Hoffrage, U. (2002) Teaching Bayesian reasoning: an evaluation of a classroom tutorial for medical students. Medical teacher. [Online] 24 (5), 516–521. Available from: doi:10.1080/0142159021000012540 [Accessed: 9 September 2011].
16
Lave, J. & Wenger, E. (1991) Situated learning: Legitimate peripheral participation. [Online]. Available from: doi:10.2307/2804509.
Liberati, A., Altman, D., Tetzlaff, J., Mulrow, C., et al. (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. BMJ (Online). [Online] Available from: doi:10.1136/bmj.b2700 [Accessed: 14 March 2014].
Mamede, S., Schmidt, H.G. & Penaforte, J.C. (2008) Effects of reflective practice on the accuracy of medical diagnoses. Medical education. [Online] 42 (5), 468–475. Available from: doi:10.1111/j.1365-2923.2008.03030.x [Accessed: 2 September 2010].
McMillan, W.J. (2010) Teaching for clinical reasoning - helping students make the conceptual links. Med Teach. 32 (10), e436–42.
Moher, D., Liberati, A., Tetzlaff, J. & Altman, D.G. (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. [Online] 339 (august), 332–336. Available from: doi:10.1136/bmj.b2535 [Accessed: 11 March 2014].
Newman, M., Learning, C.C.S.R.G. on the effectiveness of P.B. & University, M. (2003) A pilot systematic review and meta-analysis on the effectiveness of Problem Based Learning. [Online]. p.74. Available from: doi:0 7017 0158 7.
Norman, G. (2009) Dual processing and diagnostic errors. Advances in health sciences education : theory and practice. [Online] 14 Suppl 1 (0), 37–49. Available from: doi:10.1007/s10459-009-9179-x.
Norman, G.R. (2006) Building on experience--the development of clinical reasoning. The New England journal of medicine. [Online] 355 (21), 2251–2252. Available from: doi:10.1056/NEJMe068134.
Norman, G.R. (2005) Research in clinical reasoning: past history and current trends. Medical education. [Online] 39 (4), 418–427. Available from: doi:10.1111/j.1365-2929.2005.02127.x.
Norman, G.R., Barraclough, K., Dolovich, L. & Price, D. (2009) Iterative diagnosis. BMJ. [Online] 339 (sep22 1), b3490–b3490. Available from: doi:10.1136/bmj.b3490.
Norman, G.R. & Eva, K.W. (2010) Diagnostic error and clinical reasoning. Medical education. [Online] 44 (1), 94–100. Available from: doi:10.1111/j.1365-2923.2009.03507.x [Accessed: 2 September 2010].
Norman, G.R. & Eva, K.W. (2005) Does clinical experience make up for failure to keep up to date? Evidence Based Medicine. [Online] 10 (3), A8–9. Available from: doi:ACPJC-2005-142-3-A08 [pii].
Norman, G.R., Young, M. & Brooks, L.R. (2007) Non-analytical models of clinical reasoning: the role of experience. Medical education. [Online] 41 (12), 1140–1145. Available from: doi:10.1111/j.1365-2923.2007.02914.x [Accessed: 9 March 2012].
Patel, V.L., Evans, D.A. & KAUFMAN, D.R. (1990) Reasoning strategies and the use of biomedical knowledge by medical students. Medical Education. [Online] 24 (2), 129–136. Available from: doi:10.1111/j.1365-2923.1990.tb02511.x [Accessed: 2 September 2010].
Patrício, M. & vaz Carneiro, A. (2012) Systematic reviews of evidence in medical education and clinical medicine: is the nature of evidence similar? Medical teacher. [Online] 34 (6), 474–482. Available from: doi:10.3109/0142159X.2012.682186 [Accessed: 28 January 2014].
17
Radomski, N. & Russell, J. (2010) Integrated case learning: teaching clinical reasoning. Advances in health sciences education : theory and practice. [Online] 15 (2), 251–264. Available from: doi:10.1007/s10459-009-9194-y [Accessed: 30 August 2010].
Reiter, H.I., Eva, K.W., Hatala, R.M. & Norman, G.R. (2002) Self and peer assessment in tutorials: application of a relative-ranking model. Academic Medicine. [Online] 77 (11), 1134–1139. Available from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve&db=PubMed&dopt=Citation&list_uids=12431928.
Rencic, J. (2011) Twelve tips for teaching expertise in clinical reasoning. Medical Teacher. [Online] 33 (11), 887–892. Available from: doi:10.3109/0142159X.2011.558142.
Rikers, R.M.J.P. & Verkoeijen, P.P.J.L. (2007) Clinical expertise research: a history lesson from those who wrote it. [Online] 41 (12), 1115–1116. Available from: doi:doi:10.1111/j.1365-2923.2007.02920.x.
Schmidt, H.G., Norman, G.R. & Boshuizen, H.P.A. (1990) A Cognitive Perspective on Medical Expertise: Theory and Implications. Acad Med. 65 (10), 611–621.
Sherbino, J., Dore, K.L., Wood, T.J., Young, M.E., et al. (2012) The relationship between response time and diagnostic accuracy. Academic medicine : journal of the Association of American Medical Colleges. [Online] 87 (6), 785–791. Available from: doi:10.1097/ACM.0b013e318253acbd [Accessed: 3 February 2013].
Smith, M., Higgs, J. & Ellis, E. (1991) Factors influencing clinical decision making. 89–100.
Stieger, S., Praschinger, A., Kletter, K. & Kainberger, F. (2009) Diagnostic grand rounds: A new teaching concept to train diagnostic reasoning. European journal of radiology. [Online] In Press,. Available from: doi:10.1016/j.ejrad.2009.05.015.
Stiegler, M.P., Neelankavil, J.P., Canales, C. & Dhillon, a (2011) Cognitive errors detected in anaesthesiology: a literature review and pilot study. British journal of anaesthesia. [Online] 108 (December 2011), 229–235. Available from: doi:10.1093/bja/aer387 [Accessed: 17 January 2012].
Struyf, E., Beullens, J., Van Damme, B., Janssen, P., et al. (2005) A new methodology for teaching clinical reasoning skills: problem solving clinical seminars. [Online] 27 (4), 364–368. Available from: doi:RK674L2J7875NNLX [pii] 10.1080/01421590500046411.
Thompson, C. & Stapley, S. (2011) Do educational interventions improve nurses’ clinical decision making and judgement? A systematic review. International Journal of Nursing Studies. [Online]. 48 pp.881–893. Available from: doi:10.1016/j.ijnurstu.2010.12.005.
Trowbridge, R.L., Dhaliwal, G. & Cosby, K.S. (2013) Educational agenda for diagnostic error reduction. BMJ quality & safety. [Online] 22 Suppl 2ii28–ii32. Available from: doi:10.1136/bmjqs-2012-001622 [Accessed: 15 February 2014].
Windish, D., Price, E., Clever, S., Magaziner, J., et al. (2005) Teaching medical students the important connection between communication and clinical reasoning. Journal of General Internal Medicine. [Online] 20 (12), 1108–1113. Available from: http://dx.doi.org/10.1111/j.1525-1497.2005.0244.x.
Woods, N.N., Brooks, L.R. & Norman, G.R. (2007) The role of biomedical knowledge in diagnosis of difficult clinical cases. [Online] 12 (4), 417–426. Available from: doi:10.1007/s10459-006-9054-y.
18
APPENDIX 1: PRISMA CHECKLIST ADAPTED FROM LIBERATI ET AL. (2009)
Milestones in the review/updates to planed review sections Description of the current review details (planned at protocol stage)
Were any changes to the
protocol requires? Provide rational
Phase 1
Phase 2
Title 1 Educational Interventions To Promote/Teach Clinical Reasoning: A Best Evidence In Medical Education Systematic Review (BEME) Abstract
Structured summary 2
We will provide a structured summary of the background, objectives, data sources, study eligibility criteria, populations, types of educational activities reviewed, health professions covered, study appraisal/quality indicators and synthesis methods, results, limitations, conclusions and implications for both future research and practice. Summary of outcomes will be grouped by type of educational activity to increase readability and usefulness of the results.
Introduction
Rationale 3 A brief rationale is provided in pages 2 and 3 of the current protocol. This section will be expanded and updated in the final report to account for possible developments, mainly those identified in another BEME review by Durning et al.2013 currently in progress.
Objectives 4 The objectives of the current review are stated on page 5 of the protocol. Methods
Protocol and registration 5 All materials produced by the current review will be available on the BEME website and written according to BEME guidance.
Eligibility criteria 6
Eligibility criteria is presented in the current protocol (page 7). No study will be excluded on the basis of its methodology or type of outcome measured, rather those will be coded and included in the review as an overview of the research in the field.
Information sources 7
Databases searched are included in section 4 along with a description of the search strategies used. The current review will focus on the period between 1994 and 2014/2015. Studies looking at all healthcare professions will be included in initial search, however if necessary, for feasibility reasons, the articles retrieved will be limited to the professions where a higher number of studies has been conducted (likely to be medicine, nursing, dentistry, veterinary, physiotherapy). This will be clearly stated and reported in the report of our review.
Search 8 Search strategy, terms and databases searched are presented on page 6 of the protocol. Study
selection 9 All study records from phase 1 and retrieved studies in phase 2 will reviewed by two independent reviewers, consensus will be achieved by discussion. The process for selecting studies screening, eligibility and decision about inclusion in the systematicreview is
19
described on page 9 of the protocol and further details will be provided in the final report. Coding sheets are presented in appendix 4. A final diagram will be presented identifying the number of records and articles identified, screened, assessed for eligibility and included in the review.
Data collection process 10
Method of data extraction, including data extraction forms (appendix 3) and any changes made to the testing phase will be reported. Software to collate data from the paired reviewers, to calculate inter-rater reliability (Cohen’s Kappa) will be identified. It is not anticipated that data from the studies will need to be obtained from investigators, however if required, the authors and editors of journals may be approached to obtain data not available on the article or the supplementary materials.
Data items 11
All classification items used to code the studies are presented in appendix 3 and 4. Any simplifications (categorisation of data or standardisation of measures) made will be presented and justified.
Risk of bias in individual studies 12
Quality of the studies will be assessed by the checklist (appendix 2) adapted from a previous BEME reviews (Buckley et al., 2009) No studies will be excluded on the basis of their methodological quality, except for those not reporting any quality measures. Quantitative, high quality studies’ risk of bias (RoB) will be assessed by an independent reviewer according to either the Cochrane Risk of Bias for Randomized or Non-Randomized studies (Kim et al., 2013). The results will be reported as a separated section within findings section.
Summary measures 13
It is expected that this review will be mainly of a narrative nature. An initial analysis of the definitions, salient features and frameworks of interventions will be carried out by content analysis (thematic analysis). If enough high quality quantitative justification and clarification studies are found with comparable measures (allowing standardization of the mean to be performed) are retrieved these will be analysed separately. In this analysis, weighted effect sizes (at 95% confidence interval) possibly using a random-effects model will be calculated and considered as the primary measure of effect has previously done by other systematic reviews in healthcare education (Cook at al. 2009).
Planned methods for analysis 14
All included studies (quantitative and qualitative) will be categorised by the type of educational activity reported, methodological choices and outcome reported using the Kirkpatrick model as adapted by Barr et al.(2000). Qualitative studies will be synthesised using thematic analysis and used to inform discussion of quantitative data (Bearman & Dawson, 2013). If enough studies retrieved allow for a meta-analysis to be performed (which we do not anticipate) heterogeneity will be analysed using I
2 statistic as described by (Liberati et al., 2009).
Risk of bias across studies 15
Possible risk of bias likely to impact on the review will be identified and discussed. At this early stage of the review, it is possible to identify a few possible risks we account for: i) publication bias and author (as pilot search shows a very high percentage of the papers has been published by only a few journals/authors therefore more likely to be susceptible to this type of bias) ii)poor indexing, may mean searches are difficult iii) possible selective reporting of findings. Careful coding of the research retrieved will be made to account for such possible sources of bias and to identify others during the review process, the steps taken will be described in the final reports.
Additional analyses 16 If during the review process additional analyses are necessary the methods used will be described in detail in this section.
Results
Study selection 17
Selection criteria and search strategies are described in page 7 of the protocol. At later stages of this review this section will report numbers of studies screened, assessed for eligibility, and included in the review, with reasons for exclusions at each stage. This will be present as a follow diagram in accordance to PRISMA guidance.
20
Study characteristics 18
Coding sheets were designed to allow characterisation of different types of studies and their methodological choices (See coding sheet/extraction form appendix 3). No studies will be excluded on due to methodological choices or quality (as long as reporting quality measures).
Risk of bias within studies 19
Study quality and risk of bias will be assessed as described in the protocol. Additionally, comment boxes will be included to allow reviewers to include extracts from the originals to support their judgement.
Results of individual studies 20
For all the outcome levels considered (Kirkpatrick model) and educational intervention type, per study, a simple summary of characteristics will be presented. Additionally, if enough high-quality quantitative studies are retrieved to justify a meta-analysis, statistical data will be presented as weighted effect estimates and a confidence intervals forest plot.
Synthesis of results 21
The summary of results will be based upon the defined aims of the current review as described in the protocol. Summary of results, both quantitative and qualitative, will take a tabular format, along with narrative description of the findings. For any meta-analyses, if done, confidence intervals and measures of consistency (I2 heterogeneity) will be reported.
Risk of bias across studies 22
Identified quality queries and risk of bias across studies will be reported. If any meta-analysis are conducted and heterogeneity levels between studies are high those should be explored using adequate methods (e.g. funnel plot).
Additional analysis 23
Currently no additional analyses are planned, but it may be that during the review process those are found to be necessary, there results will be reported here.
Discussion/conclusion
Summary of evidence 24
A summary of the main findings will be guided by the aims of the review. It will provide a overview table with results from the holistic analysis of the research in this field and the strength of the evidence found. From the subgroup of high-quality studies, those in which it is more appropriate to place greater confidence, a more detailed description of results will be provided, by type of educational intervention in relation to the key groups included in the review (e.g. different healthcare professions, levels of training), contexts and if possible the resources required for implementation. Finally, the relevance to stakeholders (such as healthcare educators and providers, institutions and policy makers) educators and researchers will be considered and recommendations drawn (items 27 and 28).
Limitations 25
A discussion of the limitations of the current review will be provided under this section. Here we will also report on a evaluation of the review process, focusing on key questions about the process of the review and how significant for practice were those findings.
Conclusions 26
In this section we will provide a general interpretation of the results in the context of other evidence, and implications for future research.
Recommendations For Practice* 27
These sections were added to the PRISMA checklist based upon BEME recommendations that the reviews conducted in this context should aim to provide guidance to those in practice. In these section we will include key messages for both practice and research in this field.
Recommendations For Research* 28
Funding 29
External funding maybe be sought to support particular activities (e.g. meetings, research support, database management). Any sources of funding, support from the institutions and possible role of funders will be described under this section.
21
* Added to original PRISMA checklist.
22
APPENDIX 2: QUALITY CHECKLIST
Adapted from Buckley et al. 2009 BEME review.
Quality Indicator Question/descriptor:
Ge
ne
ral (
all a
rtic
les)
Research question Is the research question or hypothesis clearly stated? YES / NO /UNCLEAR
Study participants Is the participant group appropriate for the study being carried out (number, characteristics, selection, and homogeneity)? YES / NO /UNCLEAR
Sampling strategy is clearly outlined. YES / NO /UNCLEAR
Methodological choices/Type of study
The methodology used is clearly stated as well as the reasons for the choice made with reference to the research questions and aims of the research. YES / NO /UNCLEAR
‘Data’ collection methods
Chosen methods are coherent and follow logically from the research questions and methodological choices made. YES / NO /UNCLEAR
Are the methods used (qualitative or quantitative) reliable and valid for the research question and context? YES / NO /UNCLEAR
Completeness of ‘data’
Applies to both qualitative and quantitative studies. Have subjects dropped out? Is the attrition rate less than 50%? YES / NO /UNCLEAR
For questionnaire based studies, is the response rate acceptable (60% or above?) YES / NO /UNCLEAR
Control for confounding
Have multiple factors/variables been removed or accounted for where possible? YES / NO /UNCLEAR
Analysis of results Are the statistical or other methods of results analysis used appropriate?
YES / NO /UNCLEAR
Reporting of results Are all measures/results of interest reported? (or is there a selective reporting of outcomes?) YES / NO /UNCLEAR
Limitations Study discusses/recognises possible sources of bias and how those were addressed? YES / NO /UNCLEAR
Conclusions Low--------------------------------------------------------------------------------- High
1 2 3 4 5
23
1 – No clear conclusions can be drawn. Not significant 2 – Results ambiguous, but there appears to be a trend. 3 – Conclusions can probably be based on the results. 4 – Results are clear and very likely to be true. 5 – Results are unequivocal.
Reproducibility Could the study be repeated by other researchers? YES / NO /UNCLEAR
Prospective Does the study look forwards in time (prospective) rather than retrospective? YES / NO /UNCLEAR
Peer review Has the paper been peer reviewed? YES / NO /UNCLEAR
Ethical issues Were all relevant ethical issues addressed YES / NO /UNCLEAR
Was ethical approval sought for the study? YES / NO /UNCLEAR
Triangulation Were results supported by data from more than one source? YES / NO /UNCLEAR
Qu
anti
tati
ve s
tud
ies
(on
ly, l
eav
e b
lan
k fo
r q
ual
itat
ive
stu
die
s)
Hypotheses Are clear and theory driven? YES / NO /UNCLEAR
Instruments psychometric characteristics
Uses valid instruments/tests to measure outcomes? YES / NO /UNCLEAR
Reports validity? YES / NO /UNCLEAR
Reports reliability data on the instruments used? YES / NO /UNCLEAR
Sampling Reports power statistics and/or sampling strategy used ? YES / NO /UNCLEAR
Confounding variables
Study presents adequate confirmation and consideration of possible confounding variables? YES / NO /UNCLEAR
Are there any confounding variable not accounted for that may have impacted the study findings. YES / NO /UNCLEAR
Replicability Would this study be replicable by others? YES / NO /UNCLEAR
Does this article report a
RCT > Use Risk of Bias Assessment tool for randomized Studies (RoBANS)
Cohort study, Non-randomized controlled trial, Case-control study or a Before-after study > Use Risk of Bias Assessment tool for randomized Studies (RoBANS) (Kim et al. 2013)
24
Qu
alit
ativ
e s
tud
ies
(on
ly)
Trustworthiness Are the procedures taken in order to ensure trustfulness clearly described? YES / NO /UNCLEAR Transferability Does the study discuss how the same ideas could be applied
more widely in an adequate manner ? YES / NO /UNCLEAR
Dependability Does the study discuss how dependability was achieved ?(audit trails, rich documentation, triangulation, etc.) YES / NO /UNCLEAR
Confirmability Does the study identifies strategies used to control for researcher bias (e.g. researcher self-reflection on potential bias, peer review, or consultations) ? YES / NO /UNCLEAR
Credibility Are the procedures taken in order to ensure credibility clearly described?
Triangulation, or multiple sources of data as evidence. YES / NO /UNCLEAR
Does the study report any of the following measure to increase quality and rigour of its findings:
Member checks, or arranging for those who provided data to evaluate the conclusions. YES / NO /UNCLEAR Saturation, or continuous data collection to the point where more data add little to regularities that have already surfaced. YES / NO /UNCLEAR Peer review, or consultation with experts. YES / NO /UNCLEAR Audit trail, or the detailed record of data collection and rationale for important decisions. YES / NO /UNCLEAR Thick description, or providing rich detail of the context of the study. YES / NO /UNCLEAR Plausible alternatives, or the rationale for ruling out alternative explanations and accounting for discrepant (negative) cases. YES / NO /UNCLEAR Inter-coder or inter-observer agreement (Cohen’s k) or and code-recode consistency. YES / NO /UNCLEAR
25
APPENDIX 3: KIRKPATRICK’S EVALUATION OF OUTCOMES
Adapted from : Gordon & Findley, 2011 and Barr et al., 2000.
Outcome level Description:
Level 1: Reaction Learners’ views and satisfaction with the educational interventions. Can include but are not limited to learns' views on organization, presentation, content, teaching methods, and aspects of the instructional organization, materials, quality of instruction.
Level 2a: modification of attitudes/perceptions
Outcomes relate to changes in attitudes or perceptions of the learns' towards the intervention's aims (e.g. learners' perceived impact on confidence in clinical reasoning ability; perception of diagnostic accuracy; perceptions of preparation for practice).
Level 2b Modification of knowledge/'skills'
Knowledge outcomes relate to the acquisition of concepts, principles or conceptual models(e.g. system 1 vs. system 2); 'Skills' outcomes relate to the impact on the ability to solve problems or engage in reasoning/ thinking about similar cases to the ones used in the intervention and/or within the same/similar context (e.g. results in high-fidelity simulation scenario - MI case- as outcome of high-fidelity simulation interventions using common MI typical presentations).
Level 3 Behavioral change Outcome here should document a transfer of learning to the workplace or application of newly acquired knowledge/clinical reasoning abilities in practice, or in a different context from the context of the interventions (e.g. performance in simulation scenario- MI case- used as a outcome measure of taught small group case discussions on models of clinical reasoning; evaluation of students differential diagnosis and clinical decisions in general practice placements as outcome measure of case based discussions).
Level 4a Change in organisational practice
Wider changes in the organizational delivery of care, attributable to an educational program.
Level 4b Benefits to patient/clients
Any improvement in the health & well being of patients/clients as a direct result of an educational program.
26
APPENDIX 4: DRAFT CODING SHEETS
Phase 1 - Abstract Screening Checklist (adapted from Gordon & Findley, 2011) Reference Number: Reviewer: Search Method: Electronic search Hand search Database: Citation (BEME style): DOI: URL:
Population: Are participants healthcare professionals? YES / NO If NO exclude
Level of training (report):
Does this study look at an educational intervention offered to the population?
YES / NO If NO exclude
Was clinical reasoning (decision-making, clinical problem solving, or reduction of cognitive bias) an explicit outcome/aim of this intervention?
YES / NO If NO exclude
Is this article reporting an assessment clinical reasoning without mentioning educational interventions (e.g. comparisons between novices and experts)
YES / NO If YES exclude
Type of study (Cook et al., 2008),
Description Justification Clarification
Decision: Include Exclude If not clear from above provide reason with reference to the inclusion/exclusion criteria.
What level of outcome are reported? (Kirkpatrick’s hierarchy)
Select Level 1: Reaction Level 2a: modification of attitudes/perceptions Level 2b Modification of knowledge/'skills' Level 3 Behavioural change Level 4a Change in organisational practice Level 4b Benefits to patient / clients Not clear from abstract
27
Data Extraction form Phase 2 full articles meeting eligibility criteria (Adapted from Gordon & Findley, 2011 and Barr et al., 2000). Reference Number: Reviewer: Search Method: Electronic search Hand search Database: Citation (BEME style): DOI: URL: Type of study: Level of outcome: (Filled in phase 1)
Background/ Question / Objective
Has a review of the literature been described? YES / NO
there a clearly defined and well described objective to the study?
YES / NO
Research design
Is the design appropriate to answer the research question?
YES / NO
Is the study design reported? YES / NO
Place an S for Stated or I for Implied in the box:- Audit Interrupted time series Survey Before and after studies Cross sectional studies Measures taken over period of time after the intervention Post intervention studies Randomized Non-randomized Case series Cohort studies Retrospective Prospective
Action research Longitudinal studies. Observational Case studies; Mixed-methods case (select above) Qualitative studies (state type of design) ___________________________________ Other please state here type of study ___________________________________
Were the learner characteristics reported (If NO continue to intervention)
YES/NO
Age Mean: SD:
Level Undergraduate Postgraduate CPD
Healthcare profession (state)
Gender Distribution:
Number of participants
Intervention
Definition of clinical reasoning (please use original text when possible):
28
Conceptual framework used and reference if provided:
Is the educational intervention clearly described (could it be replicated)?
YES / NO
Are the required resources described? YES / NO
Please record details of educational intervention (please provide details and if possible quotes from the original article to support your classification)
Outcome Single outcome (clinical reasoning) Multiple outcomes (one being clinical reasoning)
Environment 1 Face-to-face Online Mixed
Environment 2 Clinical Simulation High fidelity/Low fidelity Non-clinical Mixed (please state)
Group/participants Small group Large group Individualised Mixed
Content Clinical cases Clinical reasoning conceptual frameworks/research (specify) Biomedical knowledge Mixed
Duration (e.g. 45m session): Frequency: Time period (e.g. one month, one semester, one year):
Other described factors, external to the intervention, that impact the outcome - limitations (if necessary quotes from the original article to support your classification):
Others provided details of the intervention:
Report quality assessment measures in Quality Checklist
29
APPENDIX 5:
Below are the overall results retrieved from a pilot search using SCOPUS database. These results below are
to be understood as a first test of the search terms and strategy only.
Figure Ap5.1: Number of articles (all types considered) retrieved by year per each of the searches considered
Table Ap5.1: Types Of Articles And Subject Areas Of Papers Retrieved Per Search
Types of articles S1 Types of articles S2
Research articles 792 (70%) Research articles 204 (59%)
Reviews 215 Reviews 89
Subject area (all types considered) Subject area (all types considered)
Medicine 586 Medicine 149
Social Sciences 461 Social Sciences 109
Psychology 286 Psychology 160
Nursing 234 Nursing 28
Health Professions 78 Health Professions 9
Computer Science 73 Computer Science 9
Dentistry 26 Dentistry 2
Veterinary 11 Veterinary 1
Pharmacology, Toxicology and
Pharmaceutics 6
Pharmacology, Toxicology and
Pharmaceutics 1
Decision Sciences 5 Decision Sciences 7
Multidisciplinary 1 Multidisciplinary 1
Arts and Humanities 34
Neuroscience 31
Biochemistry, Genetics and
Molecular Biology 11
0
20
40
60
80
100
120
140
S1
S2
30
Table Ap5.2: Search terms (test of search terms and ‘limit to ’vs ‘exclude’)
Search 1 Search 2
(ALL (clinical reasoning OR clinical decision making OR clinical problems solving OR diagnostic reasoning AND development OR improvement OR learning OR teaching OR education OR educational OR interventions educational activities OR educational strategies OR educational methods OR Expertise) AND ( EXCLUDE(SUBJAREA,"ARTS" ) OR EXCLUDE(SUBJAREA,"BUSI" ) OR EXCLUDE(SUBJAREA,"NEUR" ) OR EXCLUDE(SUBJAREA,"ENGI" ) OR EXCLUDE(SUBJAREA,"ARTS" ) OR EXCLUDE(SUBJAREA,"BUSI" ) OR EXCLUDE(SUBJAREA,"NEUR" ) OR EXCLUDE(SUBJAREA,"ENGI" ) OR EXCLUDE(SUBJAREA,"MATH" ) OR EXCLUDE(SUBJAREA,"BIOC" ) OR EXCLUDE(SUBJAREA,"ECON" ) OR EXCLUDE(SUBJAREA,"AGRI" ) OR EXCLUDE(SUBJAREA,"ENVI" ) OR EXCLUDE(SUBJAREA,"CENG" ) OR EXCLUDE(SUBJAREA,"IMMU" ) OR EXCLUDE(SUBJAREA,"CHEM" ) OR EXCLUDE(SUBJAREA,"EART" ) OR EXCLUDE(SUBJAREA,"MATE" ) OR EXCLUDE(SUBJAREA,"PHYS" ) OR EXCLUDE(SUBJAREA,"MATH" ) OR EXCLUDE(SUBJAREA,"BIOC" ) OR EXCLUDE(SUBJAREA,"ECON" ) OR EXCLUDE(SUBJAREA,"AGRI" ) OR EXCLUDE(SUBJAREA,"ENVI" ) OR EXCLUDE(SUBJAREA,"CENG" ) OR EXCLUDE(SUBJAREA,"IMMU" ) OR EXCLUDE(SUBJAREA,"CHEM" ) OR EXCLUDE(SUBJAREA,"EART" ) OR EXCLUDE(SUBJAREA,"MATE" ) OR EXCLUDE(SUBJAREA,"PHYS" ) ) )
(ALL (cognitive bias cognitive errors OR heuristics clinical problems solving OR diagnostic reasoning AND Reduction OR educational interventions OR educational activities OR educational strategies ) AND ( LIMIT-TO(SUBJAREA,"PSYC" ) OR LIMIT-TO(SUBJAREA,"MEDI" ) OR LIMIT-TO(SUBJAREA,"SOCI" ) OR LIMIT-TO(SUBJAREA,"NEUR" ) OR LIMIT-TO(SUBJAREA,"NURS" ) OR LIMIT-TO(SUBJAREA,"PSYC" ) OR LIMIT-TO(SUBJAREA,"MEDI" ) OR LIMIT-TO(SUBJAREA,"SOCI" ) OR LIMIT-TO(SUBJAREA,"NEUR" ) OR LIMIT-TO(SUBJAREA,"NURS" ) OR LIMIT-TO(SUBJAREA,"HEAL" ) OR LIMIT-TO(SUBJAREA,"DECI" ) OR LIMIT-TO(SUBJAREA,"DENT" ) OR LIMIT-TO(SUBJAREA,"MULT" ) OR LIMIT-TO(SUBJAREA,"PHAR" ) OR LIMIT-TO(SUBJAREA,"VETE" ) OR LIMIT-TO(SUBJAREA,"HEAL" ) OR LIMIT-TO(SUBJAREA,"DECI" ) OR LIMIT-TO(SUBJAREA,"DENT" ) OR LIMIT-TO(SUBJAREA,"MULT" ) OR LIMIT-TO(SUBJAREA,"PHAR" ) OR LIMIT-TO(SUBJAREA,"VETE" ) ) )