Embed Size (px)
Citation preview
21/12/2015
1
Evaluation and Impact of IPE: How IPE might improve quality of careProfessor Liz Anderson
Disclaimer: PRESENTING AUTHORS HAVE NO RELATIONSHIPS TO DISCLOSE
University of Leicester
1995 2000 2002 2005 2012/3 2015
Collaborative working across three HEI’s
Our Story - South Trent (Midlands UK)
Our Local IPE Curriculum
Strand Three• Patient safety workshops/simulations• Interprofessional care planning• Polypharmacy• Mental health• Homeless project• Child health
Strand Two
• Health in the Community• Listening workshop• Workshop early years education
.
Strand One• Introduction to team working• Communication
Aims of Strand Oneo To explore what is meant by team working in
health and social careo To begin to apply a theoretical understanding to
team worko To become familiar with your chosen profession
and otherso To consider the outcomes of team working for
promoting person-centred collaborative care.
Aims of Strand Twoo To apply the theoretical basis of team workingo To gain a richer appreciation of roles and
responsibilities of practitionerso To analyse effective collaborative team practiceo To consider your future contribution to person
centred team working
Assessed for developing
interprofessional competence using an IPE Portfolio with additional professional requirements
Aims of Strand Threeo To provide context for applying working
competencies to practiceo Analyse and reflect on real situations and consider
solution to improve team-based careo To understand how individual professional
competencies complement each othero To develop an understanding of team working in
modern health and social care practice
Anderson, Ewing and Moore (2013) http://caipe.org.uk/silo/files/iperg-review-15-4-14--download-printing.pdf
21/12/2015
2
Deans of the 3 Universities with local health commissioners sign Strategy, summer 2004.
This talk
AimTo consider what it means to evaluate Interprofessional Education for improving patient care
Objectives• Evaluation: Steps to take• Assessment: What is possible
Reminder of the definition of IPE
“Occasions when two or more professions learn with, from and about each other to improve
collaboration and the quality of care”
Barr H, 2002, Interprofessional Education Today, Yesterday and Tomorrow, LTSN, Occasional Paper 1, pg 6. (Revised CAIPE 1997).UK Centre for the Advancement of Interprofessional Education (CAIPE) website: www.caipe.org.uk (2002).
The key question
The gold standard
Do we know what it will look like when we get there?
21/12/2015
3
Reflection
Helpful Reviews
• Best Evidence Systematic Review of IPE: BEME Guide no.9. Hammick et al., 2007‐ 21. High quality studies.
• Interprofessional education: Effects on professional practice and Health care outcomes. Cochrane Review. Reeves et al., 2008 ‐ 6 studies
• Interprofessional collaboration: Effects of practice‐based interventions on professional practice and healthcare outcomes. Cochrane Review. Zwarenstein et al., 2009 ‐ 5 studies
• Interprofessional education: Effects on professional practice and health care outcomes (update) Reeves et al., 2013 ‐ 15 studies
BEME Review: Getting started
• Are students learning from one another? If so what? What occasions help and what are the essential ingredients for effective IPE?
Review Outcomes: Looking at the educational processMethodology led to understanding the required ingredients for effective IPE relating to teaching, design and delivery
o Request more rigorous evaluation studies
• What does collaboration look like? Is there any measure to help us know how to improve this?
• What does quality of care mean here? Does a dose of IPE help?Outcomes: Comparing IPE with no IPE• Impact on service and service organisation (RCT, CBA and ITS)• Methodology showed impact in clinical areas e.g. diabetes• Require more than just comparative studies
o More comparisons with qualitative measureso Cost‐benefit analysis
Cochran Reviews
21/12/2015
4
Evaluation Literature
• Frye, A. & Hemmer, P. (2015). Program evaluation models and related theories: AMEE Guide No 67. Medical Teacher, e288‐299.
• Anderson, E., Smith, R. & Hammick, M. (2015). Evaluating an interprofessional education curriculum: A theory‐informed approach. Medical Teacher, 36: 495–504.
• Reeves, S., Boet, S., Zierler, B. Kitto, S. (2015). Interprofessional Education and Practice Guide No. 3: Evaluating Interprofessional Education. Journal of Interprofessional Care, 305‐312.
• Thistlethwaite, J., Kumar, K., Moran, M., Saunders, R. & Carr, S. (2015). An exploratory review of pre‐qualification interprofessional education evaluations. Journal of Interprofessional Care, 29(4), 282‐297.
• Thisthethwaite J. (2015). Assessment of Interprofessional Teamwork ‐ An International Perspective. In: Ed’s Forman, D, Jones, M. & Thistlethwaite J. Leadership and Collaboration: Further development for interprofessional Education. Palgrave Macmillan: Basingstoke Hampshire UK.
What is evaluation?
“…the systemic collection and analysis of information related to the design, implementation and outcomes of a program, for the purposes of monitoring and improving the quality and effectiveness of the program”.Accreditation council for graduates medical education glossary of terms. http://www.acgme.org/acWedsite/about/ab_ACGMEglossary.pdf
Why do evaluation?
• To discover if change has occurred– For accrediting body and funder requirements– Allows educators to learn about their teaching – To determine what makes for success
(are we reaching quality standards for IPL?)
– To consider what actions might need to be taken to make improvements (not everything that is negative means it is not worthwhile)
Steps
1. Establish an evaluation team2. Apply a theoretical framework3. Establish the aims and purpose of the evaluation4. Outline data collection methods and tools5. Ensure ethical principles and governance6. Use continuous cycles of data collection7. Feedback the findings to make changes and
disseminate
21/12/2015
5
Step 1: Establish an evaluation team
Step 1
‘Think about Evaluationas early as possible’
• Independent researcher• Research committee• External consultant• Ethical permission
Step 2. Apply a theoretical framework
Theoretical Frameworks
• Theories help us think about relationships taking place within the complexity of IPE– Offer clarity about the investigation– Critical exploration
• Theories inform the development of your evaluation framework/model
21/12/2015
6
General theories with examples
These theories look to the richness of and interconnectivity in the education process
Systems Theory: Generates the notion that change is an inherent part of a system
Complexity Theory: Acknowledges the myriad of factors that impact on an education programme
General theories with examples
• Systems Theory Logic Model: Considers Inputs – Activities ‐ Outputs and
Outcomes (Frechtling, 2007) CIPP Model: Context/input/process/product. Consider non‐
linear relationships (Stufflebeam & Shinkfield, 2007)3 P’s Model: Presage, Process and Product (Biggs, 1993).
Interplay and relationships between the different components of learning
Realist Evaluation: An in‐depth analysis of what works for whom and in what context; creating an appreciation of Context‐Mechanism‐Output (Pawson & Tiley, 1997)
Theory
• Reductionism: The whole or outcome can be understood by investigating its constituent parts. This follows a linear approach; cause and effect between programme elements and desired outcomes
• Sociological/psychological: Exploring social situations such as power relationships; in psychology concerning individual differences e.g. Witz Model of professional closure (1992)
Theory
ReductionistQuasi‐experimental models (Stufflebeam & Shinkfield
1960’S): Linear relationships pre and post testing randomised groups. Fail to tell you why particular observations were identified.
Kirkpatrick Model (1996): Clarity of focus to outcomes; hierarchical levels learner related outcomes. Adopted by the JET team and has been used for evaluating the outcomes of IPE (Barr et al., 2000; Barr et al., 1999). BEME review.
21/12/2015
7
Kirkpatrick Modified by the Joint Evaluation (JET) Team
1. Reaction Learner view on the learning experience and its interprofessional nature
2. a Modification of attitudes/perceptions
Changes in reciprocal attitudes or perceptions between participant groups. Changes in perception or attitude towards the value and/or use of team approaches to caring for a specific client group
2 .b. Acquisition of knowledge and skills
Including knowledge and skills linked to interprofessional collaboration
3. Behavioural change Identified individuals’ transfer of interprofessional learning to their practice setting and changed professional practice
4. a change in organisational practice Wider changes in the organisation and delivery of care
4. b. Benefits to patients/clients Improvements in health or well being of patients/clients
Originally designed for evaluating training in a business organisation
Step 2. Apply a theoretical framework
Curriculum evaluation model
Model of curriculum design (from Coles & Grant 1985)
Curriculum on Paper (A)
Curriculum in Action (B)
Curriculum Experienced by Learners
(C)
Our framework
Evaluation Matrix Framework
InputPresage
Process
OutputProductKirkpatrick Outcome Levels
Students2005 year one study group
StakeholdersFaculty Members,(academic and practice educators), Practitioners and patients
The IP experience
• Personal to stakeholder• Higher Education
Institution • Students
Advanced Patient Care
Validated measures
Assessments
Practice behaviour
Cost of participation
Effects of participationa) Experienced teachers and
novice educators
b) Practitioners and patients
21/12/2015
8
3. Aims and purpose of the evaluation
Aims and purpose
• Part A: Assessment of the learning (one element of evaluation)– Formative and summative
• Part B: Educational research into what is going on (educational/clinical research)
Part A: The first requirement
Knowsknowledge
Knows Howcompetence
Doesaction
Identity
Shows Howperformance
Framework for clinical assessment: Miller (1990)
Novice
Expert
Blooms learning domains (1956)
Cognitive mental knowledge
Psychomotor manual or physical skills
Affective , feeling and emotional
21/12/2015
9
We understand knowledge measurement but we are very poor on our understanding of performance behaviours.
Have we begun to tell students exactly what the goals of IPE are in terms of their behaviours in practice settings? For Example, what does becoming a team player mean exactly? Do I know my preferred Belbin role?
Assessment of IPE
‘Practice ready collaborative practitioners’ (WHO, 2010)
• Competency‐based assessmentConstructivist principles of alignment
Mastery of one domain/competence or all? Individual or within
a team?
Level of competence or capability; what is
the minimum standard?
Similarly to professionalism
abstract concepts are difficult to define and
measure
How will you measure collaboration?
Concerns about administrating valid, reliable, acceptable feasible assessments
Alignment
A: Learning Outcomes
B: Teaching
C. Assessment
of A/B
Presage Process Product
21/12/2015
10
Part B: Evaluation questions further educational research
• What impact is this teaching having on faculty members?
• Which type of learning helps students think differently?
• How is IPE benefitting patients and service design?
4. Data collection methods and toolsA ‐ assessmentB ‐ evaluation research
21/12/2015
11
A: Assessment Measures
• Teamwork assessment measures– Simulation: iT‐OSCE: Newly formed team being assessed?
– Peer and self‐assessment frameworks (SPARKplus) by Freeman and McKenzie 2002; Sheffield Peer Review Assessment Tool (SPRAT) Archer 2005
– Work Based Assessment Tools: Used in medical education not fully understood for IPE
Tools: Research and Assessment
• Recent Australian work‐based assessment: iSTAT toolhttp://www.olt.gov.au/resource‐work‐based‐assessment‐teamwork.
• Interprofessional Capability Tool (Curtin‐Australia)https://healthsciences.curtin.edu.au/local/docs/Interprofessional_Capability_Assessment_Tool_2014.pdf
Feedback: Paying more attention to learning and the need for formative assessments with dialogue between student and teacher
• Research as Inquiry: Research is iterative and depends upon asking increasingly complex or new questions whose answers in turn develop additional questions or lines of inquiry
• Scholarship as Conversation: Communities of scholars, researchers, or professionals engage in sustained discourse with new insights and discoveries occurring over time as a result of varied perspectives and interpretations
http://www.ala.org/acrl/standards/ilframework
B: Evaluation‐Research
21/12/2015
12
Research Questions
• The research we need to do…http://whoeducationguidelines.org/blog/moving‐interprofessional‐education‐forward
Research Questions
–How do collaborators think about their professions?
–How do collaborators talk about their professions?
–How do collaborators view the “good” of what they do?
–How do collaborators “conceptualise” their interactions?
Main Challenges
• More observational studies• More economics of IPE collaboration• More exploration of organisational and personal behaviour in teams (psychology and systems)
Main Challenges
• More cross‐sectional• More longitudinal studies• Studies that articulate a theory of collaboration• Studies that show the benefits for patients/communities
21/12/2015
13
Evaluation Methods
• Quantitative Data– CBA: Controlled study before/after and later time period
– RCTs: Randomised control trials– Mixed methods: Scored Likert scales with qualitative data
– Psychometric instruments ‐ Tools
Tools: Research
• Problem ‐ vast majority are for qualified teamsNeed to ask question of psychometrics
– CIHC (2012): Overview of instruments for measuring qualified team performance
– TeamSTEPPS teamwork attitudes questionnaire
http://rcrc.brandeis.edu/pdfs/Canadian%20Interprofessional%20Health%20Collaborative%20report.pdf
Methods
• Qualitative Data– Focus groups and interviews– Action Research– Ethnography– Grounded theory
21/12/2015
14
5. Ethical principles and governance
6. Use continuous cycles of data collection
Part B: What is happening (processes)
Systematic data collection to consider: Improvement, monitoring, justification and change.
21/12/2015
15
What is happening?
Why students might not like IPE?• Non‐alignment• Misunderstood• Teacher issues/faculty issues• Mature students• The myriad of issues bringing students together without forethought and understanding
Theories to help explain
• Four Channel Flow Model: Engaging learners
Anxiety Flow
Apathy Boredom
Competence/Skill
Challenge (risk)
Csikszentmihalyi, M 1990
Research Challenges
• The community of interprofessional educators has matured over the past 50 years
• Scholarship has become increasingly important• Research is being challenged to align with theoretical bases (sociology, philosophy, anthropology, economics, political science)
Final thoughts
• IPE: Iterative development, using local context and possibilities
21/12/2015
16
References• Anderson, ES., Thorpe, LN., Heney, D. & Petersen, S. (2009). Medical Students benefit from
learning about patient Safety in an interprofessional team. Medical Education, 4, 542-552• Anderson, ES., Ewing, A. & Moore, C . Our Story of Pre-registration Interprofessional
Education within South Trent: Leicestershire, Rutland and Northamptonshire. Pages 65-74. In: Barr, H., Helme, M. D’avary, L. Review of Interprofessional Education in the United Kingdom 1997-2003. CAIPE Publication. February 2014 http://caipe.org.uk/silo/files/iperg-review-15-4-14--download-printing.pdf
• Anderson, ES. & Lennox, A. (2009). The Leicester Model of Interprofessional education: Developing, Delivering and Learning from student voices for 10 years. Journal of Interprofessional Care, 23(6), 557-573
• Anderson, ES., Hean, S., O’Halloran, C., Pitt, R. Hammick, M. (2014). Faculty Development for Interprofessional Education and Practice, Chapter 14.: In Steiner, Y. (Eds). ‘Faculty Development in the Health Professions: A focus on Research and Practice’. Springer.
• Barr, H., Freeth, D., Hammick, M., Koppel, I. & Reeves, S. (2000). Evaluations for interprofessional education. A United Kingdom review for health and social care. London:ACIPE/BERA
• Barr, H., Hammick, M., Koppel, I. & Reeves, S. (1999). Evaluating Interprofessional Education: Two systematic reviews for health and social care, British Educational Research Journal, 25, 533-543.
• Barr H, 2002, Interprofessional Education Today, Yesterday and Tomorrow, LTSN, Occasional Paper 1, pg 6. Revised from CAIPE 1997. www.caipe.org.uk (2002).
• Biggs, J. (1993). From theory to practice: A cognitive systems approach . Higher Education Research and Development, 12, 73-85.
• Biggs, J. & Tang, C. (2007). Teaching for Quality Learning at University, 3rd eds. Open University Press.
• Coles, C. R. & Grant, J. G. (1985). Curriculum evaluation in medical and health-care education. Medical Education, 19(5), 405-422.
• Csikszentmihalyi, M (1990). Flow, the psychology of optimal experience. New York: Harper Perennial Modern Classics.
• Dewey, J. (1938) Experience and education. New York: Collier.• Frechtling, J. (2007). Logic Modelling in programme evaluation, San Francisco: John
Wiley & Sons.• Freeman, M. & McKenzie, J. (2002). SPARK, a confidential web-based for self and peer
assessment of student teamwork: Benefits of evaluations across different subjects. British Journal of Educational Technology, 33, 551-69
• Hammick M., Freeth, D., Koppel, I., Reeves, S. & Barr H. (2008). Best evidence systematic review of interprofessional education. BEME Guide 9.
• Hean, S., Craddock, D., O’Halloran, C. (2009). Learning theories and interprofessional education: a user’s guide. Learning in Health and Social Care, 8(4), 250-262.
• Kolb, D.A. (1984). Experiential Learning – Experience as a source of Learning and Development, Prentice Hall.
• Kirkpatrick, D. (1996). Revising Kirkpatrick four-level model. Train Dev, 1, 54-59.• Ladden M.D., Bednash, G., Steves, D.P., N., Moore, G.P. (2006). Educating
interprofessional learners for quality, safety and systems improvement. Journal of Interprofessional Care, 20(5), 497–505.
• Irvine, D (2003). The Doctors Tale: Professionalism and Public Trust. Radcliffe Books: Oxford
• Paavola & Hakkarainen (2005). The knowledge creation metaphor-an emergent epistemological approach to learning. Science and Education, 14, 537-557.
• Palaganas, J., Epps,C., Raemer, DB. (2014). A history of simulation-enhanced interprofessional education. Journal of Interprofessional Care, 28(2), 110-115
• Piaget, J. (1973). To Understand is to Invent. Grossman, New York.• Reeves, S., Zwarenstein, M., Goldman, J., Barr H., Freeth, D., Hammick, M. &
Koppel, I. (2008). Interprofessional educaion: effects on professional practice and health care outcomes. Cchhrane Collaboration: 1469-493x: 1-21.
• Reeves, S., Zwarenstein, M., Goldman, J., Barr H., Freeth, D., Hammick, M. & Koppel, I. (2008). Interprofessional educaion: effects on professional practice and health care outcomes. Cchhrane Collaboration: 1469-493x: 1-21.
• Reeves, E., Perrier, L., Goldman, J., Freeth, D. & Zwarenstein, M. (2013). Interprofessional education: effects on professional practice and healthcare outcomes (update) Cochrane Review. Chochrane Collaboration: CD002213: 1-17.
• Schőn, D.A. (1987). Educating the reflective practitioners: Towards a new design for teaching and learning in the professions. San Francisco, CA: Jossey-Bass.
• Stufflebeam, D & Shinkfield A, (2007). Evaluation theory, models and applications. San Francisco: Jossey Bass/ John Wiley & Sons, Inc.
• Vygotsky, L (1978). Mind in Society. Harvard University Press, Cambridge.
21/12/2015
17
• Wackerhausen, S. (2009). Collaboration, professional identity and reflection across boundaries. Journal of Interprofessional Care, 23, 244-473.
• Witz, A. (1992). Professions and patriarchy. London: Routledge.• World Health Organisation (2010). Framework for Action on Interprofessional
Education & Collaborative Practice. WHO, Geneva. p13.• Zwarenstein, M., Goldman, J. & Reeves, S. (2009). Interprofessional collaboration:
effects of practice-based interventions on professional practice and healthcare outcomes. Cochrane Collaboration: 1469-493X: 1-31.
THANK‐YOU
