View
5
Download
0
Category
Preview:
Citation preview
Assessment: a key tool in health professions education
Cees van der Vleuten
Maastricht University
www.maastrichtuniversity.nl/she
European Midwife Association (EMA) Education Conference
29 November 2013
Disclaimer!
Overview of presentation
Where is education going?
Lessons learned in assessment
Practical implications
Conclusions
Where is education going?
School-based learning
Discipline-based curricula
(Systems) integrated curricula
Problem-based curricula
Outcome/competency-based curricula
Where is education going?
Underlying educational principles:
Continuous learning of, or practicing with, authentic tasks (in steps of complexity; with constant
attention to transfer)
Integration of cognitive, behavioural and affective skills
Active, self-directed learning & in collaboration with others
Fostering domain-independent skills, competencies (e.g. team work, communication, presentation, science orientation, leadership, professional behaviour….).
Where is education going?
Underlying educational principles:
Continuous learning of, or practicing with, authentic tasks (in steps of complexity; with constant
attention to transfer)
Integration of cognitive, behavioural and affective skills
Active, self-directed learning & in collaboration with others
Fostering domain-independent skills, competencies (e.g. team work, communication, presentation, science orientation, leadership, professional behaviour….).
Cognitive
psychology
Constructivism
Cognitive load
theory
Collaborative
learning theory
Empirical evidence
Where is education going?
Work-based learning
Practice, practice, practice….
Optimising learning by:
More reflective practice
More structure in the haphazard learning process
More feedback, monitoring, guiding, reflection, role modelling
Fostering of learning culture or climate
Fostering of domain-independent skills (professional behaviour, team skills, etc).
Where is education going?
Work-based learning
Practice, practice, practice….
Optimising learning by:
More reflective practice
More structure in the haphazard learning process
More feedback, monitoring, guiding, reflection, role modelling
Fostering of learning culture or climate
Fostering of domain-independent skills (professional behaviour, team skills, etc).
Deliberate Practice theory
Social learning theories
Empirical evidence
Where is education going?
Educational reform is on the agenda everywhere
Education is professionalizing rapidly
A lot of ‘educational technology’ is available
How about assessment?
Overview of presentation
Where is education going?
Lessons learned in assessment
Practical implications
Conclusion
Miller’s pyramid of competence
Knows
Shows how
Knows how
Does
Miller GE. The assessment of clinical skills/competence/performance.
Academic Medicine (Supplement) 1990; 65: S63-S7.
Lessons learned while climbing this pyramid with assessment technology
Assessing knowing how
Knows
Shows how
Knows how
Does
Knows
Knows how
60-ies:
Written complex
simulations (PMPs)
Key findings written simulations (Van der Vleuten, 1995)
Performance on one problem hardly predicted performance on another
High correlations with simple MCQs
Experts performed less well than intermediate experts
Stimulus format more important than the response format
Assessing knowing how
Knows
Shows how
Knows how
Does
Knows how
Specific Lessons learned!
Simple short scenario-based formats work best (Case & Swanson, 2002)
Validity is a matter of good quality assurance around item construction (Verhoeven et al 1999)
Generally, medical schools can do a much better job (Jozewicz et al 2002)
Sharing of (good) test material across institutions is a smart strategy (Van der
Vleuten et al 2004).
Moving from assessing knows
Knows: What is arterial blood gas analysis most likely to show in patients with cardiogenic shock? A. Hypoxemia with normal pH B. Metabolic acidosis C. Metabolic alkalosis D. Respiratory acidosis E. Respiratory alkalosis
To assessing knowing how
Knowing How: A 74-year-old woman is brought to the emergency department because of crushing chest pain. She is restless, confused, and diaphoretic. On admission, temperature is 36.7 C, blood pressure is 148/78 mm Hg, pulse is 90/min, and resp are 24/min. During the next hour, she becomes increasingly stuporous, blood pressure decreases to 80/40 mm Hg, pulse increases to 120/min, and respirations increase to 40/min. Her skin is cool and clammy. An ECG shows sinus rhythm and 4 mm of ST segment elevation in leads V2 through V6. Arterial blood gas analysis is most likely to show: A. Hypoxemia with normal pH B. Metabolic acidosis C. Metabolic alkalosis D. Respiratory acidosis E. Respiratory alkalosis
http://www.nbme.org/publications/item-writing-manual.html
Maastricht item review process
anatomy
physiology
int medicine
surgery
psychology
item pool review
committee
test
administration
item analyses
student
comments
Info to users
item bank
Pre-test review Post-test review
Assessing knowing how
Knows
Shows how
Knows how
Does
Knows how
General Lessons learned!
Competence is specific, not generic
Assessment is as good as you are prepared to put into it.
Assessing showing how
Knows
Shows how
Knows how
Does
Knows how
Shows how
70-ies:
Performance
assessment
in vitro (OSCE)
Key findings around OSCEs1
Performance on one station poorly predicted performance on another (many OSCEs are unreliable)
Validity depends on the fidelity of the simulation (many OSCEs are testing testing fragmented skills in isolation)
Global rating scales do well (improved discrimination across expertise groups; better intercase reliabilities; Hodges, 2003)
OSCEs impacted on the learning of students
1Van der Vleuten & Swanson, 1990
Reliabilities across methods
Testing Time in Hours
1
2
4
8
MCQ1
0.62
0.76
0.93
0.93
Case- Based Short Essay2
0.68
0.73
0.84
0.82
PMP1
0.36
0.53
0.69
0.82
Oral Exam3
0.50
0.69
0.82
0.90
Long Case4
0.60
0.75
0.86
0.90
OSCE5
0.47
0.64
0.78
0.88
1Norcini et al., 1985 2Stalenhoef-Halling et al., 1990 3Swanson, 1987
4Wass et al., 2001 5Petrusa, 2002
Checklist or rating scale reliability in OSCE1
Test length In hours
Examiners using Checklists
Examiners using Rating scales
1 0.44 0.45
2 0.61 0.62
3 0.71 0.71
4 0.76 0.76
5 0.80 0.80
1Van Luijk & van der Vleuten, 1990
Assessing showing how
Knows
Shows how
Knows how
Does
Shows how
Specific Lessons learned!
OSCE-ology (patient training, checklist writing, standard setting, etc.; Petrusa 2002)
OSCEs are not inherently valid nor reliable, that depends on the fidelity of the simulation and the sampling of stations (Van der Vleuten & Swanson, 1990).
Assessing showing how
Knows
Shows how
Knows how
Does
Shows how
General Lessons learned!
Objectivity is not the same as reliability (Van der Vleuten, Norman, De
Graaff, 1991)
Subjective expert judgment has incremental value (Van der Vleuten et
al., 2011)
Sampling across content and assessors is eminently important
Assessment drives learning.
Assessing does
Knows
Shows how
Knows how
Does
Shows how
Does
90-ies:
Performance assessment
in vivo by judging work
samples (Mini-CEX, CBD,
MSF, DOPS, Portfolio)
Key findings assessing does
Reliable findings point to feasible sampling (8-10 judgments seems to be the magical number; Williams et al 2003; Moonen et al, 2013)
Scores tend to be inflated (Dudek, 2005; Govaerts et al 2007)
Qualitative/narrative information is (more) useful (Govaerts et al 2007; Sargeant et al 2011)
A lot of the feedback given is poorly (Pelgrim et al, 2011)
Non-credible feedback is ignored (Watling et al, 2011)
Feedback works best in a dialogue (Watling et al., 2012).
Reliabilities across methods
Testing Time in Hours
1
2
4
8
MCQ1
0.62
0.76
0.93
0.93
Case- Based Short Essay2
0.68
0.73
0.84
0.82
PMP1
0.36
0.53
0.69
0.82
Oral Exam3
0.50
0.69
0.82
0.90
Long Case4
0.60
0.75
0.86
0.90
OSCE5
0.47
0.64
0.78
0.88
Practice Video
Assess- ment7
0.62
0.76
0.93
0.93
1Norcini et al., 1985 2Stalenhoef-Halling et al., 1990 3Swanson, 1987
4Wass et al., 2001 5Petrusa, 2002 6Norcini et al., 1999
In- cognito
SPs8
0.61
0.76
0.92
0.93
Mini CEX6
0.73
0.84
0.92
0.96
7Ram et al., 1999 8Gorter, 2002
Assessing does
Knows
Shows how
Knows how
Does Does
Specific Lessons learned!
Reliable sampling is possible
Qualitative information carries a lot of weight
Assessment impacts on work-based learning (more feedback, more reflection…)
Validity strongly depends on the users of these instruments and therefore on the quality of implementation.
Assessing does
Knows
Shows how
Knows how
Does Does
General Lessons learned!
Work-based assessment cannot replace standardised assessment (yet), or, no single measure can do it all (Tooke report, UK)
Validity strongly depends on the implementation of the assessment (Govaerts et al 2007)
But, there is a definite place for (more subjective) expert judgment (Schuwirth
& van der Vleuten, 2012; Govaerts et al 2013).
Competency-frameworks
CanMeds Medical expert
Communicator
Collaborator
Manager
Health advocate
Scholar
Professional
ACGME Medical knowledge
Patient care
Practice-based learning & improvement
Interpersonal and communication skills
Professionalism
Systems-based practice
GMC Good clinical care
Relationships with patients and families
Working with colleagues
Managing the workplace
Social responsibility and accountability
Professionalism
Measuring the unmeasurable
Knows
Shows how
Knows how
Does
“Domain independent” skills
“Domain specific” skills
Measuring the unmeasurable
Importance of domain-independent skills
If things go wrong in practice, these skills are often involved (Papadakis et 2005; 2008)
Success in labour market is associated with these skills (Meng 2006)
Practice performance is related to school performance (Padakis et al 2004).
Measuring the unmeasurable
Knows
Shows how
Knows how
Does
“Domain independent” skills
“Domain specific” skills
Assessment (mostly
in vivo) heavily relying on professional judgment and
qualitative information
Measuring the unmeasurable
Self assessment Peer assessment Co-assessment (combined self, peer, teacher
assessment) Multisource feedback Log book/diary Learning process simulations/evaluations Product-evaluations Portfolio assessment
Eva, K. W., & Regehr, G. (2005). Self-assessment in the health professions: a reformulation and research agenda. Acad Med, 80(10 Suppl), S46-54.
Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis
comparing peer and teacher marks. Review of Educational Research, 70(3), 287-322.
Driessen, E., van Tartwijk, J., van der Vleuten, C., & Wass, V. (2007). Portfolios in medical education: why do they meet
with mixed success? A systematic review. Med Educ, 41(12), 1224-1233.
Overview of presentation
Where is education going?
Lessons learned in assessment
Practical implications
Conclusions
General lessons learned
Competence is specific, not generic Assessment is as good as you are prepared to put
into it Objectivity is not the same as reliability Subjective expert judgment has incremental value Sampling across content and judges/examiners is
eminently important Assessment drives learning No single measure can do it all Validity strongly depends on the implementation of
the assessment
Practical implications 1
Competence is specific, not generic One measure is no measure Increase sampling (across content, examiners,
patients…) within measures Combine information across measures and across
time Be aware of (sizable) false positive and negative
decisions Build safeguards in examination regulations.
Practical implications 2
Assessment is as good as you are prepared to put into it Train your staff in assessment Implement quality assurance procedures around
test construction Share test material across institutions Reward good assessment and assessors Involve students as a source of quality assurance
information
Practical implications 3
Objectivity is not the same as reliability Don’t trivialize the assessment (and compromise
on validity) with unnecessary objectification and standardization
Don’t be afraid of holistic judgment Sample widely across sources of subjective
influences (raters, examiners, patients)
Practical implications 4
Subjective expert judgment has incremental value Use expert judgment for assessing complex skills Who is an expert depends on assessment context
(i.e. peer, patient, clerk, etc) Invite assessors to provide qualitative information
or mediation of feedback
Practical implications 5
Sampling across content and judges/examiners is eminently important Use efficient test designs: use single examiners
per test item (question, essay, station, encounter…) and different examiners across items
Psychometrically analyse sources of variance affecting the measurement to optimise the sampling plan and sample sizes needed
Practical implications 6
Assessment drives learning For every evaluative action there is an educational
reaction Verify and monitor the impact of assessment
(evaluate the evaluation); many intended effects are not actually effective -> hidden curriculum
No assessment without feedback! Embed the assessment within the learning
programme Use the assessment strategically to reinforce desirable learning behaviours
Practical implications 7
No single measure can do it all Use a cocktail of methods across the competency
pyramid Arrange methods in a programme of assessment Any method may have utility (including the ‘old’
assessment methods depending on its utility within the programme)
Compromises on the quality of methods should be made in light of its function in the programme
Compare assessment design with curriculum design
Responsible people/committee(s) Use an overarching structure Involve your stakeholders Implement, monitor and change (assessment programmes
‘wear out’)
Practical implications 8
Validity strongly depends on the implementation of the assessment Pay special attention to implementation (good
educational ideas often fail due to implementation problems)
Involve your stakeholders in the design of the assessment
Many naive ideas exist around assessment; train and educate your staff and students.
Overview of presentation
Where is education going?
Lessons learned in assessment
Practical implications
Conclusions
Conclusions
Assessment in health education has a rich history of research and development with clear practical implications (we’ve covered some ground in 40 yrs!)
We are moving beyond the psychometric discourse into an educational design discourse
We are starting to measure the unmeasurable
Expert professional judgment is reinstated as an indispensable source of information both at the method level as well as at the programmatic level
Lots of exciting developments lie still ahead of us!
“Did you ever feel you’re on the verge ofan incredible breakthrough?”
www.fdg.unimaas.nl/educ/cees/ema
Literature
Cillier, F. (In preparation). Assessment impacts on learning, you say? Please explain how. The impact of summative assessment on how medical students learn.
Driessen, E., van Tartwijk, J., van der Vleuten, C., & Wass, V. (2007). Portfolios in medical education: why do they meet with mixed success? A systematic review. Med Educ, 41(12), 1224-1233.
Driessen, E. W., Van der Vleuten, C. P. M., Schuwirth, L. W. T., Van Tartwijk, J., & Vermunt, J. D. (2005). The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study. Medical Education, 39(2), 214-
220. Dijkstra J, Van der Vleuten CP, Schuwirth LW. A new framework for designing programmes of assessment. Adv Health Sci Educ
Theory Pract. 2009 Oct 10. Eva, K. W., & Regehr, G. (2005). Self-assessment in the health professions: a reformulation and research agenda. Acad Med,
80(10 Suppl), S46-54. Gorter, S., Rethans, J. J., Van der Heijde, D., Scherpbier, A., Houben, H., Van der Vleuten, C., et al. (2002). Reproducibility of
clinical performance assessment in practice using incognito standardized patients. Medical Education, 36(9), 827-832. Govaerts, M. J., Van der Vleuten, C. P., Schuwirth, L. W., & Muijtjens, A. M. (2007). Broadening Perspectives on Clinical
Performance Assessment: Rethinking the Nature of In-training Assessment. Adv Health Sci Educ Theory Pract, 12, 239-260. Hodges, B. (2006). Medical education and the maintenance of incompetence. Med Teach, 28(8), 690-696. Jozefowicz, R. F., Koeppen, B. M., Case, S. M., Galbraith, R., Swanson, D. B., & Glew, R. H. (2002). The quality of in-house
medical school examinations. Academic Medicine, 77(2), 156-161. Meng, C. (2006). Discipline-specific or academic ? Acquisition, role and value of higher education competencies., PhD
Dissertation, Universiteit Maastricht, Maastricht. Norcini, J. J., Swanson, D. B., Grosso, L. J., & Webster, G. D. (1985). Reliability, validity and efficiency of multiple choice
question and patient management problem item formats in assessment of clinical competence. Medical Education, 19(3), 238-247.
Papadakis, M. A., Hodgson, C. S., Teherani, A., & Kohatsu, N. D. (2004). Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board. Acad Med, 79(3), 244-249.
Papadakis, M. A., A. Teherani, et al. (2005). "Disciplinary action by medical boards and prior behavior in medical school." N Engl J Med 353(25): 2673-82.
Papadakis, M. A., G. K. Arnold, et al. (2008). "Performance during internal medicine residency training and subsequent disciplinary action by state licensing boards." Annals of Internal Medicine 148: 869-876.
Literature
Petrusa, E. R. (2002). Clinical performance assessments. In G. R. Norman, C. P. M. Van der Vleuten & D. I. Newble (Eds.), International Handbook for Research in Medical Education (pp. 673-709). Dordrecht: Kluwer Academic Publisher.
Ram, P., Grol, R., Rethans, J. J., Schouten, B., Van der Vleuten, C. P. M., & Kester, A. (1999). Assessment of general practitioners by video observation of communicative and medical performance in daily practice: issues of validity, reliability and feasibility. Medical Education, 33(6), 447-454.
Stalenhoef- Halling, B. F., Van der Vleuten, C. P. M., Jaspers, T. A. M., & Fiolet, J. B. F. M. (1990). A new approach to assessign clinical problem-solving skills by written examination: Conceptual basis and initial pilot test results. Paper presented at the Teaching and Assessing Clinical Competence, Groningen.
Swanson, D. B. (1987). A measurement framework for performance-based tests. In I. Hart & R. Harden (Eds.), Further developments in Assessing Clinical Competence (pp. 13 - 45). Montreal: Can-Heal publications.
van der Vleuten, C. P., Schuwirth, L. W., Muijtjens, A. M., Thoben, A. J., Cohen-Schotanus, J., & van Boven, C. P. (2004). Cross institutional collaboration in assessment: a case on progress testing. Med Teach, 26(8), 719-725.
Van der Vleuten, C. P. M., & D. Swanson, D. (1990). Assessment of Clinical Skills With Standardized Patients: State of the Art. Teaching and Learning in Medicine, 2(2), 58 - 76.
Van der Vleuten, C. P. M., & Newble, D. I. (1995). How can we test clinical reasoning? The Lancet, 345, 1032-1034. Van der Vleuten, C. P. M., Norman, G. R., & De Graaff, E. (1991). Pitfalls in the pursuit of objectivity: Issues of reliability. Medical
Education, 25, 110-118. Van der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005). Assessment of professional competence: from methods to programmes.
Medical Education, 39, 309-317. Van der Vleuten, C. P. M., & Schuwirth, L. W. T. (Under editorial review). On the value of (aggregate) human judgment. Med
Educ. Van Luijk, S. J., Van der Vleuten, C. P. M., & Schelven, R. M. (1990). The relation between content and psychometric
characteristics in performance-based testing. In W. Bender, R. J. Hiemstra, A. J. J. A. Scherp bier & R. P. Zwierstra (Eds.), Teaching and Assessing Clinical Competence. (pp. 202-207). Groningen: Boekwerk Publications.
Wass, V., Jones, R., & Van der vleuten, C. (2001). Standardized or real patients to test clinical competence? The long case revisited. Medical Education, 35, 321-325.
Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine, 15(4), 270-292.
Recommended