Embed Size (px)
Citation preview
Universiteit Maastricht
Barcelona, 6 – 9 July 2004
11th Ottawa conference onMedical Education
Universiteit Maastricht
where do we go from here?
Where do we go from here?
Lambert Schuwirth
new challenges in assessmentof clinical competence
Universiteit Maastricht
overview of the presentation
- why this presentation
Where do we go from here?
- developments in competence assessment
- implications for research
- developments in performance assessment
Universiteit Maastricht
overview of the presentation
Where do we go from here?
- why this presentation
- developments in competence assessment
- implications for research
- developments in performance assessment
Universiteit Maastricht
experts vs novices: context vs concept
Where do we go from here?
Fz
Ff
1,25 kg
Universiteit Maastricht
overview of the presentation
Where do we go from here?
- why this presentation
- developments in competence assessment
- implications for research
- developments in performance assessment
Universiteit MaastrichtWhere do we go from here?
simple isn’t it?
test
pass
fail
competent
incompetent
Universiteit MaastrichtWhere do we go from here?
simple, it isn’t
competent incompetent
pass
fail
modern approaches to assessment
Universiteit Maastricht
1 re-evaluate the nature of medical competence
2 shift towards more student-centred assessment
3 include general professional competencies
4 broaden the measurement time frame
Where do we go from here?
1 trait orientated view on medical competence
Universiteit Maastricht
knowledge
instrument
instrument
skills
instrument
instrument
problem solving
instrument
instrument
instrument
attitudes
instrument
instrument
instrument
Where do we go from here?
one instrument for one trait
Universiteit MaastrichtWhere do we go from here?
instruments
traits
A B C D
1
2
3
4
one instrument for one trait
Universiteit MaastrichtWhere do we go from here?
old model of expertise: problems
Universiteit MaastrichtWhere do we go from here?
more variation within than between methods
1 trait orientated view on medical competence
Universiteit Maastricht
knowledge
instrument
instrument
skills
instrument
instrument
problem solving
instrument
instrument
instrument
attitudes
instrument
instrument
instrument
Where do we go from here?
1 trait orientated view on medical competence
Universiteit Maastricht
knowledge
instrument
instrument
skills
instrument
instrument
problem solving
instrument
instrument
instrument
attitudes
instrument
instrument
instrument
Where do we go from here?
instrumentinstrument
old model of expertise: problems
Universiteit MaastrichtWhere do we go from here?
more variation within than between methods
content determines the assessed competence
Universiteit Maastricht
knows
knows how
shows how
does
Miller's pyramid
Where do we go from here?
Universiteit Maastricht
2 shift towards student-centred assessment
- self-assessment
- peer-assessment
- co-assessment
self-evaluation skills
assessing the unmeasurable
assessing the unmeasurable
peer pressure?
peer-evaluation skills
self, peer and co-evaluation skills
peer pressure?
joint responsibility
Where do we go from here?
Universiteit Maastricht
3 include general professional competencies
dealing with tasks
dealing with others
dealing with oneself
planning and time management
information gathering
co-operation
communicating
giving feedback and criticising
self-responsibility
self-evaluation
self-reflection
Where do we go from here?
Universiteit Maastricht
4 broaden the measurement time frame
specific test occasions - specific preparation of students
- inflexibility
longitudinal testing
- unwanted influence on learning
- certain aspects unmeasurable
- no specific preparation
- congruent with flexible curricula
- wanted influence on learning
Where do we go from here?
Universiteit Maastricht
lessons learnt
Where do we go from here?
good assessment draws the whole picture
no single method is a panacea
the content is important, not the format
too much structure does not work
sampling strategies are essential sampling strategies are essential
Universiteit Maastricht
concept and context: subjectivity
Where do we go from here?
experts
LambertMozart
Universiteit MaastrichtWhere do we go from here?
sampling strategies are essential sampling strategies are essential
how do you feel now
how did you feel 5’ ago
how did you feel 10’ ago
how will you feel in 5’
how will you feel in 10’
what does your partner about think how you feel
Lambert Neil Dale
7
7
7
7
7
7
5
5
5
5
5
5
9
9
9
9
9
9
Universiteit Maastricht
lessons learnt
Where do we go from here?
sampling strategies are essential sampling strategies are essential
good assessment shows the whole picture
no single method is a panacea
the content is important, not the format
too much structure does not work
Universiteit Maastricht
overview of the presentation
Where do we go from here?
- why this presentation
- developments in competence assessment
- implications for research
- developments in performance assessment
Universiteit Maastricht
definitions
Where do we go from here?
competencewhat professionals can do in controlled representationsof professional practice, performing at the maximum of their ability
performancewhat professionals actually do in their professional practice
Universiteit MaastrichtWhere do we go from here?
1 clinical expertise
2 communication skills
3 collaboration skills
4 management skills
6 educational skills
7 professional attributes
8 personal health & self care
5 personal development
current approaches: still trait orientated
Universiteit MaastrichtWhere do we go from here?
instrumentschart review
data review
interviews
rating scales
surveys
video
portfolios
incognito simulated patiens
critical incident analyses
Universiteit MaastrichtWhere do we go from here?
but…….
so…….
no direct link between instrument and trait
domains are not discrete entities
trait-orientated approach did not work in competence assessment
modern approaches define competence and performance in terms of professional roles
teacher
Universiteit MaastrichtWhere do we go from here?
patient care
person
professional role-orientated goals
scientist
health care management
Universiteit MaastrichtWhere do we go from here?
1 role – 1 instrument?
instruments
roles
A B C D
1
2
3
4
Universiteit MaastrichtWhere do we go from here?
multi-modal assessment
roles
A B C D
1
2
3
4
sources of information
Universiteit MaastrichtWhere do we go from here?
how: seeing the whole picture
Universiteit MaastrichtWhere do we go from here?
how: approaches
- variation in instruments needed
- performance often involves subjective aspects
-collation of information based on content-equal bits and not on method-equal bits
no flight into structuring, but in sampling
Universiteit MaastrichtWhere do we go from here?
how: organisationplan
purposes
elements of defensibility
quality control measures
pathways
implementation
administration
consequences
assessee
Universiteit MaastrichtWhere do we go from here?
observations
assessment is not mainly a measurement problem but more an (educational) design problem.
instrument is not synonymous with a competence
no single instrument is the panacea, a variety isneeded: an assessment programme
content is more important than format:it is the concept that counts, not the format
Universiteit MaastrichtWhere do we go from here?
suggestionsassessment is not mainly a measurement problem but more an educational designproblem.
- built-in quality control procedures and bodies
- sampling in stead of (over) structuring
- variation of methods, compensation, regulations
- saturation of information approach
- training of students, teachers, examiners, board members
masterplan of the assessment system
Universiteit Maastricht
overview of the presentation
Where do we go from here?
- why this presentation
- developments in competence assessment
- implications for research
- developments in performance assessment
Universiteit MaastrichtWhere do we go from here?
trait orientated
assessment as a psychological testing problem
- highly quantitative methodologies
- reliability of methods
- construct validity
+++
+++
+++
Universiteit MaastrichtWhere do we go from here?
trait orientated
assessment as a instructional design problem
- quantitative methodologies
- reproducibility of methods
- construct validity
- qualititative methodologies
- consequential validity
- research into assessment systems
- direct validity
+
+
+
±
++
+++
++++++
Universiteit MaastrichtWhere do we go from here?
observations
research of assessment systems is more importantnow than research into single methods.
methodology is not synonymous to research question
no single one study/methodology will provide thedefinitive answer, a variety is needed
the methodological concept is important not the specific analysis or method
Universiteit MaastrichtWhere do we go from here?
research agenda
what constitutes good assessment programmes?
which building blocks should be included?
how should information be collated:qualitatively / quantitatively?
assessment as a programme:
who should do what?
Universiteit MaastrichtWhere do we go from here?
research agenda
g-analyses
reproducibility of (portfolio) judgments?
other statistical approaches
sequential testing?
organisational approaches?
Universiteit MaastrichtWhere do we go from here?
Research agenda
how to “score” qualitative judgments?
quality elements of multi-modal assessment
identifiers for profile scores
what influences the thinking of judges?
fraud/plagiarism (internet)
flexible adaptation of teaching to assessment outcomes
Universiteit MaastrichtWhere do we go from here?
research agenda
roles of format, content, scheduling and regulations
consequential validity / educational impact
what influences teacher behaviour
Universiteit MaastrichtWhere do we go from here?
research agenda
consequential validity / educational impact
Universiteit MaastrichtWhere do we go from here?
research agenda
how to change a programme?
research into assessment programmes
stakeholder involvement
how are the instruments combined?
how to use parts in stead of whole instruments?
rules of design
Universiteit MaastrichtWhere do we go from here?
SUMMARYSUMMARY
overview of the presentation
- why this presentation
- developments in competence assessment
- implications for research
- developments in performance assessment
Universiteit MaastrichtWhere do we go from here?
conclusions: in (education), assesment and research
strength comes from collation of results
the method is not synonymous with the concept
no single method is a panacea
translation of concepts into methods is useful
Universiteit MaastrichtWhere do we go from here?
Thank you
