Faculty of Medicine and Health Sciences Exploring new approaches to assessing attitudes and behaviours Dr Sandra Gibson Dr John Winpenny 6 th February

Faculty of Medicine and Health Sciences

Exploring new approaches to assessing attitudes and behavioursDr Sandra GibsonDr John Winpenny

6th February 2012


The plan

• Introductions• Discussion of attitudes and behaviours• Item types used in medical training• Transferability• Lunch • Item writing• Future plans• Closing discussion


Outcomes

To discuss the theory and application of assessing attitudes and behaviours

To discuss the methods of assessing behaviours used in medical training and their transferability to other disciplines

To try writing assessment items in chosen disciplines

To develop a plan of action to follow up questions/ideas raised during the workshop


Why are attitudes and behaviours important to you?

• Take a minute and write down why measuring attitudes and behaviours is of interest/importance to you


What do we mean by ‘attitude’?

‘a psychological tendency that is expressed by evaluating a particular entity with some degree of favour or disfavour’

Eagly and Chaiken (1993)


Attitudes

• How an attitude is expressed (or not) can be influenced by– Context (specificity)– Reinforcement (behavioural control)– External drivers- guidelines (subjective norm)– Reflection (Schön)

Attitudes can change


Attitudes vs. Behaviours

A student I supervised repeatedly mentioned she held very strong personal and religious views against the termination of pregnancy.

I knew she had just been placed in a family planning clinic.


Our approach

• Attitudes do not determine behaviour

• Students who hold suitable attitudes will experience less cognitive dissonance

• Seletion procedures are increasingly incorporating the assessment to try and assess attitudes/behaviours.


Tommorow’s DoctorsThe graduate will be able to behave according to ethical

and legal principles. The graduate will be able to:

(c) Be polite, considerate, trustworthy and honest, act with integrity, maintain confidentiality, respect patients’ dignity and privacy, and understand the importance of appropriate consent.

(d) Respect all patients, colleagues and others regardless of their age, colour, culture, disability, ethnic or national origin, gender, lifestyle, marital or parental status, race, religion or beliefs, sex, sexual orientation, or social or economic status.


Why are attitudes and behaviours important to you?

• Before I describe how we measure behaviours, lets revisit your interest in measuring them attitudes and behaviours.


Medicine

• Fostering professional attitudes

• Measuring Professional behaviours• Observed Structured Clinical Exams (OSCEs)• Situation Judgment tests• Script concordance tests


Dealing with uncertainty• Studies with Foundation Year Doctors are

consistently finding that dealing with uncertainty is an issue that they are concerned about.

• Use of these item types can be helpful in preparing and reassuring students about their behaviour in situations where decision making is not clear cut.


Observed Structured Clinical Exam (OSCE)

• Sit back and enjoy the movie.• osce.mp4



• Greater reliability that traditional clinical assessments.• The more realistic the better• Feasibility can be an an issue• Generalisibility Analysis used to optimise design



• The mark scheme– Global judgment used to set boundary between

minimum competence and non competence– Can be used to assess professional behaviour

• Dealing with difficulty patients• Honesty• Dealing with uncertainty/ambiguous situations

– Multiple mini interview


Situation judgment tests

• Another assessment.

• For further examples of SJT’s see the Medical School Council- Assessment Alliance Project page

• Improving Selection to the Foundation Programme• http://www.isfp.org.uk/SJT/Pages/Example-

questions.aspx


Situation Judgment Tests

• Vary widely in format and application• Format can have an impact on what they test.• Scoring be complex• Experts can score similarly to novices, but with smaller

standard deviation.• Faking


Situation judgment Tests

• Used in General Practitioner (GP) selection• Will be used in Foundation Year selection in the near

future• May be introduced into MBBS selection (UKCAT)


Script concordance

• Another test to try.


Script concordance

• Based on script theory• Ddesigned to assess decision making in ambiguous

situations.• Studies confirm acceptable reliability and validity• Novices consistently less well than experts• Case specificity and generalisabiity need to be

considered• Importance of reference group


Script concordance

• Currently used in Masters in Oncoplastic Breast Surgery on a formative basis

• Plan to introduce in undergraduate course for formative assessment


Questions


Writing workshop

• Ask group what they want to do• Split into groups each try each item type, or groups

doing different types?• Spend half time facilitating writing, then presenting

items to other groups for review.


The critical incident

• To help identify potential areas to write items in, no matter what the format, it can help to start with a critical incident/significant event.

• Take a few minutes and jot down an event were you felt that your judgment felt uncertain.


Planning

• What have you got out of the workshop?• Are there any areas that we could take forward as a

group/subgroup?• Do you have any suggestions for future workshops?


Thank you

• Thank you to the HEA for supporting this workshop• Feedback forms will be sent to you individually after

the session. • Contact details

– [email protected]– [email protected]


ReferencesBland AC, Kreiter G & Joel A (2005). The psychometric properties of five scoring methods applied to the script

concordance test. Academic Medicine 80:395-9

Charlin B, et al. (2000). The script concordance test: a tool to assess the reflective clinician. Teach Learn Med 12: 189-95

Creighton P, Scott N. (2006). An introduction to situational judgement inventories. Sel Dev Rev 22:3-6

Eagly HA & Chaiken S, (1993). The Psychology of Attitudes, Fort Worth, TX: Harcourt

Feltovich, PJ.et al (1984). LCS: The role and development of medical knowledge in diagnostic expertise. In W.J. Clancey & E.H. Shortliffe (Eds.), Readings in medical artificial intelligence: The first decade (pp. 275-319). Reading, MA: Addison Wesley

Fournier JP, et al (2008). Dcript Concordance Testis: Guidelines for construction BMC Midecal Informatics and Decision Making 8:18

Gagnon R (2005). Assessmet in context of uncertainty: how many members are needed on the panel of reference of a script concordance test? Med Educ 39: 28

Humbert AJ, Besinger B & Edward JM (2011). Assessing clinical reasoning skills in scenarios of uncertainty: convergent validity for a script concordance test in an emergency medicine clerkship and residency. Academic Emergency Medicine 18: 627-634

Mc Daniel MA, & Nguyen NT (2001). Situational Judgment Tests: A Review of Practice and Constructs Assessed. International Journal of Selection and Assesment 9(1-2): 105-113

McDaniel MA, Hartman NS & Grubb W L (2003). Situational Judgment Tests, Knowledge, Behavioral Tendency, and Validity: A Meta-Analysis. Paper presented at the 18th Annual Conference of the Society for Industrial and Organizational Psychology. Orlando.

McDaniel MA, & Whetzel DL (2005). Situation Judgment Tests, An IPMAAC Workshop.

Motowidlo S. Borman W & Schmit M (1997). A theory of individual differences in task and contextual performance. Human Performance 10(2): 71-83

Schön D. (1983) The reflective practitioner: how professionals think in action. New York: Basic Books


Exploring new approaches to assessing attitudes and behavioursDr Sandra GibsonDr John Winpenny

Documents

Faculty of Medicine and Health Sciences Exploring new approaches to assessing attitudes and behaviours Dr Sandra Gibson Dr John Winpenny 6 th February