Embed Size (px)
Citation preview
Adapting Situational Judgement Tests for Cross Cultural Assessment
Tao Li & Wendy Lord
Hogrefe Ltd. UK
The 7th Conference of the International Test Commission
Hong Kong July 2010
Traditional personality tests vs. Situational judgement tests (SJTs)
• Traditional personality tests measure
— preferred or typical behaviour
• SJTs measure flexibility of behaviour
—Present work related situations and potential response —Present work related situations and potential response
options
―identify what behaviours are most effective for a given
situation
The Leadership Judgement Indicator (LJI)
• M. Lock & R. Wheeler, 2005. © by Hogrefe
• Measures effectiveness at flexibly adapting
leadership style to suit the situationleadership style to suit the situation
• Consider the nature of the task and the
characteristics of the people involved to
determine different level of participation
Styles of leadership
DIRECTIVE
“I make the
decisions based on
my ideas”
CONSULTATIVE
“I make the decision
based on our ideas”
CONSENSUAL
“We make the
decision based on
our ideas”
DELEGATIVE
“You make the
decision based on
your ideas”
Example LJI item
You manage a small business in which you have three employees. The
office needs covering over the forthcoming school holidays and last
year there were problems. All three employees seemed only interested
in their own self-interest and their choice of annual leave time created
considerable disagreement. You anticipate similar difficulties this year,
which could place you under pressure.
a) Do not risk debate this year, but tell them when they can have leave.a) Do not risk debate this year, but tell them when they can have leave.
b) Tell them the office should always be covered and get them to come up
with a holiday schedule.
c) Call a meeting with all present and come to an arrangement that is
acceptable to everyone.
d) Find out each person’s preferred holiday time and then you decide on
the holiday times.
Issues in adapting LJI
• The transportability of the scenarios across
national cultures
• The extent to which the response alternatives • The extent to which the response alternatives
represent the same intended leadership style
across cultures
The transportability of LJI scenarios across cultures
• Does the theoretical model apply across cultures?
• Cross cultural studies showed that the model
works very well in developed and developing
countries.countries.
Adaptations of LJI in progress
• France
• Germany
• Italy
• Bulgaria
• Czech Republic• Czech Republic
• Slovakia
• Russia
• Brazil
CFA modelling
i3 i4i1 i2 i3 i4i1 i2 i3 i4i1 i2 i3 i4i1 i2
scenario1 scenario2 Scenario3 scenario4
directive consultative consensual delegative
Model fit
CountryCountryCountryCountry NNNN CFICFICFICFI RMSEARMSEARMSEARMSEA
UK 1345 0.912 0.078
French 1255 0.933 0.060
• Satisfactory fit indicates basic structural equivalence
i.e. the two versions measure similar constructs.
Item difficulty comparisonCorrect identification of Correct identification of Correct identification of Correct identification of
best stylebest stylebest stylebest styleCorrect identification of Correct identification of Correct identification of Correct identification of
worst styleworst styleworst styleworst style
UK French UK French
scenario1 0.65 0.71 0.32 0.35
scenario2 0.77 0.66 0.81 0.61
scenario3 0.81 0.64 0.63 0.61
scenario4 0.73 0.45 0.57 0.47
scenario5 0.51 0.53 0.93 0.85
scenario6 0.65 0.59 0.72 0.76
scenario7 0.94 0.86 0.74 0.57scenario7 0.94 0.86 0.74 0.57
scenario8 0.72 0.46 0.91 0.85
scenario9 0.16 0.51 0.84 0.77
scenario10 0.76 0.78 0.87 0.72
scenario11 0.82 0.77 0.85 0.74
scenario12 0.69 0.54 0.56 0.43
scenario13 0.49 0.33 0.70 0.53
scenario14 0.83 0.74 0.72 0.61
scenario15 0.97 0.95 0.92 0.85
scenario16 0.81 0.73 0.88 0.77
Average 0.710.710.710.71 0.640.640.640.64 0.750.750.750.75 0.660.660.660.66
Differential item function
Mantel Mantel Mantel Mantel LOR ZLOR ZLOR ZLOR Z COX ZCOX ZCOX ZCOX Z
scenario1 0.02 -0.14 -0.14
scenario2 14.11 3.70 3.76
scenario3 22.91 4.81 4.78
scenario4 0.64 0.81 0.79
scenario5 17.34 -4.08 -4.16
scenario6 14.55 -3.86 -3.82
scenario7 13.48 10.69 10.66
scenario8 23.13 4.78 4.81scenario8 23.13 4.78 4.81
scenario9 68.83 18.34 19.20
scenario10 11.21 -3.28 -3.36
scenario11 68.63 -8.19 -8.29
scenario12 50.14 6.99 7.07
scenario13 4.69 2.15 2.16
scenario14 5.82 2.43 2.41
scenario15 3.07 -1.78 -1.75
scenario16 9.32 3.04 3.05
Mantel: Mantel chi-square
LOR Z: Standardized Liu-Agresti cummulative common log-odds ratio
COX Z: Standardized Cox’s noncentrality parameter estimator
Summary
• This pioneering study raises some key issues
regarding adapting SJTs
• Modelling measurement equivalent for SJTs • Modelling measurement equivalent for SJTs
• SJTs have more parameters than traditional
test formats that are prone to culture influence
