Predicting in-tray performance: The effect of cognitive ability vs. personality

Your presenters

2

Kim Dowdeswell Senior Researcher, TTS

Dr Sebastian Clifton Research Consultant,

TTS

The AC method

¡ AC methodology involves the focused observation, measurement, and systematic evaluation of a person’s behaviour in a simulated context.

¡ Attractive to organisations because:¡ Provides insight into selection, development and promotion decisions¡ Provide richness of information for high-stakes applications¡ Proven to predict subsequent job performance

¡ Attractive to participants because:¡ High fidelity nature of assessments clearly related to work¡ Can provide realistic job previews¡ Opportunity to build insights into one’s own performance during simulations

3

Rationale for this study

¡ Addressing a practical challenge: How to advise clients considering replacing measures of cognitive ability with an inbox exercise

¡ Especially within the selection context for senior-level managers and executives

¡ Key question: What’s the best predictor of overall inbox performance?

4

Highlights from the literature

¡ Research illustrates that performance on an inbox exercise is typically more cognitively loaded:

¡ Furnham et al. (2008) found a measure of ability the best predictor of AC performance

¡ Specifically concerning inbox exercises, Hoffman et al. (2015) found cognitive ability to be most strongly related to candidate performance in relation to measures of personality

¡ However, the majority of research evidence concerning the AC method originates from outside of South Africa

5

Methodology

Methodology

¡ Sample of 113 South African candidates who completed an online virtual inbox, a cognitive ability test, and a personality questionnaire

¡ Identified a priori hypotheses identifying relevant personality questionnaire dimensions, given dimensions measured by the in-tray

¡ Conducted a series of multiple regression analyses, testing two models where the order of entry of variables varied

7

¡ Saville Wave Professional Styles

Assessment instruments

8

¡ Saville Swift Analysis Aptitude

Assessment instruments

¡ Virtual inbox exercise from a&dc’s VirtualACTM offering, designed for use at middle management levels in an organisation

9

Dimensions measured by the virtual inbox

Dimension Definition A priori hypotheses

Delegation Effective allocation of decision making and other responsibilities to the appropriate person, inside or outside the team.

Empowering IndividualsDirecting People

Organisational Sensitivity

Capacity to perceive the impact and implications of decisions and activities on other parts of the organisation.

Adopting Practical ApproachesExploring Possibilities

Interpersonal Sensitivity

Awareness of other people and environment and own impact on these. Actions indicate a consideration for the feelings and needs of others).

Valuing IndividualsUnderstanding People

Initiative Actively influencing events rather than passively accepting, sees opportunities and acts on them. Originates action.

Taking ActionSeizing Opportunities

10

Dimensions measured by the virtual inbox

Dimension Definition A priori hypotheses

Decisiveness Readiness to make decisions, state opinions, take action or commit oneself.

Making DecisionsTaking Action

Strategic Perspective

Takes account of a wide range of longer-term issues, opportunities and contingencies. Identifies the means of implementing plans in line with the vision and direction.

Adopting Practical ApproachesDeveloping Strategies

Judgement Ability to evaluate data and courses of action and to reach logical decisions. An unbiased, rational approach.

Examining InformationInterpreting Data

Problem Solving & Analysis

Effectiveness in identifying problems, seeking pertinent data, recognising important information and identifying possible causes of problems.

Examining InformationInterpreting Data

11

Sample description

Gender

Male Female

First Language

African Language AfrikaansEnglish Other / Not Provided

Sample description

13

African languages included- Northern Sotho- Sotho- Tsonga- Tswana- Xhosa- Zulu

High School Diploma Degree ProfessionalQualification

Postgraduate Other / NotProvided

0

5

10

15

20

25

30

35

Percentage

Highest Qualification

Sample description

14

Manager Senior Manager Executive Leader Other / Not Provided05

101520253035404550

Percentage

Managerial Responsibility

Sample description

15

Results

0

1

2

3

4

5

6

7

8

9

10

Swift Ana

lysis

Aptitud

e

Adopti

ng Prac

tical

Approa

ches

Develo

ping S

trateg

ies

Directi

ng Peo

ple

Empoweri

ng In

dividu

als

Examini

ng In

formati

on

Explor

ing Pos

sibiliti

es

Interp

reting

Data

Making

Dec

isions

Seizing

Opp

ortunit

ies

Taking

Acti

on

Unders

tandin

g Peo

ple

Valuing

Indiv

iduals

Mea

n S

ten

Sco

re

Mean assessment scores (cognitive ability & personality)

17

0

1

2

3

4

5

6

7

8

9

10

Delegation

Organisati

onal S

ensitivi

ty

Interpersonal

Sensitivit

y

Initiativ

e

Decisive

Strateg

ic Persp

ectiv

e

Judg

ement

Problem Ana

lysis

IB O

verall S

core

IB C

ognitiv

e Composit

e

IB Behav

ioural C

ompo

site

Mea

n S

ten

Sco

re

Mean assessment scores (inbox)

18

Model 2 R R2 Sig.

Step 1:Personality 0.505 0.255 0.002

Step 2: Ability 0.550 0.303 0.000

Ability adds 4.8% incremental variance in participants’ inbox overall scores, over personality

Inbox overall score regressed on cognitive ability and a priori personality dimensions

19

Dependent variable: Inbox overall score

Model 1 R R2 Sig.

Step 1:Ability 0.272 0.074 0.004

Step 2: Personality 0.550 0.303 0.000

Personality explains 22.9% incremental variance in participants’ inbox overall scores, over ability

Model 2 R R2 Sig.

Step 1:Personality 0.360 0.129 0.049

Step 2: Ability 0.433 0.187 0.009

Ability adds 5.8% incremental variance in explaining participants’ behaviourally-oriented composite inbox scores over personality

In-tray behavioural composite score regressed on cognitive ability and a priori personality dimensions

20

Model 1 R R2 Sig.

Step 1:Ability 0.271 0.073 0.004

Step 2: Personality 0.433 0.187 0.009

Personality explains 11.4% incremental variance in participants’ behaviourally-oriented composite inbox scores over ability

Dependent variable: Inbox behaviourally-oriented dimensions composite score

Model 2 R R2 Sig.

Step 1:Personality 0.277 0.077 0.197

Step 2: Ability 0.344 0.119 0.060

Entering the a priori personality dimensions first resulted in a non-significant regression model when predicting the cognitively-oriented in-tray composite score.

Inbox cognitive composite score regressed on cognitive ability and a priori personality dimensions

21

Model 1 R R2 Sig.

Step 1:Ability 0.219 0.048 0.020

Step 2: Personality 0.344 0.119 0.060

When adding the a priori personality dimensions in the second step, the resulting regression model predicting the cognitively-oriented inbox composite score was non-significant.

Dependent variable: Inbox cognitively-loaded dimensions composite score

What do these findings mean?

Dimensions commonly assessed in inbox exercises (Thornton et al., 2015)

Behavioural vs. Cognitive Orientation

Dimensions assessed by the in-tray utilised in this research

Problem Analysis Cognitively-loaded JudgementProblem Analysis

Strategic Perspective

Decision Making Cognitively-loaded Decisiveness

Planning & Organising Cognitively-loaded

Written Communication Cognitively-loaded

Leadership Behaviourally-oriented Initiative

Empathy Behaviourally-oriented Interpersonal SensitivityOrganisational Sensitivity

Delegation Behaviourally-oriented Delegation

The nature of dimensions typically measured in inbox exercises are fairly balanced in terms of their orientation

23

It’s not WHAT or HOW we are measuring, but HOW we are evaluating the observed data

Examples from the scoring guide:

24

Problem AnalysisThe participant, in dealing with the

“Potential Restructuring” email, understands that reorganisation may

mean possible redundancies

JudgementThe participant, in tackling the “Partnership” email, suggests

solutions to overcome the difficulties

Strategic PerspectiveThe Participant, in tackling the

“Partnership” email, thinks about the wider issues of partnering with the

proposed agency

Recommendations for practice

¡ Limitation of this study: Absence of reliable criterion performance data to evaluate the incremental validity of each measure included

¡ Still, our findings highlight that inbox exercises are not always as cognitively loaded as we may think

¡ Serves as a reminder of the importance of:¡ Reviewing content validity (of the inbox tasks & behavioural anchors) in relation

to job requirements¡ In line with best practice, regular training of assessors and reviewing scoring

guidelines / the identification of behavioural anchors

25

References

References

¡ Arthur, W., Day, E. A., McNelly, T. L., & Edens, P. S. (2003). A meta-analysis of the criterion-related validity of assessment center dimensions. Personnel Psychology, 56, 125–154.

¡ Furnham, A., Taylor, J., & Chamorro-Premuzic, T. (2008). Personality and intelligence correlates of assessment center exercises. Individual Differences Research, 6(3), 181–192. Retrieved from www.idr-journal.com

¡ Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., & Bentson, C. (1987). Meta-analysis of assessment centervalidity. Journal of Applied Psychology, 72(3), 493–511.

¡ Hoffman, B. J., Kennedy, C. L., LoPilato, A. C., Monahan, E. L., & Lance, C. E. (2015). A review of the content, criterion-related, and construct-related validity of assessment center exercises. Journal of Applied Psychology, 100(4), 1143–1168. https://doi.org/10.1037/a0038707

¡ Kuncel, N. R., & Sackett, P. R. (2014). Resolving the assessment center construct validity problem (as we know it). Journal of Applied Psychology, 99(1), 38–47. https://doi.org/10.1037/a0034147

¡ Krause, D. E., Rossberger, R. J., Dowdeswell, K. E., Venter, N. & Joubert, T. (2011). Assessment centerpractices in South Africa. International Journal of Selection and Assessment, 19(3), 262-275. New York, NY: Routledge.

27

References

¡ Mulder, G. (2014). Effectiveness of assessment centres as a managerial tool in organisations. In 40th Annual Conference of the Assessment Centre Study Group (ACSG), February 28 - March 1, 2014, Stellenbosch, South Africa. Stellenbosch, South Africa.

¡ Rupp, D. E., Thornton, G. C., & Gibbons, A. M. (2008). The construct validity of the assessment center method and usefulness of dimensions as focal constructs. Industrial and Organizational Psychology, 1(1), 116–120. https://doi.org/10.1111/j.1754-9434.2007.00021.x

¡ Sackett, P. R., Shewach, O. R., & Keiser, H. N. (2017). Assessment centers versus cognitive ability tests: Challenging the conventional wisdom on criterion-related validity. Journal of Applied Psychology, 102(10), 1435–1447. https://doi.org/10.1037/apl0000236

¡ Thornton, G. C., & Gibbons, A. M. (2009). Validity of assessment centers for personnel selection. Human Resource Management Review, 19, 169–187. https://doi.org/10.1016/j.hrmr.2009.02.002

¡ Thornton, G. C., Rupp, D. E., & Hoffman, B. J. (2015). Assessment center perspectives for talent management strategies (2nd ed.).

28