Low Cost Assessment Center

A custom tailored experimental and

diagnostic Assessment Centre experience

in education industry.

Low Cost Assessment Centre

Shirshendu Pandey

NIPM, 2011

The contents of the presentationAgenda

Shirshendu Pandey,

NIPM, 2011

Brief on Company

Recommendations and Limitations

Phases of the Project

Problem Definition

Methodology

Findings

88

77

55

33

22

11

Assessment Centers

Phases of the Project44

Shirshendu Pandey

NIPM, 2011

•History of the company

•The Education Industry

•Its vision

•The relevance of the Project

An emerging player in the

education industry

Primary and Secondary Problems

Problem Definition

Using analytical reasoning to

diagnostically link the

developmental areas and

competency levels to

company’s problems.

Secondary Problem

Designing and executing a

low cost and yet, validated

Assessment Centre in order

to analyse the competency

levels of managers.

Primary Problem

Shirshendu Pandey,

NIPM 2011

1 22

Shirshendu Pandey

NIPM 2011

• Assessment Centres are battery

of multiple tests which are used to

assess individual potential,

competencies, skills and traits.

• ACs are considered to have high

predictive validity.

• ACs typically have multiple tests,

multiple assessors and multiple

participants.

• ACs have evaluative,

developmental and diagnostic

functions.

Assessment CentresWhat are Assessment Centres?

Phases of ProjectRefined Competency Framework

Refined Competency Framework

Detailed Competency Based

Assessment Tool


Assessment Tool

Assessment Centre Design


Tests and Instruments


Assessor TrainingAssessor TrainingExecutionExecutionValidationValidation

The Competency Framework was made by earlier projects at the company and had to be weeded for refinement and quality.

The new framework used Behavioral Event Interviews, Critical Incident Interviews, Vision and mission Interviews.

The Competency Framework was made by earlier projects at the company and had to be weeded for refinement and quality.

The new framework used Behavioral Event Interviews, Critical Incident Interviews, Vision and mission Interviews.

Shirshendu Pandey NIPM 2011

Old Vs New Competencies

Shirshendu Pandey

NIPM, 2011

•Customer Focus

•Perceptive to Change

•Communicating Effectively

•Result Oriented

•Managing Tasks

•Decision Making and Risk

Taking

•Strategic Congruence

•Awareness

•People Focus

•Brand Quality Association

New Competency FrameworkOld Competency Framework

•Customer Focus

•Responsive to Change

•Awareness

•Result Orientation

•Team Work

•People Focus

•Brand Focus

•Quality Focus

•Establishing Direction


Phases of ProjectDetailed Competency Based Assessment Tool



Competency Based Detailed

Competency Based Assessment Tool






A total of ten competencies were identified for which a BARS based Competency Assessment Tools was made.

Each Assessment tool consisted of

A total of ten competencies were identified for which a BARS based Competency Assessment Tools was made.

Each Assessment tool consisted of behavioural indicators of each competency.



Phases of ProjectAssessment Centre Design




Assessment Tool


Assessment Tool

Assessment Centre Assessment Centre Design




An AC design was created which categorically mentioned which competency would be assessed via which test.

Due to constraints of time and resources, it was decided that each competency would be measured twice in different tests. Also each competency would be assessed by two different assessors in the two different tests.

An AC design was created which categorically mentioned which competency would be assessed via which test.

Due to constraints of time and resources, it was decided that each competency would be measured twice in different tests. Also each competency would be assessed by two different assessors in the two different tests.


S.no. Competency SAQ GCP ICP BEI IBT FAT CT P

1 Communication Skills

2 Task Management

3 Awareness

4 Decision making and Risk Taking

5 People Focus

6 Strategic Business Acumen

7 Result Oriented

8 Customer Focus

9 Brand Quality Association

10 Perceptive to change

SAQ= Self Assessment QuestionnaireGCP= Group Case ProblemICP= Individual Case ProblemBEI= IBT= In Basket TestFAT= Focused Apperception TestCT= Compromise TestP= Performance Scores form Company based on peer and superior feedback.

SAQ= Self Assessment QuestionnaireGCP= Group Case ProblemICP= Individual Case ProblemBEI= Behavioural Event InterviewIBT= In Basket TestFAT= Focused Apperception TestCT= Compromise TestP= Performance Scores form Company based on peer and superior feedback.


Phases of ProjectTests and Instruments




Assessment Tool


Assessment Tool





A set of six tests were one questionnaire were developed for the benefit of the AC.

Of these four were a part of AC while two were being tested on an experimental basis.

A set of six tests were one questionnaire were developed for the benefit of the AC.

Of these four were a part of AC while two were being tested on an experimental basis.


Individual Case ProblemIndividual Case Problem

Group Case ProblemGroup Case Problem

Behavioural Event InterviewBehavioural Event Interview

In Basket TestIn Basket Test

Self Assessment QuestionnaireSelf Assessment Questionnaire

Focused Apperception TestFocused Apperception Test

Compromise TestCompromise Test

Tests And InstrumentsFour Generic Tests, Two Experimental and One Self

Assessment

A case ‘Chetan at BES’ was prepared

keeping in mind the competencies which

were to be tested in the ICP and GCP.

The case was same for the two test. In

case of ICP the managers were asked to

answer questions as regards the problems

they identify and the solutions they

propose.










Assessment

The GCP was a leaderless group

discussion which encouraged the

managers to find one solution for the same

case for which they responded in the ICP.










Assessment

A structured BEI framework was made to

help the assessors evaluate the managers

for the six competencies which the BEI

intended to assess.

The various questions were however only

indicative.










Assessment

An In Basket was created with a scenario

of an IT consulting company named,

‘Manish at NetLogic’.

The In Basket additionally consisted of a

total of 22 documents.

Managers were asked to respond to them

in order of priority, arrange them, and most

importantly act on them.

The In Basket was subjective.










Assessment

A Self Assessment question was made

based on the behavioural indicators of all

competencies with a total of 87 elements.

Managers were asked to rate themselves

on these behaviours on a scale of 1 to 5, 1

being the least demonstrated and 5 being

the most demonstrated behaviour.










Assessment

TAT conventionally uses projective tests to

delve into the subconscious mind based on

observed defensive mechanisms.

FAT was created to adapt it to industrial needs

by focusing the pictures on professional

situations and incite responses which take into

account the professional problems rather than

personal ones.

Further an incomplete story featuring a problem

was presented to ask for the climax from the

managers. This allowed the probing of specific

competencies.










Assessment

Compromise Test consisted of a rank and

an option test.

All competencies were put in generic

situations and managers were asked to

rank them and choose one, in case of rank

and option CT respectively.

The idea was to see if compromise is

better indicative of real life preferences or

competencies of an individual.


Phases of ProjectAssessor Training




Assessment Tool


Assessment Tool





Assessor TrainingExecutionExecutionValidationValidation

Due to cost constraints, the AC had to use assessors who were not professionally assessors but experienced enough to assess.

Three professors and an internal assessor (AVP, HR) were brought in to aid assessment. All of them received three to six hours of training each before the AC.

Due to cost constraints, the AC had to use assessors who were not professionally assessors but experienced enough to assess.

Three professors and an internal assessor (AVP, HR) were brought in to aid assessment. All of them received three to six hours of training each before the AC.


Phases of ProjectExecution of AC




Assessment Tool


Assessment Tool





Assessor TrainingAssessor TrainingExecutionValidationValidation

The AC was executed on 13, 14 and 15 of June 2011.


Participa

nt

11-11.20 11.20-

11.50

11.50-

12.40

12.40-

12.50

12.50-

1.50

1.50-

2.50

2.50-

3.20

3.20-

4.00

4.50-

5.00

5.00-

6.00

6.00-

6.20

P1 Briefing TAT ICP Break IBT Lunch GCP BEI Break CT SAQ



P4 Briefing TAT ICP Break IBT Lunch GCP CT Break BEI SAQ




AC Schedule

Phases of ProjectValidation of AC




Assessment Tool


Assessment Tool





Assessor TrainingAssessor TrainingExecutionExecutionValidation

Since the AC was being carried out with non standard tests, validation was a concern.

The AC assumed three key parameters for validation, i.e. Internal reliability, Inter rater reliability, and construct validity.

Since the AC was being carried out with non standard tests, validation was a concern.

The AC assumed three key parameters for validation, i.e. Internal reliability, Inter rater reliability, and construct validity.


• Reliability : Cronbach’s Alpha and Inter Rater Reliability

• Validity : Relatedness Score and Concurrent Validity

Validation Establishing Reliability and Validity

Reliability Measure SAQ GCP ICP BEI IBT FAT CT(mean) AC Overall

Cronbach’s Alpha 0.870 0.874 0.773 0.583 0.810 0.779 0.383 0.931

Inter-Rater Reliability

(Correlation Coefficients)

NA NA 0.822 NA 0.873 0.701 NA NA

Validity Measure SAQ GCP ICP BEI IBT FAT CT(mean) AC Overall

Mean Relatedness Response

Score (1 to 5) n=8

4.99 NA 4.67 NA 4.78 5.0 4.55 4.798

Concurrent Validity 0.314 0.749 0.623 NA 0.526 0.682 0.512 0.943


• The AC was a success with most parameters indicating healthy

coefficients.

• A large amount of errors can still be attributed to the rating errors of

unskilled assessors.

• The cost of the skill for AC was about Rs 30,000 as compared to a

conservative market value of such service at Rs 10Lakhs; with

reputed consulting firms charging about Rs 30Lakhs.

• The ROI of such AC could range from 110% to 150% depending on

how the company utilizes it.

• The unskilled assessors didn’t exhibit as high rating errors as

expected.

Findings The Results of AC and Conclusions


• FAT came up as a useful competency based assessment tool.

However, more skilled assessors would be needed.

• CT option did not work well showing negligible correlations with

most. CT Rank was highly correlated with SAQ.

Findings contd. The Results of AC and Conclusions


Your own footer Your Logo

Diagnostic Findings The Results of AC and Conclusions


Recommendations

Further research would be required to standardize the FAT and CT and prove their

validity and reliability esp. test retest reliability.

Further Research

Development needs of Middle and Senior managers are very critical esp. the

middle managers.

Developmental Needs of Middle Managers and Senior Managers

The company has a tacit culture of not delegating work to subordinates. Most

managers do most of the ‘difficult work’ themselves and so not giving

subordinates a chance to take up challenging work. This must be changed.

Culture of not delegating work

1

2

33


Limitations

A sample of 17 is a very small one to use the coefficients as final indicators of a

validated AC.

Small Sample

My very own expertise and experience in the field is very less and absence of

guidance from a specialist in this field was always felt.

Limited Experience and Guidance

The assessors though experienced were not professional assessors and the

training received by them delivered by me in four different briefings. My own

limitations were also a playing factor in the assessor’s transference of training.

Assessors and Assessor Training

1

2

33


Thank You

Shirshendu Pandey

NIPM, 2011

Shirshendu Pandey

[email protected]