WorkKeys Innovations: A Holistic Solution for WIRED West Michigan

WorkKeys Innovations: A Holistic Solution for WIRED West Michigan

Steve Robbins, AVP, Applied Research, ACT, Inc.

22

Overview

• Why we should care about combining cognitive- and personality-based measures

• National Career Readiness Certificate “Plus”

• What we know from workforce and educational literatures

• Differential test strategies along the entire continuum of employment

33

Why Now?

• Market Need: SHRM National Study

• Body of research informs how to optimize Personal Skill Assessments

• ACT strength in cognitive & non-cognitive assessment (John Holland, VP Research in 1960’s)

• Solution-focused approach

44

Businesses want integration of Cognitive & Personality Constructs

SHRM: Applied Skills and Basic Knowledge: Combining and Ranking

For new entrants with a two-year college/technical school diploma, applied skills are four of the top five “very important” skills in combined

ranking with basic knowledge and skills.

Rank Skill %

1 Professionalism/Work Ethic 83.4

2 Teamwork/Collaboration 82.7

3 Oral Communications 82.0

4 Critical Thinking/Problem Solving

72.7

5 Reading Comprehension 71.6

6 Written Communications 71.5

7 English Language 70.6

8 Ethics/Social Responsibility 70.6Casner-Lotto, J. & Barrington, L. (2006)

55

Solutions are Need Driven: Continuum of Employment

Purpose Needs Purpose Needs

Pre-Selection

-Screen people in most cost effective way-Find honest/dependable employees-Save time in the screening process

Coaching/

Development

-Identify other jobs that an employee can fit-Develop employees for future company needs-Employees identify areas of improvement

Recruitment -Identify people who fit the work environment-Identify people with skills that match the job

Succession/

Leadership Planning

-Identify candidates for top-level positions-Develop employees for future needs of company-Retain top performers

Selection -Select employees with skills that best fit the job-Save time in selection process-Select people in most cost-effective way-Find honest/dependable employees-Certifying employees

Training/

Development

-Identifying work readiness-Identify basic workplace skill levels-Educating about career planning-Job Placement

66

WorkKeys Assessment Solutions: Pyramid for Success

• Ensure work and training readiness (WorkKeys Foundational Skills)

• Provide employers another source of information for selection (“Performance”)

• Give a “snap shot” of strengths and areas of improvement across key response tendencies or domains (“Talent”)

• Promote career exploration of job fit (“Fit”)

77

Pyramid for SuccessFIT

•Match individual interests/values to work environment•Enhance job persistence & satisfaction•Develop Talent pool to meet needs

TALENT•Benchmarking for selection•Coaching & Development•Compound Indices: Sales, Managerial,

Leadership, Safety

PERFORMANCE•General Work Performance: Productivity, Absenteeism,

Complaints about conduct•Safety and Risk Reduction

FOUNDATIONAL SKILLS•Job Analysis – identifies the skills and skill levels needed to be

successful on the job•Assessments – show the current skill levels of an individual•Training – helps individuals and employers correct skill gaps

88

National Career Readiness Certificate

Applied Math Locating InformationReading

5

4

3

2

1

6

Worker’s Skill Level

Benchmark Levels Required

Skill Gap

33

4 4

5

Currently, the WorkKeys system assesses foundational skills, such as reading and locating information, using a three-step process of job profiling, assessment, and skill-gap training.

99

National Career Readiness Certificate “Plus”

• A flexible solution to meet state and system needs

• Georgia example

• Talent for coaching & development

• “Fit” for career exploration

• Alterable variables allow for intervention & guidance

1010

Cognitive Ability Tests“Rule”

• General vs. specific test effect sizes

• Adverse impact issues can be ameliorated with specific tests:– Task analysis– Job-specific tests– Combinational use of cognitive and non-

cognitive tests

Training Outcomes

Work Performance

GMA .54 (.55*) .62**

Math .48 (.48*) .52**

Reading .44 (.40*) .35*** Brown, Le, & Schmidt (2006) **Salgado et al. (2003)

1111

Personality Tests“Add Value”

• Validity Estimates:

1 From Schmidt et al. (2007) using indirect range restriction

Training Outcomes

Work Performance

Old Validity

New1 Validity

OldValidity

New1 Validity

GMA .526 .668 .507 .652

Conscientiousness .169 .176 .217 .230

Emotional Stability .098 .103 .112 .121

1212

What are We Talking About? Personality Test Constructs

• The Big Five:Emotional Stability, Extraversion, Agreeableness, Conscientiousness, Openness to Experience

• Specific or Facet-level Traits:Carefulness, Discipline, Influence, Order, Sociability, Drive, Creativity

• Compound Traits:Service Orientation, Integrity, Managerial Potential, Teamwork

1313

Personality Test Issues

• Selection vs. Coaching and Development as Differential Applications

• Selection Approach– Recommend multiple hurdles and/or top-down approach

• Adverse Impact Limited• Differential Outcomes (Rotundo & Sackett, 2002)

– Task Performance (Technical Core)

– Organizational Citizenship Behavior (Contextual Performance)

– Counterproductive Work Behavior

• Add Satisfaction/Tenure as another Key Outcome

1414

Combining Personality & Cognitive Ability Tests

• Level of correlations are low:GMA x C = .02 GMA x ES

= .17

Math x C = -.15 Math x ES = .17

Reading x C = -.05 Reading x ES = .11(Ackerman & Heggestad, 1997)

1515

Combining Personality & Cognitive Ability Tests

Creating opportunity for incremental validity especially as criteria vary

Correlations between general cognitive ability and personality tests and measures of job performance in Project A

Cog Pers Both Criteria

.63 .26 .67 Core technical proficiency

.65 .25 .70 General Soldiering Proficiency

.31 .33 .44 Effort and Leadership

.16 .32 .37 Personal Discipline

.20 .37 .42 Physical fitness and military bearing

(McHenry, Hough, Toquam, Hanson, & Ashworth, 1990)

1616

Educational Examples

Retention Behavior R1 Cumulative R1, 2

ACT .176 .176

Motivation .146 .227

Social Engagement .097 .200

Personality Combined .155 .2321 Correlation of fitted probabilities and dichotomous retention outcome

2 SRI scale scores and ACT Composite score used as predictors

1717

Educational Examples

Academic Performance R1 Cumulative R1, 2

ACT .464 .464

Motivation .297 .545

Self Regulation .196 .467

Personality Combined .315 .553

1 Multiple R from linear regression model2 SRI scale scores and ACT Composite score used as predictors

1818

ACT: Foundational and Personal Skills Assessments measure Different Applied Job Skills:

CommunicationProblem Solving

Interpersonal Personal

Business Writing Applied Technology

Teamwork Performance

Listening Applied Mathematics

Talent

Reading for Information

Locating Information

Fit

Writing Observation

1919

“If they staged a slowdown, how would we know?”

Harvard Business Review. March 2007. p. 90

2020

Highly desirable level of expected performance

Moderately desirable level of expected performance

Less desirable level of expected performance

WorkKeys Performance Assessment measures personal behaviors and attitudes critical to workplace success. A high score on the Performance Index indicates higher likelihood of having successful work attitudes and engaging in safety behaviors.

64 Performance Index -- The overall rating for an examinee based on the combination of General Work Attitudes and Risk Reduction scores.A moderate Performance Index (combination of General Work Attitudes and Risk Reduction) suggests a candidate may be moderately desirable. Individuals with similar scores are likely to perform at a moderate level of productivity and work safety.See below for additional interpretive information about this person’s general work attitudes and risk reduction tendencies.

What This Means:

64PERFORMANCE

10 20 30 40 50 60 70 80 90 100

50General Work Attitudes

Risk Reduction 89

50

89Risk Reduction—Tendency to avoid engaging in risky behaviors, such as noncompliance with safety rules and conflict with supervisors and coworkers.A candidate with a high Risk Reduction score may be an employee who: Consistently follows safety rules and procedures Is consistently alert to job risks Is very unlikely to engage in inappropriate interpersonal behaviors such as, aggression or hostility

General Work Attitudes—Positive and productive attitudes toward work tasks, coworkers, the organization itself, and other work-related behaviors. A candidate with a moderate General Work Attitudes score may tend to be an employee who: May be disagreeable with coworkers or supervisors under stressful circumstances Is usually conscientious about completing work on time Is usually honest with coworkers or supervisors Will make appropriate use of company assets under most circumstances

Percentile Scale Definitions

0

Performance Assessment

WorkKeys Performance Assessment Profile

Page 1 of 2

Employer Report

Examinee: Alvin C. Tracey

Examinee ID: XXXXX7890

Percentile Rank: Approximate percent at or below score

Report for: Abbatoir Industries

Site: Iowa City, IA

Test Date: 4/12/07 Performance Score Report

2121

“It was about here, wasn’t it, Ed, when you came on board as sales manager?”

Harvard Business Review. March 2007. p. 90

2222

Talent Score Report

90Stability

50Sociability

50Savvy

90Order

50Optimism

40Influence

20Goodwill

65Drive

85Discipline

35Creativity

25Cooperation

90Carefulness

0 10 20 30 40 50 60 70 80 90 100

WORKEYS TALENT ASSESSMENT PROFILE

90Stability

50Sociability

50Savvy

90Order

50Optimism

40Influence

20Goodwill

65Drive

85Discipline

35Creativity

25Cooperation

90Carefulness

0 10 20 30 40 50 60 70 80 90 100

WORKEYS TALENT ASSESSMENT PROFILE

90Carefulness: Tendency to think and plan carefully before acting.

This individual’s responses suggest that he or she is cautious, deliberate, and pays close attention to detail in the workplace. Responders at this score level tend to think carefully before acting or speaking. They always consider the consequences of theiractions, and their decisions are usually well thought-out.

85

Order: Tendency to be well organized as well as keeping the work area neat and tidy.

The examinee’s responses suggest that he or she is well-organized, and consistently keeps physical surroundings neat and tidy. Individuals who respond at this score level are always methodical in their manner and maintain a structured professional environment.

Discipline: Tendency to begin tasks and complete them without becoming distracted or bored.

The examinee’s responses suggest that he or she commits to work duties until they are complete. Individuals who respond at this score level take responsibility and can always be relied upon to get their work done on time. They are not easily distracted, and always persist through challenges until the task is done.

Talent Assessment

WorkKeys Talent Assessment measures personal and workplace behaviors and attitudes. This report is designed to help identify examinee strengths and weaknesses in order to ensure success in the workplace.

Percentile Scale Definitions

Capitalize on Individual Strengths

90

Page 1 of 3

Employer Report

Stability: Tendency to maintain composure and rationality in situations of actual or perceived stress.

This individual’s responses suggest that he or she maintains his or her composure even when faced with highly stressful situations. Individuals who respond at this score level tend to remain calm and even-tempered in their conduct, and they feel confident in their ability to handle the pressure and stress of working under deadlines..

90





Site: Iowa City, IA

Test Date: 4/12/07

2323

2424

FIT Score Report

WorkKeys Fit Assessment measures the fit between an examinee's work-relevant interests and values and the characteristics of occupations. Worker satisfaction and commitment are associated with greater fit. Examinee-specified primary occupation is in BOLD.

Fit Index for Related Occupations

Occupations specified by the examinee, as well as related occupations, are ranked by Fit Index. Examinee-specified occupations are in BOLD.

0 10 20 30 40 50 60 70 80 90 100

Fit Assessment

1 of 3

Employer Report

Code Title Fit Index Level of Fit

11-3071.00 Transportation, Storage, and Distribution Managers 86 High11-3011.00 Administrative Services Managers 80 High11-2022.00 Sales Managers 79 High11-3071.01 Transportation Managers 77 High11-3071.00 Storage and Distribution Managers 69 Moderate11-3042.02 Training and Development Managers 61 Moderate11-3040.00 Human Resources Managers 56 Moderate11-3061.00 Purchasing Managers 56 Moderate 11-3031.02 Financial Managers, Branch or Department 55 Moderate11-3041.00 Compensation and Benefits Managers 54 Moderate11-3049.99 Human Resources Managers, All Other 54 Moderate11-3031.00 Financial Managers 53 Moderate11-2011.00 Advertising and Promotions Managers 51 Moderate11-2021.00 Marketing Managers 50 Moderate 11-2031.00 Public Relations Managers 50 Moderate11-3031.01 Treasurers and Controllers 42 Moderate11-3051.00 Industrial Production Managers 29 Moderate11-3021.00 Computer and Information Systems Managers 27 Moderate11-9011.02 Crop and Livestock Managers 23 Low11-9011.01 Nursery and Greenhouse Managers 14 Low



Fit Index for Occupations Specified by Examinee

86Transportation, Storage, and Distribution Managers11-3071.00

56Purchasing Managers11-3061.00

56

61

54

Human Resources Managers

Training and Development Managers

Human Resources Managers, All Other

11-3040.00

11-3042.00

11-3049.99

86Transportation, Storage, and Distribution Managers11-3071.00

56Purchasing Managers11-3061.00

56

61

54

Human Resources Managers

Training and Development Managers

Human Resources Managers, All Other

11-3040.00

11-3042.00

11-3049.99


Fit scores are based on the examinee's responses to the Interest Inventory and Work Values Inventory.


Site: Iowa City, IA

Test Date: 4/12/07

2525

2626

Selection Solutions

• Reducing Risk– Task Competence through WorkKeys and job

profiling– General Work & Safety

• Increasing Tenure– Task Competence through WorkKeys and job

profiling– Fit

• Getting the Right Person– Talent Benchmarking– Past Work Performance Record

2727

Coaching & Development Solutions

• Career Exploration– Task Competency– Fit

• Leadership Development– Talent– Fit

• Teamwork– Talent

2828

Selection % Candidate Success %

Selected Success %

Cost per failure

ROI per 100 candidates

$5,000 $13,000 25% 54%

$10,000 $27,500

$5,000 $13,500 10%

50% 80% $10,000 $28,500

$5,000 $23,500 25% 45%

$10,000 $48,500

$5,000 $26,000 25%

50% 72% $10,000 $53,500

$5,000 $26,000 25% 36%

$10,000 $53,500

$5,000 $33,500 50%

50% 64% $10,000 $68,500

Return on Investment Approximations

under Various Scenarios

Notes: Selection % = the percentage of the candidate pool selected for hire, Candidate Success % = the percentage of the candidate pool that would be successful if hired, Selected Success % = the percentage of the selected candidate pool that will be successful, Cost per failure = the average cost for each unsuccessful employee relative to each successful employee, ROI per 100 candidates = the average return on investment for

the selection procedure assuming a $15 fee per candidate.

2929

Final Thoughts

• Adverse Impact may be reduced when combining tests– Still recommend “multiple hurdles” approach

• Incremental Validity Research underway:– Combination of Task & Personality

measures– Differential work outcomes– ROI x Solution

3030

References• Ackerman, P. L., & Heggestad, E. D. (1997). Intelligence, personality, and interests: Evidence

for overlapping traits. Psychological Bulletin, 121, 219-245.

• Brown, K. G., Le, H., & Schmidt, F. L. (2006). Specific aptitude theory revisited: Is there incremental validity for training performance? International Journal of Selection and Assessment, 14, 87-100.

• Casner-Lotto, J. & Barrington, L. (2006). Are they really ready to Work? Society for Human Resource Management. http://www.shrm.org/hrresources/surveys_published

• McHenry, J. J., Hough, L. M., Toquam, J. L., Hanson, M. A., & Ashworth, S. (1990). Project A validity results: The relationship between predictor and criterion domains. Personnel Psychology, 43, 335-354.

• Robbins, S., Allen, J., Casillas, A., Peterson, C., & Le, H. (2006). Unraveling the differential effects of motivational and skills, social, and self-management measures from traditional predictors of college outcomes. Journal of Educational Psychology, 98, 598-616.

• Rotundo, M., & Sackett, P. R. (2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach. Journal of Applied Psychology, 87, 66-80.

• Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & de Fruyt, F. (2003). International validity generalization of GMA and cognitive abilities: A European community meta-analysis. Personnel Psychology, 56, 573-605.

• Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262-274.

• Schmidt, F. L., Shaffer, J., & Oh. I. (2007). Reassessing the Relative Importance of Cognitive Ability and Personality in Job Performance and Training Performance: Some Surprising New Research Findings. Paper presented at the 2007 ATP conference, Palm Springs, CA. Feb. 6.

3131

Incorporating Foundational and Soft Skill Assessments

For questions regarding this presentation or for further information contact:

Steve Robbins at 319-337-1227 or [email protected],

Gary Nolan at 319-337-1526 or [email protected],

Documents

WorkKeys Innovations: A Holistic Solution for WIRED West Michigan