Presented by: LCdr Jeff Gauger

Prepared by: Maj Rankin & Maj Girard

Director Military Personnel Research Analysis

Job Analysis

for

Selection in the Canadian Armed Forces

IAMPS 2015 – Lisbon, Portugal

Outline

Objective

Phase 1 - Combined Job Analysis Method Overview

Process

Cognitive Task Analysis (CTA)

Phase 2 - Post Focus Groups - Rating

Phase 3 – Data Analysis Task Importance Value

Interrater Agreement (rwg)

Phase 4 – Selection Tool Selection

Objective

Create Job Description Occupation Level

Basic Qualification Level

Job specific

Determine critical KSAO’s

Create appropriate selection process

Combined Job Analysis Method (CJAM)

This process utilizes work orientated processes contained in a functional job analysis (FJA – Fine & Cronshaw, 1999) to provide information about the outputs, tasks, and relevant KSAOs required for the job.

Combined with an importance analysis, during which SMEs rate each task’s and KSAO’s importance for selection (Brannick, Levine, & Moregson, 2007).

CJAM Process – Initial Preparation

Research available materials

Determine number of Focus Groups

Request SMEs 6 to 8

Good understanding of the current job

CJAM Process – Day 1 - Staging

Provide SMEs with overview of: Why they are there

The process over the 2 days

Scoring

End result

CJAM Process – Day 1 Generation of Outputs

Outputs: a functional category of work under which various task fall “What do you get paid to do?”

A baker bakes bread.

May be started with a suggested list (2 schools of thought)

Provides a framework

Recorded with a computer / screen

Group discussion – consensus is the key

CJAM Process – Day 1 Generation of Tasks

Tasks: collection of activities that are directed toward the achievement of a specific output.

All tasks required to complete output

Recorded by computer / screen

Group discussion – consensus is the key

CJAM Process – Day 2 KSAO Generation

KSAOs are generated for all tasks under each output

Group discussion – consensus is the key

Less of focus on knowledge Focus is on the knowledge required for selection

Cognitive Task Analysis (CTA)

Object-orientated cognitive task analysis and design (Wei & Salvendy, 2006)

Based on human information processing theory

Decompose tasks identified as crucial for selection entered into the model to identify the cognitive facets of interest (Girard, 2009)

Process tracing

CTA - Process

Tasks identified

Identify heavy cognitive loaded tasks

Process tracing SMEs describe in detail the steps taken to complete the task

No more than 6 to 8 steps

Checklist of steps is not cognitive heavy load

Identify specific cognitive facets

Selection tests can be identified to measure cognitive domains of interest

TASKS

Observe battle ground

Attend to multiple stimuli

Identify information related to plan

Analyzes information

COGNITIVE FACETS

Working memory

Sensory memory

Long term memory

Example of CTA

Post Focus Group – Task Rating

Serving members are sent surveys for ratings

Serving members rate all tasks and sub-tasks Task Difficulty: level of difficulty in performing task (7 point scale)

Task Criticality: degree that incorrect performance of task will result in negative consequence (7 point scale)

Task Importance Value (Barrick et al., 2007) Sum of task difficulty and task criticality

Sum of greater than 6 considered important

Post Focus Group– KSAO Rating

SMEs rate final list of KSAOs: Necessary for applicant to possess? (yes/no)

Practical to expect in applicant pool? (yes/no)

To what extent is trouble is likely if it is ignored in selection? (Scale of 1 – 5)

To what extent do the KSAOs distinguish from the superior and average worker? (Scale of 1 – 5)

Criteria for KSAO into Selection

A clear majority of SMEs must indicate that it was necessary for an applicant to possess the KSAO for selection

A clear majority of SMEs must indicate it is practical to expect KSAO in applicant pool

Mean rating on “trouble-likely” must be 3 or greater

(Brannick et al., 2007)

Criteria for KSAO into Selection

Majority defined as m > .75

Trouble likely defined as m > 3 m > 1. 5 recommended by Brannick et al.

Superior / average worker distinction Mean rating of 2 or less used for screen out measures (pass/fail)

Mean rating of 2 or more used to rank order applicants

Interrater Agreement (rwg)

Disagreement among raters on the importance of a task/KSAO is crucial to the evaluation of ratings

Demaree & Wolf’s (1993) single item index of agreement known as within group agreement (rwg) is calculated to identified critical tasks and KSAOs

Higher values of (rwg) reflect higher agreement

rwg > .70 are acceptable levels of within-group agreement (Lance, Butt, & Michels, 2006)

Summary

Combined Job Analysis Method Identify outputs

Identify tasks

Identify KSAO

Cognitive Task Analysis (CTA) Identify cognitive heavy tasks

Process tracing to identify underlying cognitive process

Post Focus Groups Identify critical tasks & KSAOs

Interrater Agreement (rwg)

Questions