Conducting a Job Analysis to Establish the Examination

Content Domain

Patricia M. Muenzen

Associate Director of Research Programs

Professional Examination Service

What is Job Analysis?

• Set of procedures to used to describe a job or profession

• Foundation for licensure and certification examination development

• Key source of validity evidence for the interpretation of test scores

Job Analysis Process

• Create initial description of the job or profession

• Develop and administer job analysis survey• Analyze survey data to identify the

examination content domain• Create written report of job analysis

procedures and findings

Create Initial Description of the Job or Profession

• Collect information about the job from a variety of sources– subject-matter expert committee– written documents– observations– individual and group interviews

• Translate information into domains, tasks, and knowledge, skills, and abilities (KSAs)

Develop Job Analysis Survey

• Develop rating scales for domains, tasks, and KSAs. Rating scales might include:– Domains: time spent and criticality to public

protection– Tasks: how frequently performed, criticality to

public protection, performance at entry-level– KSAs: frequency of use, criticality, required at

entry level

• Create demographic questionnaire

Administer Job Analysis Survey

• Conduct pilot test of survey instrument• Select survey sample - typically job

incumbents, but could also include educators, supervisors

• Distribute survey: invitation, cover letter, follow-up mailing

Analyze Survey Data

• Review characteristics of respondents• Calculate means, standard deviations,

frequency distributions for rating scales• Compare ratings of subgroups based on

demographic variables of interest, such as practice setting, academic background

Identify the ExaminationContent Domain

• Create decision rules to identify key tasks– for example, tasks rated as (1) performed at entry

level and (2) at least moderately critical to public protection

• Create decision rules to identify key KSAs – for example, KSAs rated as (1) used frequently

and (2) necessary at entry level

• Calculate test weightings for domains, key tasks and/or KSAs using questionnaire data

The Job AnalysisWritten Report

• The job analysis written report documents the process by which the examination content domain was identified. Thus, it is an important source of evidence for the content validity of the exam.

• The examination content domain/test plan documented in the written report guides all item writing and test construction activities.