7/30/2019 Steps in Design of Test

1/2

1. Define the Outcome

Determine what on the job attribute you need to measure (e.g. Job Performance).

2. Define the Psychometric Construct

Define and research the trait characteristic that will predict the desired business / on the job outcome. (e.gGeneral

Cognitive Ability).

3. Write Items to Test

Design draft questions to measure the target psychometric attribute. Writing good test items is difficult and the

guidelines used are based on published research. Initially, many questions are drafted. Only 10-30% of the initial

items will typically pass the various quantitative and qualitative quality control processes and make it into the final

test.

4. Choose the Response Format

Response formats are chosen and designed on the basis of the test function, structure and delivery method.

5. Scoring & Reporting

Scoring thresholds and reporting are determined. Drafts of both are created.

6. Administration

Instructions and the administration process are determined. In some cases new technology to aid delivery is required.

7. Trial Population

A group of people who are considered representative of the final population are selected to test the draft

assessment. Consideration needs to be given to how you will access and incentivise this group, as well as thelogistics of the administration process.

8. Conduct Trial

The trial population completes the full set of draft questions in conditions designed to mirror the environment the final

assessment will be delivered in (as closely as possible).

9. Question Analysis

The results of the trial are analysed to demonstrate the psychometric properties of the questions. That is, how

effective the questions are at measuring the desired attribute or trait. Stages 6 and 7 are repeated until a final shortlist

of questions is produced.

10. Full Form Validation

Once finalised, the resulting shortlist of items is now considered the final full form. The full form is tested on a new

trial population to determine how effectively the questions work together to measure the desired outcome. This is

done by comparing the results of the assessment to either an existing measure of the same construct or to job

metrics such as existing performance data.

http://www.onetest.com.au/home/Cognitive-Abilityhttp://www.onetest.com.au/home/Cognitive-Abilityhttp://www.onetest.com.au/home/Cognitive-Abilityhttp://www.onetest.com.au/home/Cognitive-Abilityhttp://www.onetest.com.au/home/Cognitive-Abilityhttp://www.onetest.com.au/home/Cognitive-Ability

7/30/2019 Steps in Design of Test

2/2

11. Full Form Review

The effectiveness and order of questions is analysed to ensure that, in combination, they provide the most effective

measure of the desired outcome. Stages 8 and 9 are repeated until the final set of questions meet necessary

psychometric standards.

12. Normative Data

Relevantnormative datais collected for score comparison following assessment publication.

13. Publish

The assessment is released and ready to use. Supporting technical reports, collateral, example reports and training

resources are developed.

14. Ongoing Maintenance and Validation

Item exposure and performance is monitored, normative data is reviewed, and the assessment and reporting is

modified as required. Ongoing validation studies are conducted to ensure the assessment continues to measure the

required trait.

So, next time you are evaluating psychometric testing options, ask to review each tests development and validation

documentation to ensure the test you choose actually predicts real life outcomes and isnt just smoke and mirrors'.

http://www.onetest.com.au/home/Normative-Groupshttp://www.onetest.com.au/home/Normative-Groupshttp://www.onetest.com.au/home/Normative-Groupshttp://www.onetest.com.au/home/Normative-Groups