Download pdf - Nlng,Ats Shl

8/18/2019 Nlng,Ats Shl

1/28

© SHL GROUP plc, 2000

The Technical Test BatteryManual & User’s Guide

INTRODUCTION TO THE TTBRationale

Short Descriptions of the Tests

Choice of Tests

Administration

Scoring

Technical Support

Distribution Controls

RATIONALE

The use of ability tests in the technical and engineering sectors of the economy is wellestablished. Since its launch in 1979, SHL’s Technical Test Battery (TTB) has been widelyused for the assessment and selection of engineering and craft apprentices as well astechnicians and process operators. The emphasis in the TTB has always been on theassessment of relevant work-related skills rather than the measurement of abstractconceptions of human intelligence. Accordingly, the TTB has been updated so as to reflect

changes that have taken place in the workforce since its original launch. Followingconsultations with major users and extensive re-analysis (including sex and ethnic bias), wehave replaced or modified several of the original tests. The new tests are an improvement infour main respects:

1. They are more up to date

2. More reliable

3. More valid

4. Less prone to bias.

Each of the eight tests in the new edition of the TTB is presented separately to ensuregreater flexibility, enabling users to select the most appropriate mix of abilities to be tested for

a given job. Additionally, two versions of the most popular tests will be available to allow useof parallel versions on different occasions. This is valuable in helping to avoid any potentialproblem which may result from the over-exposure of individual tests, as well as in theretesting of individual candidates.


2/28


2

As previously mentioned, the TTB can be used for the assessment and selection of staff for awide range of technical jobs. The types of industry for which the TTB has been used include:paper and printing, gas, railways, heavy engineering, glass, the motor industry, chemicalsand pharmaceuticals, the building industry and metal refining and processing. This list is byno means exhaustive of the areas in which the tests have been or could be used. The list

should also be considered to include jobs in both the public and private sectors.

SHORT DESCRIPTIONS OF THE TESTS

The updated TTB is composed of eight tests, and these are summarised below. Parallelversions of four tests, namely, NT2.2, MT4.2, VT5.2 and ST8.2 will be available a fewmonths after the initial launch of the updated TTB.

Verbal Comprehension (VT1.1) – measures vocabulary and basic word skills usinglanguage which reflects the requirements of technical occupations.

Numerical Computation (NT2.1 and NT2.2) – assesses basic number skills with theemphasis on straightforward calculations and the understanding of arithmetical operations.

Visual Estimation (ET3.1) – measures spatial perception involving the comparison ofshapes and lengths.

Mechanical Comprehension (MT4.1 and MT4.2) – measures understanding of basicmechanical principles.

Technical Understanding (VT5.1 and VT5.2) – measures a candidate’s understanding ofwritten passages containing the type of verbal material likely to be encountered in a technicalsetting, such as equipment manuals and operating instructions.

Numerical Reasoning (NT6.1) – measures more advanced number skills, with the emphasison reasoning and recognising short cuts to reach solutions, rather than complex calculations.

This test has been developed specifically for use with hand-held calculators.

Fault Diagnosis (FT7.1) – measures the candidate’s ability at finding faults in systems. Thecandidate must locate which element of an arrangement of processors is not working asspecified.

Spatial Recognition (ST8.1 and ST8.2) – measures the ability to recognise shapes in twodimensions, even after rotation.

CHOICE OF TESTS

Choice of tests should be founded on careful job analysis, which will identify the abilities and

attributes required to perform the job. The resulting, well researched person specification,read in conjunction with test descriptions, validity evidence, etc, presented in this manual andelsewhere, should enable the selection of an appropriate combination of TTB tests.Wherever possible, validity research should be conducted prior to the introduction of tests toensure that they are appropriate for their intended purpose and fair to all applicants.


3/28


3

ADMINISTRATION

The administration of the TTB tests is a straightforward procedure through the use ofseparate administration cards. The total testing time for this battery is 1 hour and 50minutes. With administration and example questions, it would take just over 3 hours for all

the tests to be completed. These tests can, however, be used singly or in variouscombinations to establish a profile of an individual’s abilities depending on the skills requiredfor a particular job.

Experience has shown that a selection of three or four of the tests is typically moreappropriate, and most combinations of this number of tests can be completed in about 1½hours.

SCORING

The tests in the TTB can be scored either by hand or machine. They are all in multiple-choice, paper and pencil format. The multiple-choice format and special answer sheetspermit quick scoring using plastic scoring key overlays. In additions, the tests can be scored

using the NCS “Opscan 5®” optical mark reader, connected to a personal computer.

Machine scoring is particularly effective where a large number of applicants need to betested. (Please contact SHL’s Computer Services Department for further information.)

TECHNICAL SUPPORT

The Technical Test Battery is supported by SHL’s ongoing research and developmentprogramme and test users are regularly updated with recent norms and validation studiesfrom a range of organisations.

Norms for particular organisations can be provided on receipt of raw scores (and basic bio-

data such as age, sex, educational qualifications), and test users are encouraged to senddata to us. Advice or assistance with job analysis and validation studies is also available.Such research is important to ensure test relevance as only relevant measures contribute tothe effectiveness and intrinsic fairness of selection decisions.

DISTRIBUTION CONTROLS

These tests are not freely available to the general public, but are distributed under strictcontrol to suitably trained and qualified test users. SHL runs training courses in occupationaltesting to standards laid down by international psychological associations.


4/28


4

2. DESCRIPTION OF THE TTB TESTS

Overview

Verbal Comprehension

Numerical Computation

Visual Estimation

Mechanical Comprehension

Technical Understanding

Numerical Reasoning

Fault Diagnosis

Spatial Recognition

OVERVIEW

The SHL Technical Test Battery is a collection of tests designed specifically for the selectionand allocation of a wide range of apprentice and technical personnel.

The eight tests in the TTB can be divided into two groups. NT2.1, MT4.1, VT5.1 and ST8.1made up a group of core skills, while the remaining tests measure more specific aptitudes.Together, the tests provide for a wide ability range from those with no formal educational

qualifications through to GCSE or ‘O’ level and ‘A’ level standard.

All the tests are available in individual booklets and are designed to be used separately or inany combination. The choice of test or tests to be used should be based on the results of a

job analysis, a detailed job description or a validation study. This will identify the attributesrequired to perform effectively in the job.


5/28


5

The time limits and number of items in each test are given below:

TEST NUMBEROF

ITEMS

TIMELIMIT

(minutes)

APPROX. ADMIN TIME

(minutes)

VT1.1NT2.1

ET3.1

MT4.1

VT5.1

NT6.1

FT7.1

ST8.1

Verbal ComprehensionNumerical Computation

Visual Estimation



Numerical Reasoning

Fault Diagnosis

Spatial Recognition

3636

36

36

36

25

36

36

1210

10

18

20

12

18

10

2020

20

30

30

20

30

20

More detailed descriptions of the individual TTB tests follow. In order to provide a clearunderstanding of the item format, copies of the example pages are also included. Thesereflect only the easier items. For an overview of the complete test, the question booklet itselfshould be consulted.

The indications given in this manual of the jobs for which these tests may be applicable areintended as guidelines only; test relevance should be examined in relation to each individual

job. Users who have concerns about test choice should seek professional advice beforeimplementing the tests.

SHL does not accept liability for any loss of whatsoever nature suffered by any person orentity as a result of placing reliance on this Manual and User’s Guide.

VERBAL COMPREHENSION VT1.1

The TTB Verbal Comprehension test (VT1.1) has been designed as a measure of basic wordskills suitable for a wide range of occupations, from basic craft apprentices up to those oftechnician or supervisory level.

The item format of VT1.1 is varied and includes sentence completion, analogies andsynonyms. The language, which has been selected to reflect the requirements of technicaltraining, is relevant to most semi -skilled jobs within the technical sector; however, no priorknowledge of ‘technical’ words is assumed. Indeed, even though many of the sentencecompletion items are set in a technical context, no specialist knowledge is required to selectthe missing word. Some of the language used for other questions in this test is deliberately

of a more general nature, since, in any work environment, a reasonably wide vocabulary isessential if an individual is to communicate effectively. In addition, the test is not highlyspeeded, with the majority of candidates being expected to complete the test within the timeallowed.

Time limit: 12 minutes

Number of items: 36


6/28


6

Range: GCSE level up to ‘A’ level.

Total administration time (including instructions) approximately 20 minutes


7/28


7

NUMERICAL COMPUTATION NT2.1 and NT2.2

The Numerical Computation tests have been designed to measure the four basic arithmeticoperations of addition, subtraction, multiplication and division.

In most modern technical environments, calculators are frequently used to perform basiccalculations, so saving time and reducing the number of computational errors. This tendency


8/28


8

to use calculators at every opportunity, however, increases people’s dependence upon themand reduces their ability to spot any errors which may occur.

The emphasis in this test is placed on the understanding of the relationships betweennumbers rather than straightforward computational ability. The applicant is required not onlyto calculate but also to follow and choose the correct sequence of mathematical operations in

order to derive the correct answer: skills which remain of crucial importance even in the ageof the calculator.

Candidates are required to work through problems where one operation or number has beenomitted from a sequence and to select the missing term from five alternatives. Percentages,decimals and fractions are also covered in this test but there is a deliberate avoidance of themore advanced number-skills which are covered in the Numerical Reasoning test (NT6.1).


Number of items: 36

Range: No formal qualifications up to ‘A’ level.

Total administration time (including instructions) approximately 20 minutes.


9/28


9

VISUAL ESTIMATION ET3.1

The Visual Estimation test has been designed to measure a candidate’s ability to accuratelyestimate length and breadth as well as to identify minor differences in a variety of basicshapes. Some examples of activities where visual estimation skills are useful are: a fitterchecking parts prior to assembly, a turner machining a precision component, a joiner makinga close fitting joint and a heating engineer estimating pipe runs.

In this test, the candidate is asked to choose the two figures from a set of five which areidentical. The test is arranged so that progressively finer discriminations are required inorder to select the two identical figures. These differences range from 3 mm in the earlyitems to 1 mm towards the end of the test.


Number of items: 36

Range: Suitable for any educational level.



10/28


10


11/28


11

MECHANICAL COMPREHENSION MT4.1 and MT4.2

Unlike many previous tests of mechanical comprehension which purport to measure anindividual’s understanding of basic mechanical principles, the emphasis here is on morespatially orientated items, which require the candidate to visualise movement, rather than

recall situations obviously based on ‘textbook’ laws of physics. An effort has been made toselect items aimed at identifying aptitude and potential rather than current knowledge whichmay be better measured by other means, eg GCSE results.

The test is made up of 36 questions, each relating to a three-dimensional drawing. Mostitems involve simple mechanical devices, such as gears, pulleys, levers and liftingmechanisms.

Due to different levels of interest and involvement between men and women and betweenpeople of different ethnic backgrounds, mechanical comprehension is an area where groupdifferences are likely to occur. Therefore, when employing these tests, the user shouldalways ensure that these skills are relevant to the job or function in question.


Number of items: 36




12/28


12

TECHNICAL UNDERSTANDING VT5.1 and VT5.2

The Technical Understanding tests measure the ability to follow written instructions in atechnical context. Candidates are presented with passages of written information. Afterreading each passage, they have to answer four multiple choice questions which test theirunderstanding.

To use new equipment and appliances properly often requires an operator to read, and followprecisely, written instructions laid out in detailed manuals. Even the most basic of writteninstructions are capable of misinterpretation and, apart from avoidable delays, mis-operationof equipment could lead to its damage or compromise the safety of the operator or others.

The topics covered include operating instructions and procedures associated with a widerange of equipment and technology, just the kind of materials to be found in technicalmanuals or system guides. No prior knowledge of technical words or jargon is assumed andso applicants with little or no technical background can be assessed.


13/28


13

This ability is relevant to a wide range of technical work, particularly that involving writtendocumentation for the installation and maintenance of machinery and also in the training oftechnical personnel.

Research into VT5.1 has shown group differences which appear to reflect the amount of

verbal information contained in the test. This is particularly true for individuals for whomEnglish is not the first language. Users must therefore ensure that the verbal loadingintroduced by this test is justifiable against job demands. See section 10, ‘Testing and EqualOpportunity’ for further details.


Number of items: 36

Range: From no formal qualifications through to GCSE or ‘O’ level.



14/28


14


15/28


15

NUMERICAL REASONING NT6.1

In this test the emphasis has been placed on reasoning skills rather than on the simplernumber calculations covered in the Numerical Computation tests (NT2.1 & NT2.2). TheNumerical Reasoning test comprises written problems, expressed within the context of

technical work, which involve percentages, fractions, decimals and diagrams, as well asknowledge of the four basic rules of arithmetic.

The need for long and tedious calculations has been avoided in NT6.1. It is expected thatcandidates will spend more time in generating and choosing strategies to solve numericalproblems rather than in calculating their answers. Candidates are allowed to use calculatorsin this test to reflect the wide availability of such aids at virtually every workplace. The testhas also been designed to give better discrimination at high levels compared to theNumerical Computation tests NT2.1 and NT2.2. Nevertheless, for those with still highernumerical skills, such as graduates, the Numerical Critical Reasoning tests from the MGIBand CRTB will be more suitable.


Number of items: 25

Range: Suitable for GCSE and GCE ‘O’ level groups up to GCE ‘A’ level candidates.



16/28


16

FAULT DIAGNOSIS FT7.1

This test is designed to measure candidates’ ability to locate faults in systems.

The growing complexity of equipment, especially that associated with electronics andcomputing, brings with it greater demands in terms of maintenance and servicing. The abilityof an individual to isolate faults and problems in a system will therefore be of increasingimportance in reducing maintenance costs and equipment downtime.

This test measures the ability to locate faults in a logical system. The test requires anindividual to locate which element in an arrangement of coded symbols is not working asspecified. The test is based upon a PROCESS CONTROL PANEL, which contains tenbuttons or processors. The control panel, together with a complete listing of all processorsand their functions, is shown on each double page. No specialised knowledge of fault-finding

or electronics is required and so applicants with little or no technical background can beassessed.

The ability to find and locate faults is relevant in a wide range of areas both in machinemaintenance and electronics, and also in software, process control systems andcommunications.


17/28


17

This test could be compared with FTS4 (from the Applied Technology Series) but, since it isof a slightly lower level, it is suitable for candidates with little educational attainment up to ‘A’level standard. The test could also be used for candidates with some work experience.


Number of items: 36

Range: From GCSE upwards to ‘A’ level.



18/28


18


19/28


19


20/28


20


21/28


21

SPATIAL RECOGNITION ST8.1 and ST8.2

These tests measure the ability to recognise and match shapes in various orientations. Thisskill is of crucial importance in this design, manufacture and assembly of individualcomponents and machinery.

The candidate is asked to choose the one figure from a set of five which is identical to agiven shape. The choices are often rotated relative to the given pattern and the test isorganised so that progressively finer discriminations are required in order to select theidentical figure.

This ability is likely to feature in the design, production and maintenance of machine tools,engineering components, mechanical sub-assemblies and other activities involvingdesigning, fitting and inspection.

The level of spatial aptitude measured by this test is probably slightly lower than thatmeasured by STS5 (from the Applied Technology Series) or SIT7 (from the InformationTechnology Test Series).


Number of items: 36




22/28


22


23/28


23


24/28


24

3. THE DEVELOPMENT OF THE TTB

Background

Defining the Concept

Test and Item Design

Test and Item Review

Item Trialling

Item Analysis

Final Form Trials and Analysis

Manual Writing and Ongoing Technical Reports

BACKGROUND

Like all SHL test batteries, the TTB has been through a lengthy development process. Theoriginal TTB tests were first developed in 1979. A full review of all the TTB materials wasthen carried out in 1991. Although some of the present test content is based on the 1979work, this has all been revised, amended and supplemented by new material in addition, allthe tests have been retrialled. The most relevant description of the development process forthe present battery is therefore this current revision rather than the original development, and

it is this which is mainly reported here.

DEFINING THE CONCEPT

The original TTB was developed after extensive research in the technical/engineering sector.Several key skills, required by those employed in this sector, were identified. These skillswere reviewed and refined in 1991 on the basis of further research, recent validation studiesand consultation with many companies and training organisations with an interest in thisarea. Broadly speaking, the skills which were found to be important twelve years ago are stillrelevant today.

TEST AND ITEM DESIGN

The rationale for each of the TTB tests, including information on the sources used for testcontent, is summarised below.


25/28


25

Verbal Comprehension

Designed to measure basic work skills, Verbal Comprehension is a vocabulary based testwhich uses a variety of item types to measure a candidate’s understanding or application ofwords. The language used has been chosen to reflect that which may be encountered in a

‘technical’ working environment although obscure, out-dated, and jargonistic words havebeen avoided. The test has generous time limits which allows most candidates to complete itwithin the time allowed.

Numerical Computation

These tests measure the ability to identify and carry out standard arithmetical operationswithout the aid of a calculator. Although it is recognised that aids to calculation are readilyavailable in most work places, there are still situations where dependence on them for simplecomputation could lead to significant (and often unacceptable) reductions in efficiency.

The test content has been made context -free to ensure applicability across a wide range of jobs. Most items contain operations on whole numbers, but decimals, fractions and

percentages are also included in the test.

The format of the items requires candidates to demonstrate an understanding of therelationships between numbers and arithmetical operations rather than relying purely oncalculation alone. This results in a measure which samples a more complete set of basicnumber skills.

Visual Estimation

The ability to make accurate visual comparisons between various components is an essentialpart of many manufacturing jobs, as is the ability to gauge size without the constant need totake repeated measurements. Visual Estimation is specifically designed to measure thisimportant ability. The original test format has been completely revised, with uniform

rectangles replacing lines and items involving angles being dropped. The test getsprogressively more difficult and requires candidates to make finer and finer discriminations.Visual Estimation is a straightforward test to administer and is suitable for groups of anyeducational level.


This test is designed to measure the ability of candidates to understand the workings ofsimple mechanical devices by recognising and applying appropriate mechanical principles.The aim of Mechanical Comprehension is to identify aptitude rather than current knowledgeas this may be better measured by academic examinations. Items obviously based on ‘textbook’ laws of physics have, as far as possible, been avoided with many items requiringcandidates to visualise movement in order to find the solution to a problem.

Whilst every effort has been made to choose items which are fair and equally predictive forall groups, our experience of this test is that males are generally more likely to score higherthan females. This difference is usually smaller when males and females from actualapplicant groups are compared. However, it is important that users of this test are able to

justify its use in relation to the requirements of the job, ie, is it relevant? does it measure akey skill in the job?


26/28


26


The ability to read, understand and act on written information is a fundamental skill in thetechnical workplace. Technical Understanding is designed to measure this ability usingpassages of text drawn from a variety of sources including technical manuals, operating

instructions, maintenance schedules and fault finding routines, etc.

Each passage is followed by four questions aimed at testing the candidates’ understanding.

Although there are generous time limits for this test, the amount of verbal material involvedmay lead to adverse impact against some ethnic minority groups. Users need to ensure,therefore, that the test is directly relevant to the job requirements before using it.

Numerical Reasoning

This test is designed to measure candidates’ reasoning ability when handling numericalinformation. The data is laid out in the form of short written problems, some of which aresupported by diagrams. The candidate must first decide how to solve the problem and then

correctly apply the relevant arithmetical operations to complete it.

Since the emphasis here is on understanding and reasoning with numerical information, thistest has been developed specifically for use with a hand-held calculator.

This reduces the computational load of the test and ensures that those candidates who haveadequate reasoning ability but not always the computation skills to match, are not penalisedunfairly.

Fault Diagnosis

This test measures the ability to locate faults in a complex but logical system. Thisdiagnostic ability has become increasingly important in recent years with the rapid rise in new

technology. Individuals who are able to isolate faults and take the necessary remedial actionare likely to reduce maintenance costs and equipment downtime.

The test requires candidates to identify, by working back from a given output, the oneprocessor in a sequence which is not working as specified. Fault Diagnosis is a taskorientated test unlike the more abstract Diagrammatic Reasoning (DT8) from the originalTTB, which it replaces. As such, it is more directly relevant to the requirements of modernindustry and is likely to have increased validity.

Spatial Recognition

This test is designed to measure the ability of candidates to recognise complex shapes(components) even after they have been rotated. This is an essential skill in many areas of

manufacturing and production and is relevant to a wide range of occupations includingmachinists, fitters, motor mechanics and service engineers as well as draughtspeople anddesigners.

The item format is based on a selection of shapes which have been chosen to reflect moderncomponent design. Candidates are given a shape on the left hand page and are asked tochoose from five alternatives on the right, the one which is identical. Items toward the end of


27/28


27

the test get progressively more difficult, as the differences between the given shape and fivealternatives become less obvious.

TEST AND ITEM REVIEW

As mentioned earlier, the original TTB tests were developed and trialled in 1978/9. In the1991 revision, all of this original material was thoroughly reviewed and sufficient additionalmaterial written to enable parallel forms to be constructed. (The provision of sufficient extraitems enabled those not performing to acceptable standards to be rejected at a later date.)

All of the material was then further reviewed and amended by a team of experiencedpsychometrists within SHL, at two levels. First, the overall design of each test wasconsidered in relation to its credibility(ie, face validity) and job relevance (ie, content validity).

Secondly, within each test, every item was reviewed individually in order to eliminate thosewhich did not appear to be rationally related to other test content, those that were ambiguousin any manner or those that were judged to be in any other way unsuitable.

Finally, the material was reviewed by equal opportunities specialists and members of minoritygroups to eliminate any items which could be identified as having a potential to create genderor ethnic bias.

ITEM TRIALLING

All of the items which met the reviewing criteria were assembled into trial test versions in anorder which was a close as possible to the intended final forms. Each test was trialled on asample of around 200 people drawn from various companies, applicant groups and final yearschool and further education students; a sample intended to be as representative as possibleof those on whom the tests might be used. In total, some 800 people took part in these trialswhich were held between March and July 1991.

ITEM ANALYSIS

The next stage was item analysis. Here statistical procedures were used to investigate thepattern of responses to each item in detail. Each item was considered individually and inrelation to the rest of the test from which it was drawn.

Items were rejected if there was inconsistency in response patterns, if they did not relate tothe other items in the test or to the construct being measured, or if there appeared to be adifficulty or ambiguity which resulted in otherwise high scoring trial participants respondingincorrectly.

The response patterns of different ethnic and gender groups were also investigated at this

stage using methods such as the Mantel-Haenszel procedure. Items were rejected wherethey were found to show differential difficulty for a particular group unrelated to the abilitybeing measured by the test.

(Full detail of the results of the item analyses are shown in Section 12, “Additional TechnicalData”.)


28/28

© SHL GROUP plc 2000

FINAL FORM TRIALS AND ANALYSIS

The remaining items meeting the item analysis criteria were then selected for inclusion in thefinal forms of the tests. In choosing these items a number of considerations were addressed.The items which were selected had to reflect the full range of content of each test and

provide a span of difficulty levels from very easy items through to those which the trial groupsfound most difficult. Within these constraints, the items with the best psychometric propertieswere chosen.

The final form of each test was re-trialled during the period October 1991-April 1992. Additional groups were used in order to confirm that the tests performed psychometrically asanticipated and to provide standardisation (norm) data. A further aim of these trials was tocollect calibration data to allow scores on earlier versions of each test to be directlycompared with those from new and updated versions. This ensured that norm data collectedon one form of the test could be converted for use with other forms. For this purpose, groupsof trial subjects were asked to complete more than one form of a test, wherever possible.

Standardisation data was collected from a sample made up principally of apprentices in and

apprentice applicants to the engineering industry (76%). Their educational level varied fromno formal qualifications up to ONC level. The remainder (24%) of the sample was made upof final year students (studying a range of subjects) and work experienced adults in a varietyof disciplines within the technical field.

In terms of ethnic origin, 87% were classified as white European and 13% belonged to acombined ethnic minority group (Asian, African, Afro-Caribbean – according to self-description). In terms of gender, 89% of the group were male and 11% were female.

Altogether 1 937 people took part in these trials.

MANUAL WRITING AND ONGOING TECHNICAL REPORTS

The first (1979) edition of the TTB manual contained detailed information on administration,scoring and norming as well as technical information on all eight tests. Since originalpublication, much more information on the tests has been collected through technical studiesand follow up on the tests in use.

In 1982, a Normline supplement was published containing norm data for a variety of groupsand details of test validation. In addition, a further spatial test (ST9) was added to thebattery.

Later, a practice leaflet was produced to allow users to provide candidates with informationabout the tests in advance and some sample questions. In 1987 and 1991, further NormlineUpdates were published and made available to holders of the original TTB manual.

This present (1992) manual contains original material and incorporates new norm data,details a new validation studies and an expanded discussion of Testing and EqualOpportunity.

Monitoring the TTB is an ongoing process and we will continue to publish new informationfrom time to time. Test users are urged to contribute to the process by supplying SHL withscores and details of their own findings with the tests which can then be shared with otherTTB users through the publication of appropriate norms and technical updates.