RAG - Measuring Employee Motivation

7/28/2019 RAG - Measuring Employee Motivation

1/70

JUNE 2011 Draft v6 REVISED Report 1

Measuring Employee Motivation:

Design and Cognitive Test of Employee Motivation Questions

A report for:

Presented by:

Your contact person:

GfK Business

Colin Hand

Research Manager

Phone: +44 (0)20 7890 9654

Fax: +44 (0)20 7890 9222

E-mail: [email protected]

GfK. Growth from Knowledge


2/70


CONFIDENTIALITY

Please note that the copyright in the attached report is owned by OME and the

provision of information under Freedom of Information Act does not give the

recipient a right to re-use the information in a way that would infringe

copyright (for example, by publishing and issuing copies to the public).

Brief extracts of the material may be reproduced under the fair dealing

provisions of the Copyright, Designs and Patents Act 1988 for the purposes of

research for non-commercial purposes, private study, criticism, review and

news reporting.

NOTE ON QUALITATIVE FINDINGS

The findings in this document are qualitative in nature. They are derived from

a small sample and are not projectable in numeric terms to a larger

population. As such, these findings are directional and should be considered

indicative rather than conclusive.


3/70


CONTENTS

Page

1. BACKGROUND TO THE RESEARCH ........................................................... 4

2. RESEARCH APPROACH ............................................................................... 6

3. THE PROPOSED QUESTION SET FOR TESTING ........................................ 8

4. OVERVIEW ................................................................................................... 20

5. QUESTION 1: MEASURING MOTIVATION ................................................. 23

6. QUESTION 2: SELECTING FACTORS ........................................................ 26

7. THE LIST OF MOTIVATIONAL FACTORS .................................................. 32

8. QUESTION 3: RANKING MOTIVATIONAL FACTORS ................................ 40

9. QUESTION 4: ASPECTS OF PAY AND BENEFITS .................................... 45

10. CONCLUSION .............................................................................................. 51

11. APPENDIX I: RECOMMENDED QUESTION SET ........................................ 52

12. APPENDIX II: SIGNIFICANCE TESTS ......................................................... 59

13. APPENDIX III: COGNITIVE TESTING TOPIC GUIDE .................................. 62


4/70


1. BACKGROUND TO THE RESEARCH

The Office of Manpower Economics (OME) provides support for the six independent

public sector Pay Review Bodies, and the Police Boards. OME was set up to provide an

independent Secretariat to the Review Bodies, free from Government direction as to the

advice and support offered. It aims to provide high quality analytical, administrative andtechnical support.

The Review Body process is evidence-driven: stakeholders are invited to submit

evidence each year which the Pay Review Bodies consider when deciding upon their

recommendations, often alongside other, independently-provided information. Included

in most Pay Review Bodies terms of reference among several other important

considerations is to consider, when forming their pay recommendations, the need to

recruit, retain and motivate suitably able and qualified people.

In 2009 the Office of Manpower Economics (OME) commissioned GfK NOP to conduct a

research study aimed at providing authoritative guidance on, and a suite of standard

questions to measure, employee motivation. These questions and the attending

guidance were intended to be passed on to Review Bodies as a set of standard,

harmonised questions that could be used to provide in evidence to Review Body

Stakeholders comparable data across Review Body remit groups.

Three standard questions were presented to Review Bodies, however, they wereconcerned that the questions would not provide enough usable detail, and did not

differentiate between factors that motivate and demotivate. Also, the relative effect of

aspects of remuneration on motivation could not be drawn out, and placed in context of

motivation overall.

Since the first submission of questions, revisions have been made to the standard

questions, and a revised set was developed by OME. These questions underwent further

review by GfK before a new set of questions was proposed.

These new questions however were untested and required piloting with potential

respondents to identify unforeseen problems, and provide evidence for further

refinement.

1.1 Objectives

A cognitive testing phase was carried out to evaluate a draft set of employee motivation

questions, by observing potential respondents complete the questions, identifying

problems and gathering evidence for further revisions. The objectives of the testing

phase were:


5/70


To conduct a cognitive test of a revised set of employee motivation questions

amongst public sector workers

To test recommended approaches alongside alternative approaches

To produce an agreed, concise set of questions with the potential to improve the

quality of evidence on employee motivation available to Review Bodies

To document the conduct of the testing in order to provide justification for the final

proposed question set.


6/70


2. RESEARCH APPROACH

2.1 Cognitive depth interviews

Following the review process, 13 cognitive-depth interviews were conducted with

respondents representing the range of pay review body remit groups. Cognitive testing

explores the mental process by which respondents formulate their response to a

question. By investigating how respondents interpret questions, checking their

understanding of key terms, and exploring how judgments about answers are made, it is

possible to test whether the questions are working as intended.

The questions were administered as a pen and paper self completion survey, whilst an

interviewer observed how respondents answered questions. Respondents were asked to

think out loud, by vocalising their considerations as they completed the questions. This

enabled the interviewer to probe concurrently on problems as they arose, and explore

how respondents overcame them, as well as check that key terms were being registered

and understood as intended.

2.2 Sample

The majority of participants were recruited from GfK NOPs ePanel, with 3 respondents

being contacted through free found sources. Since respondents were not contacted via

their workplace or employer, the interviews were conducted in respondents homes or

other convenient place, outside of work hours.

The interviews sought to gain a representation of the Pay Review Bodies remit groups,

at junior and senior levels. The final achieved sample is summarised below:

Junior Senior

Armed Forces 1 1

NHS 1 1

Doctors and Dentists 1 1

Prison Services 1 1

School Teachers 1 1

Police Force 1 1

Senior Salaries 1

Each respondent was offered an honorarium of 50 as a token of thanks for their time. It

is not common practice to offer incentives to employees for employee research.

However, for this study we were not interviewing respondents as employees, but in their


7/70


capacity as people working in the public sector. The key differences to employee

research that give good reason for an honorarium were:

Interviews took place during their free time and likely in their homes

We were not introduced to respondents by an employer, and their employer was

not informed of their participation

Respondents were primarily recruited from GfKs panel of respondents, and

therefore it is common practice to offer an honorarium as a thank you for the

additional time they have given to any research study.

We also expected more success in recruiting willing participants if an incentive payment

were to be offered. Also, by using a cash incentive, this allows participants to donate this

to a charity of their choice or use it to replace their lost hour.


8/70


3. THE PROPOSED QUESTION SET FOR TESTING

The questions taken into testing followed the general structure proposed by OME.

Question 1. Identifying level of motivation on a scale of 1 to 10:

This question intended to identify where respondents currently are in terms of

motivation and provide a way of tracking motivation levels over time.

Question 2. Identify factors that contribute to motivation:

The question retained the original approach of identifying what contributes to

levels of motivation, but from a neutrally worded list of factors. Respondents

were asked to identify whether each factor, raises, has no influence, or lowers

their motivation.

Question 3. Ranking the most important factors:

The question then asked respondents to rank up to three of the most important

factors that currently raise motivation, and then up to three of the most important

factors that currently lower motivation.

Question 4. Aspects of pay and benefits:

Where respondents identified pay and benefits as a factor that either raises or

lowers motivation, they were routed to a question asking them to rank up to

three of the most important aspects of pay and benefits that either raised or

lowered motivation.

Alternative answer frameworks and question wordings were also tested during the

interviews. The full set of questions as they were tested is set out below. The final

revised set of questions is set out in Appendix I.


9/70


TEST 1) In relation to your current working life, on a scale of 1 to 10 where 1 is extremely

low and 10 is extremely high, how motivated are you to do a good job?

1 Extremely low

2

3

4

5

6

7

8

9

10 Extremely high


10/70


Below is a list of factors that may affect how motivated you are.

TEST 2) Again, in relation to your current working life, overall, how do the following

factors affect how motivated you are to do a good job?

Please provide an answer for each factor

Raises my

motivation

Neither

raises nor

lowers my

motivation

Lowers my

motivation

Dont

know

a. My Work Life Balance

b. The extent to which I am treated with respect in my workplace

c. My colleagues/team

d. The level of pride I have in working for this organisation

e. The level of control I have over my own work

f. The extent to which I am valued in my workplace

g. Material reward (salaries, pension etc.)

h. Access to flexible working policies

i. My line manager

j. The extent to which my work makes a difference to people

k. My prospects for promotion/progression

l. Status

m. Being challenged by my work

n. Relationship between my work and my personal values

o. Level of variety in my work

p. Extent to which I enjoy my work

q. The extent to which I am treated fairly and equally in my

workplace

r. Exercising authority

s. Job security

t. Extent to which I am able to develop new skills on the job

u. Opportunities for training and development

v. Communication at work

w. Other (please specify) _____________________________


11/70


TEST 3 a) Of the factors that you have identified that currently raise your motivation to

do a good job; please rank up to three in order of importance

Please rank only those factors that you indicated raise your motivation

Insert 1 for most important factor, 2 for the second, 3 for the third.

Raises my

motivation


b. The extent to which I am treated with respect in myworkplace







i. My line manager

j. The extent to which my work makes a difference topeople


l. Status





q. The extent to which I am treated fairly and equally in my

workplacer. Exercising authority

s. Job security

t. Extent to which I am able to develop new skills on thejob



w. Other (please specify) _____________________________________


12/70


TEST 3 b) Of the factors that you have identified that currently lower your

motivation to do a good job; please rank up to three in order of importance

Please rank only those factors that you indicated lower your motivation

Please insert 1 for most important factor, 2 for the second, 3 for the third.

Lowers my

motivation


b. The extent to which I am treated with respect in myworkplace







i. My line manager

j. The extent to which my work makes a difference topeople


l. Status





q. The extent to which I am treated fairly and equally in myworkplace


s. Job security

t. Extent to which I am able to develop new skills on thejob



w. Other (please specify) ______________________________________


13/70


ANSWER QUESTION 4 only if material reward is selected in response to

Question 2

You identified "material reward" as a factor that contributes to your current

level of motivation

Below is a list of aspects that make up material reward

TEST 4a) Please rank up to threeaspects of material reward that currently

raiseyour motivation to do a good job, in order of importance


Raises my motivation

1. Basic salary level

2. Latest pay settlement / review

3. Performance-related pay elements, e.g. bonuses, profit Sharing

4. Access to pay increments/ structured pay progression

5. Other pay elements, e.g. overtime pay, shift pay premiums,

unsocial hours payments

6. Pension benefits being built up

7. other non-pay benefits, e.g. health insurance, car/car allowance

8. other (please specify) e.g. season ticket loan, childcare vouchers,

subsidised canteen___________________


14/70


TEST 4b) Please rank up to threeaspects of material reward that currently

loweryour motivation to do a good job, in order of importance


Lowers my motivation1. Basic salary level




5. Other pay elements, e.g. overtime pay, shift pay premiums,

unsocial hours payments




subsidised canteen


15/70


Alternative question 3

TEST 3a) Of the factors that you have identified that currently raise your motivation to do

a good job; please rank up to three in order of importance

Please rank only those factors that you indicated raises your motivation

Select one factor in each column

Most

Important

1

Second most

important

2

Third most

important

3









i. My line manager



l. Status





q. The extent to which I am treated fairly and equally in my workplace


s. Job security


u. Training and development


w. Other Specify _______________________


16/70


TEST 3b) Of the factors that you have identified that currently lower your

motivation to do a good job; please rank up to three in order of importance

Please rank only those factors that you indicated lower your motivation


Most

Important

1

Second most

important

2

Third most

important

3









i. My line manager



l. Status





q. The extent to which I am treated fairly and equally in my workplace


s. Job security


u. Training and development


w. Other Specify _______________________


17/70


Alternative Question 4

You identified "material reward" as a factor that contributes to your current level of

motivation


TEST 4a) Please rank up to threeaspects of material reward that currently raise

your motivation to do a good job, in order of importance


Most

Important

1

Second most

important

2

Third most

important

3





5. Other pay elements, e.g. overtime pay, shift pay premiums, unsocial

hours payments




subsidised canteen______________________________________


18/70


You identified "material reward" as a factor that contributes to your current level of

motivation


TEST 4a) Please rank up to three aspects of material reward that currently lower

your motivation to do a good job, in order of importance


Most

Important

1

Second most

important

2

Third most

important

3





5. Other pay elements, e.g. overtime pay, shift pay premiums, unsocial

hours payments




subsidised canteen____________________________________


19/70


Alternative Question 4

4 Please rank up to three aspects of material reward that currently raiseyour

motivation to do a good job, and up to threeaspects of motivation that

currently loweryour motivation to do a good job in order of importance.


Raises my

motivation

Lowers my

motivation



3. Performance-related pay elements, e.g. bonuses, profit

Sharing


5. Other pay elements, e.g. overtime pay, shift pay

premiums, unsocial hours payments


7. other non-pay benefits, e.g. health insurance, car/car

allowance

8. other (please specify) e.g. season ticket loan, childcare

vouchers, subsidised

canteen____________________________________


20/70


4. OVERVIEW

Throughout testing, the questions performed well, with the majority of issues being

around understanding of key terms, signposting tasks clearly, and complex question

wording that led to confusion when ranking motivational factors. The questions provoked

thoughtful deliberation, and respondents commented that providing answers was not aneasy task. This was attributed to the challenge of considering how the range of factors

affected their working life rather than issues around understanding of the questions

themselves. However, testing revealed recurring issues that impacted on the immediacy

of understanding and ability to complete the questions in a way that felt satisfactory to

respondents:

The impact of mixed considerations: when deciding how one of the

factors affects motivation, the final answer may be subject to several

considerations or experiences, which can affect motivation both positively and

negatively.

Use of technical language: where terms such as factorand rankadded

an additional interpretative task on respondents

Grammatical complexity of questions: where questions were difficult to

follow, and had to be interpreted rather than immediately understood

Understanding the ranking up to three tasks: there were insufficient

instructions to make it clear to respondents how to progress where less than

three answers had been provided at previous questions.

Each of these issues affected all respondents to a greater or lesser degree, and had

significant impact on the ease with which respondents were able to navigate through the

questions. These recurring issues are explained and explored in further detail below.

Impact of mixed considerations

It was anticipated that the questions would present a challenge to respondents to

answer, particularly as the factors and aspects of pay identified could potentially both

raise and lower motivation over a period of time.

The purpose of currentand on balancein the questions was intended to help minimise

this difficulty and help respondents towards their answers. However, presented with

either a list of factors, or aspects of pay and benefits, respondents still found that they

had a mix of experiences and considerations that they could bring to bear on the

questions.

Further probing revealed that the terms current and on balance were not being

immediately registered within the question, diluting the impact each had on how

respondents formulated their answers. This was due to respondents interpretation of the

terms, the fit with the mental process of formulating an answer, and the grammatical

complexity of the questions themselves.


21/70


Respondents dealt with this dilemma in one of several ways:

Respondents reflected on their current situation in comparison with a

memorable period where the factor affected their motivation in a

particularly extreme way (either positively or negatively). For example a

police officer recently returned to work after maternity leave felt that her

Work/Life Balance raised her motivation since she was now able to benefit

from family friendly policies, whereas in the past, shift work has been

particularly disruptive to her work/life balance.

Respondents selected the neither raises nor lowers answer, as

considerations pulling each way neutralised the affect on motivation, although

this was considered to be an unsatisfactory answer.

Answered dont know. Again this was considered an unsatisfactory

answer, because You should know these things, [factor] either does affectmotivation or it doesnt. There was a mild preference for rewording dont

knowto cant say.

Use of technical language

Respondents were given pause by four terms used throughout the questionnaire;

factors, indicated, column and rank. However, these were not considered difficult

terms in themselves, but they did require interpretation within the context of the

questionnaire to be fully understood. For example, on seeing factorsfor the first time,

respondents immediately looked for any clue as to what the word factor was referring to.

Grammatical complexity

Respondents did not consider the questions to be difficult to understand and were able to

articulate what they were being asked to do in their own words. However, there was

evidence that questions were not immediately grasped, and often required re-reading

before respondents were comfortable they had understood the question correctly.

This was particularly the case with question 2a) asking respondents to identify how

factors affect their motivation and 3a) and 3b), asking respondents to rank factors in

order of importance:

2a) Again, in relation to your current working life, on balance, how do each of the

following factors affect how motivated you are to do a good job?

3a) Of the factors that you have identified that currently raise your motivation to do

a good job; please rank up to three in order of importance

Difficulties were caused by the presence of several concepts competing within thequestion wording. Terms such as currently, working life, on balance, up to three,


22/70


contributed to a degree of grammatical complexity that made the question difficult to

follow. This in turn led to any one or more of these terms being overlooked when

considering an answer.

In addition, this perceived complexity of the question, implied that providing answers

should be an equally complex task, prompting respondents to regularly double check to

ensure their understanding of the question was correct. However, even when doing so,

the key concepts; currentlyand on balance were overlooked by some respondents, as

they attempted to interpret a question as a whole.

Understanding the ranking upto three task:

This was an issue that affected both the ranking of motivational factors and the ranking of

aspects of pay and benefits. Essentially, respondents were unclear as to how to progress

where less than three answers had been provided at previous questions, or where less

than three answers were applicable. Again, due in part to the grammatical complexity of

the questions, the clue- up to threewas not always interpreted as no more than three.

Interestingly, where up to three was understood, in some instance respondents

uncomfortable with the notion of moving on to the next question, as it somehow felt

incomplete or unsatisfactory not to give a full answer.

4.1 Overall Recommendations

The way each question is currently worded does little to help respondents over the

difficult task of deciding how to answer where there are several considerations that affect

how they might respond. As a result, respondents are taking different approaches to

overcoming the dilemma, which risks introducing an element of instability into responses.

The recommendations below outline how each question has been revised to address

these overarching issues. The key overall recommendations that have arisen from the

cognitive test are:

Recommendation 1. Simplify the questions to make them easier to follow, by drawing

out key terminology into a separate clear instruction

Recommendation 2. Ensure simple clear instruction at each question and ensure that

guidance on completing the ranking questions is clear.

Recommendation 3. Place greater emphasis on directing respondents to answer on

the basis of how they feel at the current time

The following sections of the report will discuss each question in turn and provide more

detailed recommendations for revisions to each question.


23/70


5. QUESTION 1: MEASURING MOTIVATION

The first question required respondents to provide a self assessment of their motivation

at work along a 10 point scale.

CHART 1 TESTED QUESTION

The following questions are about your motivation at work.

1) In relation to your current working life, on a scale of 1 to 10 where 1 is extremely low and 10 is

extremely high, how motivated are you to do a good job?

1 Extremely low

2

3

4

5

6

7

8

9

10 Extremely high

Respondents had no problems in providing an answer for this question. The 10 point

scale was appropriate, with respondents tending to begin by answering at the positive

end of the scale and adjusting their answer as particular issues were recalled.

5.1 Motivation to do a good job

As in the previous research respondents had no problems understanding that thequestion related to their work, and no definition of motivation needed to be given. The

question taken into testing included the phrase to do a good jobwhich was added as a

result of the initial review of the question prior to testing. This addition was crucial to

ensuring respondents interpreted the question consistently. To do a good job was

associated with performance and outcomes, was very much considered to be an

objective focus of motivation. Alternative wordings were tested; to do your best in your

joband to do your bestat work, both of which seemed more subjective and associated

with effort, and self motivation. Respondents felt that to do a good job was both the most

appropriate measure, being the more tangible concept to consider, and easier to placeon a scale.


24/70


Recommendation 4. Retain the words to do a good jobas preference to to do your

best at work, or to do your best in your job.

5.2 Time frame: Current working life

The review process recommended that the word currentwas added to tighten the time

frame of reference around the term in relation to your working life. Respondents

interpreted current working life in one of two ways; either narrowly as my job, or in

slightly broader terms including my job as affected by mypersonal life. The addition of

currentalongside working life went some way to narrowing the frame of reference in

terms of theircurrent job, but did not explicitly encourage reference to the current point in

time.

As such, respondents answered the question based upon a broad range of time frames,from the time spent in their current job, to an arbitrary measure of the last 2 weeks. This

meant that respondents were applying an inconsistent time frame to this question, which

was then carried through and applied to the following questions.

Recommendation 5. Reposition currentwithin the question to break up the phrase

current working life.

5.3 Fit between question and answerAlthough not an issue raised by respondents it was apparent there was a mismatch

between the question and the answer framework. The question and answer can be read

out as follows:

Q. How motivated are you to do a good job...A. Extremely low / extremely high.

Although this was not an issue raised by respondents in this test, it may well become a

point of dispute in the future.

Recommendation 6. Reword the question to match the answer scale, by using the

phrase how would you describe your current motivation.


25/70


CHART 2 RECOMMENDED QUESTION

The following questions are about your motivation at work.

1) In relation to your working life, on a scale of 1 to 10 where 1 is extremely low and 10 isextremely high, how would you describe your current motivation to do a good job?

1 Extremely low

2

3

4

5

6

7

8

9

10 Extremely high


26/70


6. QUESTION 2: SELECTING FACTORS

The second question asked respondents to identify factors that influenced their

motivation positively and negatively from a neutrally worded list of factors. An answer

framework was designed that collected positive, negative and neutral views in one

question, rather than having to repeat the list of factors for each direction of influence.

CHART 3 TESTED QUESTION

Below is a list of factors that may affect how motivated you are.

2) Again, in relation to your current working life, on balance, how do each of the following factors

affect how motivated you are to do a good job?

Please provide an answer for each factor

Raises my

motivation

Neither

raises nor

lowers my

motivation

Lowers my

motivation

Dont

know


...................................

w. Other Specify______________

6.1 Question and instruction wording

As discussed above, this question was considered difficult to follow and suffered from a

degree of grammatical complexity caused by inclusion of several concepts in the

sentence. Since respondents were continually referring back to the question if factors

required more consideration, this grammatical complexity failed to provide respondentswith the clear immediate understanding they were looking for.

The question and instructions sought to convey three key concepts:

To consider on balance where mixed considerations (i.e. where a factor

could both raise and lower motivation) could influence response

Consideration of theircurrentsituation, that is how they feel now

To answer once for each factor.


27/70


Minimis ing m ixed considerations:On balance

The review process recommended that a term such as overall or on balance, was

included at appropriate questions to overcome the anticipated difficulty of factors both

raising and lowering motivation. When probed, respondents were clear as to the meaning

of on balance within the questions, providing interpretations including all things

considered, oroverall. However, respondents also commented that balance may not

be the most appropriate concept. Those who found themselves bringing several

considerations to bear on a particular factor suggested that they were not weighing up

considerations, but referring to how they have felt at different times. When prompted,

respondents found it easier to reflect on their current situation, that is how they felt at

that moment in comparison to an earlier positive/negative period, than to attempt answer

on balance.

Reference period: current situation

As discussed at section 5.2, the placement of current alongside working life did not

explicitly encourage reference to the current point in time. As with question 1,

respondents responded to each factor based upon a broad range of time frames, most

commonly between 3 months (marking the New Year as a reference point) and 12

months (marking this time last year, as a reference point). When prompted to answer on

the basis of how they felt at that moment, respondents were able to come to a judgement

more easily, since it negated conflicting considerations originating from issues occurring

in the past. That is not to say past issues were discounted, as respondents noted that

events occurring even beyond a year ago still affect how they felt now.

Response instructions

The instruction to answer once for each factor was understood, and respondents all

provided an answer for every factor. Respondents assumed that only one answer was

allowed, but this was associated with the common convention of answeringquestionnaires rather than any interpretation of the instruction.

However, two issues emerged from testing question 2:

Factorwas found to be an overly technical term. It was understood, and it

was eventually clear what factors referred to, however it was an additional

interpretative task that respondents had to go through each time the question

was considered. Respondents own language commonly referred to issuesthat

affect their motivation


28/70


The instructions did not direct respondents consideration towards how they

felt at the current time. As discussed above, this was a key piece of guidance

that helped overcome difficulties formulating a response.

6.2 Answer framework

In general, the question approach worked well, and combining positive and negative

considerations into one question helped respondents make judgements about how to

answer against each factor. The impact of mixed considerations on response suggests

that to separate the question into what raises and what lowers risks encouraging

responses that indicate the same factors influencing motivation in both directions.

6.3 Neutral response options and opt outs

The answer framework included a neutral response Neither raises nor lowers my

motivationand an opt out to response as dont know. It was clear that the neutral

option was crucial to allowing respondents to answer both easily and in a way that

reflects how the factor actually affects them. There were however, subtle differences as

to how the neutral option, the dont knowoption, and a potential not applicable option

were interpreted.

Neither raises nor lowers my motivation

The neutral option was used in two ways:

The factor does not affect my motivation at all

The factorboth raises and lowersmy motivation.

This latter interpretation was symptomatic of the difficulty of having severalconsiderations for a factor, and was in effect, being used to express a polarising effect of

those considerations. However, used this way, the neutral option was the best fit as

without it, the only other way of responding would be to force respondents to answer

either raisesor lowers, or to respond as dont know.

Respondents who answered neutrally were presented with the prospect of having the

neutral option removed. Selecting instead, either raises or lowers felt to some

respondents to be a false response, and they did not feel comfortable with their answers.

They felt that that it discounted considerations being made at the other end of the scale

which may mislead those acting on the results. The dont knowoption was considered

Raises my

motivation

Neither

raises nor

lowers my

motivation

Lowers

my

motivation

Dont

know


29/70


by respondents as an entirely false alternative to a neutral option, as quite simply, they

did know.

Dont know

The dont knowoption was used in four ways:

As an alternative to the neutral option. In this case dont know became a

negative response, and there was preference for a clarification suggesting

cant say.

To express a genuine dont know response, particularly where respondents

felt they had too little information to give an informed judgement. For

example, at access to policies that allow flexible working, where respondents

were unaware of what policies were available to them, and what provision

was made for flexible working1.

Where the factor was not completely understood. An example here was

communication at work which was considered perhaps too ambiguous to

provide a certain answer to.

Where the factor did not apply to them at all, a good example being,

exercising authoritywith respondents suggesting they had no authority so

could not answer. In these cases, respondents suggested a response option

for not applicablewould be a better response.

Not applicable

A not applicable response option would certainly fit the way respondents wanted to

answer for some of the factors. The argument for its inclusion is that it would provide

some differentiation within the dont knowresponses. However, as a response itself, it

may also be of interest to understand what factors respondents think are not applicable

to them, for instance why is work/life balance not applicable to some employees?

The argument against its inclusion is its overlap with the neutral category. When probed

as to what not applicablemeans, respondents stated that it means it does not affectthem, or it is not relevant to their current job or lifestyle, both very similar to the

interpretation of the neutral category which was interpreted as having no effect on my

motivation.

Recommended approach for neutral and opt out options

The neutral and opt out options need to allow respondents to answer in a way that

expresses that a factor does not apply, that they are unable to provide an answer, or that

it has no affect. The evidence from the testing suggests that a not applicable response

1Ambiguities in factor wording are discussed at section 7.


30/70


may be confused or overlap with the neutral response (neither raises nor lowers).This

could be overcome by prefacing the neutral option with the issue applies to me but...

and including a not applicable answer option. This would distinguish the purpose of the

neutral option as being to identify where a factor has a neutral affect on motivation.

The question arises as to whether a not applicableoption should replace or sit

alongside a dontknowoption. The not applicableresponse option would address the

overlap in interpretation between dont know and the neutral option but removing the

dont knowoption does not allow a response where there is too little information or

knowledge to provide an answer.

The argument for including a separate dont knowresponse is to allow respondents to

answer where they have a limited knowledge or frame of reference to inform their

judgement, and want to opt out.

However, the argument for replacing a dont knowoption is that a not applicableoptionremoves the need for a dont knowoption. Arguably, if an issue is properly understood,

then it should either be relevant to a respondent or not. If relevant then, respondents

should be able to judge whether this issues raises or lowers motivation. As discussed

above, the dont knowoption in testing was used where a judgement could not be made,

but there was ambiguity as to whether the neutral or dont knowoption was the better

answer (driven in part by respondents aversion to answering dont know.

Therefore, the recommendation for these options is to replace the dont knowoption

with not applicable.

6.4 Recommendations

Recommendation 7. Replace the term factorwith issue, as this reflects respondents

own language. Reference should also be made to selecting a box

as opposed toproviding an answer.

Recommendation 8. Simplify the question to remove the grammatical complexity that

makes it difficult for respondents to follow. Remove the terms

currentand on balance.

Recommendation 9. Include an instruction explicitly guiding respondents to focus on

how they feel at the moment. The instruction should also

acknowledge that a bestanswer will enable respondents to

decide between the positive/neutral/negative responses, and

discourage ticking more than one response.

Recommendation 10. Ensure the instruction highlights that respondents should answer

for each issue, and make it clear that one box should be marked

for each issue.


31/70


Recommendation 11.At this point we would recommend replacing dont knowwith a not

applicable option so as to differentiate between neutral and not

applicable responses. Although there is potential for overlap and

confusion between the two, including the precursor Applies to me

but...resolves that ambiguity.

CHART 4 RECOMMENDED QUESTION

Below is a list of issues that may affect how motivated you are in your current job.

2) In relation to your working life, how do each of the following issues affect how motivated you

are to do a good job?

For each issu e, please tick the bo x that best descr ibes you r current s i tuat ion

Select one box only for each issue

Raises

my motivation

Applies but

neither

raises nor

lowers my

motivation

Lowers

my

motivation

Not

applicable


...................................



32/70


7. THE LIST OF MOTIVATIONAL FACTORS

The factor list was considered long and somewhat daunting when first encountered.

However, once each factor had been considered, it was described as comprehensive,

relevant and resonated with the issues that respondents face in their own working lives.

7.1 Length of list and ordering of the motivational factors

The problems arising from the length of the list relates to respondents bringing mixed

considerations to bear on each factor. Where this was the case, respondents tended to

glance down, or back up, the list to see if their considerations had been covered

elsewhere in another factor. This did extend the time taken for respondents to complete

the list of factors.

There seemed to be few issues with ordering of the factors, but again there may be some

anticipation amongst respondents that like factors would be grouped together. However,

this may have an effect on response as respondents may be inclined to respond in the

same way within groups of factors. For this reason we recommend that, if used in online

surveys, the factors are rotated / randomised to ensure any potential order effect within

the list is minimised.

The testing did not find any evidence of ordering effects with the list on a paper based

questionnaire. Consideration has been given to grouping factors around similar themes.

However, this risks encouraging respondents to answer in the same way for groups

factors. Therefore, considering that no ordering affects were observed during testing, the

current list ordering is appropriate for paper versions.

Recommendation 12. Online surveys should rotate/randomise the order of the factors to

ensure any order effect within the lists is minimised. Paper

questionnaires should retain current ordering.


33/70


7.2 Wording of the motivational factors

The grid below outlines the original tested wording of the factors and any recommended

revision, along with commentary on how each factor was interpreted by respondents.

Tested Factor Revised Factor

My Work Life Balance No revisions

Respondents had no issues understanding this factor.

The extent to which I am treated with respect

in my workplace

No revisions

Respondents had no issues understanding this factor

My colleagues/team No revisionsRespondents had no issues understanding this factor

The level of pride I have in working for this

organisation

No revisions

There is a risk of inconsistent interpretation within review body remits as to what the

organisation refers to. Respondents felt that this organisation could be interpreted to

represent different levels of their workplace (for example, the NHS, this PCT, this hospital).

However, alternative approaches are likely to attract similar issues (such as ...in working

here ..in my place of work may be too limiting and still be ambiguous as to the level atwhich they should respond).

The level of control I have over my own work No revisions

More junior grades interpreted control to mean having control overwhat /how muchwork

they receive, whereas senior grades interpreted control to mean dealing with the work they

have. However, there was far greater variation in interpretation of an alternative managing

factor. The extent I am able to manage my work was interpreted to mean

competency/ability to manage workload.

The extent to which I am valued in my

workplace

No revision


Material reward Pay and benefits

Before entering the testing phase OME suggested that material reward might not be well

understood and suggested testing pay and benefits as an alternative. Respondents


34/70


understanding of material rewardvaried, interpreting it both narrowly and broadly. Narrow

interpretations focussed on basic pay as the reward, whilst others included other elements

of pay such as bonuses, allowances and pension benefits. Amongst these respondents

material reward was a familiar term and was associated with pay. However, some

respondents were less familiar with the phrase, did not immediately associate the factor

with their pay or benefits. Instead, material reward, was interpreted as recognition or actual

awards presented by their employer, or even uniforms provided by the employer.

Pay and benefits was interpreted as being about pay, and essentially what respondents get

at the end of the month. This tended to include pension benefits, overtime payments and

allowances. It tended not to relate to bonuses or performance related pay, however these

are less frequently paid in the public sector.

Access to flexible working policies The extent to which I am allowed to

work flexibly

Respondents interpreted this factor in two ways. Flexible working policies were

understood to mean access to policies that allowed/promoted flexible working hours, home

working etc. However, because of the way the factor is phrased, it was also understood to

be referring to the flexibility of working practices in general.

An alternative suggestion made by OME was to revise to The extent to which I am allowed

to work flexibly, whilst an alternative generated from testing suggested Access to policies

allowing flexible working. The issue is what the factor is intending to measure: I can wo rk

flexiblyorthere are pol ic ies that al low me to work f lexib ly. The problems identified with the

former approach were that

1) when considering working flexibly, this was interpreted as a range of activities that

could or could not be conducted flexibly, as some aspects of a task are flexible, such as

managing workloads, whilst others are not, such as shift patterns and working hours. This

added to the difficulty making judgement as to how to answer.

2) that some professions such as policing, nursing, teaching work to stricter shift patterns,

and operational approaches making flexibility, (as interpreted as a work/life balance

concept) is not as relevant on a day to basis. However, such issues should be either

captured by a not applicable response, or as lowering motivation where the absence of such

flexibility impacts negatively.

The key to flexible working as a motivational factor was their organisations comm itment to

providing opportunities to employees to work flexibly if they needed it rather than the extent

to which they can actually do so themselves. This was the rationale behind a previous

suggested revision on Access to p ol ic ies allowing f lexib le working.

However OME have indicated that being allowed to work flexibly is likely to be a more direct

driver of motivation than whether flexible working policies are in place. Therefore the

recommended revision is The extent to which I am allowed to work flexib ly.

My line manager No revision

Respondents had no issues understanding this factor. Those respondents that did not have

a line manager as such, (for example Head Teachers) understood what the factor referredto and were able to transpose the concept onto a similar individual(s) that monitors their


35/70


performance.

The extent to which my work makes a

difference to people

No revision

Respondents had no issues understanding this factor. This was a key positive motivator forrespondents, and reflected respondents earlier consideration of what affects their

motivation.

My prospects for promotion/progression My prospects for promotion and

progression

Promotion and progression were understood to mean different things. Promotion referred to

a formal step up to a new pay scale, or new grade, and may be accompanied by a change of

title. Progression was interpreted as developing or moving up within a pay scale or grade.

Despite the two meanings, both were highly relevant, particularly for those respondents that

had no further immediate avenue for promotion. However, this broad interpretation comes

from placing the two words together. Respondents felt that two separate factors for

promotion and progression were unnecessary, as individually they were likely to be

understood to be referring to the same idea.

Status Status in the workplace

Status was considered vague, ambiguous and attracting a broad range of interpretations,

ranging from status in the workplace, status in the community, to current employment

status (e.g. contractor, temporary, full-time, part-time). Therefore, consideration needs to be

given as to whether the factor should refer to status in the work place or include status

outside of work.

If the factor is to refer to community status, the issue is whether this refers to a

respondents own community, or the community in which they work. Respondents

considered status in the workplace to be a relevant factor to motivation, and in some cases

more so than their social status in the community. Status in the community was also felt to

overlap with the factor referring to making a difference to people.

Being challenged by my work No revision


Relationship between my work and my

personal values

No revision

Respondents had no issues understanding this factor. This was a key factor that

respondents were looking for, as it reflected their sense of self motivation to do their work

within the public sector. However, it was thought that any disconnect between personalvalues and work would need to be severe in order to warrant a negative response.


36/70


Respondents in this research found it difficult to visualise that kind of situation. Despite

this, there was no indication that the neutrality of the factor was compromised, as a negative

response could be envisaged, however severe the circumstances needed to be.

Level of variety in my work No revisionRespondents had no issues understanding this factor

Extent to which I enjoy my work No revision


The extent to which I am treated fairly and

equally in my workplace

No revision

Respondents had no issues understanding this factor. Respondents tended to consider

whether they were treated fairly and equally, and if so, suggested that this should raise their

motivation, since it is naturally a positive thing to be treated as such. Equally, if they felt

they were not treated fairly and equally, this would lower their motivation. Respondents then

considered whether it actually affected motivation, or had become something that they

expected, and so had become a factor with little impact on motivation.

The difficulty appeared to be in mapping their self assessment of their treatment against its

affect on their motivation, rather than understanding of the factor itself.

Exercising authority No revision

Respondents had no issues understanding this factor; however, it was not thought to berelevant to more junior grades. In such cases respondents tended to answer in the neutral

as something that did not affect their motivation. Other alternatives were suggested, such as

making decisions that affect other people, as this was more inclusive and perhaps more

neutral. However, since this changes the meaning of the factor, (decision making vs.

authority) consideration will need to be given as to whether this alternative is an appropriate

measure.

Job security No revision

Respondents had no issues understanding this factor. Interestingly, a poor outlook on job

security, for some was described as a positive motivator. This was explained by one

respondent as a stick rather than acarrot, for example where employees have been warned

of a reduction in headcount, this can motivate to do a good job to ensure that they are

included in those that remain.

Extent to which I am able to develop new

skills on the job

No revision

Respondents had no issues understanding this factor. This was understood to mean

informal training and developing within a role, or learning from a close colleague or line

manager.


37/70


Opportunities for training and development No revision

Respondents had no issues understanding this factor. A key differentiatorbetween training

and development and new skills on the job is whether there is advance planning or a need

to book a course. There was potential overlap between development and skills on the job,

however respondents differentiated again by understanding development to be planned, and

perhaps in preparation for/enabling advancement, whereas skills on the job was

interpreted as being about developing skills to do their current job.

Communication at work Communication in my workplace

Communication at work was considered vague and potentially covers a range of

communication types. Foremost in respondents minds was work related communication

between colleagues and other teams. Respondents had to be prompted to consider whether

this could refer to internal communications about whats going on in the organisation, or

communication from the leadership in the organisation. Respondents felt the bigger issue

for their motivation is what they are told by their senior managers, particularly at a time of

change and uncertainty within the public sector.

However, Senior Managers interviewed, stated this either excluded them from the question,

or was asking them to evaluate their own communications, and that of their counterparts

within their organisation.

Despite some ambiguity/vagueness around the wording of the factor, all respondents were

able to provide and answer. Therefore the current recommendation is to adjust the wording

to reflect communications within the workplace, to give some clarity/direction as to what is

being asked,

Recommendation 13. Make changes to the factor list as indicated in the factor grid

above

Recommendation 14. Consideration needs to be given to four factors to confirm what

needs to be measured before agreeing the factor wording:

Exercising authority, (authority vs. decision making)

Status (workplace vs. community)

Communication at work (work related communication vs.

internal communication about the organisation/from

leadership)

Access to policies allowing flexible working

(organisations commitment to flexible working vs. ability to

take advantage of f lexible working)

7.3 Additional factors

Respondents struggled to suggest any further factors to add to the list. However, a small

number of factors emerged that were considered noticeable omissions or commonlycited by respondents. These were:


38/70


My personal well being: there was a strong feeling from one respondent that health and

well being will have an effect on motivation. Considering the focus that government has

given to the cost of sickness and absence on the economy, we recommend this factor is

given consideration for inclusion. OME suggested that this could be extended to My

personal well being and that of my family. However, there was no evidence from thetesting phase that suggests this is an addition that warrants inclusion.

The leadership of this organisation: The factors ask about relationships with

colleagues and line managers, but respondents noted an absence of opportunity to

comment on how senior management decision making and leadership affects their

outlook on their work. Again, in the current context, we recommend this factor is given

consideration for inclusion.

My working environment: This was not a factor mentioned explicitly by respondents,

but it has been suggested amongst the project team that working environment may be an

issue to cover. Since interviews were conducted at respondents home, their working

environment may not have been at the forefront of their minds. We agree that this

recommendation should be given consideration.

Recommendation 15. Give consideration to additional factors, given that respondents

considered the list reasonably long already:

My working environment

My personal well being

Leadership of this organisation

At present, on the basis of a small number of interviews, and in the

absence of quantitative testing we do not recommend that these

are included in the factor list; however, these issues may become

more relevant over time.

7.4 Other specify

The other specify option suffered from two issues:

i) Visibility: it did not stand out and was almost entirely overlooked, particularly after

what seemed to respondents an exhaustive list. Therefore pertinent information would

not have been collected, without being prompted to do so.

ii) Multiple entries: the succinct one line specifier and answer boxes assume that only

one other will be provided. This lead to respondents manually drawing in lines and

boxes, in order to complete the question.


39/70


Recommendation 16. Encourage respondents to give consideration to other factors, by

expanding the other specify factor, setting it apart from the list,

and provide space for more than one other to be recorded.

CHART 5 REVISED OTHER SPECIFY

w. If there are any other factors that affect your motivation pleasespecify these below


40/70


8. QUESTION 3: RANKING MOTIVATIONAL FACTORS

Questions 3a and 3b asked respondents to select and rank the three most important

factors from their previous answer (to question 2) those that raise motivation (3a) and

those that lower motivation (3b).

As with the task of identifying the direction of influence of each of the motivational

factors, ranking the top three was considered a challenging task. However, this difficulty

was exacerbated by:

confusion when trying to follow and unpick the question wording

understanding what the task required them to do

and how to record responses.

It was at this point in the questionnaire that some frustration set in amongst respondents

who did not have three answers to select from.

CHART 6 TESTED VERSION 1

3a) Of the factors that you have identified that currently raise your motivation to


Please rank only those factors that you indicated raise your motivation

Insert 1 for most important factor, 2 for the second, 3 for the third.

Raises my

motivation


...................................



41/70


8.1 Understanding the question

When asked about their initial response to seeing the question, the majority of

respondents were checking whether the list of motivation factors was the same or had

changed in any way. Respondents could spend up to a minute checking to see if the

factors were the same, supposedly in case of any tricksor attempts to trip them up.

However, this action did have the effect of drawing the eye down the list and away from

the question and instructions. Essentially, the question can be the last piece of

information on the page to be read. This, perhaps combined with a familiarity with the

factor list, led to a tendency to skip read the question and move straight on to recording

responses in the answer grid. Both versions, either the different response options in

Version 2 or the single column in Version 1 prompted these respondents to check the

question.

The question itself was understood, with respondents picking up that they needed to

refer back to their previous answers. Respondents could rarely recall how they answered

and so needed to physically refer back to check.

The term rankwas understood, but considered an extremely rigid term to apply to a

series of motivational issues that essentially could not easily be ordered along a single

continuum. Respondents articulating their understanding of the question referred tochoosingwhich factors were most important to them.

It also required an additional step of interpretation that contributed to respondents

consistently overlooking the crucial phrase up to three, seemingly distracted by either

trying to recall how they answered the previous question, or searching back to check,

and then beginning to decide which where the most important answers.

Recommendation 17. Introduce the question with a bold statement indicating that this

question is asking respondents to do something different to

previous questions

Recommendation 18. Replace the word rankwith choose, reflecting the language

used by respondents.

Recommendation 19. Reword the question by removing up to threefrom the question

and create an explicit note to say they can select up to three.

Recommendation 20.Add the word issueto the end of each response option for

clarification


42/70


8.2 Understanding the task

In general the task was understood. Respondents referred back to previous answers,

they deliberated as to which were most important and they recorded answers

appropriately. In the main this was completed successfully by most respondents for

raising motivation, yet respondents were still misdirected by confusion as to how torecord answers and what to do if they could not. The problems encountered related to:

Not being able to rank their answers, either because they are all equally

important or unimportant in one way or another

Failing to give consideration to their other options and including them in the

ranking process

In Version 1, marking the boxes with a tick or cross, rather than marking

numerically

Ticking more than one box per column, essentially expressing multiple factorsas the most important, second most important etc.

In all cases respondents eventually realised their mistake without probing, usually by

noticing an instruction or registering threein the question.

A more severe issue was generally encountered at the lowers motivation question. The

majority of respondents had marked fewer than three factors that lowered their

motivation. By this point respondents understood the task, and were aware that threefactors is what the question was looking for. However, without three factors to refer to,

respondents looked to the questionnaire for guidance on what to do next. It was clear

that up towas not clear or explicit enough to prompt respondents to move onto the next

question once none, one or two factors had been ranked, where less than three factors

were available. Respondents reacted to this by:

Closing the questionnaire, and not moving onto question 4 until prompted to

do so by the interviewer

Marking however many factors they had selected at question 2 and moving to

question 4. Although this was what the question intended them to do, this

made these respondents feel that the question and their response was

incomplete.

In either event, respondents looked to the question and failed to immediately register the

significance ofup toin the question.


43/70


Recommendation 21. Reiterate the consistency check with an instruction that these

most important factors should be selected from factors selected at

question 2 (for respondents who do not read the question first).

Recommendation 22. Provide an instruction that permits leaving the question

unfinished, and give clear sign posting telling respondents whatto do next

8.3 The answer frameworks

A preferred version was presented, followed by an alternative version with a different

answer framework. The order in which each approach was presented was rotated across

interviews. The questions below illustrate the raise motivation questions. This was

followed by an identical question asking about factors that lowermotivation.


3a) Of the factors that you have identified that currently raise your motivation to


Please rank only those factors that you indicatedraiseyour motivation

Select one factor only in each column

Most

Important

1

Secondmost

important

2

Third most

important

3


...................................


Respondents expressed no strong preference for either of the two answer frameworks.

On balance, version 2 was considered a better approach as it appeared more familiar

and closer to questionnaires or forms that they had seen before. Of the two, version 1

was found to have the greatest risk of response error. Version 1 gave an impression of

being incomplete, due to the replication of a singular response category from Q2 (i.e.

raises motivation for question 3a and lowers motivation for question 3b) and the area of

blank space next to the answer column. This encouraged one respondent to duplicate

responses from question 2 before realising the illogic of that. The response categories at

version 2 were thought to prevent that, and clearly stated that something different was

being asked at this question prompting respondents to read the question and instruction

with greater care.


44/70


Recommendation 23. Version 2 is the recommended approach as this minimises

potential response error, and allows answers to be recorded in a

way consistent with the rest of the survey.

Recommendation 24. Reformat the other specify factor to encourage consideration of

those issues when choosing the most important.

CHART 8 RECOMMENDED VERSION

We would now like you to choose the three most important issues that ra ise your

motivation, and place them in order of importance.

3 a) Thinking of the issues that you have selected that raiseyour motivation to do a goodjob, please select the issues that affect your motivation the most in order of

importance to you.

You can select up to three issues

Please select from the issues that you s elected that ra ise your m ot ivat ion in quest ion 2 I f you cannot select three issues, please answer as best you c an, then go to qu est ion 3b

Tick one box only in each column

The most

Important issue

1

The second

most important

issue

2

The third

most important

issue

3


-----------------------------------------------------------------------------------------------------------------

w. Any other issues that raise your motivation


45/70


9. QUESTION 4: ASPECTS OF PAY AND BENEFITS

The final questions asked respondents to select the aspects of their pay and benefits that

raised (4a) and lowered (4b) their motivation to do a good job.

On the whole the question structure was familiar, and so there were fewer issues with

completing the question at this point. The main issues encountered were;

understanding of the aspects of pay and reward listed;

mapping their own pay package to the aspects listed;

understanding how to answer where less than three aspects apply.

For this question, a preferred answer framework was presented to respondents, with two

alternative approaches tested after the questions were completed. The order in which the

answer frameworks were presented was rotated for each interview. Illustrated below are

the tested versions of the questions for raising motivation. Versions 1 and 2 were

followed by identical questions, asking for aspects that lower motivation.


Answ er quest ion 4 only i f youidentified material rewardas a factor that either raises or

lowers your cu rrent level of mot ivat ion in quest ion 2

Below is a list of aspects that make up pay and benefits

4a) Please rank in order of importance up to threeaspects of pay and benefits that currentlyraiseyour motivation to do a good job,

Please insert 1 for most important aspect, 2 for the second, 3 for the third.

Raises my motivation



subsidised canteen____________________________________


46/70



Answer quest ion 4 only i f youidentified material reward as a factor that either

raises or lowers your c urrent level of motivat ion in quest ion 2.

Below is a list of aspects that make up pay and benefits

4a) Please rank in order of importance up to three aspects of pay and benefits

that currently raiseyour motivation to do a good job.

Select one aspect only in each column

Most

Important

1

Second

most

important

2

Third most

important

3


10. None of the above


4 Please rank in order of importance up to threeaspects of material reward that

currently raiseyour motivation to do a good job, and up to three aspects ofmotivation that currently loweryour motivation to do a good job.

Please insert 1 for most important aspect, 2 for the second, 3 for the third.

Raises my

motivation

Lowers my

motivation


10. None of the above

9.1 Routing

Question 4 versions 1 and 2 were tested with a routing introduction, instructing that only

those who answered that pay and benefits either raised or lowered their motivation at

question 2 should proceed. However, those that gave a neutral option at Q2 felt that theyshould also be answering the Q4, since there may have been mixed considerations in


47/70


their response at question 2 that polarised their response to neither raises nor lowers.

This was proven to be the case as respondents who answered neutrally at question 2

were able to select answers that both raise and lower at the pay and benefits question.

Subsequent interviews were tested with the routing instructions removed, and

respondents responding neutrally at question 2 still felt the pay and benefits question tobe relevant to them.

Recommendation 25. Route all respondents through the pay and benefits question

9.2 Answer framework

Of the three versions of the answer framework, version 3 was considered to be the most

difficult to understand. Respondents attributed this to a long, wordy question and an

involved task of:

Sorting through aspects to check which apply

Deciding which aspects raised and which aspects lower

Deciding which are the most important before recording an answer

This was also combined with a certain level of frustration that across the four question

types this would introduce a fourth style of answer framework to interpret. There was a

clear preference to use the answer framework that was used at question 3. Therefore,

respondents generally found version 2 to be the preferred approach.

9.3 Understanding of task

The question was well understood by this point, being described as having to choose

three again. However, once again, there were issues registering up to, with

respondents focus on picking out three aspects. When probed, respondents revealed

that they felt the question was actually asking for three, and felt almost compelled to

provide three answers even though they understood up toto be allowing fewer.

However, it was very difficult for some respondents to identify three aspects, and so

there was a potential for their answers to reflect how an aspect could potentiallyaffectmotivation. For example ifbasic salary levelwas to fall, then it could become something

that lowers motivation, but at present it may have no effect.

Consequently, there is a risk that respondents may select the factors for both raising and

lowering motivation. This was attributed to this being the final question of the set, and a

lack of a clear instruction to end if less than three aspects apply. During the interviews,

suggesting that respondents consider how they felt about each aspect at the current time

clarified the intention of the question and enabled respondents to move through the

factors more easily.


48/70


Recommendation 26. Provide an instruction that permits leaving the question

unfinished, and give clear sign posting telling respondents what

to do next.

Recommendation 27. Include an instruction explicitly guiding respondents to focus on

how they feel at the moment. The instruction should also

acknowledge that a bestanswer will enable respondents to

decide between the positive/neutral/negative responses, and

discourage ticking more than one response.

9.4 Understanding aspects of pay and benefits

A key issue with the question was the relationship between the list of aspects and

respondents understanding of their pay and benefits. When probed at question 2, as to

what was included in material reward/pay and benefits, the top of mind aspects werebasic pay, pension package and allowances. On further probing, season ticket loans,

subsidised travel and canteens and performance related pay were common additions.

This indicated that the problem was not understanding the aspects, but recalling their

own pay and benefits package, and mapping that to the appropriate aspects in the

question. An example of this is to consider subsidised canteens. Since it was listed as an

example, this prompted a respondent to include it in their answer. On further probing it

was revealed that they would have been unlikely to include subsidised canteens if it had

not been stated within the response option. This indicated that respondents were notrecognising any logic behind how the other pay aspects were named, but using the

examples within the heading as a prompt to direct their response.

Since the aspects themselves were reasonably well understood, few changes have been

suggested to the list of aspects.

Tested Aspects Revised Aspects

Basic salary level Basic salary level

Latest pay settlement / review Latest pay settlement / review

Performance-related pay elements, e.g.bonuses, profit sharing

Performance-related pay, for example:bonuses, profit sharing

Access to pay increments/ structuredpay progression

Access to pay increments/ structured payprogression

Other pay elements, e.g. overtime pay,shift pay premiums, unsocial hourspayments

Other pay elements, that is paymentsreceived on top of your basic pay, forexample: allowances, overtime pay, shift

premiums, unsocial hours pay


49/70


Pension benefits being built up Your pension package

Other non-pay benefits, e.g. healthinsurance, car/car allowance

Other benefits, that is benefits that are paidfor by your employer, for example: healthinsurance, use of a car, subsidised canteen

Other (please specify) e.g. season ticketloan, childcare vouchers, subsidisedcanteen

If there are any other aspects of your payand benefits that raise your motivationplease specify these below

One significant addition to the list of aspects is an option to select none of the above.

Respondents who felt unable to answer either of the

Documents

RAG - Measuring Employee Motivation