Designing Questionnaires

Designing QuestionnairesDr. Roger Harris

Visiting ProfessorResearch and Innovation Management Centre (RIMC)

andInstitute of Social Informatics and Technological Innovations (ISITI)

[email protected]://www.rogharris.org/

@HarrisrwhRogerSeptember 2013

1. Survey Research

Dr. Roger HarrisVisiting Professor

Institute of Social Informatics and Technological [email protected]://www.rogharris.org/


Survey questionnaires

• If the researcher is looking to test and quantify hypotheses and the data is to be analysed statistically, a formal standardised questionnaire is required.

• If the data to be collected is qualitative or is not to be statistically evaluated, it may be that no formal questionnaire is needed

• The heart of a survey is its questionnaire; the main means of collective quantitative primary data

• Survey results depend crucially on the questionnaire• Best practices stem from experience and

methodological research

Questionnaires

• A questionnaire is a formatted set of questions for obtaining information from respondents.

• The objective of questionnaire design is to translate the researcher’s information needs into a set of specific questions that respondents are able and willing to answer.

• Questionnaires collect quantitative data in a standardised way so that the data is internally consistent and coherent for analysis.

• There is no scientific principle to guarantee an optimal or ideal questionnaire.

Survey implementation choices

• Direct interview– costly; slow; high quality data

• Telephone interview– Costly; needs expertise.

• World Wide Web– Efficient; cheap; limited range of question types; hard to apply full

range of implementation procedures; respondents need access.• Self-administered by Mail

– Well studied; moderately expensive; can do well• Self-administered by eMail

– Cheaper; efficient; respondents need access• Multi-method

– Complex.

Quiz 1: True or False?

A questionnaire can be used to test and quantify a hypothesisTrue

Survey results depend on questionnairesTrue

There is a scientific method to design questionnairesFalse, best practices stem from experience and methodological research

The objective of questionnaire design is to translate the researcher’s information needs into a set of specific questions

True

Direct interviews are cheap to conductFalse

Telephone interviews are costlyTrue

The 5 most important elements for achieving high response rates

• Respondent-friendly questionnaire• The 4 contacts;

– Pre-notice invitation– The questionnaire– Thank you– (Replacement questionnaire reminder)– Final outcome notification

• For mail – return stamped envelopes• Personalised correspondence• Prepaid token incentive

Three types of survey information

• Dependent variables– Information we are primarily interested in

• Independent variables– Information which might explain the dependent

variables • Confounding variables

– Other factors related to both dependent and independent factors which may distort the results and have to be adjusted for

Characteristics of survey questionnaires

• Prescribed wording and order of questions, – to ensure that each respondent receives the same

stimuli• Prescribed definitions or explanations for each

question, – to ensure interviewers handle questions consistently

and can answer respondents' requests for clarification if they occur

• Prescribed response format, – to enable rapid completion of the questionnaire during

the interviewing process.

Quiz 2: True or False?1. Questionnaires should be respondent-friendly

True

2. There are 5 contacts with respondents during a questionnaire survey

False; there are 4

3. The three types of variables in a survey are; dependent, independent and confusing

False, they are dependent, independent and confounding

4. Dependent variables explain the independent variablesFalse, independent variables contain information that might explain the dependent variables

5. Prescribed wording and order of questions enables rapid completion of the questionnaire during the interviewing process.False. Prescribed wording and order of questions, to ensure that each respondent receives the same stimuli

However

• Given the same task and the same hypotheses, six different people will come up with six different questionnaires that differ widely in their choice of questions, line of questioning, use of open-ended questions and length.

• There are no hard-and-fast rules about how to design a questionnaire, but there are a number of points that can be borne in mind

A well-designed questionnaire should :

• Meet the research objectives• Obtain the most complete and accurate

information possible• Ensure that respondents fully understand the

questions • Be organised and worded to encourage

respondents to provide accurate, unbiased and complete information

• Be easy for the interviewer to record the answer• Keep the interview brief and to the point

How people answer questions; ideally

1. First, they interpret the question and deduce its intent, next ,

2. They must search their memories for relevant information, and,

3. Then integrate whatever information comes to mind into a single judgment.

4. Finally, they must translate the judgment into a response, by selecting one of the alternatives offered by the question.

This process is known as optimizing

How people may answer questions;

1. By automatic compliance; with no intrinsic motivation to make the answers of high quality

2. They may become fatigued, disinterested, or distracted and may be less thorough in comprehension, retrieval, judgment, and response selection

3. Instead of attempting the most accurate answers, respondents settle for merely satisfactory answers

4. Respondents may interpret each question superficially and select what they believe will appear to be a reasonable answer

This process is known as satisficing

Minimising satisificing

• The likelihood of satisficing is thought to be determined by three major factors:– task difficulty,

• question-specific attributes, e.g., the difficulty of interpreting a question and of retrieving and manipulating the requested information

• attributes of the questionnaire’s administration, e.g., the pace at which an interviewer reads the questions and the presence of distracting events

– respondent ability, • the extent to which respondents are adept at performing complex mental operations,

practiced at thinking about the topic of a particular question, and equipped with pre-formulated judgments on the issue in question

– respondent motivation• Motivation is influenced by need for cognition, the degree to which the topic of a question

is personally important, beliefs about whether the survey will have useful consequences.

• Efforts to minimize task difficulty and maximize respondent motivation are likely to pay off by minimizing satisficing and maximizing optimising.

Remedies for satisficing• Minimise task difficulty

– Minimise the number of words in questions– Avoid double-barreled questions– Decompose questions when needed

• Use ratings not rankings• Label response options• Minimise ‘response effects’

– Avoid blocks of ratings on the same scale (prevents ‘straight lining’)– Avoid ‘no response’ options– Avoid ‘agree/disagree’, ‘yes/no’, ‘true/false’ questions

• Maximise motivation– Describe purpose of the survey– Provide instructions to think carefully– Include random probes (“give your reason”)– Keep surveys short– Put important questions early.

Quiz 3: True or False?1. Given the same task and the same hypotheses, six different people

will come up with the same questionnaire False

2. There are no hard-and-fast rules about how to design a questionnaire

True3. The 4 steps in respondent optimising are; interpret, search,

integrate and translateTrue

4. Satisficing includes when respondents interpret each question superficiallyTrue

5. The likelihood of satisficing is thought to be determined by 3 factors; task difficulty, respondent ability and the weather.False. The likelihood of satisficing is thought to be determined by 3 factors; task difficulty, respondent ability and respondent motivation

Example of a survey1. Adoption rates of personal

computers varies among people2. Knowledge workers

3. Computer anxiety4. “Respondents with higher levels of

computer anxiety will be less likely to use personal computers”

5. The Computer Anxiety Rating Scale with 19 items; able to learn, confidence, apprehension, etc.

6. “I feel insecure about my ability to interpret a computer printout “ “I am confident that I can learn computer skills “

1. Review the literature to define the research question

2. Define the population to be surveyed3. Isolate the variables of interest4. Organise the variables into

hypotheses that reflect the research question

5. Operationalise the variables into tangible indicators

6. Develop questions that will enable measurement of the indicators

7. Formulate a questionnaire8. Carry out the survey9. Analyse the data

2. Purpose and Types of Questionnaires




Approaches

• Structured– All respondents provide answers to a set of questions

on a standardised questionnaire. Results can be compared to each other and easily summarised

• Unstructured– Allows the respondent to talk about whatever aspects

of the survey topic they wish. Best suited for qualitative data

• Semi-structured– Mixes both. Uses qualitative data to supplement

quantitative.

Administering questionnaires

• Self administered questionnaires may be sent by post, email, or electronically online.

• Advantages– Cheap and easy to administer.– Preserve confidentiality.– Can be completed at respondent’s convenience.– Can be administered in a standard manner.

• Read out by interviewers. Questionnaires may be by telephone or face to face.

• Advantages – Allow participation by illiterate people.– Allow clarification of ambiguity.– More reliable response rate

• The method depends on who the respondents are. For example, university lecturers may be more appropriately surveyed by email; older people by telephone interviews; train passengers by face to face interviews.

Comparisons of data collection modesParameter Interview Telephone Mail/Self

Cost Most expensive Medium expensive Cheapest

Standardisation Depend on interviewer

Depend on interviewer Good & reduces bias

Coverage Depend on personal contact

Depend on personal interviewer

Wide and scattered reach

Observation Listen, watch, assess Listen, assess None

Probing Allow Allow Not possible

Literacy Not required Not required Required

Respondent skills Language Language Not needed

Response rate Highest Medium Lowest

Privacy Difficult Some anonymity Good

Consent Easy to gain Possible Difficult

3. Asking Questions




Common Wisdom

1. Use simple, familiar words 2. Avoid technical terms, jargon, and slang3. Use simple syntax4. Avoid words with ambiguous meanings5. Use wording that all respondents will interpret in the same way6. Strive for wording that is specific and concrete as opposed to general

and abstract7. Make response options exhaustive and mutually exclusive8. Avoid leading or loaded questions that push respondents toward an

answer9. Ask about one thing at a time 10. Avoid double-barreled questions (if there’s an ‘and’ in the question,

there are probably two questions11. Avoid questions with single or double negations.

Optimize question order

1. Early questions should be easy and pleasant to answer, and should build rapport between the respondent and the researcher.

2. Questions at the very beginning of a questionnaire should explicitly address the topic of the survey, as it was described to the respondent prior to the interview.

3. Questions on the same topic should be grouped together.4. Questions on the same topic should proceed from general to

specific.5. Questions on sensitive topics that might make respondents

uncomfortable should be placed at the end of the questionnaire.6. Filter (branching) questions should be included, to avoid asking

respondents questions that do not apply to them.

Open versus closed questions

• Open questions permit respondents to answer in their own words

• Closed questions require respondents to select an answer from a set of choices

• The vast majority of survey questions are closed• Closed questions must be grouped into a

relatively small number of categories• This requires the development of a coding

scheme

Open versus closed questions

• Open questions are usually preferable for measuring quantities, e.g. age, number of doctor visits, hours devoted to housework,

• Closed quantity categories for this type of open question (e.g., less than 1 h, 1–3 h, more than 3 h) can produce errors

Comparison of open and closedOpen questions

Advantages• Allows exploration of the range of possible

themes arising from an issue• Respondent is asked to give a reply to a

question in his/her own words• Can be used even if a comprehensive range

of alternative choices cannot be compiled• Reveal the issues which are most important

to the respondent• Respondents can 'qualify' their answers or

emphasise the strength of their opinionsDisadvantages

• Respondents may find it difficult to 'articulate' their responses

• Respondents may not give a full answer if they may forget to mention important points

• Verbatim comments have to be interpreted, coded and reduced to manageable categories

Closed questionsAdvantages

• Easy and quick to fill in, less reliance on memory

• Can specify answer categories most suitable for the purposes

• Minimize discrimination against the less literate (self administered questionnaire) or the less articulate (interview questionnaire)

• Easy to code, record, and analyze results quantitatively

• Easy to report resultsDisadvantages

• Do not allow the respondent to give a different response to those suggested.

• Suggest answers that respondents may not have considered before.

Categorical judgments

• Such as the ‘‘what is most important problem’’

• Closed categorical question can be used only if its answer choices are comprehensive.

• Usual to include other category.• Respondents tend to restrict their answers to

the substantive choices that are explicitly offered.

Open response-option questions

• Both open-ended and also include specific response-options as well. For example,

• What features of this implement do you like?– Performance– Quality– Price– Weight– Others mentioned:

• Eliminates the disadvantages

Types of closed questions format

• Choice of categories, for example, What is your marital status?– [ ] Single– [ ] Married– [ ] Divorced– [ ] Widowed

• Likert style scale, for example, Statistics is an interesting subject– [ ] Strongly disagree – [ ] Disagree – [ ] Cannot decide – [ ] Agree Strongly – [ ] Agree

• Differential scales, for example, How would you rate the presentation?Extremely interesting [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] Extremely dull

• It may also be diagrammatic; • Extremely interesting Extremely dull

Types of closed questions format

• Checklists, for example, Circle the clinical specialties you are particularly interested in:

General medicine Obstetrics and gynecologyGeneral surgery OrthopaedicsOphthalmology Accident and emergencyPaediatrics General practice

• Ranking, for example, Please rank your interests in the following specialties(1= most interesting, 8=least interesting)

– [ ] General medicine – [ ] Obstetrics and gynecology– [ ] General surgery – [ ] Orthopaedics– [ ] Ophthalmology – [ ] Accident and emergency– [ ] Paediatrics – [ ] General practice

Question checklist1. Is this question sufficient to generate the required information?2. Can the respondent answer the question correctly?3. Are there any external events that might bias response to the

question?4. Do the words have the same meaning to all respondents?5. Are any of the words or phrases loaded or leading in any way?6. Are there any implied alternatives within the question?7. Will the question be understood by the type of individual to be

interviewed?8. Is there any ambiguity in my questions?9. Are any words or phrases vague?10. Are any questions too personal or of a potentially embarrassing

nature?11. Do questions rely on feats of memory?

Put questions into a meaningful order and format

• Opening questions– Opening question easy and pleasant to answer– Not threatening– Questions of special importance should be earlier

• Question flow– Flow in some kind of logical order– One leads easily and naturally to the next.– Questions on one subject should be grouped together

• Question variety– Vary the respondent's task from time to time– An open-ended question here and there (even if it is not analysed)– Questions involving showing cards/pictures to respondents can help

vary the pace and increase interest.

The effect of one word

Balanced questions

• Do you support India taking military action against Pakistan?• Do you support or oppose India taking military action against

Pakistan?• If you heard that candidate X supported a higher minimum

wage, would that make you more likely to vote for her?• If you heard that candidate X supported a higher minimum

wage, would that make you more likely to vote for her, less likely to vote for her, or wouldn’t it make much difference?

• Did you happen vote in the presidential election in 2010?• Did you happen vote in the presidential election in 2010, or

not ?

“Don’t know” – offered or volunteered

Using scales to get to “the most”

How questions can affect one another

• Funnelling. Ask general questions before specific priority questions in order to obtain unbiased responses.

• Pay attention to how one question might influence answers to the next:– In considering whether or not to vote for a political

candidate, how important is the candidate’s position on HIV/AIDS in making your choice?

– What do you think are the two most important health care issues for the government to address?

4. How to use Scales




Types of scales for closed questions

• The four levels of measurement: • Nominal• Ordinal• Interval, and • Ratio

Nominal scales

• Simplest form of measurement scale • Classifies individuals, companies, products, brands or other entities into categories

where no order is implied. • Often referred to as a categorical scale. • It is a system of classification and does not place the entity along a continuum. • Involves a simply count of the frequency of the cases assigned to the various categories• Example of a nominal scale• Which of the following food items do you tend to buy at least once per month? (Please

tick)1. Okra2. Palm Oil3. Milled Rice4. Peppers5. Prawns6. Pasteurized milk

• The numbers have no arithmetic properties and act only as labels.

Ordinal scales

• Involve the ranking of individuals, attitudes or items along the continuum of the characteristic being scaled.

• For example, if a researcher asked farmers to rank 5 brands of pesticide in order of preference.

• Example of an ordinal scale used to determine farmers' preferences among 5 brands of pesticide.

• Brand Order of preference1. Rambo2. R.I.P.3. Killalot4. D.O.A.5. Bugdeath

• Provides the order of preference but nothing about how much more one brand is preferred to another - no information about the interval between any two brands.

Interval scales

• The interval scale has equal units of measurement, thus making it possible to interpret not only the order of scale scores but also the distance between them.

• Allows the use of the arithmetic mean as the measure of average. • Two respondents with scale positions 1 and 2 are as far apart as two

respondents with scale positions 4 and 5• Cannot be said that a respondent with score 10 feels twice as strongly as

one with score 5. • Temperature is interval scaled, being measured either in Centigrade or

Fahrenheit. We cannot speak of 50°F being twice as hot as 25°F. No fixed origin.

• Interval scales may be either numeric or semantic. • Most of the common statistical methods of analysis require only interval

scales in order that they might be used.

Ratio scales

• The highest level of measurement• Has the properties of an interval scale together with a

fixed origin or zero point. • Examples; weights, lengths and times. • Permit the researcher to compare both differences in

scores and the relative magnitude of scores. • For instance the difference between 5 and 10 minutes

is the same as that between 10 and 15 minutes, and 10 minutes is twice as long as 5 minutes.

• Virtually all statistical operations can be performed on ratio scales.

Likert-type scales• Most often uses 5 points; • “I would use this tool in my work” • [Strongly Disagree ] [Disagree] [Neither Agree Nor Disagree] [Agree] [Strongly Agree]• Semantic scale• “Was the product easy or difficult to use?” Very Difficult [ ] [ ] [ ] [ ] [ ] Very Easy• Scales might have 7or 11 points.• The points offered should cover the entire measurement continuum, leaving out no regions• Points must appear to be ordinal, progressing from one end of a continuum to the other• The meanings of adjacent points should not overlap. • Each respondent must have a relatively precise and stable understanding of the meaning of each

point on the scale.• Most or all respondents must agree in their interpretations of the meanings of each scale point• Offering a neutral midpoint may encourage satisficing, or• Eliminating the midpoint will force people with no opinion to pick a point measurement.• Cross-sectional validity and test–retest reliability increases from 2- to 3- to 5-point scales but

were equivalent thereafter for 7-, 9-, and 14-point scales.

Some labels that work

Bipolar intervals

• Extremely bad• Moderately bad • Slightly bad• Neither good nor bad• Slightly good• Moderately good• Extremely good

• Dislike a great deal• Moderately dislike• Dislike a little• Neither like nor dislike• Like a little• Moderately like• Like a great deal

Unipolar intervals

• Not at all• Slightly• Moderately• Very• Extremely

• Definitely won’t• Probably won’t• Might or might 7not• Probably will• Definitely will

• Never• Sometimes• About half the time• Most of the time• Always

Quiz 5: True or False?1. There are three levels of measurement; Nominal, Ordinal, and

IntervalFalse. There are four levels of measurement; Nominal, Ordinal, Interval, and Ratio

2. Ordinal scales involve the ranking items along the continuum of the characteristic being scaledTrue.

3. In an interval scale, two respondents with scale positions 1 and 2 are as far apart as two respondents with scale positions 4 and 5

True. 4. Interval scales can only be numeric

False. Interval scales may be either numeric or semantic5. A ratio scale has the properties of an interval scale together with

a fixed origin or zero pointTrue

5. Questionnaire Appearance




Physical appearance

• The physical appearance of a questionnaire can have a significant effect upon both the quantity and quality of data obtained

• Ill-designed questionnaires can give an impression of complexity, and too big a time commitment

• Unnecessarily confusing layouts making it more difficult for interviewers, or respondents in the case of self-completion questionnaires, to complete

• Keep questionnaires as short as possible• In a rural situation an interview should not last

longer then 30-45 minutes.

Format

• Booklets– In the place of loose or stapled sheets of paper,

make it easier for interviewer or respondent to progress through the document. Fewer pages tend to get lost

• Simple, clear formats– The clarity of questionnaire presentation can also

help to improve the ease with which interviewers or respondents are able to complete a questionnaire.

Design tips

• Creative use of space and typeface– Tendency to put too much information on a page. This is counter-

productive since it gives the questionnaire the appearance of being complicated.

– Questionnaires that make use of blank space appear easier to use, enjoy higher response rates and contain fewer errors when completed.

• Colour coding– Can help in the administration of questionnaires. It is often the case that

several types of respondents are included within a single survey (e.g. wholesalers and retailers). Printing the questionnaires on two different colours of paper can make the handling easier.

• Interviewer instructions– Place alongside the questions to which they pertain. Instructions on

where the interviewers should probe for more information or how replies should be recorded are placed after the question

Distributing questionnaires

• If you’re sending a cover letter, be sure to use letterhead stationary and have the principal investigator or someone of renown from the funding agency sign it to add prestige.

• Your cover and advance letters should include a brief explanation of the study, an explanation of how you obtained the respondent’s name, why it is important that each "sampled" respondent cooperate, and a short assurance of confidentiality.

• Include a stamped self-addressed envelope for the respondent to return the completed form.

Design tips

• Clear instructions how to mark answers:– “Please tick the box that best describes your

answer”• Instructions for each question:

– “Tick one box only”– “Write the number inside the box”– “Circle or underline the choice that corresponds

most closely to your preference”

Design tips

• Questions in a meaningful order; natural flow, progression

• Place definitions in the question• Respondents rarely read instruction booklets• Indicate prominently which questions are

mandatory• Include a “thank you” at the end• Ensure the respondent or interviewer knows

what to do with the completed questionnaire.

Design elements for a mail questionnaire

• Booklet• Descriptive title• No questions on front or back• Size appropriate for mail envelope• Easy to read font; sans serif, 10 pt+, Ariel.

(this is sans serif this is serif)• Colour• Attractive, inviting design• Number questions• Embedded instructions

6. Testing and Evaluating Questionnaires




Piloting

• Until it has actually been used in interviews and with respondents, it is impossible to say whether it is going to achieve the desired results.

• Pre-test the questionnaire before it is used in a full-scale survey, to identify any mistakes that need correcting.– whether the questions as they are worded will achieve the desired results– whether the questions have been placed in the best order– whether the questions are understood by all classes of respondent– whether additional or specifying questions are needed or whether some

questions should be eliminated– whether the instructions to interviewers are adequate.

• Usually a small number of respondents are selected for the pre-test. The respondents selected for the pilot survey should be broadly representative of the type of respondent to be interviewed in the main survey.

Testing methods

• Expert review– one or more experts critiques the questionnaire. The

experts are typically survey methodologists, but they can be supplemented with specialists in the subject matter(s) of the questionnaire. Reviews are done individually or as part of a group discussion.

• Pretesting – involves administering a questionnaire to a small sample

of the relevant population under conditions close to, or identical to, those of the main survey. Ask them how the questions worked, which are typically reported at a group debriefing discussion (cognitive interviewing).

Cognitive interviewing

• After testing participants have completed the questionnaire, ask them:– In your own words, what do you think the

questionnaire is trying to discover?– What was the question asking?– What did you consider in determining your

response?– Was there anything difficult about this question?– Do you feel that something important might be

missing?

What am I measuring? Reliability

• Questionnaire reliability has to do with the quality of measurement

• Reliability is a measure of the consistency or repeatability of your measures

• Internal consistency is one type of reliability measure. Cronbach’s alpha measures it by how well a set of indicators measures a single uni-dimensional variable.

What am I measuring? Validity

• Construct validity– The degree to which inferences can legitimately

be made from the operationalisations in the questionnaire to the theoretical constructs on which they are based

• External validity– The degree to which the conclusions from the

questionnaire data would hold for other persons in other places and at other times

Quiz 6: True or False?1. The physical appearance of a questionnaire has no effect on either

the quantity and quality of data obtainedFalse.

2. In a rural situation an interview should not last longer then 30-45 minutes.True.

3. Expert review and pre-testing are two methods of piloting a questionnaire

True. 4. Cognitive interviewing involves asking test participants about the

questionnaireTrue.

5. Reliability is a measure of how truthful respondents areFalse. Reliability is a measure of the consistency or repeatability of your measures

7. Survey Conduct




Getting respondents to respond

• Introduce your questionnaire • Motivate people to complete it• Write a few lines to introduce your survey. • Introduction should be short and includes:

– Purpose of the survey – Why it is important to hear from the respondent – What may be done with the results – Promise of confidentiality– What possible impacts may occur with the results. – Person to contact for questions about the survey. – Due date for response

Steps in designing a survey

1. Review the literature to define the research question2. Define the population to be surveyed3. Isolate the variables of interest4. Organise the variables into hypotheses that reflect the

research question5. Operationalise the variables into tangible indicators6. Develop questions that will enable measurement of

the indicators7. Formulate a questionnaire8. Carry out the survey9. Analyse the data

Steps in designing a questionnaire

1. Decide the information required.2. Define the target respondents.3. Choose the method(s) of reaching your target

respondents.4. Decide on question content.5. Develop the question wording.6. Put questions into a meaningful order and format.7. Check the length of the questionnaire.8. Pre-test the questionnaire.9. Develop the final survey form.

Decide the information required.

• Do not start by writing questions• Decide 'what are the things I need to know

from the respondent in order to meet my survey objectives?‘

• From the research brief and the research proposal a small number of preliminary informal interviews with target respondents will give a glimpse of reality

Define the target respondents

• Must define the population about which the researcher wishes to generalise from the sample data to be collected

• E.g., existing users of a product type and/or non-users

• Take into account factors such as the age, education, etc. of the target respondents.

Choose the method(s) of reaching your target respondents

• The method of contact will influence the questions and the phrasing of those questions

• The main methods available in survey research are:· personal interviews· group or focus interviews· mailed questionnaires· telephone interviews.

Decide on question content

• Always ask, "Is this question really needed?“• No question should be included unless the data it gives rise

to is directly of use in testing one or more of the hypotheses

• Occasions when seemingly "redundant" questions might be included:– Opening questions that are easy to answer and which are not

perceived as being "threatening", and/or are perceived as being interesting, can greatly assist in gaining the respondent's involvement in the survey and help to establish a rapport.

– "Dummy" questions can disguise the purpose of the survey and/or the sponsorship of a study

– Validating questions to test the validity of other answers

. Develop the question wording

• Three forms of question,–closed, –open-ended and –open response-option

questions.

Put questions into a meaningful order and format

• Opening questions: – Easy to answer – Not threatening– The first question is crucial as the respondent's first exposure to the interview

and sets the tone for the nature of the task to be performed.• Question flow:

– Questions should flow in some kind of psychological order, so that one leads easily and naturally to the next. Questions on one subject, or one particular aspect of a subject, should be grouped together.

• Question variety:– Respondents become bored quickly and restless when asked similar questions.

Vary the respondent's task from time to time. An open-ended question here and there (even if it is not analysed) may provide much-needed relief from a long series of questions in which respondents have been forced to limit their replies to pre-coded categories. Questions involving showing cards/pictures to respondents can help vary the pace and increase interest.

Quiz 4: True or False?1. Surveys can be structured, unstructured or stratified

False. Surveys can be structured, unstructured or semi-structured2. Self administered questionnaires are relatively expensive

False.3. Double-barreled questions ask about guns

False. Double- barreled questions contain two questions 4. Questions on sensitive topics that might make respondents

uncomfortable should be placed at the end of the questionnaireTrue

5. The vast majority of survey questions are closed.True

6. Closed questions allow respondents to give a different response to those suggestedFalse

8. Automated Techniques




Internet questionnaires

• Facilitates – Design; templates available– Distribution– Completion– Analysis

• Lowers costs• Accelerates survey• Only for computer literate online respondents• No face-to-face• Concerns with confidentiality and privacy

Aspects of online questionnaires

• Graphical user interface– Control appearance, colours, fonts for attractive

design• Layout

– Paging from screen to screen– Scrolling simplifies navigation

• Push buttons– Buttons for specified actions and options; ‘Submit’

‘Next’


• Status bar– Visual indicator of respondent’s progress

• Radio button– Circular icon to activate one response choice and

de-activates the others• Drop-down box

– Space-saving device to present choices when they are needed but hides them when they aren’t


• Check boxes– Small boxes next to question that the respondent

clicks on to select their answer; when an x appears in it.

• Open-ended boxes– Boxes in which respondents can type in their own

answers to open-ended questions• Pop-up boxes

– Boxes that appear at selected points containing information or instructions to respondents.

Software aspects of online questionnaires

• Variable piping– Allows variables to be inserted into a questionnaire as a

respondent is completing it.• Error trapping

– Controls the flow• Forced answering

– Prevents respondents from continuing with a questionnaire if they fail to answer a mandatory question

• Interactive help desk– Live, real-time support for respondents with a problem.

Example: Survey Monkey

• Founded in 1999 • In 2013,

SurveyMonkey had 15 million users

• Over 200 employees• Used a lot for market

surveys, and by SMEs

Survey Monkey pricing options

Quiz 7: True or False?1. Construct validity degree to which inferences can legitimately be

made from the operationalisations in the questionnaire to the theoretical constructs on which they are based True.

2. External validity is a measure of the appearance of the front page.False. The degree to which the conclusions from the questionnaire data would hold for

other persons in other places and at other times3. Automated techniques facilitate design, distribution, completion

and printingFalse. Automated techniques facilitate design, distribution, completion and analysis

4. The status bar is a measure of how important the respondent isFalse. It is a visual indicator of respondent’s progress

5. Survey Guerrilla is an example of an online survey tool.False. Survey Monkey is an example of an online survey tool.

9. Examples of Question Design




A poor question The problem Improved version

Do you support gun control?

Yes No Don’t know

Unclear question because of vague words

Do you believe that guns do not belong in schools? Yes No Don’t know

Do you believe that guns and knives do not belong in schools? Yes No Don’t know

Two or more questions Do you believe that knives do not belong in schools?

Yes No Don’t know

Whenever violence occurs in schools, weapons are typically found in school lockers. Do you believe that students should keep guns in their lockers? Yes No Don’t know

Wordy or lengthy question Should students keep guns in their lockers?

Yes No Don’t know


Should students pack a 45 at school? Yes No Don’t know

Question contains jargon Should students carry a handgun to school? Yes No Don’t know

Students should not carry weapons and should not have them in their lockers. Do you agree? Strongly agree Agree Undecided Disagree Strongly disagree

Question contains negatives Should students keep guns in their lockers?

Yes No Don’t know

How many times have you seen a student carry a handgun? 0 times 1-2 times 2-3 times More than 3 times

Response categories overlap How many times have you seen a student carry a handgun? 0 times 1-2 times 3-4 times More than 4 times


How often have you seen students carry semi-automatic weapons at school? None 1 time 2 times 3 or more times

Respondent does not understanding to answer the question

How often have you seen students carry a rifle at school? None 1 time 2 times 3 or more times

To what extent do you feel that handguns are a problem at your school? A great extent Some Not very important Not a problem

Unbalanced response option To what extent do you feel that handguns are a problem at your school? A great extent Some extent Little extent

No labels

19 indicators

What does the * mean?

http://www.questionpro.com/survey-templates/training-services-evaluation-survey/

Bit confusing here; age/sex

What’s this?

Instructions

Definition

Overlapping< > ?Two questions

Two questions

Uneven frequency flow

Two questions

Two questions

Two questions

Two questions

Dense appearanceConfusing layout

What this?

Yes or no?

No instructions

What does it mean?Not much space

Comments about what?

No instructions what to put here

No alternatives or graduated response

Wordy

Some observations

• Almost every questionnaire you come across has something wrong with it

• Most are the result of inappropriate approaches;– Starting with the questions– No clear articulation of the purpose– Not following simple guidelines– Insufficient thought put into the process– Underestimation of what’s required

10. Recording and Coding Questionnaire Responses




• Closed question answers should be coded in order to facilitate recording.

• Number all questions• Record coded answers in a spreadsheet• For ease of recording, show the code next to the

response:1. Strongly disagree 2. Disagree 3. Cannot decide 4. Agree Strongly 5. Agree

Recording responses

Q1 Q2 Q3 Q4 Q5 Q6 Q71 Respondent2 Respondent3 Respondent4 Respondent5 Respondent6 Respondent7 Respondent8 Respondent9 Respondent

Sample spreadsheet recordingeBorneo Question 1 2 3 4 5 6 Gender Ethnicity Age Education Employment Income

10-20=1 21-30=2 None=N Unskilled=U 31-40=3 Primary=PI Skilled=S 41-50=4 Junior High=J Farmer=F Up to RM250 =1 Lun Bawang=L 51-60=5 Senior High=S Business=B RM251-RM500=2 Kelabit=K 61-70=6 College=C Government=G RM501-RM750=3 M Penan=P 71-80=7 University=U Housewife=H RM751-RM1,000=4

Resp. No. Village F Other=O 81+=8 Post-Graduate=PG None/other=O RM1,001+=5

1 Bakelelan F L 4 J F 22 Bakelelan F L 2 S B 23 Bakelelan F L 5 PI F 24 Bakelelan M L 5 PI F 15 Bakelelan M L 6 J F 16 Bakelelan F L 3 PI F 27 Bakelelan F L 4 J H 28 Bakelelan F L 3 U B 59 Bakelelan F L 3 J F 2

10 Bakelelan F L 4 J F 111 Bakelelan M L 4 PI F 112 Bakelelan M L 5 J B 513 Bakelelan F L 2 S B 214 Bakelelan F L 3 PI B 315 Bakelelan M L 3 C B 216 Bakelelan M L 4 PI F 2

Spreadsheet analyses

Male Female0

102030405060708090

100

62

33

Gender

Gender

Quantity

10-20 21-30 31-40 41-50 51-60 61-70 71-80 81+0

102030405060708090

100

1

25 25 24

127

1 0

Age

Age Range

Quantity

None

Primary

Junior High

Senior H

igh

College

Universi

ty

Post-Grad

uate0

20

40

60

80

100

2

2819 24

9 121

Education

Education Level

Quantity

Unskilled

Skilled

Farmer

Business

Governmen

t

Housewife

None/other

0102030405060708090

100

6 8

35

923

5 9

Employment

Employment Category

Quantity

SPSS for statistical analyses

11. Group Exercises




Exercise

• Nomophobia is the fear of being out of mobile phone contact.• A research organisation looked at anxieties suffered by mobile

phone users. • The study found that nearly 53% of mobile phone users in

Britain tend to be anxious when they "lose their mobile phone, run out of battery or credit, or have no network coverage".

• The psychographic variables were:– Attitude towards usage of cellular phones,– Mobile phones dependence, and – Associated anxiety

• Devise a set of questions to measure nomophobia.

The questionnaire focusing on nomophobia had 7 components

1. Duration of having mobile phone with self; 2. Anxiety and stress experienced because of faulty

connections; 3. Loss of mobile and battery discharge; 4. Amount spent per month on recharge 5. Reaction shown to phone ringing at inappropriate

times; 6. Frequency of change of phone/sim cards7. Reactions to inability of using the phone for a

period of one week.

Answers

Physical sensations 1. Physical sensations

Never Seldom Sometimes

Often Always

Accelerated heart beat, heart pounding

Face reddening or hot flashes

Tightness of the chest, chest pain or pressure

Numbness or tingling sensations of arms or legs

Lightheadedness, dizziness or feelings of unsteadiness

Shortness of breath or difficulty breathing

Sweating

Shaking or trembling

Intense fear, panicky feelings, anxiety

Thoughts 2. Thoughts Never Seldom Some

timesOften Always

I worry that something bad might happen to myself

I worry that something bad might happen to someone else

I worry that something bad might happen to the device (getting lost, stolen, or broken)

I worry about others being unable to contact me

I worry about missing important news

I worry that I might get lost

I am preoccupied with thoughts about how to complete tasks without the device

I am preoccupied with thoughts about how I could have acted differently to be able to access the device

Emotions 3. Emotions Never Seldom Some

timesOften Always

I feel lonely

I feel anxious and/or embarrassed because I feel unprepared

I feel anxious because I cannot access certain information

I feel stressed when I am not able to check on certain things on my device

I feel uncomfortable communicating with friends, family, colleagues, etc. without my device (I prefer to avoid communicating face-to-face)

I feel guilty for not contacting someone

I feel guilty because I am unable to do work or other important tasks

I feel angry that I allowed myself to lose access to the device

I feel angry that someone else led me to lose access to the device

I feel sad that I will not be able to use my device

Other 4. Other Never Seldom Some

timesOften Always

I feel bored

I just don’t feel “right”

Review Questions

1. Summarise the qualities of a good questionnaire.2. Where should interviewer instructions pertaining to responses to a

particular question be placed on the questionnaire?3. The textbook says that one does not start by writing questions. How should

the researcher begin?4. What are two occasions when apparently "redundant" questions should be

found in a questionnaire?5. Name the three advantages of open-ended questions.6. What are three reasons why a respondent might be unable to answer a

question?7. What is the recommended duration of interviews carried out in rural

situations?8. What are the key characteristics of opening questions in a questionnaire?

Documents

Designing Questionnaires