Survey design workshop

Lecture 9 Week 14

Survey Design Workshop

Inter-University Research Workshop Program

Dr. James Neill
Centre for Applied Psychology
University of Canberra
1 February, 2011

Survey Design WorkshopUniversity of Canberra, ACT, AustraliaJames T. Neill

This workshop was previously presented by Dr. Brent Ritchie, 2008 who is now at the School of Tourism at The University of Queensland he kindly gave me a copy of his slides, which have been adapted and expanded each year since.

Image sources:Questionnaires are by James Neill (License: Public domain)

Scissors are by Gracenotes - http://commons.wikimedia.org/wiki/File:Edit-cut-mod.svg - (License: - Creative Commons by SA 2.5)

Further info: http://en.wikiversity.org/wiki/Survey_design/Workshop

Outline

Objectives

Introductions

Logins & Resources

Research methods

Questionnaire design

Levels of measurement

Sampling

Evaluation

Objective 1

Understand the importance of a rigorous, step-by-step process in planning, developing & implementing research questionnaires

Objective 2

Consider the pros and cons of common methods for survey administration Face-to-face interview

Telephone survey

Mail survey

Internet/mobile survey

Objective 3

Examine nuts & bolts of questionnaire design e.g.,Question style,

Response formats,

Layout, and

Pre- and pilot testing

Objective 4

Consider implementation issues Sampling methods

Sample size

Objective 5

Critically review example surveys.Existing examples

Student in-progress examples

with a view towards planning, drafting, and/or revising of an initial draft (pilot) survey.

Resources

Survey Design Workshop Notes (Wikiversity)

Readings

Books about surveys design and survey research (check library)

Image: James Neill, from Flickr, cc-by-a

Introductions

Introductions

Types of Research
(Research Methods)

There are 3 main research methods:Experimental

Quasi-experimental

Non-experimental

Image source: http://en.wikiversity.org/wiki/File:Nuvola_apps_edu_science.svgLicense: GFDL

Types of Research -
Experimental

Characterised by:Random assignment

Control over extraneous variables

Types of Research -
Quasi-experimental

Characterised by:Non-random assignment

Control over some extraneous variables

Groups are naturally occuring

Types of Research -
Non-experimental

Characterised by:No groups or conditions are created or used (i.e., no full experimental or quasi-experimental groups)

Minimal control over extraneous variables

Survey Research Characteristics

Surveys are widely used in
non-experimental social science research.

Often use interviews or questionnaires.

Involve real-world samples.

Often quantitative, but can be qualitative.

History of Survey Research

Survey research methodology was initially developed in the 1940's 1960's.

Since the 1980's, theories and principles evolved to create a unified perspective on the design, conduct, and evaluation of surveys.

Groves et al. (2004)

8 Survey Research Characteristics
Backstrom & Hursh-Csar (1981, pp. 3-4)

Systematic: follows a specific set of rules, a formal and orderly logic of operations

Impartial: selects units of the population without prejudice or preference

Representative: includes units that together are representative of the problem under study and the population affected by it

Theory-based: operations are guided by relevant principles of human behaviour and by mathematical laws of probability (chance).

Quantitative: assigns numerical values to nonnumerical character

Self-monitoring: procedures can be designed in ways that reveal any unplanned and unwanted distortions (biases) that may occur


Contemporary: it is current, more than historical, fact-finding

Replicable: other people using the same methods in the same ways can get essentially the same results


Advantages of
Survey-Based Research

Ecological validity

Access to wide range of participants

Potentially large amounts of data

May be more ethical
(than experiments)

Image source: http://commons.wikimedia.org/wiki/File:Crystal_Clear_action_edit_add.pngLicense: GFDL

Disadvantages of
Survey-Based Research

Lack of control
less internal validity

Data may be 'superficial'

Can be costly to obtain representative data

Self-report data only

Potentially low compliance rates

Image source: http://commons.wikimedia.org/wiki/File:Crystal_Clear_action_edit_remove.pngLicense: GFDL

Example Surveys

Unit Satisfaction Survey

Community Library Use

Image source: http://commons.wikimedia.org/wiki/File:Hyperlink-internet-search.svgLicense: GFDL

The research process

1. Establish
need for info/
research2. Problem
definition/
Hypotheses3. Researchdesign4. Sampling/Data collection5. Data
analysis6. Reporting

Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/

The research process

Image source: http://www.socialresearchmethods.net/kb/Assets/images/hourglas.gif

Survey construction:
Overview

What is a survey?

Types of questionnaires

Questionnaire development

Writing questions

Types of questions

Response formats LOM

Survey formatting

Image source: Unknown.

A standardised stimulus

A measuring instrument

A way of converting fuzzy psychological stuff
into hard data
for analysis

What is a survey?


Types ofsurveysSelf -administeredInterview -administeredPostalDelivered andcollectedTelephoneFace to facestructuredinterview

Web-basedTypes of surveys

1. Formulate
generic questionnaire2. Expand
the
questionnaireTurn into separatesections based onstudy objectives. Draft qs &
response
formats

4. Finalise
questionnaire
& implementQuestion
order &
funnel qs

Questionnaire development

3. Pre-test,
pilot test,& redraft

Formulate Generic Questionnaire

Turn objectives into sections of the survey

Ensure all questions relate to research objectives

For explanatory objectives or hypotheses ensure both dependent and independent variables exist

Cover Letter & Ethics Statement

Introduction or cover letter:Outline details of research project and allow for ethical informed consent.

Few will read it without good prompting and being easy-to-read

Instructions

Provides consistency - helps to ensure standard conditions across different administrations

Explain how to do the survey in a user-friendly manner

Example:
Life Effectiveness Questionnaire

Instructions: Example

Image source: Life Effectiveness Questionaire (Neill, 2009)

Cover letter / ethics statement checklist

Outline details of research project e.g.,:Who are you? Are you bona fide?

Purpose of survey?

What's involved?

Explain any risks/costs/rewards

How will results be used?

Human ethics approval #

How is consent given / not given?

Voluntary - can choose not to continue anytime

More info: Complaints, how to obtain results, contact details etc.

Font (type, size)

No. of pages

Margins

Double vs. single-siding

Colour, etc.

Layout

Image sources:Questionnaires are by James Neill (License: Public domain), based on the Life Effectiveness Questionnaire

Demographics single section, usually at beginning or end of questionnaire

only include relevant questions

Layout


Space for comments?

Indicate the end

Say thanks!

Layout


Flow/Structure

Logical order of questions
(use sections)

Ask screening questions first, rather than later. Does the participant qualify for the survey? (esp. for internet surveys)

Use funneling/branching questions to move respondents through survey

Start off with easy to answer and engaging questions

More controversial questions in middle section

Expanding the Survey

Image source: http://www.flickr.com/photos/peretzpup/3059447579/

Personal Information

Generally, researchers put personal information at beginning of survey (if required). However, this may put off respondents, so also consider uncluding at towards end.

Consider response format e.g, Income in categories or ranges

Personal Information

More likely to respond to personal questions for anonymous or mail surveys as opposed to face to face or telephone

Show cards for responses may help for face to face interviews

Pre- & pilot-testing

Pre-test items on convenient others - ask for feedback

Revise items e.g.,

Which dont apply to everybody

Are redundant

Are misunderstood

Are non-completed

Reconsider ordering & layout

Pilot test on a small sample from the target population, analyse, & revise

e.g., skewed response items

Types of Questions

Be able to justify and defend your choices...

Imag sourcese: http://commons.wikimedia.org/wiki/File:Aiga_information_.svghttp://www.flickr.com/photos/laffy4k/404298099/

Writing questions - Dos

Define target constructs

Check related research & questionnaires

Draft items
(for important, fuzzy constructs aim to have multiple indicators)

Pre-test & revise

Writing questions - Dos

Focus directly on topic/issue

Be clear

Be brief

Avoid big words

Use simple and correct grammar

Writing questions Don'ts

Inapplicable
must apply to all respondentsOver-demanding
e.g., recall of detail or time-consuming, unnecessary questioningAmbiguous
meaning must be clear to all respondentsDouble negative
e.g., Do you not disapprove of tax reforms?

Writing questions Don'ts

Double-barrelled -
e.g., Do you think speed limits should be lowered for cars & trucks?Leading -
e.g., dont you see some danger in the new policy?Loaded
e.g., Do you advocate a lower speed limit
to save human lives? vs Does traffic
safety require a lower speed limit?

Response biases

Social desirability

Acquiescence yea- and nay-saying

Self-serving bias

Order effects

Social desirabilityAcquiescence or Yea- and Nay-saying - tendency to agree or disagree with everything, use reversed items to controlSelf-serving bias - tendency to enhance selfOrder effects - routine, fatigue

Demand characteristics

InterviewHigh demand characteristics

Can elicit richer information

QuestionnaireLower demand characteristics

Information may be less rich

Types of Questions

Image: http://commons.wikimedia.org/wiki/File:Aiga_information_.svgThis example actually has many elements of a well-structued/designed survey what are they?

Accuracy of recall
decreases over time

Image source:http://books.google.com/books?id=E-8XHVsqoeUC&pg=PA58&lpg=PA58&dq=survey+design+types+of+questions+objective+subjective&source=bl&ots=fwFJdFznwn&sig=7Sil7P1uq4j6ctHzw3oKqliUhB4&hl=en&ei=KBSqSd6NJpm0sQPI3pzlDw&sa=X&oi=book_result&resnum=2&ct=result#PPA59,M1

Objective questions

A verifiably true answer (i.e., factual information) exists for each unit.

The question could be accurately answered by an observer.

e.g.,How times during 2009 did you visit a G.P.? ______

Subjective questions

Asks about fuzzy personal perceptions.

There is no true, factual answer.

Many possible answers per unit.

Can't be accurately answered by an observer. e.g.,

Think about the visits you made to a G.P. during 2010. How well did you understand the medical advice you received?perfectly very wellreasonably poorly not at all

See alos: The Power of Survey Design By Giuseppe Iarossihttp://books.google.com/books?id=E-8XHVsqoeUC&pg=PA58&lpg=PA58&dq=survey+design+types+of+questions+objective+subjective&source=bl&ots=fwFJdFznwn&sig=7Sil7P1uq4j6ctHzw3oKqliUhB4&hl=en&ei=KBSqSd6NJpm0sQPI3pzlDw&sa=X&oi=book_result&resnum=2&ct=result#PPA59,M1

Objective vs. subjective questions

Both types of questions may be appropriate; depends on the purpose of the study.

One criticism of this distinction: There is no such thing as objective and that all responses are subjective.

Open-ended Questions

Rich information can be gathered

Useful for descriptive, exploratory work

Difficult and subjective to analyse

Time consuming

Open-ended questions

Rich information can be gathered

Useful for descriptive, exploratory work

Difficult and subjective to analyse

Time consuming

Open-ended questions: Examples

What are the main issues you are currently facing in your life?

How many hours did you spend doing university study last week? _________

Closed-ended questions

Important information may be lost forever

Useful for hypothesis testing

Easy and objective to analyse

Time efficient

Levels of Measurement
=
Type of Data

Stevens (1946)


Levels of measurement

Nominal / Categorical

Ordinal

Interval

Ratio

Nominal/Category - measures identify categories e.g., sex, ethnicity.

Ordinal - relative ordering of responses e.g., rankings in an exam

Interval - scores stand in a quantitative relationship to one another, adjacent scores are separated by an equal interval

Ratio - like interval but with a true zero value e.g., height, speed

Discrete vs. continuous

Discrete- - - - - - - - - -

Continuous___________

Discrete data: finite options (e.g., labels)Continuous data: infinite options (e.g., cms)Discrete data is generally only whole numbers, whilst continuous data can have many decimalsDiscrete: nominal, ordinal, intervalContinuous: ratio

Each level has the properties of the preceding
levels, plus something more!


Categorical / nominal

Conveys a category label

(Arbitrary) assignment of #s to categories

e.g. Gender

No useful information, except as labels

Categorical / nomimal example:
Phrenological labels

Image source:L.N Fowler & Co. c. 1870.

Ordinal / ranked scale

Conveys order, but not distance

e.g. in a race, 1st, 2nd, 3rd, etc. or ranking of favourites or preferences

Image: Cropped version of http://www.flickr.com/photos/beatkueng/1350250361/?addedcomment=1#comment72157605326099631CC-by-A by Beat - http://www.flickr.com/photos/beatkueng/

Ordinal / ranked example:
Ranked importance

Rank the following aspects of the university according to what is most important to you (1 = most important through to 5 = least important) Quality of the teaching and education Quality of the social life Quality of the campus Quality of the administration Quality of the university's reputation


Interval scale

Conveys order & distance

0 is arbitrary

e.g., temperature (degrees C)Usually treat as continuous for > 5 intervals

Interval example:
8 point Likert scale


Ratio scale

Conveys order & distance

Continuous, with a meaningful 0 point

e.g. height, age, weight, time, number of times an event has occurredRatio statements can be made

e.g. X is twice as old (or high or heavy) as Y

Ratio scale:
Time


Why do levels of measurement matter?

Different analytical procedures
are used for different
levels of data.

More powerful statistics can be applied to higher levels


Levels of measurement:
Revision question

Fill in all cells

Descriptive Statistics

Ratio

Interval

Ordinal / Rank

Nominal / CategoricalGraphsExamplesProp-ertiesLevel

Closed-ended rating scales

Dichotomous

Multichotomous

Verbal frquency scale

The list (multiple response)

Ranking

Likert scale

Graphical rating scale

Semantic differential

Non-verbal (idiographic)

Image source: Questionnaire by TuppusLicense: Creative Commons Attribution 2.0

Dichotomous

2 response options e.g.,

Excluding this trip, have you visited Canberra in the previous five years?__ Yes __ No

Provides the simplest type of quantification

Multichotomous

How many hours did you spend doing university study last week?__ less than 5 hours__ > 5 to 10 hours__ > 10 to 20 hours__ more than 20 hours

eg. Which of the following statements best describes your reasons for taking a holiday to Canberra? (please tick one only) to visit friends and relatives for business for educational purposes for holiday/ sightseeing

Multichotomous

More than two possible answers e.g.,What type of attractions in your current trip to Canberra most appeal to you?__ historic buildings__ museum/art galleries __ parks and gardens

eg. Which of the following statements best describes your reasons for taking a holiday to Canberra? (please tick one only) to visit friends and relatives for business for educational purposes for holiday/ sightseeing

Verbal frequency scale

Over the past month, how often have you argued with your intimate partner?1. All the time2. Fairly often3. Occasionally4. Never5. Doesnt apply to me at the moment

Consider number of points (avoid over ~10)Consider directionConsider layout

The list (multiple response)

Provides a list of answers for respondents to choose from e.g.,Tick any words or phrases that describe your perception of Canberra as a travel destination:__ Exciting __ Important__ Boring __ Enjoyable__ Interesting __ Historical

The list (multiple-response)

What are the main issues that you are currently facing in your life? (tick all that apply) __ financial __ physical / health __ academic __ employment / unemployment __ relationships __ other (please specify)

Ranking

Helps to measure the relative importance of several itemsRank the importance of these reasons for visiting Canberra (from 1 (most) to 4 (least)):__ to visit friends and relatives__ for business__ for educational purposes__ for holiday/ sightseeing

Likert scale

Measures strength of feeling or perception.Indicate your degree of agreement with this statement:I am an adventurous person.
(circle the best response for you)

12345stronglydisagreeneutral agree stronglydisagreeagree

12345stronglyagreeneutral disagree stronglyagree disagree

Consider number of points (avoid over ~10)Consider directionConsider layout

Graphical rating scale

How would you rate your enjoyment of the movie you just saw?
Mark with a cross (X)

not enjoyable very enjoyable

What is your view of smoking? Tick to show your opinion.

Bad ___:___:___:___:___:___:___ GoodStrong ___:___:___:___:___:___:___ WeakMasculine ___:___:___:___:___:___:___ FeminineUnattractive ___:___:___:___:___:___:___ AttractivePassive ___:___:___:___:___:___:___ Active

Semantic differential

Non-verbal scale

Point to the face that shows how you feel about what happened to the toy.

Also called an idiographic scale.


Verbal frequency scale

Over the past month, how often have you argued with your intimate partner?1. All the time2. Fairly often3. Occasionally4. Never5. Doesnt apply to me at the moment

Sensitivity & reliability

Scale should be sensitive yet reliable.

Watch out for too few or too many options.

How many response options?Minimum= 2

Average= 3 to 9

Maximum= 10?

Basic guide: 7 +/- 2

Number of response options?

AGREEMENT ABOUT SOMETHING2-CategoriesDISAGREE AGREE3-Categories DISAGREE NEUTRAL AGREE

4-Categories STRONGLY MILDLY MILDLY STRONGLYDISAGREE DISAGREE AGREE AGREE

5-Categories STRONGLY MILDLY MILDLY STRONGLY DISAGREE DISAGREE NEUTRAL AGREE AGREE

Number of response options?
Likert scale example

Watch out for too many or too few response options

Capital punishment should be reintroduced for serious crimes

1 = Agree2 = Disagree

1 = Very, Very Strongly Agree 7 = Slightly Disagree2 = Very Strongly Agree 8 = Disagree3 = Strongly Agree 9 = Strongly Disagree4 = Agree 10 = V. Strongly Disagree5 = Slightly Agree 11 = V, V Strongly Disagree6 = Neutral

Example: How could
this question be improved?

How old are you?
___ 18-20
___ 20-22
___ 22-30
___ 30 and over

Are you satisfied with your marriage and your job?__________________________

Example: How could

You didnt think the food was very good, did you?_____ Yes _____ No

Example: How could

Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation?
____ Yes ____ No

Example: How could

What information sources did you use to locate your restaurant for todays meal?
(please tick appropriate spaces)
____ yellow pages
____ Internet
____ word of mouth

Example: How could

Comparison of
data collection methods

Alreck and Settle (1995:32)Alreck & Settle (1995; 32)

Finalise Questionnaire Draft

Questions need to be exhaustive and mutually exclusiveInclude other (please specify)

Ensure categories do not overlap

Finalise Questionnaire Draft

LengthTry to keep them as short as possible

Only ask questions that relate to objectives

Tricks? Font size/double sided photocopying/numbering sections

Maximising Response Rate

Layout and design is key

Respondents level of interest

Colour of paper

Accompanying letter / introduction

Mail surveys - self-addressed stamped return envelope

Rewards

Reminders or follow up calls

Pre-testing and Pilot testing

Pre-test try out on convenient others & revise

Pilot test try out on a small sample from the target population & revise

Be assertive and interactive about seeking feedback ask questions & observe

The customer is always right.

Pre-test & Revise

Pre-test items and ask for feedback

Revise:

items which dont apply to everybody

redundancy

skewed response items

misinterpreted items

non-completed items

Reconsider ordering & layout

LUNCH BREAK

Survey Design Critique

In pairs, look through the example questionnaires, and highlight aspects which:could be improved

are particularly good

you would like to ask about

Examples

Examine questionnaire examples

Examine structure, design issues and question styles

Note cover page and details provided

Sampling

Image source: http://commons.wikimedia.org/wiki/File:Marbles_canicas.PNG

Sampling:
Overview

Sampling terminology

What is sampling?

Why sample?

Sampling methods

Example: Shere Hites survey


Sampling terminology

Target population To whom you wish to generalise

Sampling frame Those who have a chance to be selected

Sample Those who were selected and responsed

Representativeness The extent to which the sample is a good indicator of the target population

Population - set of all individuals having some common characteristic, e.g., Australians

Sampling Frame subset of the population from which the sample is actually drawn e.g., White pages

Sample set of people who actually contribute data to e.g., Every 1000th person in the white pages who answers the phone and responds

Representativeness How similar is the sample to the population with regard to the constructs of interest?

What is sampling?

Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.- Trochim (2006)

Why sample?

Reduces cost, time, sample size etc.

If the sample is representative, the sample data allows inferences to be drawn about the total population.

Representativeness of a sample depends on:

Adequacy of sampling frame

Sampling method

Adequacy of sample size

Response rate both the % & representativeness of people in sample who actually complete survey

It is better to have a small, representative sample than a large, unrepresentative sample.

Sampling methods

Probability sampling Random

Systematic

ClusterMulti-Stage Cluster

Non-probability samplingQuota

Convenience

Snowball

Probability sampling - each member of population has a specific probability of being chosen.Random Sampling - everyone in population has an equal chance of being selected.

Systematic Sampling - e.g., every 10th student ID number

Stratified Random Sampling - population divided into strata, then random sampling from within each stratum (e.g., an equal number of males/females are selected)

Cluster Sampling - identify clusters of individuals & sample from these (e.g., 1 person per household)Multi-Stage Cluster Sampling (e.g., 1 person per selected household per selected suburb)

Non-probability sampling - arbitrary, sample not representative of populationQuota Sampling - e.g., 50% psychology students, 30% economics students, 20% law students

Convenience Sampling - take them where you find them method e.g., at shopping mall

Snowball Sampling - ask each respondent if they know someone else suitable for survey e.g., studying drug-users.

Random/probability sampling

Each unit has an equal chance of selection

Selection occurs entirely by random chance

Also called representative sampling

Simple random sampling

Everyone in the target population has an equal chance of selection

Useful if clear study area or population is identified

Similar to a lottery:List of names are assigned #s and randomly select #s of respondents

Randomly select # through table of random #s or by computer

Systematic random sampling

Selecting without first numbering

Respondents (units) selected from a list/file.

Useful when survey population is similar e.g. List of students

Select sample at regular intervals from the population e.g., every 5th person on a list, starting at a random number between 1 and 5

Sometimes called file sampling

Stratified random sampling

Sub-divide population into strata (e.g., by gender, age, or location)

Then random selection from within each stratum

Improves representativeness

e.g., Telephone interviews using post-code strata

Non-random / non-probability

Also called purposive or judgemental sampling

Useful for exploratory research and case study research

Able to get large sample size quickly

Limitations include potential bias and non-representativeness

Convenience sampling

Sampling is by convenience rather than randomly

Due to time/financial constraints

e.g. surveying all those at a tourist attraction over one weekend

Purposive sampling

Respondents selected for a particular purpose e.g., because they may be typical respondentse.g., select sample of tourists aged 40-60 as this is the typical age group of visitors to Canberra

e.g., Frequent flyers to contact regarding service quality in an airline setting

Snowball sampling

Useful for difficult to access populations e.g., illegal immigratnts, drug users

Respondents recommend other respondents

e.g., in studying ecstasy users, gain trust of a few potential respondents and ask them to recommend the researcher to other potential respondents

Sampling process

Identify target population and sampling frame

Select sampling method

Calculate sample size for desired power.

Maximise return rate

Summary of sampling strategy


Selection sampling method

Calculate required sample size


Sampling Example:
Shere Hite
American Sexology

Image sources: Unknown.

Hite's survey of American
male-female relations (early 1980's)

Shere Hite doyenne of sex polls

Media furors & worldwide attention

127-item questionnaire about marriage & relations between sexes

Sample: 4500 USA women, 14 to 85 years

Conclusion: Society and men need to change to improve lives of women

Some of Hites findings about American women....

Only 13% married for 2+years were still in love

70% married for 5+ years were having affairs... usually more for 'emotional closeness than sex

76% of these women did not feel guilty

87% had a closer female friend than husband

98% wanted basic changes to love relationships

84% were emotionally unsatisfied

95% reported emotional & psychological harassment from their men

Some of the critical comments....

She goes in with
prejudice & comes
out with a statistic.The survey often
seems merely to
provide an occasion
for the authors
own male-bashing
diatribes.Hite uses statistics to bolster her
opinion that American women are
justifiably fed up with American men.

Hite's response rate &
selection bias

100,000 questionnaires were sent to a variety of womens groups
(feminist organisations, church groups, garden clubs etc.)

4,500 replied
(4.5% return rate)

We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.
- Regina Herzog, University of Michigan Institute for Social Research

Hite's response rate &
selection bias

Regina Herzog, University of Michigan Institute for Social ResearchTo learn more about Shere Hites research, visit her website: http://www.hite-research.com

Sample size it's not how big, it's how representative

Objectivity watch out for manipulating the survey questions and results interpretation to suit your personal conjectures

Lessons from Hite's male-female relations survey

Regina Herzog, University of Michigan Institute for Social ResearchTo learn more about Shere Hites research, visit her website: http://www.hite-research.com

Measurement
Error

Image source: http://commons.wikimedia.org/wiki/File:Noise_effect.svgImage license: GFDL 1.2+

Measurement error

Observed score =

true score + measurement errorMeasurement error =

systematic error + random errorAny deviation from the true value caused by the measurement procedure.

Sources of measurement error

Non-sampling
(e.g., unreliable
or invalid
tests)Sampling
(e.g., non-rep. sample)Personal bias
(e.g., researcher favours a hypothesis)Paradigm
(e.g., Western focus on individualism)

Image source:s Unknown.Paradigm (e.g., assumptions, focus, collection method)

Personal researcher bias (conscious & unconscious)

Sampling (e.g., representativeness of sample)

Non-sampling (e.g, non-response, inaccurate response due to unreliable measurements, misunderstanding, social desirability, faking bad, researcher expectancy, administrative error)

To minimise measurement error

Use well designed measures:Multiple indicators

Sensitive to target constructs

Clear wording on questions/instructions

Reduce demand effects:Train interviewers

Use standard protocol



Maximise response rate:Pre-survey contact

Minimise length / time / hassle

Offer rewards / incentives

Coloured paper

Call backs / reminders

Ensure administrative accuracySet up efficient coding, with well-labelled variables

Check data


Summary of sampling strategy


Selection sampling method

Calculate required sample size


Sampling Task

A research project's aim is
To identify the behaviour and attitudes of UC students with regard to its computing services.What is the research population?

How might you get hold of a sample frame?

What sampling technique would you use?

Confidence Levels
/ Margins of Error

Relates to representativeness of a sample of a target population - to what extent can we be confident about the results?

Gives the estimated range of values into which we expect other samples to fall say 95% of the time

Social sciences = at least 90% or above, preferably 95%

Example 95%
Confidence Level Graphs

Source: http://en.wikipedia.org/wiki/File:Marginoferror95.PNG

Example 95%
Confidence Level Table

Confidence Level Example

Survey of visitors to Canberra between June 1 - August 31, 2004 = 10,000 visitorsWant to work with 95% confidence level and 3% margin of errorHow many to survey?Answer: Sample size = 964 peopleQ: What are your favourite attractions?

Following Results

+ / - 3% error @ 95% confidence level

Confidence Intervals /
Margins of Error

One time out of 20, we expect the answers may be greater than +/- 3%Establish the confidence level, margin of error you want to work with

Identify the number of surveys to be done

Once you have completed that number, do not do any additional surveys

HOWEVER.

Margins of Error

Sometimes we do surveys and we do not know how many will be returned until later, as with postal surveys

Thus you have to calculate the margin of error afterwards.Count up # of returned surveys

Identify target population and confidence level

Calculate margin of error

Margins of Error

All this assumes you know your target population and can get a sample frameIf not, a non-random sampling technique is best

Also consider whether you want to report results for sub-groups the margins of error will be widerConsider stratified random sampling

Summary - 1

Survey research has developed into a popular research method since the 1940's.

A survey is a standardised stimulus designed to convert fuzzy psychological phenomenon into hard data.

Summary - 2

Survey development - types of questions and response formats.

Sampling - probability & non-prob.

Levels of measurement & parametric / non-parametric stats

Ethical considerations

Sources of measurement error

Evaluation

Please complete the workshop evaluations

References

Alreck, P. & Settle, R. (1995). The survey research handbook (2nd ed.). New York: Irwin.Stevens, S.S. (1946). On the theory of scales of measurement. Science, 103, 677-680.Trochim, W. M. K. (2006). Sampling. In Research Methods Knowledge Base.Wikipedia (2009). Shere Hite - Methodology.See also Readings

Image source: Questionnaire by TuppusLicense: Creative Commons Attribution 2.0

Open Office Impress

This presentation was made using Open Office Impress.

Free and open source software.

http://www.openoffice.org/product/impress.html

Click to edit the outline text formatSecond Outline LevelThird Outline LevelFourth Outline LevelFifth Outline LevelSixth Outline LevelSeventh Outline LevelEighth Outline LevelNinth Outline Level

Click to edit the title text format


Click to edit the title text format


Education

Survey design workshop