© 2009 by The McGraw-Hill Companies, Inc. Research Methods in Psychology Survey Research

© 2009 by The McGraw-Hill Companies, Inc.

Research Methods in Psychology

Survey Research


Survey Research

Survey results• describe

opinions, attitudes, preferences

• allow predictions about behavior

Survey research uses• questionnaires

predetermined set of questions

• a sample represents a population


Survey Research, continued

Surveys can be• limited, specific in scope• more global in their goals

Are surveys always biased?• Don’t assume bias just because a specific

organization or company has sponsored the survey

• Examine the survey procedures and analyses


Correlational Research

Assess relationships among naturally occurring variables• for example: attitudes, preferences, personality traits,

feelings, age, sex Correlation coefficients

• strength and direction of predictive relationship between two variables

–1.00 0 +1.00

negative no positive relationship


Correlational Research, continued

Do these relationships indicate positive or negative correlations?• As the number of years in which individuals smoke

cigarettes increases, the likelihood of lung cancer increases

• As the frequency of participating in volunteer activities increases, occasions of depressed mood decrease

• As arousal level increases, the likelihood of retaliation following an offense increases


Obtaining a Sample

Researchers are not interested simply in the responses of those who complete a survey• They seek to describe the larger population

from which the survey was drawn

Careful selection of a sample allows researchers to generalize findings from the sample to the population


Basic Terms of Sampling

Population• set of all cases of interest• examples:

current students at your school current residents of your city citizens of the United States

Sampling Frame• list of the members of a population

example: registrar’s list of registered students


Basic Terms of Sampling, continued

Sample• subset of the population used to represent the

entire population example: students in this class as a sample of all

students at this school (or this city)

Element• each member of the population


Goal of Sampling

sample should represent the population• characteristics of participants in sample

should be similar to those of the entire population example: Which sample represents a population that is 30%

freshmen, 30% sophomore, 20% junior, 20% senior?

Sample A Sample B30 freshmen, 30 sophomores, 60 freshmen, 60 sophomores,

20 juniors, 20 seniors 40 juniors, 40 seniors


Biased Samples

A biased sample • occurs when characteristics of the sample

differ systematically from those of the population

• samples can overrepresent or underrepresent a segment of a population samples made up of psychology students overrepresent

college students and underrepresent people not in college most research underrepresents individuals from diverse

cultures


Biased Samples

Two sources• Selection bias

occurs when a researcher’s procedures for selecting a sample result in one or more segments of the population being under- or overrepresented

• Response-rate bias occurs when individuals selected for the initial

sample do not complete and return the survey• lack of interest, worried about privacy, don’t have time


Approaches to Sampling

“Sampling” refers to procedures used to obtain a sample

Two basic approaches:• nonprobability sampling• probability sampling


Approaches to Sampling, continued

Nonprobability sampling• no guarantee each member of population has

an equal chance to be in sample• “convenience sampling”

researcher selects individuals who are available and willing to respond to the survey

example: magazine surveys, call-in radio surveys



Probability sampling• all members of population have an equal

chance of being selected for the survey• “simple random sample”• Two common methods:

choose randomly from the sampling frame of the population

random-digit dialing



Probability sampling• Stratified random sample: divide population

into subpopulations, called strata• Draw random samples from the strata

best to select samples proportional to the strata size

• Stratified random sampling increases the likelihood the sample will represent the population


Survey Methods

Four methods for obtaining survey data:• mail surveys• personal interviews• telephone interviews• Internet surveys

Each method has advantages and disadvantages

Choose method based on research question


Survey Methods, continued

Mail surveys• quick, convenient, self-administered, best for

highly personal or embarrassing topics• problem of response rate:

people selected for sample don’t complete and return the survey

final sample may be biased—not representative of the population

• little control over how people respond to the questions



Personal interviews• researchers gain more control over how

survey is administered• interviewers can seek clarification of answers,

ask questions potential problem: interviewer bias

• occurs when interviewers record only selected portions of answers or changes wording of questions and answers.

• interviews are costly; interviewers must be highly motivated, carefully trained, supervised



Telephone interviews• complete brief surveys efficiently and with greater

access to population• random-digit dialing to select random samples• supervise interviewers easily• Problems

selection bias: no phone or multiple phone numbers response-rate bias: willingness to answer questions on

phone interviewer bias: changes in survey questions and responses



Internet surveys• efficient, low-cost, potential for very large

samples• samples can be very diverse and access

typically underrepresented samples• problems:

selection bias: access to Internet response-rate bias: willingness to respond lack of control over research environment



Ways to increase response rate• questionnaire has a “personal touch”

use name, not “Dear student”

• responding requires minimal effort• topic of survey is interesting to respondents• respondents identify with organization or

sponsor of survey


Survey Research Designs

“Research design”• a plan for conducting a research project• choose research method best suited for

answering a particular question

Three types of survey designs• cross-sectional design• successive independent samples design• longitudinal design


Survey Research Designs, continued

Cross-sectional survey design• select sample from one or more populations

at one time choose population of interest use probability sampling or convenience sampling

• probability sampling leads to a more representative sample

respondents complete a survey



Cross-sectional survey design• Survey responses are used to

describe population (descriptive statistics) make predictions for the population (correlations)

at that one point in time• If samples are drawn from different

populations, compare the populations• cannot assess change over time with cross-

sectional designs



Successive independent samples design• a series of cross-sectional designs over time• a different sample from the population

completes the survey each time• each sample is selected from the same

population• responses from each sample are used to

describe the population at each point in time



Successive independent samples design• compare survey responses from each sample

to see how the population changes over time• cannot determine whether individuals change

over time• Problem of noncomparable samples

If different populations are sampled each time, responses may differ because of true changes over time or because different populations were sampled



Longitudinal survey design• same sample of individuals completes the

survey at different points in time• can assess how individuals change over time• responses from the sample are generalized to

describe changes over time in the population



Longitudinal survey design: Problems• Attrition: people drop out of the study

sample no longer represents population from which it was selected

• Reactivity: respondents may strive to be consistent over time or become sensitized to the topic

• longitudinal surveys can’t tell why people change over time (only correlations)


Measures in Survey Research

Questionnaires • most frequently used to collect survey data• measure different types of variables

demographic variables using checklists preferences and attitudes

• self-report scales• respond using rating scales (assume interval level of

measurement)


Measures in Survey Research, continued

All measures must be reliable and valid Reliability refers to consistency of

measurement• Test-retest reliability

administer measure two times to same sample individuals’ scores should be consistent over time a high correlation between the two sets of scores

indicates good test-retest reliability (r > .80) individuals’ scores need not be identical each time,

only same place in the distribution of scores



How to improve reliability?• more items• greater variability among individuals on the

factor being measured• testing situation free of distractions• clear instructions

A measure can be reliable but not valid



Validity refers to the truthfulness of a measure

A valid measure assesses what it is intended to measure• Construct validity refers to whether an

instrument measures the theoretical construct it was designed to measure



Example of construct validity:• Intelligence: Do these questions from a

common measure of intelligence assess a person’s intelligence in a valid manner? comprehension: “Why would people use a secret ballot?” vocabulary: “What does dilatory mean?” similarities: “How are a telephone and a radio alike?”



Establishing construct validity:• convergent validity

extent to which two measures of the same construct are correlated (go together)

• discriminant validity extent to which two measures of different

constructs are not correlated (do not go together)



Construct validity example• new measure of self-esteem • Which constructs should show convergent

validity with self-esteem measure and which would show discriminant validity?

Constructs:

depression, well-being, intelligence, extraversion, age, sensation-seeking, social anxiety, life satisfaction, grade point average,

reading comprehension, artistic ability



Correlations demonstrating construct validity and reliability are shown in a correlation matrix:

(1) (2) (3) (4)(1) new self-esteem, Time 1 1.0 --- --- ---(2) new self-esteem, Time 2 .85 1.0 --- ---(3) measure of positive affect .90 .90 1.0 ---(4) measure of artistic ability .10 .10 .15 1.0

test-retest reliabilityconvergent validitydiscriminant validity


Constructing a Questionnaire

Best choice for selecting a measure• use measure already shown to be reliable and

valid in previous research

If no suitable measure is found• create a questionnaire or measure

Creating a reliable and valid questionnaire is not easy


Constructing a Questionnaire, continued

Important first steps• Decide what information should be sought• Decide what type of questionnaire should be used

(e.g., self-administered?)• Write a first draft of the questionnaire• Have experts review questionnaire and then revise it

based on their suggestions• Pretest the questionnaire using sample and

conditions similar to the planned administration of the survey

• Review results and edit the questionnaire



Next steps: Establish reliability and validity• Reliability

Test and retest questionnaire using sample and conditions similar to planned survey.

• Validity Convergent: Administer questionnaire with

measures of theoretically related constructs Discriminant: Administer questionnaire with

measures of theoretically unrelated constructs



Guidelines for Writing Survey Questions• Choose how participants will respond

free-response (open-ended)• greater flexibility in responses• difficult to code

closed-response (multiple-choice, true-false)• responses are quick and easy• easy to score• may not accurately describe individuals’ responses

• Use simple, familiar vocabulary; keep questions short



Guidelines for Writing Survey Questions• Write clear and specific questions

avoid double-barreled questions place conditional phrases at the beginning of

sentences avoid leading questions avoid loaded (emotion-laded) questions



Ordering of questions• self-administered questionnaires

most interesting questions first

• personal and telephone interviews demographic questions first

• use funnel questions start with general questions and move to more

specific questions on a given topic

• use filter questions direct respondents to appropriate questions


Thinking Critically About Survey Research

Reported vs. Actual Behavior• Survey responses may not be truthful

Reactivity• not truthful because the information is being recorded

Social desirability• not truthful because responding as they “should”


Thinking Critically About Survey Research, continued

Measuring social desirability• sample items from Marlowe-Crowne (1964)

Social Desirability Scale

What is the socially desirable response?

1. No matter who I’m talking to, I’m always a good listener T F

2. I like to gossip at times T F

3. I’m always willing to admit it when I make a mistake T F

4. I have almost never felt the urge to tell someone off T F



Guidelines for social desirability• Accept people’s responses as truthful unless

there’s reason to suspect otherwise• Because actual behavior doesn’t always

match questionnaire responses, use a multimethod approach to answering research questions



Correlation and causality• “correlation does not imply causality”• example:

correlation between being socially active (A) and life satisfaction (B)

three possible causal relationships• A causes B • B causes A• variable C causes both A and B (e.g., number of friends)

= spurious relationship



Path analysis• statistical procedure to tease apart complex

correlational relationships among variables• Mediators

variables used to explain a correlation between two variables

• Moderators variables that affect direction or strength of

correlation between two variables



Example of path analysis• Evans et al. (2005) observed a positive

correlation between measures of poverty and measures of psychological distress among children

Poverty → Psychological distress

• This is called a direct relationship or path a



Path analysis example• Evans et al. sought to explain why this

relationship exists. level of chaos in the home as possible mediator chaotic living environment

• measured using concepts such as unpredictability, confusion, lack of structure, noise, overcrowding, and poor-quality housing



Path analysis example• Evans et al. observed two important

correlations Measures of poverty were positively correlated

with greater chaos in the home (Path b) Greater chaos in the home was positively

correlated with psychological distress (Path c)

• These two correlations represent the indirect relationship between poverty and psychological distress



Path analysis example• Diagram of direct and indirect relationships

Chaos

path b path c

Poverty Psychological

path a distress

“Chaos” mediates the relationship between poverty and psychological distress



Path analysis example• What if the relationships between poverty,

chaos, and psychological distress are not observed for all children?

• A moderator variable may affect these relationships moderators affect the direction and strength of

relationships



Path analysis example• Possible moderators:

sex of the child• the relationships between poverty, chaos, and

psychological distress may exist for boys, but not girls population density

• the relationships between poverty, chaos, and psychological distress may exist in urban areas, but not rural areas

personality features of children• the relationships may exist for low-resilient children, but

not high-resilient children



Path analysis• helps us to understand relationships among

variables• but these relationships are still correlational• cannot make definitive causal statements• other untested variables are related to

children’s psychological distress

Documents

© 2009 by The McGraw-Hill Companies, Inc. Research Methods in Psychology Survey Research