Copyright 2000, Richard B. Warnecke QUALITATIVE APPROACHES TO QUESTIONNAIRE DESIGN: METHODS AND APPLICATIONS RICHARD B. WARNECKE, Ph.D. UNIVERSITY OF

Copyright 2000, Richard B. Warnecke

QUALITATIVE APPROACHES TO QUESTIONNAIRE DESIGN:

METHODS AND APPLICATIONS

RICHARD B. WARNECKE, Ph.D. UNIVERSITY OF ILLINOIS AT CHICAGO

MINORITY HEALTH PROJECT

UNIVERSITY OF NORTH CAROLINA AT CHAPEL HILL

JUNE 15, 2000



ANSWERING QUESTIONS

INTERPRETATION

RECALL

JUDGMENT FORMATION

RESPONSE FORMATTING

EDITING


LABORATORY METHODS

FOCUSED INTERVIEWS THINK-ALOUDS PARAPHRASING CONFIDENCE RATINGS RESPONSE LATENCY SORTS


INTERPRETING THE QUESTION

LITERAL MEANING

INTENDED MEANING

RESPONSE ALTERNATIVES


INTERPRETATION PROBES

PARAPHRASE QUESTION. DEFINE A WORD OR PHRASE. WHAT DOES A RESPONSE MEAN TO

YOU? BOUNDARIES (WHICH OF THE

FOLLOWING WOULD ALSO BE …..?) LABEL PREFERENCE INTERPRETATION DIFFICULTY


WHAT TYPES OF PHYSICAL ACTIVITY OR EXERCISE DID YOU PERFORM DURING THE

PAST MONTH?

________________________________________________


WHAT DID YOU THINK WE MEANT WHEN WE SAID “PHYSICAL ACTIVITY?


WHICH, IF ANY, OF THE FOLLOWING WOULD YOU (ALSO) CONSIDER TO BE

PHYSICAL ACTIVITY? WHAT ABOUT . . .

WALKING,

HOUSEWORK, SUCH AS CLEANING THEBATHROOM OR KITCHEN,

WORK-RELATED ACTIVITY, SUCH AS CONSTRUCTION

YARD WORK, SUCH AS MOWING THE LAWN OR RAKING LEAVES?


LOGISTIC REGRESSION ANALYSIS OF PHYSICAL ACTIVITY QUESTION MEANING PROBES

(UNSTANDARDIZED COEFFICENTS REPORTED)

Behavioral frequency 0.22* 0.02 0.07 0.12Male (1=Yes)

Male -0.91*** -0.36** 0.41* -0.43**(1 = Yes)

African American 0.07 0.042** -0.01 0.59**(1 = Yes)

Mexican American 0.48 0.25 0.36 0.43(1 = Yes)


LOGISTIC REGRESSION ANALYSIS OF PHYSICAL ACTIVITY QUESTION MEANING PROBES

(UNSTANDARDIZED COEFFICENTS REPORTED) cont’d.

Puerto Rican 0.26 0.38* -0.09 0.34(1 = Yes)

Age 0.08** 0.01 -0.00 0.00

Education 0.12 0.10 -0.12 -0.03

Model chi-square 30.11*** 18.42** 8.21 18.00**

Sample Size 365 372 371 371


RECALLING AND RETRIEVING

RECALL PRIOR INFORMATION COMPUTE ESTIMATE USE A SCHEMA (A PATTERN OF

ASSOCIATIONS AROUND WHICH THINGS ARE RECALLED, I.E. PAP SMEAR AS PART OF A PHYSICAL EXAM).


RETRIEVAL PROBES RECALL STRATEGY RECALL INTERVAL SEARCH STRATEGY (PROXIMAL, DISTAL,

ANYWHERE) LONG TERM RECALL--LINK TO EVENTS TO

HELP REMEMBER RECALL FRAME OF REFERENCE--WHAT

KINDS OF THINGS HELPED YOU REMEMBER?


HOW MANY TIMES PER WEEK OR PER MONTH DID YOU TAKE

PART IN (FIRST ACTIVITY MENTIONED)?

_______________ TIMES PER WEEK

____________________ TIMES PER MONTH


HOW DID YOU REMEMBER THE NUMBER OF TIMES YOU DID THIS? (PERFORM PHYSICAL

ACTIVITY)


IN A TYPICAL WEEK, HOW MANY TIMES DO YOU TALK ON

THE TELEPHONE WITH FAMILY, FRIENDS, OR NEIGHBORS?

______ TIMES PER DAY

______ TIMES PER WEEK


IN RECALLING THIS INFORMATION ABOUT HOW MANY TIMES YOU TALK ON

THE TELEPHONE, DID YOU SPECIFICALLY THINK ABOUT THE MOST RECENT WEEK

OR DID YOU AVERAGE TOGETHER EXPERIENCES YOU’VE HAD OVER A

NUMBER OF WEEKS, OR DID YOU THINK ABOUT IT IN SOME OTHER WAY?

THE MOST RECENT WEEK AVERAGE WEEK OTHER (SPECIFY) __________________ DON’T KNOW


JUDGMENT FORMATION

FIT A REFERENCE PERIOD

FIT A DEFINITION

RELATE TO QUESTIONNAIRE CONTEXT-- ORDER EFFECTS


JUDGMENT PROBES JUDGMENT EXPLANATION

– HOW ARRIVED AT ANSWER JUDGMENT CERTAINTY

– SCALE OF CERTAINTY (1-7) ANCHORING STRATEGIES

– COMPARATIVE ATTITUDE CRYSTALLIZATION

– THOUGHT ABOUT BEFORE– FREQUENCY OF PREVIOUS THOUGHT– DISCUSS WITH OTHERS


HOW LIKELY DO YOU THINK IT IS THAT YOU WILL HAVE SERIOUS

HEALTH PROBLEMS FROM SMOKING? DO YOU THINK IT

IS…..

VERY LIKELYSOMEWHAT LIKELY

SOMEWHAT UNLIKELYVERY UNLIKELY

DEFINITELY NOT POSSIBLE?DON’T KNOW


BEFORE WE ASKED YOU THE QUESTION ABOUT SERIOUS

HEALTH PROBLEMS, HAD YOU THOUGHT ABOUT IT BEFORE?

YESNO


ON A SCALE FROM 1 TO 7, WHERE 1 IS ONLY RARELY AND 7

IS VERY OFTEN, HOW OFTEN HAVE YOU THOUGHT ABOUT IT

BEFORE?

1 2 3 4 5 6 7

ONLY VERYRARELY OFTEN


ORDER EFFECTS-I

MINIMIZING ORDER EFFECTS– GENERAL BEFORE SPECIFIC (FUNNEL)– SPECIFIC BEFORE GENERAL

(INVERTED FUNNEL)– USE ORDER TO REDUCE EFFECTS OR

ITEM NON RESPONSE


ORDER EFFECTS -2 DEVELOP A LOGICAL FLOW TO HELP

RESPONDENT

– SEGMENT BY TOPIC– ASK ABOUT RELATED TOPICS

TOGETHER– ASK RECALL FROM MOST RECENT TO

LEAST RECENT– USE TRANSITIONS TO CHANGE TOPICS


ORDER EFFECTS-III ESTABLISH RAPPORT

– START WITH EASY, NON-THREATENING QUESTIONS

– DON’T START WITH DEMOGRAPHICS END WITH THEM

– DON’T START WITH KNOWLEDGE AND AWARENESS


EXAMPLE OF USE OF ORDER TO RESOLVE CONFUSION

ORIGINAL QUESTION: HOW SATISFIED ARE YOU WITH YOUR RELATIONSHIP WITH YOUR SPOUSE/SIGNIFICANT OTHER?

PROBE: WHAT PARTS OF THE RELATIONSHIP DID YOU THINK ABOUT?


THE PROBLEM QUESTION WORKED WELL IN

ENGLISH. HOWEVER IN SPANISH THE WORD

FOR RELATIONS IS THE WORD FOR SEXUAL RELATIONS


SOLUTION

REVERSED ORDER OF QUESTIONS. ASKED ABOUT SATISFACTION WITH SEXUAL RELATIONS FIRST.

THEN ASKED ABOUT OTHER ASPECTS OF MARITAL RELATIONSHIP.


FORMATTING A RESPONSE

RESPONSE CATEGORIES

RATING SCALES

RESPONSE SET– EXTREME CHOICES– ACQUIESCENCE


AfricanAmerican(N=102)

MexicanAmerican(N=106)

PuertoRican(N=84)

Whitenon-Hispanic(N=101)

Mean** 7.78 8.09 8.69 6.7

Standarddeviation

4.65 3.99 4.20 4.42

**F-ratio p< .01.

Note: Means with different superscripts differ significantly(p< .05) on the basis of multiple range test.

Mean Group Responses to Measure ofExtreme Response Style


AfricanAmerican(N=102


PuertoRican(N=84)

Whitenon-Hispanic(N=101)

Mean** 10.00 9.96 9.43 8.76

Standarddeviation

2.84 2.91 3.12 2.71

**F-ratio p< .01.

Note: Means with different superscripts differ significantly (p< .05) on thebasis of multiple range test.

Mean Group Responses to Acquiescence Scale


AfricanAmerican(N=110)


PuertoRican(N=87)

Whitenon-

Hispanic(N=108)

Mean** 5.57 5.92 5.47 4.85

Standarddeviation

2.47 2.30 2.16 2.29

**F-ratio p<.01.

Note: Means with different superscripts differ significantly (p<.05) on the basis of multiple range test.

Mean Group Responses to the Marlowe-CrowneSocial Desirability Scale


SCALE OF SATISFACTION WITH QUALITY OF LIFE

QUALITY OF LIFE SCALE WITH RANGE OF VALUES.– VERY SATISFIED– SATISFIED– SOMEWHAT SATISFIED– A LITTLE SATISFIED– NEITHER SATISFIED NOR DISSATISFIED– A LITTLE DISSATISFIED– SOMEWHAT DISSATISFIED– DISSATISFIED– VERY DISSATISFIED


ASSESSMENT

SATISFACTION RANGE

VERY 100-60

SATISFIED 100-50

SOMEWHAT 90-45

A LITTLE 70-30

DISSATISFIED RANGE

VERY 0-80

DISSATISFIED 0-80

SOMEWHAT 10-54

A LITTLE 10-53


SOLUTION

USED A BAR GRAPH WITH SINGLE SCALE WHERE BLANK EQUALS “NOT AT ALL SATISFIED” AND 6 EQUALS “VERY SATISFIED.”

SIMPLIFIED RESPONSE ON ONE DIMENSION

AVOIDED LABELS THAT WERE CONFUSING.


EDITING A RESPONSE

SOCIAL DESIRABILITY SELF PRESENTATION EFFECT OF INTERVIEWER

– GENDER– AGE– CULTURE/ ETHNICITY


RESPONSE EDITING PROBES

PERCEIVED SOCIAL DESIRABILITY PERCEIVED EMBARRASSMENT PERCEIVED HONESTY PERCEIVED INTERVIEWER EFFECTS

– ETHNICITY– AGE – GENDER


INTERVIEWER EFFECTS: AGE ORIGINAL QUESTION: HAVE YOU EVER

SMOKED MARIJUANA, HASH OR HASH OIL OR TAKEN THEM IN SOME OTHER WAY?

PROBE: DO YOU FEEL THIS IS A QUESTION THAT A PERSON YOUR AGE WOULD BE COMFORTABLE OR UNCOMFORTABLE TALKING ABOUT TO A SURVEY INTERVIEWER OF THE SAME AGE OR WHO WAS MUCH YOUNGER OR OLDER?


Interviewer age

Respondent age

Younger Older (18-29) (30-50)

Same age 77.5 78.6

Younger orOlder

30.3 54.6

Projected Comfort Level Discussing Marijuana Use with HypotheticalInterviewer of Same Age/Different Age


INTERVIEWER EFFECTS: GENDER

ORIGINAL QUESTION:DURING THE PAST WEEK, HOW

OFTEN HAVE YOU HAD CRYING SPELLS? PROBE:

DO YOU FEEL THIS IS A QUESTION THAT (RESPONDENT’S GENDER) WOULD BE COMFORTABLE TALKING ABOUT TO A SURVEY INTERVIEWER OF THE SAME SEX? OF THE OPPOSITE SEX?


Interviewergender Male Female

Same gender 34.1 93.3

Opposite gender 28.1 41.3

Projected Comfort Level Discussing Crying Spells with Hypothetical Interviewer of Same Sex/Opposite Sex


INTERVIEWER EFFECTS: ETHNICITY

ORIGINAL QUESTION: DO YOU THINK OF YOURSELF AS HETEROSEXUAL, HOMOSEXUAL OR BISEXUAL?

PROBE: DO YOU FEEL THIS IS A QUESTION THAT (RESPONDENT’S ETHNIC GROUP) WOULD BE COMFORTABLE DISCUSSING WITH AN INTERVIEWER OF THE SAME (ETHNIC GROUP)? A SURVEY INTERVIEWER OF A DIFFERENT ETHNIC GROUP?


Comfort LevelAfricanAmerican(N=101)


PuertoRican(N=82)

WhiteNon-Hispanic(N=99)

Topic: SexualorientationInterviewerrace/ethnicity: Samerace/ethnicity

72.3 64.0 65.4 82.8

Differentrace/ethnicity

61.0 51.5 54.9 71.4

Projected Comfort Level of Respondents by Topic, By Cultureof Hypothetical Interviewer, And By Culture of Respondent

(Percent Comfortable)

Documents

Copyright 2000, Richard B. Warnecke QUALITATIVE APPROACHES TO QUESTIONNAIRE DESIGN: METHODS AND APPLICATIONS RICHARD B. WARNECKE, Ph.D. UNIVERSITY OF