Carma internet research module n-bias

N-BIAS Nonresponse BiasAssessment Techniques

CARMA Internet Research ModuleJeffrey Stanton

May 15-17, 2008 Internet Data Collection Methods (Day 2-2)

N-BIAS Methods

• Rogelberg, S. G, & Stanton, J. M. (2007). Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10, 195-209.


N-BIAS Methods

Detect, estimate, and optionally compensate for the presence, direction, and magnitude of non-response bias. Gives clearer picture of extent of bias but few good options for eliminating it.

1. Wave Analysis2. Archival Analysis3. Follow-up survey of

non-respondents4. Passive non-response

analysis (time-to-respond vs. substantive)

5. Active non-respondentpre-study analysis

6. Interest level analysis7. Worst case resistance8. Benchmarking9. Triangulation


N-BIAS Methods

• Archival Analysis• Follow-up Approach• Wave Analysis• Passive Nonresponse Analysis

• Interest Level Analysis• Active Nonresponse Analysis• Worst Case Resistance• Demonstrate Generalizability

N-BIAS comprises eight techniques


Technique 1: Archival Analysis

•Most common technique•The researcher identifies an archival database that contains the members of the whole survey sample (e.g. personnel records).•That data set, usually containing demographic data, can be described:

–50% Female; 40% Supervisors, etc

•After data collection, code numbers on the returned surveys (or access passwords) can be used to identify respondents, and by extension nonrespondents. Using this information, the archival database can be partitioned into two segments: 1) data concerning respondents; and 2) data concerning nonrespondents.


Population Respondent Non-Respondent

50% female40% supervisors60% U.S.



So, if you found the above do you have nonresponse bias?


Technique 2: Follow-up Approach

• Using identifiers attached to returned surveys (or access passwords), respondents can be identified and by extension nonrespondents.

• The follow-up approach involves randomly selecting and resurveying a small segment of nonrespondents often by phone. The full or abridged survey is then administered.

• In the absence of identifiers, telephone a small random sample and ask whether they responded or not to the initial survey. Follow-up with survey relevant questions


Technique 3: Wave Analysis

• By noting in the data set whether each survey was returned before the deadline, after an initial reminder note, after the deadline, and so on, responses from pre-deadline surveys can be compared with the late responders on actual survey variables (e.g. compare job satisfaction levels).


Technique 4: Passive Nonresponse Analysis

• Rogelberg et al. (2003) found that the vast majority of nonresponse can be classified as being passive in nature (approx. 85%).

• Passive nonresponse does not appear to be planned.• When asked (upon receipt of the survey), these individuals

indicate a general willingness to complete the survey – if they have the time. Given this, it is not surprising that they generally do not differ from respondents with regard to job satisfaction or related variables.


Technique 5: Interest Level Analysis

• Researchers have repeatedly identified that interest level in the survey topic is one of the best predictors of a respondent’s likelihood of completing the survey.

• As a result, if interest level is related to attitudinal standing on the topics making up the survey, the survey results are susceptible to bias.

– E.g., if low interest individuals tend to be more dissatisfied on the survey constructs in question, results will be biased “high”


Technique 6: Active Nonresponse Analysis

• Active nonrespondents, in contrast to passive nonrespondents, are those that overtly choose not to respond to a survey effort. The nonresponse is volitional and a priori (i.e. it occurs when initially confronted with a survey solicitation).

• Active nonrespondents tend to differ from respondents on a number of dimensions typically relevant to the organizational survey researcher (e.g. job satisfaction)


Technique 7: Worst Case Resistance

• Given the data collected from study respondents in an actual study, one can empirically answer the question of what proportion of nonrespondents would have to exhibit the opposite pattern of responding to adversely influence sample results.– Similar philosophy as what occurs in meta-analyses when considering

the “file-drawer problem”

• By adding simulated data to an existing data set, one can explore how resistant the dataset is to worst case responses from non-respondents.

Technique 8

BenchmarkingUsing measures with norms for the population

under examination, compare means and standard deviations of the collected sample to the norms


Technique 9: Demonstrate Generalizability

• By definition, nonresponse bias is a phenomenon that is peculiar to a given sample under particular study conditions.

• Triangulating with a sample collected using a different method, or varying the conditions under which the study is conducted should also have effects on the composition of the nonrespondents group.


Exercise 2: Item Writing

• Write three Likert scaled items (each) to tap into the following response-related constructs:

Busyness (how busy the respondent feels)Topical interest (how interested the respondent is in the topic of

the survey)Satisfaction with the sponsoring entity (the group who is running

the survey)Response modality preference (e.g., got paper survey but

preferred web)


Demographic Profile:

1) Archival analysisAttitudinal Profile:

1) Follow-up approach

2) Wave analysisWarning Signs:1) Interest level analysis

2) Passive NR analysis

3) Active NR analysis

4) Worst-case analysis

Establish Generalizability


Important Footnote

• In many of the N-BIAS techniques described above, one is amassing evidence for the absence of non-response bias by making comparisons and tests that preferably lead to non-significant statistical results.

• What is the problem with this?

Documents

Carma internet research module n-bias