The Unrecognized Interviewer

Studying Respondent Behavior in an

Establishment Survey of U.S. Academic

Institutions

Presented at the Third International

Conference on Establishment

Surveys (ICES-III)

June 20, 2007

Scott D. Crawford Survey Sciences

Group, LLC

Emilda B. RiversNational Science

Foundation

2

Our Establishment Survey

• Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS)

• Surveys U.S. academic institutions granting graduate degrees in science, engineering (S&E), and selected health-related fields

• Jointly sponsored by NSF, the National Institutes of Health and the Department of Energy

3

Brief History of the GSS

4

What is collected in the GSS?

• GSS produces national estimates for S&E and selected health-related fields on: fall graduate enrollment counts

by demographic categories by main mechanisms and sources of financial support

postdoctoral (postdoc) appointment counts by demographic categories by main sources of support by first professional degrees in medical and related fields

non faculty research staff counts by demographic categories by first professional degrees in medical and related fields

5

How is the GSS data collected?

6

What potential errors exist in this effort?

• Errors of NonobservationSamplingCoverageNonresponse

• Observational ErrorsRespondent InstrumentMode

Very little is known!

7

The GSS Respondent: Very Little is Known

• Are they knowledgeable about both postdocs and graduate students? If so, are they the best person to respond

to these requests?• What resources do the respondents rely upon

to provide a response?• Do they have direct access to the source of

these data?• How are the data organized?• How do respondents translate definitional

issues in their response?

8

The Establishment Response Process

9

The Establishment Response Process

Response Behavior Survey

10

The Response Behavior Survey (RBS)

• Unique characteristics that make this approach feasibleHigh response rates to GSS

Demonstrates strong motivation to complete

Institutional nature of the data collection Many respondents have it as part of their job

responsibilities to respond

Survey design unchanged in years may have provided a culture where it is easy for respondents to critique

Comfort with the web in the GSS Allows for a rapid (web) follow-up

11

RBS Sample

• Frame GSS department contact person list

• Things considered Consistency with GSS design Large enough to identify trends and comparison

cells in specific response characteristics (quantitative baseline study)

Small enough to minimize burden on respondents in an ongoing study

Some GSS departments do not have postdocs Given the respondent-department link, we needed

to consider the fact that some respondents had multiple departments

12

RBS Questionnaire Content

• Respondent characteristics• Involvement in the GSS• Institutional characteristics• Identifying resources for the GSS

response• Data sources• Barriers to access of data sources• Assessment of data quality• Survey topic definitions – their use and

understanding

13

GSS / RBS Survey Flow

14

RBS Contact Strategy

• Data and mode of each contact effort for the early and late sample• Approximately 45 days of data collection required, most of it was

during the primary study (GSS) data collection period.

Contact Mode Days After Prev. Contact

1st Contact Heads up email from NSF NA

2nd Contact Invitation letter mailed 14 days

3rd Contact Invitation email sent 7 days

4th Contact 1st email reminder sent 6 days

5th Contact Letter reminder mailed 3 days

6th Contact 2nd email reminder sent 5-7 days

7th Contact 3rd email reminder sent 4 days

8th Contact Telephone calls All conducted in late Sept. and early Oct.

15

RBS Response & Completion Rates

Contact Mode Cumulative Response Rate (RR2)

Completion Rates

1st Contact Heads up email from NSF

0.0% NA

2nd Contact Invitation letter 1.8% NA

3rd Contact Invitation email 24.9% NA

4th Contact 1st email reminder 38.8% NA

5th Contact Letter reminder 45.9% NA

6th Contact 2nd email reminder 54.6% NA

7th Contact 3rd email reminder 59.4% NA

8th Contact Telephone calls 72.1% NA

OVERALL 72.1% 84.5%

EARLY SAMPLE 74.7% 85.3%

LATE SAMPLE 61.6% 80.7%

16

The “It’s Not Me” Phenomenon

• Right from the start the RBS uncovered something amiss in the response behavior of the departmental person “of record” 6.5% of the sample reported that they were not

involved in the 2005 GSS

• Potential causes (unknown) Sample frame problem – ineligible department Institutional responder using departmental responder

information for something other than originally intended – potentially indicates a need for a 2nd layer RBS

17

Who are our GSS respondents?

• DemographicsGender, 26.2% are maleEducation

29.0% have less than a BA/BS degree 25.4% only a BA/BS

• Position (top 4 categories)27.8% administrative24.2% administrative support10.9% department chair9.5% faculty

18

Who are our GSS respondents?

• Involvement in maintaining records56.6% maintain graduate school records18.5% maintain postdoc records Percent for maintaining postdoc records

only increases to 37.8% when looking at those in postdoc only departments

19

Who are our GSS respondents?

• Are professional survey respondentsHave been involved in the GSS for a mean

of 6.5 years29.5% respond to other NSF surveys, 52.3%

respond to other surveys in generalHowever, only 7.1% have been involved

with IPEDS• May not be the most knowledgeable

Only 58.9% believe they are the most knowledgeable person to answer questions about postdocs

• 19.9% are users of GSS data

20

What do respondent characteristics tell us?

Education• While the survey is meant to collect

data on grad students and postdocs, the responder is not likely to be of same educational status

Assumptions on definitions may be incorrect

Interpretations that are effecting counts may not be appropriate

21

What do respondent characteristics tell us?

Professional respondents • These are likely to be people with job

responsibilities to complete surveys

• With 52.3% of the GSS responders also responding to other general surveys, the potential exists for a more streamlined format that may rely on these professionals’ experience and resources

22

What do respondent characteristics tell us?

Most knowledgeable • With nearly half not the most

knowledgeable respondent, this raises questions about potential problems in:

The departmental respondent selection process

How the survey is structured

23

What did the GSS respondents report on their own?

• When asked about reporting all of the data for specific data types

The majority reported all of the data on graduate students

However, most were not able to report all of the data on postdocs

Relied on assistance and other resources to report

They reported that they had less knowledge of the computer systems that manage such data

24

RBS Measures of Data Quality

• Actual measures (break-off rates, item missing data in the GSS) were not sufficient to be used as a measure of data quality (few beak-offs and few item missing data)

• Self-reported estimate of measure quality was used – respondent provided their thoughts on the quality of the data they were providing.

Some internal validity was found as our expectations of correlations with self-assessed data quality panned out through further analyses.

25

Response Behavior Effects on Data Quality

Impact on Perception of Data Quality

Positive No Impact NegativeWhen there is a person who always responds to requests for student data (Postdoc counts andfinancial support data only)

Whether the respondent is usually selected by position or on an individual basis.

Whether the respondent is one of the people always responsible for responding to student data requests or the GSS

Whether respondents have been involved in providing data for other surveys

26

Response Behavior Effects on Data Quality

Impact on Perception of Data Quality

Positive No Impact NegativeData sources is easily accessible by respondents

With how the primary data source is maintained or stored

When GSS respondents do not know whether the department has an official postdoc definition

Data source is considered accurate by the respondent

The format of the primary data source (being aggregate or individual)

Respondents who consider their data source as “complex”

Respondent is familiar with how the data is entered in the primary data source

27

Response Behavior Effects on Data Quality

• It is clear that the current GSS departmental contact is not the right one to provide postdoc data in most cases. Future evaluations of postdoc data collection should focus on the development of protocols for the identification of the correct person.

Postdoc offices appear to be a key starting point for this exploration

Knowledge of postdoc policies, as well as knowledge and accessibility of data sources, should be explored as a way to quickly identify the correct postdoc reporting individual

28

Instrument Interaction in the Response Behavior Context

• A little more than 1/2 of GSS respondents indicated that they had read the instructions for the criteria of what constitutes a postdoc

Their perceived data quality was much higher than for those who had not read the criteria.

29

Instrument Interaction in the Response Behavior Context

• Less than half of departments’ postdoc definitions are consistent with the GSS’ definition.

• However, there is no significant difference in self-assessed data quality between departments whose definition is consistent with the GSS and those whose is not.

• While departments/institutions have different definitions of postdocs, if the respondent read the GSS definition, they were able to provide good quality data (perceived). Quality problems emerged when the definition was not used.

30

Instrument Interaction in the Response Behavior Context

RECOMMENDATIONS

• Definitions and their usability must be improved

• Efforts to improve postdoc data should examine why foreign counts are the easiest data

element to provide – something about that data makes it easier to report

31

The Practice of Responding

• Does time of response (early, middle or late) correlate with perceived postdoc data quality?

• What specific recommendations does the respondent have about improving data quality?

32

9 out 10 Reported that Current Data Collection Timing was Adequate

Those who didn’t, reported that these months would be better

0

5

10

15

20

25

30

35

40

Better Month 2.4 1.4 2.2 33.6 39.3 35.7 36.9 19.4 8.7 6.6 3.7 4.5 3.3 2.7 3.1 28.6

Jul-04

Aug-04

Sep-04

Oct-04

Nov-04

Dec-04

Jan-05

Feb-05

Mar-05

Apr-05

May-05

Jun-05

Jul-05

Aug-05

Sep-05

Don't

Kno

33

How did timing of response play a role in data quality?

• Middle responders provided lower perceived quality postdoc data than early and late responders.

• There was no difference between self-assessed data quality between early responders and late responders.

34

Postdoc Counts Varied by Response Timing to the GSS

• Does lower perceived data quality (middle responders) translate into meaningful or significant survey response differences?

Mean Count of Postdocs Reported per Department

3.01

4.03

4.74

00.5

11.5

22.5

33.5

44.5

5

EarlyResponder

MiddleResponder

LateResponder

35

Responses:The Web Mode

• Most GSS respondents used the web survey as their primary mode of providing data

• The majority of respondents reported a preference for the web mode

36

Responses:The Paper Mode

• The paper survey was used more as a worksheet to collect responses prior to submission.

Approx. one-third of the respondents rely on the paper survey to use as a worksheet for completing the Web later.

37

Top Responses for Improving Response Time & Data Quality

Regarding GSS submission Providing more time

– Earlier distribution of the GSS– Extending the deadline

Improving data contacts (frequency and the right person)

Regarding their own institutions Improving their databases Improving support, reducing workload, expanding

personnel

Regarding the format of the GSS Make the format more user-friendly, simplify the form Clarify wording and definitions Designate a contact person for support

38

Study Recommendations

• Consider additional efforts to support institutions and responders in their process of providing GSS and postdoc data

• Given differences in response patterns uncovered in the RBS, a more extensive nonresponse analysis of GSS data is recommended.

• Research the design of paper worksheet versions of the GSS rather than a form intended to be submitted.

39

Is the RBS an effective tool?

• Potential strengthsAssisted in the identification of potential

frame issues (“it’s not me”)Effectively described the demographic

characteristics of the individual responderProvided some understanding of how the

respondent – establishment relationship can effect data quality

Identified at least one area (timing of GSS) where conflicting results show that the answer for a redesign is not so simple

40

Is the RBS an effective tool?

• Potential weaknessesDid not capture much information about

other “people” who were involved in the response process

Self reported measure of quality has little hard evidence so far – should be validated

41

Is the RBS an effective tool?

• Overall, the answer is yes.

• Further research is required to further develop models of the response process when an individual is responding for and within the context of an establishment.

42

Thank you.

Questions / Comments:Scott D. Crawford

scott@surveysciences.com