Slide 1

American Educational Research Association April 2013 Angie L. Miller, Ph.D. Amber D. Lambert, Ph.D. Center for Postsecondary Research, Indiana University Open-Ended Survey Questions: Non-Response Nightmare or Qualitative Data Dream?

Slide 2

Literature Review There is an increasing trend for requiring colleges and universities to show measures of their effectiveness (Kuh & Ewell, 2010) Combination of the struggling economy, funding cuts to higher education, and the evolution of the traditional higher education model (i.e. distance education, MOOCs, etc.) Alumni surveys are an important tool for assessment, but often have lower responses rates (Smith & Bers, 1987) Due to bad contact information, suspicion of money solicitation, and decreased loyalty after graduation

Slide 3

Literature Review Despite lower response rates, qualitative data from open-ended questions can still provide rich information from relatively few respondents (Geer, 1991; Krosnick, 1999) Disadvantages of open-ended questions: Heavy burden on respondents (Dillman, 2007) Some personal characteristics, such as language fluency and positive affect, can impact likelihood of responding to open-ended questions (Wallis, 2012)

Slide 4

Research Questions Do open-ended responses represent the opinions of the entire sample? Are some types of respondents more likely to complete these questions? Does question placement on the survey impact responses? The purpose of this study is to explore whether those with certain demographic and personal characteristics, including gender, age, cohort, number of children, marital status, citizenship, race, current employment status, income, and institutional satisfaction level, are more or less likely to respond to open-ended questions placed at the beginning, middle, and end of an online alumni survey.

Slide 5

Method: Participants Data from the 2011 administration of the Strategic National Arts Alumni Project (SNAAP) Participants were 33,801 alumni from 57 different arts high schools, arts colleges, or arts programs within larger universities Sample consisted of 8% high school level, 70% undergraduate level, and 22% graduate level alumni 38% male, 62% female,.2% transgender Majority (87%) reported ethnicity as Caucasian Average institutional response rate: 21% Only used those who completed the entire survey (i.e. did not drop out before the end) N = 27,212

Slide 6

What is SNAAP? On-line annual survey designed to assess and improve various aspects of arts-school education Investigates the educational experiences and career paths of arts graduates nationally Questionnaire topics include: Formal education and degrees Institutional experience and satisfaction Postgraduate resources for artists Career Arts engagement Income and debt Demographics

Slide 7

Method: Open-Ended Measures From beginning of survey (Question 17 of 82) Is there anything that [your institution] could have done better to prepare you for further education or for your career? Please describe. From middle (Question 44 of 82) Please describe how your arts training is or is not relevant to your current work. From end (Question 80 of 82) If there are additional things you would like to tell us about your education, life, and/or career that were not adequately covered on the survey, please do so here.

Slide 8

Method: Demographic Measures Demographic information collected for: Gender (Categorical- 3) Age group (Ordinal ranges) Graduation cohort (Ordinal ranges) Number of children (Ordinal ranges) Marital status (Categorical 4) Citizenship (Binary) Race/ethnicity check all (Binary 7 total) Current employment status (Categorical 7) Income (Ordinal midpoints of ranges) Institutional satisfaction level (Ordinal - 4-point Likert)

Slide 9

Analyses Series of 14 chi-squared analyses was done for each of the 3 open-ended question binary variables For gender, age group, graduation cohort, number of children, marital status, citizenship, each race/ethnicity option, and current employment status 3 independent samples t-tests completed for institutional satisfaction level and each of the 3 open-ended variables 3 non-parametric Mann-Whitney U tests were completed for income comparisons Skewed variance violated t-test assumptions

Slide 10

Results: Descriptive Statistics Much higher percentages of responses for beginning and middle questions than for the end question:

Slide 11

Results: Chi-Squared Analyses

Slide 12

Females more likely to respond Over 50 years of age more likely to respond Graduating in or before 1990 more likely to respond Singles less likely to respond Those with no dependent children more likely to respond Unemployed, retired, or with other employment status more likely to respond U.S. citizens more likely to respond Race: Asians less likely to respond, but those with other race/ethnicity more likely to respond

Slide 13

Results: Means and Other Ordinal Comparisons Those answering beginning and end questions were significantly less satisfied with overall institutional experience Didnt Answer Item: Mean Answered Item: Meant valuedf Effect Size (d) Near-Beginning Item3.573.4020.33***19914.16.26 Middle Item3.443.45-.90727082.01 Near-end Item3.463.424.20***9767.14.06 *p