Achieving High Student Evaluation of Teaching Response Rates

2

OverviewRationale

Pilot Project

Process

Lessons Learned

3

Who we areThe Office of Quality, Teaching, and Learning is a service unit for academics and students. We are made up of three services – Quality Assurance (QA), Teaching and Learning Services (TLS), and Student Learning Support (SLS).

Services we provide:

• Teaching approaches, strategies and ideas

• Professional development as a teaching academic

• Student learning support

• Assessment, assignments, and marking

• Applying for Teaching and Learning Awards and Grants

• Student evaluation of teaching (TEVAL) processes and applying student feedback

4

Fast factsBond University received the most five-star ratings out of any university in Australia in the independent 2012 Good Universities Guide

Bond had an average student enrolment of 4,480 students in 2011, 1,449 of which were international students representing 85 nationalities.

There are four Faculties and one Institute:• Faculty of Business: School of Business; School of Information Technology; School of Hotel, Resort & Tourism

Management

• Faculty of Health Sciences & Medicine

• Faculty of Humanities & Social Sciences

• Faculty of Law

• Institute of Sustainable Development & Architecture

5

Background• Traditionally decentralised paper-based system

• New electronic format inspired by former executive BUSA members

• Mandatory evaluations voted in by University Teaching and Learning Committee and Academic Senate

• Pilot project Semester 3, 2009 - Success

6

ResearchSource Response Rate (%)

Paper-Based SETsResponse Rate (%)

Online SETs

Baruch (1999) - 40Ballantyne (2005) 55 47Brigham Young* 71 50Cook et al. (2000) 56 -Cornell University* 78 50Dommeyer et al. (2004) 75 43Griffith University (2005) 57 20Krieg and Hartsoch (2010) 74.2 56.8Nair et al. (2005) 56 31Ogier (2005) 65 30Sweep (2006) 56 23University of North Carolina* 92 59Watt et al. (2002) 33 33Average 64.7 40.7

Table 2: Published response rates

Table adapted from Nulty, D. (2008). ‘The adequacy of response rates to online and paper surveys: What can be done?’ Assessment & Evaluation in Higher Education, 33(3), 303. *Results cited in Hartshorne, R., Lambert, R., Algozzine, B., McAlpin, V., Algozzine, M., Norris, L. & Pyke, J.G. (2011). A comparison of web-based and paper-based course evaluations at UNC Charlotte: A report prepared by The Center for Educational Measurement and Evaluation and The Center for Teaching and Learning. Retrieved from University of North Carolina Charlotte website: http://studentcourseevals.uncc.edu/sites/all/files/Website%20Report%20-%20Course%20Evaluation%20Study.pdf

7

Instrument - EvaluationKit

1. Blackboard building block – single sign-on with LMS and access to Blackboard groups/class data

2. Highest security rating

3. Extensive troubleshooting & customer support

http://www.evaluationkit.com

8

Key Functional Requirements/Consultations

• Mandatory evaluations - LMS pop-up module to encourage participation

• Skip/Opt-out option

• Groups tool – maintains accuracy of class allocations

• Administrator level automated report builder

• Content Analysis Software integration

9

Process - Overview

• eTEVALs administered at the end of each semester

• Timeframe: Week 10 - 14

• Two separate evaluations: Subject and Educator evaluations (5 point Likert)

• Subject Evaluation: 5 questions, 1 open-text comment box; Educator Evaluation: 10 questions, 2 open-text comment boxes

• Instructor/Administrator reports available after student results are released

10

The Process• Access via email

11

The Process• Access via iLearn (LMS)

12

The Process• eTEVAL Dashboard

13

The Process• Skip/Opt out option

14

ResultsImposed Week 12 sanction increased response rate

Table 1: Week 12 Response Rate Results - (30/03/2012)

Response Rates (%)Level Subject evaluation Educator evaluation

Overall Bond Educator Evaluation 50.66% (+12.91%) Bond Subject Evaluation 55.72% (+13.06%) Faculty of Business

Business 58.98% (+13.82%) 55.54% (+14.03%) IT 55.95% (+10.41%) 51.38% (+11.32%) HRTM 56.69% (+14.01%) 51.59% (+13.37%)Institute of Sustainable Development and Architecture School of Sustainable Development 55.30% (+17.25%) 49.77% (+18.94%) School of Architecture 46.10% (+10.39%) 43.51% (+8.45%)Faculty of Humanities & Social Sciences 53.27% (+11.98%) 46.13% (+11.72%)Faculty of Health Sciences & Medicine School of Health Sciences 58.13% (+7.72%) 54.31% (+10.96%) School of Medicine 60.49% (+3.70%) 53.81% (+3.01%)Faculty of Law 54.27% (+15.74%) 50.80% (+16.96%)Bond College/BUELI 71.38% (+21.69%) 69.47% (+28.84%)

Opt-Out Option ResponsesOpt-Out 400 (+312) 970 (+630)

15

ResultsResponse rates of nearly 90% on the Likert scale items of online student evaluation of teaching in the first semester of whole-of-university implementation.

Table 2: Final Response Rate Results - (23/04/2012)

Response Rates (%)Level Subject evaluation Educator evaluation

Overall Bond Educator Evaluation 86.04% (+0.03%) Bond Subject Evaluation 89.22% (+0.23%) Faculty of Business

Business 91.54% (+0.08%) 90.48% (+0.10%) IT 86.61% (+0.45%) 83.03% (+0.16%) HRTM 88.54% (+0.00%) 87.90% (+0.00%)Institute of Sustainable Development and Architecture School of Sustainable Development 90.52% (+0.14%) 89.72% (+0.69%) School of Architecture 82.47% (+0.65%) 82.47% (+0.00%)Faculty of Humanities & Social Sciences 87.63% (+0.31%) 85.19% (+0.60%)Faculty of Health Sciences & Medicine School of Health Sciences 88.95% (+0.00%) 88.24% (+0.00%) School of Medicine 72.84% (+1.85%) 61.52% (+0.47%)Faculty of Law 90.94% (+0.24%) 89.51% (+0.37%)Bond College/BUELI 88.05% (+0.94%) 86.95% (+0.84%)

Opt-Out Option ResponsesOpt-Out 1097 (+6) 2514 (+13)

16

Trend of responses during the evaluation period

Results

Notable spike at Week 12 – Imposed sanction for non-respondents

17

Results

Continued success – Semester 2, 2012

• Increased awareness and support from students and faculty

• Contributing to a culture of evaluation

Subject Evaluation Educator Evaluation

Actual Response Rate 82.57% (+2%) 82.84% (+8%)

Opt Out Response Rate 1,478 (+13%) 2,516 (+13%)

Aggregated Response Rate 96.31% (+7%) 96.16% (+10%)

18

Campaigns

http://www.bond.edu.au/etevals

19

Subject Evolution Reports (SERs)

• Closing the loop on student feedback

• Inspired by former executive members of BUSA

• TEQSA - Provider Standards: - Standard 5.6 - Standard 4.2

• Bond University initiative

20

The SER Process eSETs close(Semester

a)

Quantitative and Qualitative data analysed

Administrator eSET report access(Semester b)

Administrator review and action SER

items (if applicable)

Faculty Associate Dean Teaching & Learning review and approve

Quality Assurance and approved for publication

(Week 2 Semester c)

Subject Coordinators may decide that the comments cannot be actioned (“didn’t like this subject”);

there may be no negative comments to action; Positive comments cannot be actioned (“loved this

subject”)

Subject Coordinator selects the “No action Taken”

Button and "Submit" button and SER remains unchanged

21

Subject Evolution Reports (SERs)

22

Subject Evolution Reports (SERs)

23

Things to consider

eTEVALs

CommunicationAdministration

Collaboration Publication

24

Research

Kinash, S., Naidu, V. & Wood, K. (2012). Electronic teaching evaluation: Student perceptions and teacher responses. Education Technology Solutions, 48, 60-62.

Knight, D., Naidu, V. & Kinash, S. (2012, Oct). Achieving high student evaluation of teaching response rates through a culture of academic-student collaboration. Paper to be presented at the Australasian Higher Education Evaluation Forum 2012, Rockhampton, Queensland, Australia.

Kinash, S., Knight, D. & Hives, L. (2011). Student perspective on electronic evaluation of teaching. Studies in Learning, Evaluation, Innovation and Development, 8(1), 86-97.

http://works.bepress.com/shelley_kinash

25

Questions