Student Evaluations: How Do They Influence Teacher

Thinking And Behaviour?

In search of the evidence

Overview of ProjectProject teamBack row:Stuart Terry (OP)Sarah Stein (OU)Lynley Deaker (OU)

Front row:Jo Kennedy (OU)Dorothy Spiller (WU)Trudy Harris (WU)

Overview of Project• What are teachers’ perceptions and how do these perceptions

influence teacher thinking and behaviour?• Does institutional use constrain teaching innovation?• Why do teachers undertake evaluations?• Do teachers use student evaluation feedback to improve

teaching?• Do they engage in dialogue with students about evaluation

feedback and their responses to it?• How can we use our findings to enhance the use of evaluations to

improve teaching and learning?

Progress to DateBy the 2010 AKO Colloquium:• Literature and environmental scan around evaluations• Preparation and writing of a literature review• Questionnaire sent out to academic staff

Since the 2010 AKO Colloquium:• Quantitative data analysis of the survey data• Interviews conducted at each institution• Thematic analysis of the qualitative data

Quote of the Week“I’m a lapsed Irish Catholic with massive guilt and chips on my shoulders so whenever I get an evaluation faster than any computer software you have I will gravitate towards the negative comments about what I am doing and I will, my parents and grandparents have taught me very very well to ignore compliments, so any thing that is remotely complimentary I ignore and assume that it is just someone trying to be nice and just focus on the negative things, whether it’s a number or a comment. So my in my mind I think gosh I have really let someone down and them completely ignore the other 50 odd students who have good comments.That’s the other thing about how we receive them. It’s a bit of paper, it’s a set of numbers, and we know stats are like a bikini, they show something but often hide the best bits!”

(from an interview with an Otago Polytechnic staff member)

Initial Findings - Quantitative• Frequencies• Chi Square and cross tabulations• ANOVA• Factor analysis• Cluster analysis• Regression

There is a significant difference between Otago University and the other two institutions regarding teacher perceptions that student evaluation results constrain their willingness to try new teaching methods (P-value=0.000)

15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. A great deal 1 2 3 4 5 Not at all

These findings indicate that almost 10% of teaching staff are NOT prepared to take risks with teaching in case it impacts on their appraisals/evaluation results and consequently their

promotion prospects.  % Great deal, Often

 % Missing data

University of Otago 10.5% 15.4%

University of Waikato 10.7% 2.9%

Otago Polytechnic 7.4% 8.1%

Years Teaching Most constrained (lower mean) 

Least constrained Significance between groups

University of Otago 11-15 years 21+ years p-value = 0.007*

University of Waikato 6-10years 0-5 years p-value= 0 .365

Otago Polytechnic 0-5years 6-10years p-value =0.0226

       

University of Otago Lecturer TF p-value=0.115

University of Waikato Lecturer Tutor & STutor p-value=0.158

Otago Polytechnic Principal ProgMngr &Head of School

p-value=0.307

15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. A great deal 1 2 3 4 5 Not at all

Initial Findings – Qualitative (Survey Comment Data)

15. My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals.

A great deal 1 2 3 4 5 Not at all

16. Please explain your answer to q15 (WU & OU) / 16. Any comments about question 15 (OP)

Total of 595 comments for q16 (56% of respondents made comments)

0%

5%

10%

15%

20%

25%

30%

35%

40%

5%

14%

38%

17%

14%

4%

8%

q16 Comment Data - Theme Coding - All Institutions

Percentage of Comments

q16 - Please explain your answer to q15: My willingness to try new teaching approaches is constrained by the possible negative effects on my student evaluations/appraisals. 1=A great deal, 5=Not at all

Q16 - Example QuotesCode Ref1 - constrained – absolutely - “Absolutely! If my career advancement will suffer due to lower evaluations; why would I take a chance on innovative; demanding; or difficult course material?????” (OU)

Code Ref2 - constrained - cautious about evaluations - “There appears to be greater monitoring teaching appraisals these days and I feel greater pressure to get good appraisal results than try new teaching approaches.” (WU)

Code Ref3 - not constrained – absolutely - “No, I think that by being creative in the classroom and making changes that students actually respond to this and like it - I would not hold back” (OP) “I don't really care about getting sacked or a pay rise; if I was in the job for the money I would do a different job and earn 2* as much!” (OU)

Code Ref4 - not constrained - need evaluations to learn from them - “Appraisals are good servants but bad masters. The point is that they should benefit the students, not (or only secondarily) me.” (WU)

Next steps

• Completion of the coding of the qualitative data from the survey

• Thematic analysis of interviews to be completed at all institutions

• Combination of the qualitative and quantitative data to develop a fuller understanding of teacher thinking

• Comparison of findings with literature• Writing of the AKO research and summary reports

Are There

AnyQuestions?

Are There

AnyQuestions?