1

Title: INSTRUCTOR AND STUDENT FEEDBACK ON CLICKERS IN LARGE CALCULUS COURSES

Topic: Using Technology Effectively in the Post-Secondary Classroom Authors: Petra Menz , Veselin Jungic, Amy Wiebe E-mail: pmenz@sfu.ca, vjungic@sfu.ca, amy_wiebe@sfu.ca Address: Department of Mathematics Simon Fraser University 8888 University Drive Burnaby, BC, V5A 1S6 Canada Phone #: 778.782.3070 Fax #: 778.782.4947 Objectives: In order to use technology effectively in a large math class one has to have a thorough knowledge of the curriculum, be aware of the students and their background, understand the goals of the course, and study the dynamics of the classroom. This paper is a holistic approach in the analysis of the use of personal response systems or clickers in a large calculus course.

2

INSTRUCTOR AND STUDENT FEEDBACK ON CLICKERS IN LARGE CALCULUS COURSES

Petra Menz Veselin Jungic Amy Wiebe

Department of Mathematics, Simon Fraser University, Burnaby, British Columbia, V5A

2R6, Canada

ABSTRACT

The impact of clickers on the teaching of a large calculus class is investigated. The perspectives of students and the instructor as well as the effects on classroom dynamics are taken into consideration and analyzed. This study extends some other recent work in this area by considering all of these three aspects together instead of in isolation. 1. Settings

Especially in large mathematics classes, students’ responses to questions and interaction with the class in general are kept to a minimum, simply because time is limited by the amount of curriculum material that needs to be covered (Jungic et al, 2006). This can be frustrating for instructors who strive to reach out to students during lectures. A concerned instructor wants to draw each individual student – no matter what type – into the subject matter, ask each of them lots of questions, know when the student is having problems, and be able to address these problems immediately. What is even more important is that recent data supports strongly that students who participate actively in their own learning will increase their comprehension of the course content (Lucas, 2007). For years we have used hand-raising, flash cards or other manipulatives to get immediate feedback from students and to engage them with the course material, but this exposed students’ answers to the whole class, and it was clear that not all students participated in this form of activity. Recently, personal response systems (PRS), also known as Classroom Voting Systems or clickers, are being introduced as a relatively new, innovative, technological teaching tool into classrooms representing various disciplines across the country. Clickers are used to get immediate mass answers to multiple choice questions typically displayed on computer slides whether it is in the form of quiz questions or peer instruction. We wanted to find out, how these PRS impact teaching and learning in a large math class from the point of view of both instructor and student. Two courses on integral calculus taught consecutively by the first author were selected for this study. The first course with 127 participating students was held in the spring of 2006 without the aid of personal response systems. The second course with 99 participating students was held in the summer of 2006, where 53 students were given a PRS and the remaining students had no PRS. The authors set up three similar online questionnaires through the free assessment summary tool called FAST at Mount Royal College (Ravelli and Patz, 2006), with questions from the following three pedagogical categories: motivation, learning, and interaction. The first questionnaire was meant for students in the spring course (Appendix A), the second questionnaire was meant for

3

students without a PRS in the summer course (Appendix B), and the third questionnaire was meant for students with a PRS in the summer course (Appendix C). Thomson publishers have graciously lent a classroom set of clickers based on the TurningPoint technology to conduct this study (Thomson, 2006). TurningPoint is fully integrated in Microsoft PowerPoint to create interactive slides used with clickers. In addition, TurningPoint collects all students’ input and allows for a variety of data displays in Microsoft Excel. 2. Three Frameworks of Inquiry into Clickers

There were three sets of enquiries by the course instructor regarding the use of clickers in the calculus course. The first set of questions deals with the technical issues of employing clickers. How easy to use is the PRS in the lecture hall? How easy is the analysis of data? How time consuming is the creation of slides for each lecture? The second set of questions enquires into the impact on teaching. How much time will the clicker activity take during a lecture? What balance is needed between clicker activities and lectures? How do clicker activities change the dynamics of the lectures, if at all? The third set of questions address concerns with student learning. How well do students respond to the clicker activities, are they a motivator to attend lectures and learn more or seen as a gimmick? Do clickers increase interaction between students and instructor during a lecture? Do clickers increase learning of concepts, and how should this reflect on assignments and examinations? In the remainder of the article, we will present our answers to these questions.

a. Technical Aspects of Clickers

The instructor found that setting up the laptop and receiver was straightforward and did not experience any major set-backs with the slides during a lecture. In a study on a variety of personal response systems it was noted that TurningPoint is fast to learn and easy to use (Hanley and Jackson, 2006). The initial intent was to use the PowerPoint slides provided by the publishing company; however, in our opinion these slides did not sufficiently probe students’ understanding of the concepts introduced during the lecture. Most of the work in setting up the clicker activities went into the creation of slides that would match the instructor’s vision of any particular lecture. Although it was rather time consuming to come up with appropriate questions that had multiple choice answers and to create computer graphics that went along with the questions, an added side benefit was that it allowed the instructor to delve deeper into the subject matter and how to create connections between the subject matter and the students. The software is quite flexible in that pre-made slides can be used to display mass data with less effort or use instructor-created slides that are specific to the pedagogy used.

b. Constructing Lectures with Clickers

The use of clickers greatly impacted the delivery of the material as well as the material itself. Initially, the instructor created six to eight slides, with each slide containing one question. At the beginning of the lecture the students were asked to provide answers to all

4

slides without revealing the actual answers as a motivation for the new subject and assessment of students’ background knowledge. After the material was taught, the slides were given again, this time revealing the answers. While the instructor believes that showing and discussing the slides twice during a lecture is a great concept, there simply is not enough time in a 50 minute lecture to proceed this way. Through a few trial lectures the instructor settled on a different use of the slide questions for the remainder of the term. First of all, the number of slides shown was reduced to about half. Secondly, at the beginning slides with questions from the previous lecture were given to review concepts and check if students have understood this material. This provided more continuity between lectures, allowed students to digest the material and created a second time to address any learning concerns that showed up through these slide questions. Overall, written examples demonstrated during a lecture had to be reduced by about two slides, i.e. questions, to allow for the clicker activity, but coverage of content remained the same. Lastly, no changes were made to assignment and exam content for two reasons. First these slide questions were just an extension of how the instructor conducted the teaching with the content the same but the mode of delivery being different. Second we wanted to know if the clickers affected grades. It is the instructor’s opinion that the design of the question on each slide is critical work in order to motivate and engage the student with this clicker activity and to allow for critical thinking on part of the student about the subject matter. There are six characteristics of a good question: students’ interest is stimulated, students’ understanding is monitored, discussion on the subject material is enhanced, students’ misconceptions are brought to light, student learning is formatively assessed, and active learning is fostered (Miller et al, 2006). Putting extra time into the creation of clicker questions should not be a disadvantage for the instructor (Lucas, 2007) but an opportunity to re-engage with the material and to construct meaningful material that will make the student analyse, evaluate and reflect about the newly taught material. It is not easy to come up with the right question that gets at the heart of students’ misunderstanding (Cline et al, 2007) and care needs to be taken in how the question is worded using language that is plain and familiar to students rather than mathematical (Cline et al, May/June 2007) . However, an experienced and knowledgeable instructor that is engaged with the subject material and familiar with a variety of pedagogical tools will embrace clicker activities as an innovative tool and invest the time to create new questions. Lastly, we address the students’ point of view on lectures and clicker activities. An education website lists Nine Things Students Complain about on Evaluations in Lecture Courses (Berkeley UC, 2006), which are common across North America. From a student point of view, clicker activities help combat many of these complaints: slides help emphasise what is important, by their very nature slides ask questions and therefore help the instructor find out what is going on, the clicker activities liven up the material and aid the text, and lastly these activities make the instructor pause to allow students time to think. In addition, clicker activities add variety and motivation to a lecture (Gross Davis, 1993). Most importantly, every student in a large class can give an answer to a slide question and receive a response from the instructor and thereby be engaged in his or her own learning (Gedalof, 2002).

5

c. Pedagogical Impact of Clickers

Now we address the third set of questions regarding student learning. First of all, students readily responded to the use of clickers in the calculus course as Figure 1 shows. This is

in support of other studies on clickers that point out that most students appreciate clicker activities and are kept engaged with the lecture material (Cline et al, 2007, Carnevale, 2005, Kalinowski, 2005 and Martin, 2005). We wanted to compare students’ reaction to clickers, and so half the class were given a PRS while the other half did not have a PRS but were able to see the questions and answers on the slides. Here is a quote from the survey of PRS students, “I'd just like to re-iterate that I find it extremely useful to compare my answers to the rest of the class,” and a quote from the survey of non PRS students, “It takes away from doing more examples from our notes, but at the same time addresses key concepts. I liked the PRS because it tested what I understood on an ongoing basis. I was able to see how much I knew.” Both quotes support the idea that visual feedback on student percentages of correct and incorrect answers aids students in knowing where they are in their learning. Secondly, the instructor is absolutely convinced that the slide questions increase interaction between students and instructor. Having the immediate feedback on how many students selected each multiple choice answer allowed the instructor to directly

6

address misconceptions or offer additional explanations (Cline et al, 2007, Lucas, 2007). In addition, the concerns of the class majority were addressed rather than the few brave students who speak out in class. Here is just one of many quotes along the same lines from the survey of PRS users in the course, “I really found the instant feedback helped [the instructor] pin point our difficulties in a moment and address exactly what our misunderstanding was instantly. I found this much more useful than my attempts to follow for example a response from [the instructor] to a fellow student's vocal question. The information given by [the instructor] when addressing our wrong answers to a slide is much more generally applicable and helps my understanding of the material much more. One could say the onus is on me to ask more questions in class, but I find it is more time consuming for [the instructor] to get to the bottom of what I am trying to ask vs. a self explanatory poll of answers to a slide which often includes the difficulties I might be having already.” Lastly, it is suggested that an instructor can identify weak students using the PRS data and intervene in a timely fashion (Lucas, 2007). However, in a large class of 200 or more students it becomes exceedingly difficult to get to know students on an individual basis and intervene on a personal level with their learning. The above quoted comment and many like it the instructor received support the idea that clicker activities allow students to self-assess them and take a corrective course of action themselves. Thirdly, we come to the question of enhancing learning, which is difficult to measure. What the inquiry into clickers has shown though, is that any additional meaningful learning aid during a lecture is beneficial to students. In the words of one student, “I think those questions posted on the screen are more important than caring to use the clicker.” Figure 2 shows the percentage of students to correctly answer clicker questions. In light that these are review questions, it is important feedback to the instructor that students really have not totally grasped the material yet. We emphasize that this gave the instructor a second opportunity to teach the material, which was not done in lectures without clickers. There is no conclusive evidence that the clicker activities impacted the outcomes on midterm and final examinations. We also want to address the question of rewarding students for the use of clickers. It is our opinion that we do not reward students for simply attending a lecture, or for raising a question, or for making their own lecture notes, etc. Clicker activities are seen as a type of participation activity and they should just be naturally used without attaching any artificial rewards to them. “Reward but do not grade student participation” along with supportive arguments is one of many tips listed in Tools of Teaching (Gross Davis, 1993). Students quickly catch on that the feedback given to them and the instructor is beneficial and that they are experiencing questions and their accompanying answers that may otherwise not have been asked.

7

We are aware of the possibility that some responses may arise from students’ intuition rather than from an effort to provide a justifiable answer. Our experience is supported by other studies that found that sometimes students answer an electronically posted question by “merely guessing” (Kortemeyer et al.,2005). 3. Analysis of Students’ Responses

The instructor used the clickers for 2/3 of the course and not during the last 1/3 of the course. The students were surveyed at the end of each section to get a comparison of how they perceived the impact of clickers on their learning and the teaching of the material. First, we want to have a closer look at the surveys that were conducted 2/3 through the semester at the end of the clicker activities and compare it to the spring group. Figure 3 is a comparison of means between the spring students where no clicker activities were used and the summer students that were split into PRS users and no PRS users. The numbers at the bottom correspond to the questions listed in the Appendices. The responses are also divided into the three groups: interaction, motivation and learning by vertical lines.

8

Two results are of obvious interest. All three groups of students agree on questions asked pertaining to their learning in this course. One interpretation is that the instructor’s delivery of material was consistent regardless of what format was used. Furthermore, the two summer groups agreed on all but question 7 regardless of the use of the clickers or not, with the PRS group slightly higher on the Likert scale. Students of the no PRS group identified themselves with the PRS group in terms of the responses that were displayed and the feedback that was given by the instructor. This emphasizes that the importance lies in the questions that are being given to the students; however, the technology is

needed to get the feedback on these questions. The one question that the two groups differed on is interesting in itself. The PRS group strongly thought that the PRS would aid them in getting a better letter grade while the no PRS group thought less strongly so.

9

The result that surprised the instructor is the response to the third question on whether the instructor cares about student’s learning. Amazingly, all three groups of students have almost the same mean as a response, which we interpret to mean the instructor herself plays an important role in how students perceive to be cared for. Lastly, Figure 3 shows disparity in the motivation section among the three groups as was already pointed out with question 7. Furthermore, attendance is deemed much higher by the spring group with no PRS compared to the summer group where all students were exposed to clickers.

Figure 4 is a comparison of means between the PRS group to the no PRS group in the summer and shows the results of the second surveys about the last third of lectures where no clicker activities were held. This gives further evidence that the no PRS group identified with the PRS group. Even though the students had no clicker, they still read the questions and participated, thereby benefiting from the instructor’s feedback. Hypothetically speaking, what if in a class of 500 students only a group of, say, 50 students is equipped with a clicker? We theorize that the majority of students would benefit from the clicker activities held during lectures. Questions 9 and 17 are control questions, which explain the sharp expected drops.

10

In the summer group, the questionnaires for no PRS (Appendix B) and PRS (Appendix C) students were given twice, once after the first two thirds of lectures that included

clicker activities, and then again at the last third of lectures that had no clicker activities. Figure 5 is a comparison of means within the summer group of the two surveys. Amazingly, there are only two obvious disparities in opinions: one in the interaction group and the other in the motivation group. The first disparity on question 2 does not come as a surprise, since it asks how a student perceives his or her answers compare to the rest of the class. The instructor resorted to hand-raising when the clicker technology was not available but obviously students did not get the same level of feedback. The second disparity on question 5 does come as a surprise. Question 5 queries students about attendance and sees an increase in its importance in the latter third of the course when no clicker activities were given. However, what happened during the lectures is that students started to speak up more and asked questions thereby getting similar responses. The instructor also received several emails from students asking to bring back the clicker activities as they found them helpful in their learning. 4. Conclusion

In summary, clicker activities lend themselves to yet another mode of delivery for lectures which are appreciated by students; provide fast, visual, accurate and anonymous feedback for any sized class of students and are particularly practical for reaching

11

students in a large class; allow the instructor to address misconceptions which may otherwise have gone unnoticed; that students get on-going feedback on their understanding; and that clicker activities are yet another source of a valuable learning tool for students. However, coming up with appropriate questions and creating slides can be a time consuming job in addition to preparing the lecture material and should be taken seriously in an effort to make clicker activities worthwhile. Further studies need to be conducted to answer the question of whether clicker activities substantially increase students’ learning and change test performance.

References

Berkeley University of California. Office of Educational Development. 2006. Nine Things Student Complain About on Evaluations in Lecture Courses. http://teaching.berkeley.edu/ninecomplaints.html (23/08/2006). Carnevale, D. June 2005. Run a Class Like a Game Show: Clickers Keep Students Involved. Chronicles of Higher Education. Vol. 51. Issue 42. pB3-B3 Cline, K. Zullo, H. Parker, M. Using Classroom Voting in Mathematics Courses. Proceedings of the 19th Annual International Conference on Technology in Collegiate Mathematics. New York. Addison Wesley. p35-39. 2007. Cline, K. Zullo, H. Parker, M. Teaching with Classroom Voting. FOCUS. Vol. 27. No. 5. p22-23. May/June 2007. Gedalof, A. 2002. Green Guide: Teaching Large Classes. http://www.uwo.ca/tsc/tlc/theguide.html (27/07/2006) Gross Davis, B. 1993. Good Teaching. http://teaching.berkeley.edu/bgd/teaching.html (23/08/2006). Hanley, J.T. Jackson, P. June 2006. Making It Click. Technology & Learning. Vol. 26. Issue 11. p34-40. Jungic, V. Kent, D. Menz, P. October 2006.Teaching Large Math Classes: Three Instructors, One Experience. International Electronic Journal of Mathematics Education (IEJME). Vol. 1. No. 1. p1-15. Kalinowski, T. September 2005. TV Remote-like Keypad Encourages Students to Do the Reading. Canadian Press NewsWire. Toronto. Kortemeyer, G. Hall, M. Parker, J. Minaei-Bidgoli, B. Albertelli II, G. Bauer, W. Kashy, E. June 2005. Effective Feedback To The Instructor From Online Homework. Journal of Asynchronous Learning Networks. Vol. 9. Issue 2. Lucas, Adam. Using Peer Instruction and iclickers to Enhance Student Participation in Calculus: A paper submitted for publication in PRIMUS 2007.

12

Martin, J. Mitchell, J. July 2005. Clicking for Scholars. Industrial Engineer. Vol. 37. Issue 7. p66-66. Miller, R. L. Santana-Vega, E. Terrell, M. S. "Can Good Questions and Peer Discussion Improve Calculus Instruction?" PRIMUS. Vol. 16. No. 3. September 2006. Ravelli, B. Patz, Z. 2000. Free Assessment Summary Tool (FAST). http://getfast.ca/ (14/07/2006) Thomson. 2006. JoinIn on TurningPoint. http://www.thomsonedu.com/support/customer/showProductType.do?custProductId=3&flag=customersupport (23/08/06)

13

Appendix A

Student Survey Spring – No PRS

The answer to each question is based on the Likert Scale: strongly agree agree not applicable disagree strongly disagree 4 3 2 1 0 Interaction: 1. I feel that I participate during the lectures. 2. During lectures, I know how my answers compare to the rest of the class during

participation activities. 3. My instructor cares what and how I learn during the lectures. 4. I enjoy participating. Motivation: 5. I attend this class no matter what. 6. I think that I got good marks on the midterms. 7. I think that I will get a good letter grade. 8. I use the textbook and the resources it came bundled with. Learning: 9. When the instructor asks questions, it takes too much class time. 10. I think I retain the material during the lectures. 11. I think I understand the material during the lectures. 12. I think about the newly introduced concepts. 13. I read my textbook in preparation of the lectures. 14. I stay focused in lectures. 15. The lectures seem rushed because the instructor has to hurry to cover everything. 16. The instructor focuses on topics that are the most difficult for us. 17. I think the lectures are a waste of time. 18. I think the lectures are effective for my learning.

14

Appendix B

Student Survey Summer – No PRS

The answer to each question is based on the Likert Scale: strongly agree agree not applicable disagree strongly disagree 4 3 2 1 0 Interaction:

1. Even though I had no PRS, I feel that I am able to participate more in the lecture using PRS slides.

2. The PRS helps me to know how my responses compare to the rest of the class. 3. Having the PRS makes me think that my instructor cares what and how I learn

during lectures. Motivation:

4. Even though I had no PRS, I enjoy seeing the PRS in action. 5. Even though I had no PRS, I am more likely to attend this class because of the it. 6. I think that using the PRS helps me to get a better mark on midterm exams. 7. I think that using the PRS helps me get a better letter grade. 8. Knowing that in the calculus stream we will be using the textbook bundle for

three courses, I am willing to spend on extra $20 for the textbook bundled with the PRS.

Learning:

9. Using the PRS takes too much class time. 10. I retain more in lectures when I use the PRS slides. 11. I understand more in lectures when I use the PRS slides. 12. I think more about new concepts introduced in the lecture when I use the PRS

slides. 13. I am more likely to read my textbook in preparation of the lectures because of the

use of the PRS slides. 14. The PRS helps me to stay more focused in lectures. 15. Using the PRS makes the lecture seem rushed because the instructor has to hurry

to cover everything. 16. The PRS allows my instructor to focus on topics that are the most difficult for us. 17. I think that using the PRS is a waste of time. 18. Overall, I think using the PRS is very effective for my learning. 19. I wish my other lecture courses used the PRS.

Free Response Question:

15

20. What are the benefits/problems of using the PRS that are not addressed in the previous questions?

16

Appendix C

Student Survey Summer – PRS

The answer to each question is based on the Likert Scale: strongly agree agree not applicable disagree strongly disagree 4 3 2 1 0 Interaction:

1. I feel that I am able to participate more in the lecture when I use the PRS. 2. The PRS helps me to know how my responses compare to the rest of the class. 3. Having the PRS makes me think that my instructor cares what and how I learn

during lectures. Motivation:

4. I enjoy using the PRS. 5. I am more likely to attend this class because of the PRS. 6. I think that using the PRS helps me to get a better mark on midterm exams. 7. I think that using the PRS helps me get a better letter grade. 8. Knowing that in the calculus stream we will be using the textbook bundle for

three courses, I am willing to spend on extra $20 for the textbook bundled with the PRS.

Learning:

9. Using the PRS takes too much class time. 10. I retain more in lectures when I use the PRS. 11. I understand more in lectures when I use the PRS. 12. I think more about new concepts introduced in the lecture when I use the PRS. 13. I am more likely to read my textbook in preparation of the lectures because of the

use of the PRS. 14. The PRS helps me to stay more focused in lectures. 15. Using the PRS makes the lecture seem rushed because the instructor has to hurry

to cover everything. 16. The PRS allows my instructor to focus on topics that are the most difficult for us. 17. I think that using the PRS is a waste of time. 18. Overall, I think using the PRS is very effective for my learning. 19. I wish my other lecture courses used the PRS.

Free Response Question:

20. What are the benefits/problems of using the PRS that are not addressed in the previous questions?