Group One Writeup

8/9/2019 Group One Writeup

1/31

Designing a New Course Evaluation System

For the Students, by the Students

Spring Term 2010

Group One

Michael Kiso, Aaron Bicknell, Alexsa Tucker,

Richard Bradley, Josh Counsil, Caroline Hilburn,

Matthew Nollar

Table of Contents


2/31

The Current System byJosh

Counsil...Page 3

Other University Course Evaluation Systems by Matthew

Nollar....Page 6

Instructor Interviews by Michael

Kiso.. Page 9

Methods by Caroline Hilburn

.....Page 12

The Logic behind the Madness: explaining why we made the

new system the way we did by Richard Bradley

...............................................................................Page 14

A Proposed New System byAaron

Bicknell....Page 18

Student Views of the Course Evaluation Prototype byAlexa

Tucker....Page 21

Next Steps by Caroline Hilburn.

....Page 25

hem out as a student, as well as receiving them from students of my own. The one thing th


3/31

The Current System Josh Counsil

How the System Works:

Currently, Oregon State University and many other schools around the

United States are using a paper Scantron system of teaching evaulations. At

the end of the term, teachers hand out evaluation sheets which they expect

students to fill out honestly, carefully and in full. They ask you questions

such as if you felt the class was good, and if the teacher was interested in

the students learning. It provides a very poor through exceptionalchoice system (basically a 1-5 system). The students then fill out these

sheets and individually place them in a folder while the teacher is not

present (so no tension is created between the teachers and the students).

The folder is then taken to its final destination by a student so the

information cannot be tampered with. These Scantron sheets are very

informative for teachers and administration. However, the students do not

feel that the system is effective for them. The current system could easily be

made much more cost efficient by providing it online. Also, it would be

beneficial to create a system that is accessible at the students digression

instead of wasting class time that is very valuable to a students learning.


4/31

Fig. 1. Data collected by Group One. This chart reflects the opinions of OSU students

surveyed spring term, 2010

On our random surveys we asked the question do you feel like your

feedback is being heard? Over 50% of the students reply was Sometimes

(Figure 1). In my eyes, this means that the students do not fully understand

how the system is operating and what its benefits and downfalls are. When

interviewing staff at Oregon State, we found that most of them agree that

the course evaluation system is very teacher orientated. This means they

agree that the Scantron sheets system is not providing adequate feedback

for students.


surveyed spring term, 2010.

Why the Students Feel the System is flawed

As you can see, students at OSU feel that the current evaluation

system is obviously flawed. Over 98% of the student population that were

randomly surveyed felt the system at Oregon State is Good or less (Figure

2). That leaves only 2% of the student population that felt the system was

above average. Obviously there are issues at hand. Many may argue that

this teacher evaluation system is for the teachers and administration, but

why can it not benefit the students as well? We as students are not trying to

build a hostile environment by exposing the results of the teacher evaluation

system to the student body, but exposing the positives and negatives of

each teacher and the classes they teach will help students plan their

courses. We have the right to see how other students felt about the teachers

and their classes considering we are the ones paying their salaries. We as

students have the right to see the class difficulty, teacher evaluation, last


5/31

terms class GPA, last terms syllabus, and hours of homework per week. This

is not because we are trying to expose bad teachers (even though I dont see

that as a bad thing) or pick that classes that are least difficult, but better yet

try to schedule classes accordingly. Personally, I will pick the classes with the

best teacher effectiveness because I am not the quickest learner. This could

greatly benefit many students at Oregon State and also greatly increase the

quality of their stay here by having understanding teachers and ones they

can relate to. If a teacher is getting terrible feedback upon the class then

obviously something is going wrong and we have every right to know about

it before we go setting ourselves up for failure because of a teacher who is

on a power trip or does not know how to teach. Just because a teacher has12 years of college under their belt does NOT mean they understand nor

know how to teach. Understanding the concepts is one thing, but explaining

them into words that other people can understand and comprehend is

something completely different. It is very obvious that our student body

cares about providing feedback for staff, faculty and administration, so why

are our voices of change not being heard? Many would argue that the

current system is already in place and is used almost universally by all 4-

year universities. This is true yes, but we could lead the way to bring course

evaluation online and help go green and protect the environment. We can

satisfy the needs and wants of staff and students, and create a system that

is more efficient, better quality, and easily accessible to anyone and

everyone who is paying the amounts of money that they are paying to try

and obtain a teacher and class of the highest quality possible.

We Want Change

From the random surveys we obtained from students around campus it

couldnt be more obvious that we as students are in desperate need of

change.


6/31



From the two charts above (Figures 2 and 3), we can see that 60% of

OSU students ALWAYS fill out the Scantron sheet while 30% sometimes do.

This means only 1 out of 10 students do NOT fill out the Scantron course

evaluation sheets. This means that 90% of the students are regularly filling

out the Scantrons for class and 75% of the students feel that the current

evaluation system is flawed and needs changed. This means we as students

are still filling out the evaluation sheets even though we dont agree with the

material in it, or just do not like it at all. We are the ones that pay the money

to make this school run. We are the ones that are taking the classes. We are

the ones that are here to decide our future. I figure since we are the ones

doing all these things, then we are the ones who get to decide what kind of

feedback on teachers and classes we require. We should be able to obtain

any statistics of any class at anytime.

Other University Course Evaluation Systems Matthew

Nollar

Taking a look at our system of teacher evaluations, we can come to

only one conclusion. Our current system serves the teachers and

departments that they are affiliated with. Currently our system is the

standard Scantron form with a questionnaire form that the students fill out.

In the near future, Oregon State University's system of teacher evaluations

will move to an online base. We as a group have decided to take a look at


7/31

other universities' system of teacher evaluations to see how their system

differed and find any similarities, if any.

We took a look at a large handful of universities. Our scope was

focused on universities located in the Northwest but not limited to it. The

reason behind taking a look at other universities in the surrounding area was

to see what system they were currently implementing for teacher

evaluations. If they had a different system then ours then we could pull ideas

from that system. The universities we looked at concerning the teacher

evaluations were the University of Oregon, University of Washington,

Western Oregon University, Portland State University, Arizona State

University, Boise State University, Linn-Benton Community College, ClarkCounty Community College, Portland Community College, Marylhurst

University, and Stanford University. These universities are highlighted in the

map below (Figure 4).

Figure 4. A map showing the location of Universities thatwe looked at. The map was constructed by Google Maps.


8/31

To our surprise, we found that the vast majority of the other

universities were using the same Scantron teacher evaluation that Oregon

State University was. The universities that were implementing the Scantron

system of teacher evaluations include University of Oregon, University of

Washington, Western Oregon University, Portland State University, Arizona

State University, Boise State University, Linn-Benton Community College,

Clark County Community College, Portland Community College, Marylhurst

University, and Stanford University. Only Western Oregon University uses a

different system. Their system of teacher evaluations was the same as the

Scantron system. However, the school had the students fill it out online. They

claimed that they were trying to move to a paperless system, much like whatOregon State University is going to be doing in the near future. Looking at

these systems, we found that the Scantron system was prevalent amongst

these universities. This is due to two reasons: one being that this is an

effective way for universities to conduct teaching evaluations, the second is

that it is a standardized system.

Seeing how all of these systems are very similar to Oregon States

University's system, we can safely say that their system serves the teachers.

All of the systems require that students fill out the evaluations. All of the

systems also disclose all of the information that is gathered by the student

filled out evaluations. To our groups surprise, we did not expect that all of

the universities that we looked at in the surrounding area had the same

system as Oregon State University. This led us to conclude that either the

current system is the best that it can be or the current best system that is in

existence out there is what all of the universities have implemented into

their teacher evaluations. On the other hand, having a standard

questionnaire and a comment form benefits the teachers because they can

adjust their teaching according to the results. This is why this Scantron

system is so prevalent.


9/31

Another reason why the system is the way it is in the area is due to the

fact that the staff of the departments organize it. The students take no part

in designing the survey. It is not even required of them to fill it out. It's all

voluntary. With that said, one can see why the staff of the departments

choose not to release the information. They are doing all of the work

organizing the teacher evaluations. The other reason why the universities

that we looked into choose not to release the information that is collected

from the evaluations is that the departments want to protect the teachers

reputation and/or pride.

After talking with some of the students who worked in the offices of the

other universities, some of the students showed interest in a teacherevaluation form where students had access to the results. They thought that

it would be a better system that served the students. Some also said that

they would prefer a system where the results are posted. They continued to

tell me that they would also register for their classes based on those results.

Instructor Interviews Michael Kiso

We interviewed a large number of people at the Oregon State Campus.

We tried to incorporate all the different groups in order to collect a

representative sample of our student body. Along with interviews with

students, interviews with instructors were also conducted. In total, we

interviewed two instructors and one graduate teaching assistant. All of them

teach/specialize in different subjects at Oregon State.

Heather Bremicker is a Graduate Teaching Assistant Instructor (GTA).

She specializes in writing, and has been instructing the WR 121 course at

OSU. We asked her a series of questions via email on May 17, 2010. We

decided to interview her based on the fact that she teaches in small-


10/31

classroom settings, a different instruction characteristic when compared to

our other interviewees.

We asked, What types of experiences have you had with the

Instructor Evaluation System? Were They Positive? Negative? To this, she

replied,

I have experienced the current evaluation forms on many different

levels- from administering them as office staff, to filling them out as

a student, as well as receiving them from students of my own. The one

thing that I can say about these forms is that they seem completely void

and disengaged from what the students true concerns really are. I can

say this much that the scoring system is concise- but I also do not think thatit entirely reflects the concerns that students may voice if given a

chance to do so.

Next we asked, If there was to be a new Instructor Evaluation System,

what types of information would be most helpful for you? She replied with,

Again, I think that the students who are given this as the only voice

they have, should be the ones to answer this question. In other words, the

kind of information needed, in my opinion, for a new Evaluation system,

should be one created by students- even though it is a tool that can be

used with my CV when applying for jobs- I would like the Evaluation to

reflect the genuine experience that my students had in my classroom.

Example: Was instructor able to make their subject available and

understandable to you? Did they try to reach you on a variety of different

levels with different mediums? etc....

Heather Bremicker does not see the current Instructor Evaluation

System as a benefit to the students, particularly in the sense of the students

using the Instructor Evaluations as a means of exercising their voice in the

school system. For her, the current system is a great means to build her

Cover Letter, which demonstrates how the Evaluations favor the Instructors,

not the students. She also believes that if a new Evaluation system were to


11/31

arise, that it should include information which students would benefit and

want. She made a comment that her system seemed a bit idealistic, but

achievable none-the-less.

Our second interview was conducted with Instructor Paul Paulson on

May 24, 2010 via face-to-face interactive interview. Paul Paulson is a

Computer Science instructor and has been teaching at Oregon State

University for a total of eight years. Not only is he an instructor of the

Computer Science courses, but is also a MECOP advisor for the engineering

department.

Instructor Paul Paulson believes that the current system in place atOregon State is as good as it gets, and sees it as effective. He believes

that the in-class student evaluations kind of force the students to participate

in the evaluations. He made note that if there was an online system put into

place, it would have a significantly lower number of participants when

compared to the in-class surveying method.

Instructor Paul Paulson believes it would be a positive change for the

students to post the results of the course evaluations, but it would create

resentment amongst the instructors. He believes that instructors teaching

the same course would give better grades to students and make the class

easier in order to get a higher course evaluation, and promote students to

take their course over another instructor. This creates a competitive attitude

amongst instructors and the faculty of Oregon State.

There is a section of the current Course sheets which gives the student

a chance to make comments about the instructor and/or course. Paul

Paulson says that most, if not all instructors take a look at these portions of

the evaluations, and use them to make changes to their course for the

following term. Most of the changes made by the instructor are not seen by

the students who previously took their class simply because they will not be

taking the course again. However, there is some content of courses that the


12/31

instructors are not able to change about a course. An example being the

types of information a course is required to cover.

Paul Paulson noted that if instructors receive bad scores for three

consecutive terms, that they are fired from the university faculty. This

applies to all instructors, except for those who have earned tenure.

Over-all, Instructor Paul Paulson sees the current system as a positive,

and emphasized that if the current system was changed, there would be

more cons than pros that would come from the situation.

Our next interviewee was Instructor Steve Cook. Steve Cook teaches

Geo Science courses, and strongly believes that there must be a change tothe current evaluation system at Oregon State University. We interviewed

Steve Cook on May 21st, 2010 via email.

First, we asked Steve Cook, Do you find the current system of course

evaluation useful/valuable? Why or why not? He replied with,

Not so much. I think that students don't believe that faculty pays any

attention to these forms. I've suggested on several occasions to faculty

that the results of student evaluations should be posted, but most faculty

disagree. I also know that when U of O did post evaluations, few

students perused the results. I find the responses to my qualitative

"Student Survey" to be of more value, as I can ask questions that are more

specific to the course and my presentation of it.

Next, we asked, What prompted you to publish the results of your

evaluations online? He replied with,

Perhaps because I get good student evaluations. I have no way of

knowing how many new students actually look at the results from the

previous term. I do try to put it on the screen at the first of the term and

make students aware that the info is there.


13/31

Last, we asked, What information about a course would you like

students to have before they register? He replied with, Basic outline,

grading scheme, format, and topics emphasized.

Instructor Steve Cook wanted to emphasize that OSU is indeed moving

towards an online based course evaluation system, and that contacts have

been made with a Portland company to set up the system. He also wanted to

note that he instigated the move towards a different course evaluation

system.

Methods Caroline Hilburn

Since the intention of this project was to generate a course evaluation

system that would serve the students at OSU, student input was the focal

point of the development project. Because of the diversity of student

experience and opinions we needed many responses. To this end, we

designed a survey intended to provide quantitative data. In order to increase

the response rate, all of the questions are closed ended with the option of

writing in other answers. The survey is one page long with six questions. Thesurvey is included at the end of the report as Supplemental Figure 1.

Survey Questions:

What resources do you use to decide what classes to take?

(circle all that apply)

This question was intended to determine where students

currently look to decide how to schedule their courses. Also, this

question was supposed to determine whether RateMyProfessors.com

should be considered a model for a new system of course evaluation or

as competition.

Do you feel that your feedback about instructors and classes is

valued at OSU?


14/31

The aim of this question was to determine how badly OSU needs

a new course evaluation system. The secondary aim of this question

was to motivate the university administration to accept a new system

of course evaluations.

Do you fill out the course evaluation Scantrons at the end of

each class?

One problem with the current course evaluation system is the

low response rate. This question was intended to determine what the

response rate is and what it is about the current system that convinces

people to fill out the course evaluation surveys.

What information would you like to have about a course beforeregistering? (circle all that apply)

Since the new system being designed is intended to give the

students all of the information they want, we asked students what

information would be useful. Ten individual pieces of information were

provided for students to choose from. While a write-in answer was

provided, no one chose that option indicating that the list is fairly

comprehensive.

In order to get this info, would you be willing at the end of

every course to (circle all that apply)

The purpose of this question was to determine how to administer

the new course evaluation system to maximize the response rate.

How would you rate the current course evaluation system used

at OSU?

The final question was written to echo the questions asked on

the current course evaluation Scantrons. The results were designed to

demonstrate the need for a redesigned evaluation system in a form

that the university administration would identify with.

A total of 63 responses were collected from current OSU students. Each

member of Group One collected surveys from students in different areas of


15/31

campus over the course of two weeks. By spreading out the process of

collecting surveys, we ensured that the responses were random, providing a

representative sample of the students here at OSU.

The Logic behind the Madness: explaining why we made the

new system the way we did. -- Richard Bradley

Based upon the data from our survey on the current student

evaluations system, it is safe to conclude that the current system is

insufficient to the needs of the students at Oregon State University and that

the focus of the new student evaluation survey should be on giving students

the information that is relevant to them.

We were able to assess that the quality of the current system was not

working well for students from how the respondents rated the quality of the

current system. According to our results, 71% of the students who responded

thought that our current system was of either fair or poor quality. Given the

data, it is clear that the current system is leaving a majority of students

unsatisfied and we are in need of modifying it or creating a new student

evaluation system that focuses on the needs of students.

We also found that although most students are unsatisfied with the

current system, 60% of them always fill out the evaluations, while 29% of the

respondents claimed that they filled out the current evaluations

sometimes, and the most common explanation for why they filled out the

current student evaluations was, because I was told to fill it out. This

allowed our group to reason that if students were filling out the form because

they were told to, then it is our responsibility to tell students to fill this

form out.


16/31

It was also implied that our audience was going to be very egocentric,

lazy, and that the most effective way to create a good system for students is

to tailor it for this type of audience. This altered our own aims because it

made us realize that we needed to make this new system as accessible as

possible and that we had to figure out ways of reminding people to fill out

this survey. It also suggested that we should make the new form as efficient

as possible, in that, we needed to put as much relevant information into the

new survey without making it long. We also found that no one filled in the

blank spaces on the survey where they were meant to write out an open

ended response. Rather than looking at this as a lack of data, we concluded

that this just meant that the new system would be mostly all bubbled in.


surveyed spring term, 2010

The survey also helped us conclude that the best way to disperse the

information collected from the new evaluations would be to use the

resources that are already most commonly used by students. According to

our results, the three most popular resources used to help students choose

classes are the coarse catalog, advisers, and friends. What we found most

interesting about our data was how the students found their friends just as

credible as advisers. In our findings, 56% of students claimed that they used

advisers and 53% claimed that they use their friends as credible sources, but

this difference is insignificant from a statistical standpoint because of oursmall sample size (Figure 5).

As a result, the data suggests that a student run system would be a

credible source for the students at Oregon State University. It was through

this data that we concluded that the course evaluations should be a quick


17/31

link on both Blackboard and the course catalog would be the most

convenient and efficient means to put it online and installing the new system

online would make it easy to remind students to do the evaluations.



From the survey we also found what information was relevant for

students and concluded that the most relevant information for the students

was: how difficult the course is, the effectiveness of the instructor, the

amount of time necessary for studying, what type of resources were most

helpful, and how expensive the books were (Figure 6). For our new survey we

asked them to rate the effectiveness of the instructor on a scale of 1-10; with

10 meaning that the instructor made the class clear and that they were

engaged with the class. We also wanted to rate the difficulty of the coarse on

a scale of 1-10 and 1 means that it was an easy class and 10 means that the

class was exceedingly difficult. For the new survey we also wanted to know

how many hours per week are spent doing homework or studying for the

course and figured that the most effective way to measure something like

this was to rate it on a scale of 0 10. We did these questions on a scale of

1-10 because we thought that it would be the clearest most effective way to

rate something as opinionated and varying as this. We also had people mark

what grade they thought they would receive in the class so that we are able

to gauge what type of student is mad or angry with the instructor.

In the new survey we also have a question about what resources they

found most valuable in this class. We decided that the easiest most effective

way to get these responses would be to list common resources used by

students to help them learn and then rate them on a scale of necessary,


18/31

useful, and not helpful. The resources that we decided to include were office

hours, teachers aid, MLC, Google, and nothing. We wanted students to

know if they would be able to get help when they needed it and this is also

an effective tool at gauging the best resource and learning method for the

class.

When we asked students what they used the textbook for the goal was

to phrase the answers they would give in a way that describes whether you

actually need it or not. The answers that we have listed are: what

textbook?, cramming, reading before the test, help with homework, and

reading for each lecture. Within these answers you can tell how often you

need to use the textbook and thus whether you really need to buy it. We also

have a question revolved around how much the price of the textbook is,

because it is useful to know if the book is necessary for a course and

whether or not it is in your price range.

In conclusion, we made the new student evaluation like we did so that

it would be as accessible, convenient, and efficient as possible. We were

able construct our student surveys at the convenience of students at Oregon

State University and we did it in a way that supports our two main goals of:

providing as much relevant information about classes while still having the

respondents complete the survey while using as little energy as possible.

The new course evaluation prototype is available through GoogleDocs

at this link: https://spreadsheets.google.com/viewform?

hl=en&formkey=dE0tVW8xN2xadzF3WmNtUWoxdUlHcGc6MQ#gid=0


19/31

Fig. 7. A Course Evaluation Prototype by WR214 Group One.

A Proposed New System Aaron Bicknell

Our proposed system for teacher evaluations will be a new feature for

the students and staff to both get aquatinted with, however; the benefits will

be remarkable. The proposed system will work similar to 3rd party websites

such as RateMyProfessors.com in the sense that students will be able to

look up their classes through feedback from everyone that has previously

taken that class (Figure 7). This system will incorporate the fact that both

students and staff will benefit from taking the time to actually complete the


20/31

evaluation system. One of the key problems with the current system in place

is that it doesnt benefit the students.

The evaluation forms are strictly for the professors/instructors criticism

and praise. Students showed through surveys that our team provided that

they should be entitled to at least a portion of this information, since they

are the people who take the course. Staff will no longer be the only people to

see the feedback that is given through the system,

which leaves students completely in the dark. The

information about the course, other than what is

provided in the student handbook or through other

sources such as advisors/peers/etc. will now be

available to the students as well. All of the information

gathered from the students will be available online

through a website we developed and tested. Students

may access this webpage and see highly valuable

information about courses that may possibly interest

them. The site will incorporate several key factors that

students look for in courses, such as homework

workload, cost, homework study time etc. All of this

information will be at the click of a button, taking the

pain and uncertainty of selecting a course.

The proposed system will take place online

through the blackboard system that all students and

staff at the university are already familiar with. Filling

out the evaluation will be similar to checking a course

grade or reading the syllabus. There will be a new tab

on the blackboard page labeled Evaluation (Figure

8). Students will click this link and within minutes they

will be filling out the evaluation form, and logging precious information into

. #. Mock-up of

e Blackboard

. 8. A Mock-up of

e Blackboard link.


21/31

the database for that course. This makes the process a very smooth

transition for everyone. The benefit of moving the evaluation system online

is that students and staff will save a considerable amount of time taking the

evaluations online because it can be done virtually anywhere that has

Internet access. Students will no longer have to fill out the Scantron system

in the classroom, which wastes precious time and is often considered a

waste of time, especially due to the fact that the current system of teacher

evaluation is given during "deadweek", where most students already feel the

pressure of upcoming finals. The last thing students want is the annoyance

of filling out a Scantron for the staffs benefit rather than prepare for the

upcoming final in that class by asking questions, or even possibly going overa final exam review. Eliminating downtime in classrooms is a great way to

keep both the staff and students on the task at hand, whether its an

approaching final, or other class related subject. The staff will no longer have

to leave the room while students fill out forms, and students will no longer be

responsible for the delivery of those forms to an office where they are to be

counted through an automated machine that really provides no feedback to

the people who need it: the students.

Not only will the evaluation be online through blackboard, but students

will also be able to view all of the evaluations online as well. Every evaluation

that has been filled out correctly through the blackboard link will be recorded

to an outside website that will take that information, and store it into a large

database system that students can use to navigate to find desired

information about any course offered at the university. Every course will

have a special lookup code that you can type into the database, and quicklylook up any and all information that has been previously recorded by other

students previously enrolled in that course. This system can also possibly

eliminate the waste of printing off thousands of course documents that the

university currently distributes to students and staff. Oregon State University


22/31

could add this astonishing contribution to an already outstanding green

aware campus that the university already loves to show off.

Making the evaluation from online through blackboard not only saves

time, but also money as well. The current system features generic Scantron

forms that obviously cost money to produce. If the average student takes 14-

16 credits for the fall, winter, and spring terms, at 3 credits per class, that

student is filling out anywhere between 3-5 evaluation forms per term. OSU

is proud to say that they enroll 21,996 students, making the math come out

to be anywhere between 65,998 to 109,980 Scantrons for the university to

record in just one term alone. Since the proposed system will be online,

literally tens of thousands of Scantrons will not be needed, and that money

will save the university a chunk of money every term.

While saving both time and money are huge benefits for the university,

another attractive feature of the proposed system is that it is paperless. In

todays society everyone is concerned with the state of the environment.

Going green is the new trend; saving as much paper, recycling as much as

possible etc. The proposed system will remove literally thousands of pieces

of paper a term. The university is already very proud of their efforts in going

green and the addition of anything that is beneficial to the environment

should be very welcomed here at the university. Many courses here already

have made the transition to paperless classes which is a great idea for

most classes that require any form of essay, or writing assignment. Changing

the system can be a great benefit to the environment, time management,

and even saving the university, the staff, and the students a little extra

money from printing UN-necessary documents and wasting ink.

The facts and benefits are clear that this system could be a great

addition to the campus. The time saving factors benefit not only the staff,

but the students as well. The University has already become a leader for the


23/31

green environmental movement, and quite frankly saving money is

something everyone will appreciate. Even though the system is still a

prototype, with a little time and hard work, the proposed system can be a

major advantage that Oregon State can hold over other Universities.

Student Views of the Course Evaluation Prototype Alexa

Tucker

Introduction:

In order to get further opinions and feedback on the new course

evaluation system that our group designed, we had several students take a

moment to test it out for themselves. I first showed them the display of

blackboard with the Course Evaluation tab. I told them that we chose

Blackboard since all students know how to use it and that it would provide

easy and convenient access to the Course Evaluation link. Each student then

filled out the online survey we created as if evaluating a course they are

currently taking. After completing it, I showed them the results page ofstatistics and percentages of previously completed surveys. Though these

results are hypothetical and dont accurately represent an actual course, I

explained to everyone that this is a preview of how our new proposed system

would work. I explained to them that this would be the kind of information

they would have access to when choosing classes and that their input would

contribute to these results. In total, five students were interviewed about

their experiences with the course evaluation system.

1. Ashley Milliron -- The response to this new Course Evaluation

system was very positive. She said that she would be more

willing to fill out the evaluation considering the fact that it would

be easy to access through blackboard and due to the simplicity


24/31

of the survey. It wouldnt be very time consuming and I would

rather be able to fill it out on my terms rather than in the middle

or at the end of class. Im more likely to skip filling out the in-

class evaluations if they are at the end of class because I just

figure everyone else is doing it and one less opinion from me

wouldnt matter much. Ashley stated. Like Ashley, this could be

the mindset of many other students who are asked to fill out in-

class evaluations. They may think that it doesnt matter if they

fill them out because there are plenty of other people already

doing it and itd be a waste of time since they dont even get to

see the evaluation results.

2. Kaylie Sundburg -- She said she liked the layout of the online

evaluation survey and that it was very easy to use. She also

told me that she would be more likely to fill out the evaluation if

it was posted on Blackboard and she could do it when she

wanted to. Kaylie said, I like that this system will also benefit

the students. Not only does it give results of what students are

saying about the instructor, but other aspects of the class like its

difficulty and how much the textbooks are used. She told me

that one of her favorite things about the information given from

this system was how much textbooks were used. Books are

expensive, and it sucks when you pay all of that money for it

because the syllabus says its required, but in the end, you rarely

even used it throughout the term. Most information needed

these days are posted on Blackboard through outlines and other

sources. This system could help students decide if they really

want to spend the money for a textbook based on how much

previous students actually used them throughout the course. It

can also help students decide if theyd rather share a book with


25/31

someone else and split the cost if the results say there is some

reading required.

3. Jake Goodman -- Jake told me that he always fills out the in-

class evaluations because he doesnt mind doing so, but he said

that he likes the idea of an online evaluation much better. Im a

big supporter of going green, so if we can save paper this way

and have it still be effective, Im all for it! Obviously having the

course evaluations online would cut costs on paper and printing

and would be more convenient since the internet is used for so

many things these days. I think results would get in faster andstudents would really like being able to actually see the results.

Jake said. He also stated that he feels that many other green

supporters would highly encourage this system and would be

more willing to fill it out if they knew they were going to be

cutting down on resources and money. Not to mention, the U.S.

is all about fast, easy, and convenient. Thats exactly what this

system seems to be. Though I think people are getting too lazy

these days with technology and the internet, thats just how it is

and people will be drawn to the simplicity of this system. Jake

stated.

4. Lauren Butterfield -- She agreed with other three students

that the system was simple and easy to use and that she would

be more likely to fill out an online evaluation. Lauren said, I

definitely like the idea but I think the results page needs some

work The bar graphs are a little confusing and could be

inaccurate because these are just showing what a certain

amount of people said. What happens when two hundred


26/31

students fill out this application? She said that she believes

simple numerical data would be better, something more straight

to the point than graphs. I dont like clutter, I just like things to

be straightforward. What you have now is along the right idea,

just needs some alterations. Lauren did however, like the

simplicity of the actual survey. Of course, our representation is

just an example of what our proposed system would be, but we

need this kind of feedback to decide what changes need to be

made or what needs to be added/taken away. If this system were

to be put in place, it should be tested several times by students

to make sure most everyone likes the end result.

5. Hallie Borgen -- One thing she noted on the current online

survey was that once you clicked one of the bubbles for your

answer, you cant undo it if you want to change it. This, again,

shows that there are changes and tune ups that need to be

made to the system but could be easily made when actually

putting the system to use. Hallie liked the overall easy-to-use

system but also agreed with Lauren that the bar graphs were

slightly confusing and numerical data would be better. Im also

for going green, so if we can save some trees by using this

system I think its a great idea. I asked Hallie if she was satisfied

with the information given about the course from the results. I

like everything thats given, but if I had to add another thing I

would want to know how much students enjoyed the course. I

know not all classes are going to be super fun, but if a certain

class is a total drag, I dont want to waste my time unless I know

I myself would be interested in it she stated. Other than that

Hallie was happy with the overall idea of our new system and

said she would prefer it over in-class-evaluations since she gets


27/31

to see the results and it can benefit her and other students.

Summary:

Due to the overall positive response to our system, I think that with

fine tuning and certain additions this will be a very effective system. All of

the students that tested it like the simplicity of it, supported the green

aspect of not having to use all of that paper for in-class-evaluations, and

liked that it benefited them as students. With the link to the evaluation being

on Blackboard, students would be more likely to fill it out since most

students use Blackboard regularly and they could also do it when they

wanted to. Overall, minimal negative responses were given, just suggestions

that could make the system better and more understandable. Feedback is

very important for this system because it will be the students who are using

it, therefore their opinion counts significantly in the process of completing

the overall design and function of our evaluation system. Without their

opinions, certain things may not be covered and the evaluation would

continue to have problems. Like stated before, every system proposed will

most likely have its pros and cons but our group has worked hard to create

one where the pros outweigh the cons.

Next Steps Caroline Hilburn

Before implementing any major changes to the OSU course evaluation

system, the proposed new system needs to be tested. This section outlines

the steps that still need to be taken to ensure that the new course evaluation

system is ready.


28/31

Collect student reactions to the new system.

Why? Students are most likely to take advantage of and

participate in a course evaluation system if they feel that it

benefits them.

How? Allow current OSU students to test the new system

and collect feedback. Students in different academic years,

majors and backgrounds need to be polled or interviewed

to get representative feedback about the system. Data

should be collected concerning how well the survey meets

students needs and the usability of the system.

Collect professor/instructor reactions to the new system. Why? Professor cooperation would legitimize and

streamline any new course evaluation system. If

professors understand and approve of the new system

they are more likely to participate, provide previous syllabi

and use the course evaluation system to advertise their

classes.

How?Allow professors and instructors at OSU access to

the prototype system. Interview the instructors to find out

what would make them more likely to participate in a

course evaluation system for the students. Data should

also be asked about the usability of the system.

Test the prototype evaluation system on one class.

Why? Each question of the completed survey needs to be

tested for both reliability and validity. Also, testing the

survey on a class of a known size will offer an approximate

response rate.

How? Provide one large lecture class with the opportunity

to test-drive a new course evaluation system. Use the data


29/31

collected to analyze response rate, reliability and validity

for each of the questions.

These three steps should be applied as part of an iterative cycle to refine the

final course evaluation system. Additionally, these considerations need to be

addressed before the new course evaluation system can be fully

implemented.

Tailoring the survey to the course.

Why? In order to ensure that all responses are relevant

to the course being evaluated, the survey answers need to

be designed specifically for that course. Specifically, theresources available for each course need to be listed.

How? Professors and Instructors should be able to select

which answers are relevant to their class from a master

list. A more generic, department specific template should

be available for courses in which the professor or instructor

does not respond.

Avoid legal and practical complications.

Why? Currently, the prototype system is powered by

GoogleDocs. Before testing the system, it would be wise to

read the fine print and determine whether or not

GoogleDocs can be used on a large-scale at a public

university. Also, whatever system is chosen, it is

imperative that it be able to handle the high traffic that

would occur on a university-wide scale.

How? Consult Google and the OSU legal office about the

legality of the proposed course evaluation system. OSU Net

and the OSU computer support staff should be consulted

about the amount of traffic the proposed system will be

able to handle.


30/31

Conclusion

After six weeks of research, our group has come to the following

conclusions:

The current course evaluation system used at OSU does not

serve the needs of the student body.

Most students and professors would be willing to participate in a

new system of course evaluations that were tailored to students

needs.

There are many available models and ideas to base a new

system off of.

The course evaluation system proposed in this report is just one idea.

More research needs to be done to refine the final product before it is ready

to apply on a university wide scale. However, the student body is ready for

change.

Supplemental Figure 1.

What resources do you use to decide what classes to take? (circle all that

apply)

Course catalogue Advisors Friends RateMyProfessors.com

None

Other __________________________________


31/31

Do you feel that your feedback about instructors and classes is valued at

OSU?

No Rarely Sometimes Often Yes

Do you fill out the course evaluation scantrons at the end of each class?

No, never Sometimes Yes, always NA Why?_________________________

What information would you like to have about a course before

registering? (circle all that apply)

Instructor effectiveness Class difficulty Previous term grade

average

Price of textbooks Are the textbooks used Average pages of reading per

week

Hours of homework per week TA effectiveness Ability to get extra help

Professor background Professor understandability Previous syllabus

Others ____________________________________________________

In order to get this info, would you be willing at the end of every course

to (circle all that apply)

Fill out an in-class evaluation Rate the course on blackboard

Rate the course on a student published web site Other

__________________________________

How would you rate the current course evaluation system used at OSU?

Poor Fair Good Very Good Exceptional

Documents

Group One Writeup