UW-ACE Survey Results: Students’ Experiences of Using Technology in Learning

UW-ACE Survey UW-ACE Survey Results: Students’ Results: Students’

Experiences of Using Experiences of Using Technology in Learning Technology in Learning

Vivian Schoner and Katrina StrampelVivian Schoner and Katrina Strampel

Centre for Learning and Teaching through Technology (LT3)Centre for Learning and Teaching through Technology (LT3)University of WaterlooUniversity of Waterloo

The Setting: Teaching, Technology, The Setting: Teaching, Technology, and UW-ACE and UW-ACE

Student Encounters with the Blended Student Encounters with the Blended Learning ExperienceLearning Experience

Technically Viable? YesTechnically Viable? Yes Pedagogically Sound? Yes, No, Maybe, Pedagogically Sound? Yes, No, Maybe,

Depends… Depends… Survey Research ResultsSurvey Research Results

The UW-ACE SurveyThe UW-ACE Survey

We asked students to tell us about their We asked students to tell us about their experiences withexperiences with

Online Learning ActivitiesOnline Learning Activities Online Collaborative Learning (small group)Online Collaborative Learning (small group) Technology FunctionTechnology Function Student SatisfactionStudent Satisfaction Online Instructor FeedbackOnline Instructor Feedback

Students Completing SurveyStudents Completing Survey

FacultyFaculty Quest Enrolment Quest Enrolment % Survey % SurveyAHSAHS 15251525 66ArtsArts 30413041 66ENGENG 739 739 55ENVENV 14141414 77MATHMATH 31703170 44SCIENCESCIENCE 94749474 44

Comparisons of ScalesComparisons of Scales

0

10

20

30

40

50

60

70

80

90

Cum

ulat

ive

Posi

tive

%

Technology Function

Student Satisfaction

Online InstructorFeedback

Online LearningActivities

Online CollaborativeLearning

Technology Function: Access; Technology Function: Access; Ease of Use; NavigationEase of Use; Navigation

26. logging in to be… 26. logging in to be… 27. accessing my courses to be…27. accessing my courses to be…28. navigating around the UW-ACE course pages to be…28. navigating around the UW-ACE course pages to be…29. accessing the course tools/announcements to be… 29. accessing the course tools/announcements to be… 30. posting, reading, responding to messages in the online 30. posting, reading, responding to messages in the online

discussion forum to be…discussion forum to be…31. accessing feedback on submissions to be…31. accessing feedback on submissions to be…32. accessing my grades to be…32. accessing my grades to be…33. submitting assignments to be…33. submitting assignments to be…34. accessing quizzes to be… 34. accessing quizzes to be… 35. accessing team or small group resources to be… 35. accessing team or small group resources to be…

Scale ResultsScale Results

Technology (alpha .953)Technology (alpha .953)Faculty % Very

Difficult% Difficult

%Undecided

%Easy

%Very Easy

Cum. Positive%

SCIENCE 2 4 8 32 53 85FES 1 4 12 35 38 83ARTS 3 5 9 32 51 83AHS 1 5 12 26 56 82MATH 6 6 14 34 48 74ENG 6 11 16 32 35 67Av for %Combined

Neg 9 UnD 12 Pos 79

Technology: Accessibility and Ease Technology: Accessibility and Ease of Useof Use

““quick access to grades and assignment feedback, access to quick access to grades and assignment feedback, access to past lecture notes, announcements, additional resources”past lecture notes, announcements, additional resources”

““UW-ACE helped a lot in centralizing my courses online”UW-ACE helped a lot in centralizing my courses online”

““the fact that you can connect anytime/anywhere”the fact that you can connect anytime/anywhere”

““a safe online environment”a safe online environment”

““easy to use, helpful, good interactions with profs and peers”easy to use, helpful, good interactions with profs and peers”

““I could sign in and have everything at my fingertips”I could sign in and have everything at my fingertips”

Appreciation for Technical Potential Appreciation for Technical Potential

““it enhances the learning process by offering supplemental it enhances the learning process by offering supplemental resources for course related topics”resources for course related topics”

““it is certainly appropriate to post lecture notes and other it is certainly appropriate to post lecture notes and other resources, but the absolute most important thing to remember resources, but the absolute most important thing to remember is not to invent uses just because the UW-ACE software is is not to invent uses just because the UW-ACE software is there. Legitimate uses are great, but online activities just for there. Legitimate uses are great, but online activities just for the sake of doing them are frustrating”the sake of doing them are frustrating”

““it is desirable because it gives students an alternative way to it is desirable because it gives students an alternative way to find information about the course, also allows them to find information about the course, also allows them to communicate with others who need help and who can offer communicate with others who need help and who can offer help”help”

““if you have a question, but hate raising your hand and asking if you have a question, but hate raising your hand and asking in front of all your peers, it can be asked and answered in front of all your peers, it can be asked and answered online” online”

““I believe it’s an integral part of the university to have students I believe it’s an integral part of the university to have students using technology which will be useful once we are out there in using technology which will be useful once we are out there in the workforce. Developing this skill is essential for us to be the workforce. Developing this skill is essential for us to be able to learn to be technologically diverse”able to learn to be technologically diverse”

Overall Student Satisfaction: Overall Student Satisfaction: Course Organization; Instructions; Course Organization; Instructions;

Workload; Technical SupportWorkload; Technical Support

36. the organization of the course content and 36. the organization of the course content and activities?activities?

37. the instructions for completing learning 37. the instructions for completing learning activities?activities?

38. the number of course learning activities 38. the number of course learning activities through the term?through the term?

39. the distribution of course learning activities 39. the distribution of course learning activities through the term?through the term?

40. the technical support?40. the technical support?41. the professional look of the course web 41. the professional look of the course web

page(s) page(s)


Overall Student Satisfaction (alpha .916)Overall Student Satisfaction (alpha .916)Faculty Very

UnsatisfiedUnsatisfied Undecided Satisfied Very

SatisfiedCum Positive%

AHS 2 4 17 38 39 77

SCIENCE 3 4 16 42 35 77

ARTS 5 7 14 44 30 74

FES 3 6 17 4 30 74

ENG 3 8 22 35 32 67

MATH 5 9 24 427 15 62

Av for %Combined

Neg 10

UnD 18

Pos 72

Online Instructor Feedback: Online Instructor Feedback: Helpful; Informative; Appropriate; Helpful; Informative; Appropriate;

TimelyTimely

42. instructor feedback helped me learn and 42. instructor feedback helped me learn and understand difficult concepts understand difficult concepts

43. the frequency of instructor feedback met my 43. the frequency of instructor feedback met my information needsinformation needs

44. the quality of instructor feedback was helpful 44. the quality of instructor feedback was helpful and informativeand informative

45. the timeliness of instructor feedback was 45. the timeliness of instructor feedback was appropriate for my needs appropriate for my needs

46. the amount of instructor feedback was 46. the amount of instructor feedback was appropriate for my needsappropriate for my needs


Online Instructor Feedback (alpha .972)Online Instructor Feedback (alpha .972)

Faculty Strongly Disagree

Disagree Undecided Agree Strongly Agree

Cum. Positive %

ARTS 5 6 17 39 33 72

AHS 4 7 22 36 31 67

SCIENCE 5 10 21 26 38 64

FES 13 8 24 31 24 55

ENG 2 18 26 41 13 54

MATH 23 9 25 26 17 43

Av for %Combined

Neg 18 UnD 23 Pos 59

Online Instructor FeedbackOnline Instructor Feedback ““I got assignments back sooner and easier with the site”I got assignments back sooner and easier with the site”

““the professor always kept us informed of news by posting the professor always kept us informed of news by posting announcements frequently on the UW-ACE website”announcements frequently on the UW-ACE website”

““the immediate feedback from quizzes written as well as the the immediate feedback from quizzes written as well as the correct answers is a definite asset”correct answers is a definite asset”

““took too long for marks to be posted”took too long for marks to be posted”

““not being able to see answers on actual quizzes”not being able to see answers on actual quizzes”

““grades are not properly recorded”grades are not properly recorded”

““there was an announcement that marks for an important there was an announcement that marks for an important section would be posted which they never were”section would be posted which they never were”

Online Learning Activities: Learning Online Learning Activities: Learning Support Features, Concepts, Skills, Support Features, Concepts, Skills,

Applications; Integrated Parts; Applications; Integrated Parts; Study Time; Performance LevelStudy Time; Performance Level

1. Discussion forum activities1. Discussion forum activities2. Self-Assessment (un-graded)2. Self-Assessment (un-graded)3. Self-assessment (graded)3. Self-assessment (graded)4. Collaborative/Peer Review submissions4. Collaborative/Peer Review submissions5. Individual Submissions5. Individual Submissions6. Lecture notes6. Lecture notes7. Links to external resources7. Links to external resources8. Information literacy/library modules8. Information literacy/library modules9. Lab modules/activities9. Lab modules/activities10. helped me learn the new skills and concepts for this course10. helped me learn the new skills and concepts for this course11. helped in that it reduced the amount of time spent studying text or notes in order 11. helped in that it reduced the amount of time spent studying text or notes in order

to grasp the material to be learnedto grasp the material to be learned12. helped me to better analyze and apply key points in the material to be learned12. helped me to better analyze and apply key points in the material to be learned13. encouraged me to take responsibility for my own learning13. encouraged me to take responsibility for my own learning14. I found that working online improved my performance on graded course 14. I found that working online improved my performance on graded course

assessments assessments 15. I found that the online component(s) were well integrated with the other course 15. I found that the online component(s) were well integrated with the other course

components not offered online components not offered online


Online Learning Activities (alpha .965/.935/.919)Online Learning Activities (alpha .965/.935/.919)

Faculty Strongly Disagree


Cum. % Positive

ENG 6 5 27 36 26 62

FES 6 11 21 35 27 62

AHS 7 13 22 30 28 58

SCIENCE 11 11 24 33 21 57

ARTS 11 12 20 33 24 57

MATH 12 14 25 34 15 49

Av for % Combined


Online Learning ActivitiesOnline Learning Activities ““I’m sure there are situations where it would be desirable to “use I’m sure there are situations where it would be desirable to “use

technology to support teaching and learning”, but that in itself is too technology to support teaching and learning”, but that in itself is too broad a goal. You should be using technology to solve a particular broad a goal. You should be using technology to solve a particular problem, or to compensate for a recognized weakness in existing problem, or to compensate for a recognized weakness in existing methods. In the case of my course, I can’t see how UW-ACE has methods. In the case of my course, I can’t see how UW-ACE has added anything to the course. I found the written quizzes far more added anything to the course. I found the written quizzes far more helpful than the online ones, and I prefer to read printed course helpful than the online ones, and I prefer to read printed course notes rather than getting course material from the web (you can’t notes rather than getting course material from the web (you can’t highlight/underline/scribble on a webpage)”highlight/underline/scribble on a webpage)”

““it helped me develop the habit of studying for the course regularly. it helped me develop the habit of studying for the course regularly. It allowed me to learn course material through practice questions. A It allowed me to learn course material through practice questions. A chance to do the same quizzes again and again which helped me chance to do the same quizzes again and again which helped me learn and understand course concepts better. Unlimited time to do learn and understand course concepts better. Unlimited time to do the quiz”the quiz”

““a good way to practice concepts learned in class. Particularly I a good way to practice concepts learned in class. Particularly I found the ‘fill in the blank’ type exercises to be very helpful”found the ‘fill in the blank’ type exercises to be very helpful”

““the information was very complete and organized. I feel that the the information was very complete and organized. I feel that the textbook for the course would not have been sufficient on its own to textbook for the course would not have been sufficient on its own to learn all the course material. The resources offered on UW-ACE learn all the course material. The resources offered on UW-ACE enhanced my ability to learn the material”enhanced my ability to learn the material”

Online Collaborative Learning: Online Collaborative Learning: Learning Activities; Peer Learning Activities; Peer

Interaction; Reflective Learning; Interaction; Reflective Learning; CommunityCommunity

16. Team or small group graded assignments16. Team or small group graded assignments17. Whole class discussion activities17. Whole class discussion activities18. Peer review submissions18. Peer review submissions19. Team or small group un-graded assignments19. Team or small group un-graded assignments20. Giving/receiving peer feedback 20. Giving/receiving peer feedback 21. I was able to develop a better understanding of the material 21. I was able to develop a better understanding of the material

because of my interaction with my peersbecause of my interaction with my peers22. I put more thought into my comments when interacting with 22. I put more thought into my comments when interacting with

others onlineothers online23. I felt part of a learning community when working online23. I felt part of a learning community when working online24. I found that working online encouraged me to take 24. I found that working online encouraged me to take

responsibility for my group contributionresponsibility for my group contribution25. I am more comfortable asking questions to members of my 25. I am more comfortable asking questions to members of my

group in an online settinggroup in an online setting


Online Collaborative Learning (alpha .961/.964/.895)Online Collaborative Learning (alpha .961/.964/.895)Faculty Strongly

DisagreeDisagree Undecided Agree Strongly

AgreeCum. % Positive

ARTS 10 14 22 33 21 54

AHS 10 17 20 34 19 53

SCIENCE 12 7 28 29 24 53

FES 8 16 27 43 17 49

ENG 19 13 18 35 14 49

MATH 14 12 35 35 4 39

Av for % Combined


Online Collaborative Learning: Online Collaborative Learning: BenefitsBenefits

““the group discussion page made it a lot easier to communicate with the group discussion page made it a lot easier to communicate with my group members since we are all in different classes and my group members since we are all in different classes and communication for projects is an issue”communication for projects is an issue”

““I don’t mind the discussions online, I’m more comfortable since it I don’t mind the discussions online, I’m more comfortable since it gives the sense of anonymity (privacy thing)”gives the sense of anonymity (privacy thing)”

““It depends on what type of activities: I think group tutorials are a It depends on what type of activities: I think group tutorials are a good way to learn, whereas group projects are hard to realize”good way to learn, whereas group projects are hard to realize”

““they allow me to communicate with others, evaluate their thoughts they allow me to communicate with others, evaluate their thoughts and compare them to my own”and compare them to my own”

““the addition would benefit learning because it would provide a place the addition would benefit learning because it would provide a place for quick review and knowledge testing as well as the style of the for quick review and knowledge testing as well as the style of the lecturers questions”lecturers questions”

Construct Validity: Pearson Construct Validity: Pearson Correlation MatrixCorrelation Matrix

Correlations (Unadjusted)

1 .573** .604** .572** .483** .581** .526**. .000 .000 .000 .000 .000 .000

602 561 245 267 582 571 241.573** 1 .493** .626** .371** .568** .598**.000 . .000 .000 .000 .000 .000561 726 251 287 710 693 266

.604** .493** 1 .623** .493** .477** .507**

.000 .000 . .000 .000 .000 .000245 251 265 217 258 255 165

.572** .626** .623** 1 .460** .510** .556**

.000 .000 .000 . .000 .000 .000267 287 217 296 294 289 180

.483** .371** .493** .460** 1 .642** .521**

.000 .000 .000 .000 . .000 .000582 710 258 294 812 773 273

.581** .568** .477** .510** .642** 1 .661**

.000 .000 .000 .000 .000 . .000571 693 255 289 773 778 274

.526** .598** .507** .556** .521** .661** 1

.000 .000 .000 .000 .000 .000 .241 266 165 180 273 274 275

Pearson CorrelationSig. (2-tailed)NPearson CorrelationSig. (2-tailed)NPearson CorrelationSig. (2-tailed)NPearson CorrelationSig. (2-tailed)NPearson CorrelationSig. (2-tailed)NPearson CorrelationSig. (2-tailed)NPearson CorrelationSig. (2-tailed)N

S1OnlineEnvironment

S2OLEHelpedMe

S3CAExperience

S4CALearning

S5EaseOfUse

S6Satisfaction

S7FeedbackExperience

S1OnlineEnvironment

S2OLEHelped

MeS3

CAExperienceS4

CALearningS5EaseOf

UseS6

SatisfactionS7FeedbackExperience

Correlation is significant at the 0.01 level (2-tailed).**.

Identifying PatternsIdentifying PatternsScience:Science:Technology Technology (85)(85) and Student Satisfaction and Student Satisfaction (77)(77) 81% 81% Instructor Feedback Instructor Feedback (64)(64), Learning Activity , Learning Activity (57),(57), Collaborative Learning Collaborative Learning (53)(53) 57%57%

Applied Health Sciences:Applied Health Sciences:Technology Technology (82)(82) and Student Satisfaction and Student Satisfaction (77)(77) 80%80%Instructor Feedback Instructor Feedback (67)(67), Learning Activity , Learning Activity (58),(58), Collaborative Learning Collaborative Learning (53)(53) 59%59%

Environmental Studies:Environmental Studies:Technology Technology (83)(83) and Student Satisfaction and Student Satisfaction (74)(74) 79%79%Learning Activity Learning Activity (62),(62), Instructor Feedback Instructor Feedback (55)(55), Collaborative Learning , Collaborative Learning (49)(49) 55%55%

Arts:Arts:Instructor Feedback Instructor Feedback (72)(72), Technology , Technology (83),(83), Student Satisfaction Student Satisfaction (74)(74) 76%76%Learning Activity Learning Activity (57)(57) and Collaborative Learning and Collaborative Learning (54)(54) 56%56%

Math:Math:Technology Technology (74)(74) and Student Satisfaction and Student Satisfaction (62)(62) 68%68%Learning Activity Learning Activity (49),(49), Instructor Feedback Instructor Feedback (43)(43), Collaborative Learning , Collaborative Learning (39)(39) 44%44%

Engineering:Engineering:Technology Technology (67),(67), Student Satisfaction Student Satisfaction (67),(67), Learning Activity Learning Activity (62)(62) 65%65%Instructor Feedback Instructor Feedback (54)(54) and Collaborative Learning and Collaborative Learning (49)(49) 52%52%

All Faculties, All ScalesAll Faculties, All Scales

General PatternsGeneral Patterns Technology function and student satisfaction scales Technology function and student satisfaction scales

represent the more straightforward scales group represent the more straightforward scales group addressing technical support and student workload in all addressing technical support and student workload in all FacultiesFaculties

Online Learning Activity, Online Collaborative Learning Online Learning Activity, Online Collaborative Learning (Small Group) and Instructor Feedback scales represent (Small Group) and Instructor Feedback scales represent the teaching and learning pedagogy group in four of the the teaching and learning pedagogy group in four of the six Facultiessix Faculties

Exceptions Exceptions Instructor Feedback, a pedagogy scale, grouped with technical Instructor Feedback, a pedagogy scale, grouped with technical

support and student workloadsupport and student workload Learning Activity, a pedagogy scale, grouped with technical Learning Activity, a pedagogy scale, grouped with technical

support and student workloadsupport and student workload

Faculty ScalesFaculty ScalesAHS: All Categories

0102030405060

StronglyDisagree

Disagree Undecided Agree StronglyAgree

Ratings

Stud

ent R

espo

nses

Online Learning Activity

Collaborative Learning

Technology Function


Online Instructor Feedback

Arts All Categories

0102030405060

StronglyDisagree


Ratings

Stud

ent R

espo

nses

Online Learning Activity

Collaborative Learning

Technology Function


Online Instructor Feedback

Faculty ScalesFaculty ScalesEngineering All Categories

01020

304050

StronglyDisagree


Ratings

Stud

ent R

espo

nses

Online Learning ActivityCollaborative LearningTechnology FunctionStudent SatisfactionOnline Instructor Feedback

FES: All Categories

0102030405060

StronglyDisagree


Ratings

Stud

ent R

espo

nses Online Learning Activity

Collaborative LearningTechnology FunctionStudent SatisfactionOnline Instructor Feedback

Faculty ScalesFaculty ScalesMath: All Categories

01020

304050

StronglyDisagree


Ratings

Stud

ent R

espo

nses

Online Learning ActivityCollaborative LearningTechnology FunctionStudent SatisfactionOnline Instructor Feedback

Science: All Categories

0102030405060

StronglyDisagree


Ratings

Stud

ent R

espo

nses Online Learning Activity

Collaborative LearningTechnology FunctionStudent SatisfactionOnline Instructor Feedback

Survey Results: 865 Students; 113 Survey Results: 865 Students; 113 Courses; 6 Faculties Courses; 6 Faculties

UW-ACE All Faculties All Scales

05

10152025303540

StronglyDisagree


Ratings

Stud

ent R

espo

nses

by

Facu

lty

AHSARTSENGESMATHSCI

Percent Student Response Ratings Percent Student Response Ratings by Faculty by Faculty

Faculty StronglyDisagree


Total # Responses

Total #Course

Total #StudentsTaking Survey

AHS 5 9 18 33 35 1924 12 90

ARTS 7 9 16 36 32 4013 42 185

ENG 7 11 22 36 24 904 6 37

ES 6 9 29 36 29 1836 16 96

MATH 12 10 25 35 18 2628 9 120

SCI 7 7 20 32 34 6299 28 337

Av. % 7.33 9.16 20.17 34.67 28.67 17,604 113 865

% Rounded

7 9 20 35 29

Overview of Student CommentaryOverview of Student Commentary Students are aware of the benefits of technology in the Students are aware of the benefits of technology in the

classroom and commented frequently on instructors classroom and commented frequently on instructors under useunder use of this resource. of this resource.

They responded positively to technology use perceived They responded positively to technology use perceived to add real value to learning outcomes as evidenced in to add real value to learning outcomes as evidenced in improved performance on quizzes, tests, assignments or improved performance on quizzes, tests, assignments or projects.projects.

Students provided suggestions for improvements and/or Students provided suggestions for improvements and/or described new features they would like to see developed described new features they would like to see developed for UW-ACE. for UW-ACE.

They did not see value-added in working in small groups They did not see value-added in working in small groups or teams online when they could readily interact with or teams online when they could readily interact with their peers in person. their peers in person.

Recommendations for Online Recommendations for Online PedagogyPedagogy

We need to further explore ways to improve We need to further explore ways to improve instructional design and teaching techniques for instructional design and teaching techniques for online learning and collaborative learning online learning and collaborative learning activities. (LT3 Centre interface)activities. (LT3 Centre interface)

Some of the needed changes suggested by the Some of the needed changes suggested by the students should be considered for students should be considered for implementation either in the technology itself, implementation either in the technology itself, using the technology more fully to support using the technology more fully to support learning, and reducing the ambiguity learning, and reducing the ambiguity (undecided) noted in all the Scales except (undecided) noted in all the Scales except Technology Function.Technology Function.

Looking AheadLooking Ahead We need to think how to reduce the “undecided” We need to think how to reduce the “undecided”

rates, rates, Rethink providing this as a response choice Rethink providing this as a response choice Forced choice is one optionForced choice is one option Use “intermediate” instead of “undecided”Use “intermediate” instead of “undecided” Provide instructors with key information on the Provide instructors with key information on the

pedagogy of online teaching and learning pedagogy of online teaching and learning Encourage instructors to direct their students to Encourage instructors to direct their students to

the online survey available in students’ course the online survey available in students’ course pagespages

Analysis of the features of individual course Analysis of the features of individual course profiles should be undertaken (Liaisons)profiles should be undertaken (Liaisons)

THE Capstone StatementTHE Capstone Statement

Learning technologies should be used to Learning technologies should be used to support, enhance, supplement and/or support, enhance, supplement and/or expand learning opportunities; Technology expand learning opportunities; Technology should NEVER be used to replace should NEVER be used to replace teaching and learning in the classroom teaching and learning in the classroom setting. setting.

Documents

UW-ACE Survey Results: Students’ Experiences of Using Technology in Learning