40
Preliminary findings on quantifying how engineering students use webcasts of class lectures Cheryl Schramm Alan Steele

Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Preliminary findings on quantifying how

engineering students use webcasts of class

lectures

Cheryl Schramm

Alan Steele

Page 2: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Introduction

• Goal: Provide systematic evidence to help

assess the use of video-casts as a teaching

tool in order to strengthen and improve

future academic experience of students

• Method: Formative Hybrid [Program]

Evaluation of providing webcast lectures in

engineering courses.

• Implementation Issues: Preliminary run on

ECOR 1606 Summer 2011

• Impact Issues: Ongoing work

Page 3: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Presentation Overview

1. The Context of the Engineering Webcast

Program

2. Literature Background

3. Program Evaluation: Webcasting ECOR 1606

4. Preliminary Data Analysis – Early Results

5. Research Questions

Page 4: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Organizational Context at Carleton

Faculty of Engineering & Design

Faculty of Science

Faculty of Arts & Social Science

Faculty of Public Affairs

Department of Mechanical & Aerospace Eng

Department of Electronics Alan Steele

Department of Civil Eng

Department of Systems & Computer Eng Cheryl Schramm

Interactive Media Services (IMS)

Education Development Centre (EDC) CUOL

Provost & Vice President Academic

Carleton Centre for Research on Engineering Education (CCEE)

Students of ECOR 1606

Associate VP (Teaching & Learning)

Page 5: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

CUOL – Carleton University on Line

“…initiative to provide alternative and flexible access to many Carleton University

credit courses. Now in its 33rd year of operation, more than 80 000 students from

the Ottawa area, and around the world have used CUOL distributed courses to learn

and complete a University degree program”

“CUOL courses are recorded on-campus during a regular class, and the lectures are

then made available to local and distance students … CUOL classes are as close

as possible to being in the classroom – you can listen to classroom discussions,

hear questions, and see the lectures just as they are delivered to fellow students.”

http://www2.carleton.ca/cuol/

Page 6: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

My Virtual Classroom

Using multiple teaching technologies

• Lecture Materials Powerpoint Slides

• Laptop Demonstrations (Code Development

and Debugging)

• Chalk board problem solving

• Overhead projectors (Card games, midterm

reviews)

• Classroom Discussions

Page 7: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Background - Engineering

Faculty of Engineering and Design

• Engineering: 4 departments

• Civil and Environmental Eng., Electronics,

Mechanical & Aerospace Eng., Systems & Computer

Eng.

• 14 programs, including: Mechanical Eng., Electrical

Eng., Engineering Physics, Software Engineering,

Sustainable and Renewable Engineering.

• Engineering programs are accredited.

• Contact hours are important.

Page 8: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Background – ECOR 1606

ECOR 1606 [0.5 credit]

Problem Solving and Computers

Introduction to engineering problem solving. Defining and

modeling problems, designing algorithmic solutions, converting

algorithms to C programs, testing, debugging. Program style,

documentation, reliability. Numeric methods: representation of

data, rounding and truncation errors, root finding, curve fitting.

Lectures three hours a week, laboratory three hours a week.

Page 9: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Background – ECOR 1606 SUMMER

Summer Offering is a Condensed Course

• 13 week course in 6.5 weeks (May/June)

• Schedule: Tuesday and Thursday:

Lectures: 9 AM – 12 noon

Labs: 12:30 PM – 3:30 PM

Department

Mech & Aero

Electronics

Systems

Civil

Course History

1st time

2nd time

>2nd time

Page 10: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Literature Background

“Only a limited amount of empirical research has

addressed the educational advantages and

disadvantages of webcasting and podcasting, although

one good example of such a study was performed by

Brotherton and Abowd (2004).”

(Traphagan et al, 2010)

Impact of class lecture webcasting on attendance and learning

Brotherton et al. (2004) “Lessons learned from eClass:”

Gump, S.E. (2005). The cost of cutting class:

Attendance as a predictor of student success.

von Konsky et al. (2009). Lecture attendance and web

based lecture technologies

Page 11: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Program Logic Model

Program Needs

Eng students need

flexibility,

access,alternative

learning

Failure & drop-out

rates in ECOR 1606

Intermediate Outcomes

Improved Program

Implementation

Increased and more

diverse student enrolment

Increased adoption for

other engineering courses

Immediate Outcomes

Increase student success

in ECOR 1606.

Reliable & Valid

Program Evidence

Long Term Outcomes

Increased student

success/satisfaction/

retention/graduation

Extend Carleton’s

reputation for innovation

Program Activities

Daily Lecture Capture

Daily Lecture Editing &

Posting

Daily physical attendance

check

Mid & Post Surveys

LMS Webcast Monitoring

Program Inputs

Instructors

Media Services –

Equipment &

Personnel (Camera

Operators, Editing

Technicians)

Learning

Management System

(LMS) (Password-

Encoded for Webcast

storage &usage

monitoring)

Financial support

Program Outputs

Course Events Calendar

LMS WebCast Usage

Statistics

Attendance Stats

Grades

Survey Stats

Email Correspondence

and/or Discussion Groups

for Lecture Questions

Library of Course

Materials

Page 12: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Implementation Evaluative Issues

A.1 To what extent is the online captured lectures

relevant to the needs of the engineering students?

In what ways do online captured lectures

supplement their course experience?

A.2 How, and how often, are the online captured

lectures used? When? At what times? What was

their experience with the use and delivery of the

online captured lecture?

A.3 To what extent will online captured lecture have

an impact on the experience of the professors?

Page 13: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Impact Evaluative Issues

B.1. Are a greater percentage of students performing

at passing level with access to webcasts?

B.2 To what extent has motivation, satisfaction and

educational engagement changed as a result of

having access to webcasts?

B.3: What are the unintended consequences of

webcasts ? Are there positive or negative

consequences? For whom?

Page 14: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Data Collection

1. Attendance: Self-reported

2. Moodle LMS: Password protected, usage

statistics

3. Surveys: Likert-scale questions on habits

and perceptions, hosted on getFAST.ca

(Canadian hosting)

4. Final Grades from student records

Page 15: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Data Analysis

Class size: 94 (103 on May 10th)

Participants: 56

Concerns:

1. Sample size

2. Summer Demographics

3. Unreliable Attendance Records

Page 16: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Data Analysis

1. Describe Accesses to the Webcast Lectures

When were the lectures accessed? ( Range of

purposes … what are their needs?)

Continuously over the term.

Sporadically (e.g. Occasional Absence or before

deliverables and exams)

Delta Time

How were the lectures accessed?

View entire lecture or skip through them

Repeated views of same lecture (identify difficult

subject)

Page 17: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Data Analysis

2. Describe Students Who Do/Not Use

• How many students used them? (A Measure of

usefulness, uptake)

• What are the attendance records of those

who did (not) use the webcasts?

• Can we describe two constructs: Physical

(F2F) and/or Virtual Attendance?

• What are the academic standings of those

who did (not) use the webcasts?

Page 18: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

ECOR 1606 – Summer 2011

Week Monday Tuesday Wednesday Thursday Friday

1 9 10

Lecture 1

Lab 1

11 12

Lecture 2

Lab 2

13

2 16 17

Assignment 1

Lecture 3

Lab 3

18 19

Lecture 4

Lab 4

20

3 23

Holiday

24

Assignment 2

Lecture 5

Lab 5

25 26

Lecture 6

Lab 6

27

May June

4 30 31

Assignment 3

Lecture 7

Lab 7

1 2

Lecture 8

Lab 8

3

5 6 7

Assignment 4

Midterm

Lab 9

8 9

Lecture 9

Lab Midterm

10

6 13 14

Assignment 5

Lecture 10

Lab 10

15 16

Lecture 11

Lab 11

17

7 20 21

Assignment 6

Review Session

Lab 12

22 23 24

June 26, Exam

Page 19: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

A measure of uptake – How often were

the lectures used?

Midterm

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10 11

Nu

mb

er

of

Stu

den

ts W

ho

...

Lecture

Attendance and Number of Viewers – Per Lecture

Attended

Viewed

Page 20: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Which lectures were used the most?

Midterm

Topics

1 Computer Basics

2 Programming Overview

3 Algorithms

4 Variables

5 Conditionals

6 Program Development

7 Loops

8 Functions

9 File I/O

10 Simulation

11 Array Sorting

0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8 9 10 11

Lecture

Viewers and Total Viewings Over Term - Per Lecture

Viewings

Viewers

Page 21: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

When are lectures watched?

0

1

2

3

4

5

6

7

8

9

9-M

ay

10-M

ay

11-M

ay

12-M

ay

13-M

ay

14-M

ay

15-M

ay

16-M

ay

17-M

ay

18-M

ay

19-M

ay

20-M

ay

21-M

ay

22-M

ay

23-M

ay

24-M

ay

25-M

ay

26-M

ay

27-M

ay

28-M

ay

29-M

ay

30-M

ay

31-M

ay

1-J

un

2-J

un

3-J

un

4-J

un

5-J

un

6-J

un

7-J

un

8-J

un

9-J

un

10-J

un

11-J

un

12-J

un

13-J

un

14-J

un

15-J

un

16-J

un

17-J

un

18-J

un

19-J

un

20-J

un

21-J

un

22-J

un

23-J

un

24-J

un

25-J

un

26-J

un

Daily Access of Selected Lectures (Num Viewers)

Lecture 2 (May 12) Lecture 4 (May19) Lecture 6 (May 26)

Lecture 7 (May 31) Lecture 8 (June 2)

Page 22: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

To what extent are attendance and

viewing related?

0

2

4

6

8

10

12

0 1 2 3 4 5 6 7 8 9 10

Num

ber

of

Lectu

res V

iew

ed

Number of Lectures Attended

Attendance & Viewing

Attend

Correlation=0.08

Page 23: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

To what extent do webcasts affect

attendance?

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11

Attendance = {Physical F2F, Virtual}

Attend Only

Attend and View

View Only

Absent

Lecture

Num

ber

of

Stu

dents

who …

Page 24: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

To what extent do webcasts affect

attendance?

Lecture

Num

ber

of

Stu

dents

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10 11

Composite Attendance

Absent

Virtual Attendance

Physical Attendance

Page 25: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

What kind of student uses webcasts?

Student Code

0

5

10

15

20

25

30

35

100 102 104 106 108 110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 147 149 151 154

Nu

mb

er

of

Which Students Viewed?

Viewed Lectures

Viewings

Grades (GPA)

Page 26: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

0

2

4

6

8

10

12

0 5 10 15 20 25 30

GP

A

Number of Views

Achievement vs Use of Webcasts

Lectures Viewed

Viewings

What kind of students use webcasts?

Correlation Lectures: 0.06

Correlation Viewings: 0.08

Page 27: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

0

2

4

6

8

10

12

0 2 4 6 8 10

GP

A

Number of Lectures Attended

Achievement vs Attendance

Physical Attendance

Composite Attendance

What kind of student attends?

Correlation Physical: 0.34

Correlation Composite: 0.22

Page 28: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Mid-Term Survey

1. Have other courses made video recordings of the lectures

available? If 'yes' please state name of the course.

2. Approximately how many lecture videos have you watched?

3. How do you normally watch the videos?

4. How best describes how you use the lecture videos?

5. How best describes why you watch the videos?

6. Do you plan on watching videos of future lectures?

7. Do you plan on re-watching videos you have already viewed?

8. Do you plan on using the videos in preparation for the final exam?

9. On a scale of 1 (lowest) to 5 (highest) please rate how useful you

find having the lecture videos available?

10. Have you missed an ECOR1606 class (not due to sickness)

because you knew it would be available on video?

Response Rate: 35 out of the 56 participants

Page 29: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Post-Term Survey

1. Did you find having the video recordings available useful?

2. Approximately how many lecture videos have you watched?

3. On what device did you normally watch the videos?

4. Check all the different devices you used to watch the videos.

5. How best describes how you used the lecture videos?

6. How best describes why you watched the videos?

7. At what point did you start watching the videos?

8. On a scale of 1 (lowest) to 5 (highest) please rate how useful

you found having the lecture videos available?

9. Did you miss an ECOR1606 class (not due to sickness)

because you knew it would be available on video?

10. How do you agree with the following statement? "All

lectures should be video recorded“

Response Rate: 14 out of the 56 participants

Page 30: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Mid-term Survey - Results

Mid-Term

Post

Page 31: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Mid-term Survey - Results

Mid-Term

Post

Page 32: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Mid-Term Survey – Plans

Page 33: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Post Survey

Page 34: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Comparing Mid and Post Surveys

Mid-Term

Post

Page 35: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Post-Survey - Opinions

Page 36: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Research Lessons Learned

• Ethics Procedure

• Data Collection

Automate Data Collection

Attendance: Self reporting does not work

Surveys: Difficult to get follow-through

• Consider established surveys on

engagement and study habits

Need continual promotion throughout term to

maintain initial interest

Page 37: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Research Questions

Measuring Academic Success

Grades and/or Engagement

Measuring Course Success

Different Learning Supports

Identifying Difficult Topics

Measuring Other Stakeholders

Instructors

Tackling Full Implementation

• Regular Term Offering (Not condensed)

• Longitudinal Offerings (Courses in later years)

Page 38: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

References

J.A. Brotherton and G.D. Abowd, “Lessons learned from eClass:

Assessing automated capture and access in the classroom,” ACM

Trans. Comput.-Hum. Interact., vol. 11, 2004, pp. 121-155.

Gump, S.E. (2005). The cost of cutting class: Attendance as a predictor

of student success. College Teaching, 53(1), 21-26.

Traphagan, Tomoko & John V. Kucsera, John V. & Kishi, Kyoko (2010).

Impact of class lecture webcasting on attendance and learning,

Education Technology Research Development (2010) 58:19–37.

von Konsky, B. R., Ivins, J. & Gribble, S. J. (2009). Lecture attendance

and web based lecture technologies: A comparison of student

perceptions and usage patterns. Australasian Journal of Educational

Technology, 25(4), 581-595.

http://www.ascilite.org.au/ajet/ajet25/vonkonsky.html

Page 39: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Bibliography

Barrington and Johnson (2006). The Relationship between Lab

Attendance and Academic Performance in a Computer Information

Systems Course. Information Systems Education Journal, 4 (99).

http://isedj.org/4/99/. ISSN: 1545-679X. (Also appears in The

Proceedings of ISECON 2005: §3573. ISSN: 1542-7382.).

Retrieved October 2011 from http://isedj.org/4/99

Kybartaite, A., Nousiainen, J., & Malmivuo, J. (2009). Evaluation of

Students’ Attitudes towards Virtual Learning Objects for Biomedical

Engineering. IEEE Multidisciplinary Engineering Education

Magazine, 4 (4), December 2009, 102-108.

http://www.ewh.ieee.org/soc/e/sac/meem/index.php/meem/article/vi

ewFile/88/65

Page 40: Preliminary findings on quantifying how engineering ...isotlsymposium.mtroyal.ca/2011Symposium/docs/SchrammISOTLBanff2011.pdf · Preliminary findings on quantifying how engineering

Bibliography

Scutter, Sheila & Stupans, Ieva & Sawyer, Tim & King, Sharron

(2010). How do students use podcasts to support learning?

Australasian Journal of Educational Technology 2010, 26(2), 180-

191

Sinclaire, Jollean K. & Simon, Judith C. & Campbell, Charles J. &

Brown, Judith C. (July 2011). Skills versus Concepts: Attendance

and Grades in Information Technology Courses. International

Journal of Education. 2011, Vol. 3, No. 2: E1.

Stodel, E. J., Thompson, T. L., & MacDonald, C. J. (2006). Learners’

perceptions on what is missing from online learning: Interpretations

through the community of inquiry framework. International Review

of Research in Open and Distance Learning, 7(3), 1-24. Retrieved

October 2011 from

http://www.irrodl.org/index.php/irrodl/article/view/325/743