1

Evaluating the Effectiveness of Problem Based Learning

John Sedgwick and Guglielmo VolpeDepartment of Economics, Finance and International Business

London Metropolitan Universityj.sedgwick@londonmet.ac.uk, g.volpe@londonmet.ac.uk

DEE Conference – Cambridge 6-7 September 2007

Research funded and supported by the Economics Network of the Higher Education Academy www.economicsnetwork.ac.uk

2

Aims of Project

Evaluating Problem Based Learning: the literature

Structure of PBL Project

Evaluation of Experience Evaluation of Performance Students’ Perception Staff Evaluation

Project’s Next Steps

Structure of Talk

3

Evaluating PBL: the literature

Evaluation of Outcomes

Knowledge: mixed results or insignificant differences between PBL and other methods (Albanese and Mitchell, 1993); greater knowledge with PBL (Smits, 2002), (Farrell, 2003)

Lifelong Learning Skills: PBL superior to conventional methods (Bransford et al., 1989)

Learning Process: deeper approaches to learning ( Sobral, 1995)

Team Skills: PBL facilitates development of collaborative skills (Cockrell et. al., 2002)

Control Group Analysis: in some contexts PBL may lead to worse outcomes for some students (Newman, 2004)

4

Structure of Project Industrial Economics Module

Spring Semester Module Taught at North and City Campus to two separate cohorts of

students PBL at North Campus while Traditional Approach at City

Campus Different lecturers in the two campuses Same assessment but slightly broader syllabus at City

campus Control for students’ background and characteristics

Evaluate and compare students experience and performance

Control Group Analysis

5

Methodology: evaluation framework (Newman, 2004)

Control Group Analysis

Objective Measure

Students participation Class registers

Class participation

Logs of meetings

Students satisfaction Questionnaires at end of semester

Focus group

Module evaluation form

Students outcomes and students performance

Assessment performance

End of semester questionnaires

Statistical analysis of performance

Follow up to test knowledge retention

6

Students’ Performance

Average Median F D C B A

Overall 57.9 59.5 15.0% 5.0% 30.0% 40.0% 10.0% City

55.7 58.0 7.9% 13.2% 39.5% 23.7% 15.8% North

Exam 57.3 63.5 12.5% 12.5% 16.7% 33.3% 25.0% City

54.6 56.5 10.5% 18.4% 26.3% 28.9% 15.8% North

Coursework 53.9 51.0 15.0% 25.0% 15.0% 25.0% 20.0% City

56.0 56.5 10.0% 5.0% 42.5% 20.0% 22.5% North

7

Students’ Performance

  Coefficients t Stat P-value

Intercept 55.90439 17.11719 2.38E-17

Difference in Year 2&3 average 0.900969 5.500858 5.11E-06

PBL in Semester A 7.079753 2.183598 0.036683

Mature student -5.45856 -1.68865 0.101323

International student -4.16997 -1.4207 0.165385

Disability -8.72376 -1.52794 0.136667

Semester B starter 4.960657 1.216063 0.23314

Regressor: Overall performance in module by North Campus studentsOLS method; R2=0.650, Adjusted R2 = 0.582; Observations: 38

Does experience in PBL lead to a better performance?

8

Students’ Performance

  Coefficients t Stat P-value

Intercept 60.76463 27.27263 1.78E-32

Difference in Year 2&3 average 0.947212 7.512146 7.56E-10

PBL at North Campus 0.598059 0.223027 0.824388

Mature students -8.10651 -3.36355 0.001451

International students -4.69812 -1.75774 0.084677

Semester B starters 1.736257 0.457838 0.648976

Regressor: Overall performance in module by all studentsOLS method; R2=0.576, Adjusted R2 = 0.535; Observations: 58

Is PBL correlated with a better performance in the module?

9

Students’ Evaluation

North campus (PBL) City Campus

Overall Score 75% 8% 68% 10%

1. Lecturer well organised 97% 0% 83% 0%

2. Timely assessment information 97% 0% 92% 8%

3. Clear presentation 83% 0% 69% 15%

4. Appropriate level of difficulty 68% 6% 77% 8%

5. Lecturer encourages questions 90% 0% 85% 0%

6. Clear syllabus and assessment 77% 0% 54% 8%

7. Right number of topics 84% 0% 62% 8%

8. Module recommended 70% 7% 54% 15%

9. Tutorial complements lecture 77% 0% 77% 15%

Students evaluation at end of semester

10

Students Evaluation – end of semester questionnaire

City Campus Students (15) North Campus Students (33)

SA & A SD & D S A & A S D & D

I would have liked to have experienced PBL (traditional method) in this module 53% 33% 33% 30%

I prefer the ‘traditional’ lecture/seminar (PBL) approach to any other method 53% 20% 42% 18%

A PBL (traditional approach) approach would be appropriate for this module 53% 33% 52% 24%

Final year students should be required to learn more independently (less lectures/seminars) 43% 43% 64% 24%

I expect to learn more than the North (City) campus students 33% 13% 33% 6%

I will be able to retain more knowledge of IE than the NC (CC) students in the future 20% 13% 47% 6%

All students should follow the same teaching method 67% 13% 70% 9%

A PBL approach is more appropriate for final year modules 53% 20% 70% 12%

How much/well you learn does not depend on the teaching method 47% 47% 23% 47%

11

Students’ evaluation

Students’ views from questionnaires and focus group (PBL students) “You are understanding while learning, as usually in lectures you do not

retain the information”

“I hear and I forget, I see and I remember, I do and I understand”

“I believe with PBL students are forced to independently attain a wider knowledge because they need to research more and by so doing we come across more things than we would get in class teaching”

“PBL is good as it relates to real world issues and that tends to stay in my head. Group learning is also important as it is easier to understand a discussion than a lecture”

“PBL is not necessarily better. The tutor who is delivering the module and his ability to teach is important!”

“PBL is not good at all especially for final year students who need much more support at that time”

12

Staff Evaluation

“For me, these results are much stronger than I was getting in the module a few years ago, so I feel very encouraged”

“The experience was such a good one in terms of classroom buzz”

“I enjoyed the experience and will repeat it next year”

“The PBL experience induced me to introduce the approach in other modules”

13

Knowledge Retention

Evaluation of post exam knowledge retention test (mark out of 5)

Module’s mark

City Tutor North Tutor Average

City Student 69 2 2 2

North Student 1 67 5 4.6 4.8

North Student 2 67 2.5 3.3 2.9

North Student 3 58 2.5 3 2.8

•Both City and North Campus students were contacted after completion of moduleThey were asked to answer three questions without consulting any notesAttempts were marked blindly by both City and North campus tutors

14

Overall Evaluation

Green Yellow Red Positive correlation

between PBL and performance

Students’ appreciation of approach

Students’ perception of deeper / independent / dynamic learning

Students’ desire to be involved with approach

Students’ perception that approach pushes them more than other approaches

Staff satisfaction and appreciation of its value

Students’ development of transferable skills:

Working with othersSelf-directed learning

Knowledge retention and deeper understanding

Differences in CW and exam performances but not statistically significant

PBL students seems to retain more knowledge

Statistical correlation between performance and PBL not significant

Lack of prior experience in PBL can hinder performance

Difficulty in handling a large class

15

Steps to complete research…

Still two steps to complete research:

Full analysis of focus group with identification of key themes

Further evaluation of questionnaires by distinguishing responses by gender and age

16

Overall Evaluation

The PBL students perform better in the coursework but less well in the exam compared to the non-PBL students

Overall the non-PBL students have performed better but the difference in performance is not statistically significant

Students who experience PBL in the first semester perform better than those without such an experience

PBL is a positive but not significant determinant of performance across the two campuses

The majority of students appreciate the value of the PBL approach in particular for final year students

About half of the City students would have liked to experience PBL in the module

Students express some concerns about their ability to engage with PBL without having any prior experience

Staff enjoy the approach and the ability to be much closer to the students’ learning process