Learning Set liaison:Felicity Rose, Pharmacy

Learning Set members:John Harris, BiosciencesSarah McMullen, BiosciencesRong Qu, Computer Science & ITAngus Davison, BiologyMartin Gering, BiologyRichard Roberts, Biomedical SciencesJim Maas, Biosciences

In Memory of our Learning Set Advisor, Dr Martin Willis

Evaluating the Learning Outcomes of the Undergraduate Project:

Perceptions vs. Reality.

Background

• The final year undergraduate research project (UGRP) is:– An opportunity to carry out an independent, intensive

piece of original work.

• The project may take different forms (Cowie, 2005, Hollingsworth, Mahon & Thomas, 2004), but the main criteria are the same:– Independent Working.– Problem Solving.– Critical Analysis (literature, data).– Apply research approaches and methods.– Communication Skills.

• QAA encourages inclusion of UGRP & is required by certain professional bodies (e.g. RPSGB).

Previous Work

• Ryder & Leach, 1999 (University of Leeds) – Research environment affects motivation of students.

• Murphy & Valenzuela, 2002 (Pharmacy USA), Ali & Seville, 2005; Sandhu & Seville, 2005 (Aston University)– UGRP was a valuable learning tool and should be kept as

part of the course.

• Orsmond et al, 2004 (Staffordshire University)– Pre-project perceptions differed from experience. – Students’ perception of the project’s learning outcomes

were rarely in line with their supervisors’ perceptions.

• Originality of our project:- Few studies in this area.- Application of background ideas to courses at Nottingham

that are revising the UGRP.

Background

• The UG research project makes up between 10 and 30% of the final degree mark:– Biology: 25%– Biomedical Sciences (BMedSci): 30%– Biosciences: 23%– Computing and IT: 20%– Pharmacy: 13%

• Forms of undergraduate research projects:– Laboratory or field based research projects.– Bioinformatics projects.– IT.– Survey/questionnaire.– Literature-based projects.– Clinical Audit.

Background

• Laboratory-based projects:– Original research carried out in lecturer’s lab space using

departmental equipment & funding.

• Literature-based projects:– Systematic analysis of research literature.

• In comparison to literature-based projects, lab-based projects:– Require greater supervision (indirect or direct).– Place a strain on research equipment and lab space.– In general, cost more.

• With increasing numbers of students we may have to:– Offer more literature based projects.– Offer more group projects.

PGCHE Project Aims

• Do literature-based projects yield training outcomes equivalent to those of laboratory-based projects?– What are the learning objectives of the UGRP?– Do staff and students feel that the lab-based and

literature-based projects we offer yield expected learning outcomes?

• Would staff and students welcome more group projects?

• Are staff and students satisfied with the way we currently assess the UGRP?

• What is the impact of the research project on future career choices?

Methods

To answer these questions:

• Consulted the module handbooks to find out whether learning objectives were defined.

• Gave an anonymous questionnaire (5 point Likert scheme) to students and staff at the end of the UGRP in 5 different Schools:

- Biology.- Biomedical Sciences.- Biosciences.- Pharmacy.- Computer Science and IT.

• Data input by MEADS Ltd and data analyzed by group.

• Use of questionnaire enabled large sample set to be surveyed anonymously, and consistently across Schools.

Methods

Methods

Results

• Total response rate: Students 456 (59%), Staff 150 (67%).

• Among the students– Laboratory Projects: 72%; Literature Projects: 27%, unknown: 1%– Literature based projects:

– Biology 22%– Biomedical Sciences 29%– Biosciences 37%– Pharmacy 12%– Computer Science and IT 33%

– Individual Projects: 74%, Group Projects: 24%, unknown: 2%– Group Projects:

– Biology 10%– Biomedical Sciences 35%– Biosciences 0%– Pharmacy 41%– Computer Science and IT 30%

Learning Objectives

• We consulted the module handbooks:– For example, in the School of Biology & Biomedical

Sciences:• Learning objectives are not defined explicitly.

– In the School of Pharmacy:• Read, consolidate and contextualize information from scientific

journals or reference material.• Contribute to the design and execution of data gathering-and-

analysis, based in laboratories or elsewhere.• Develop strong problem-solving skills.• Carry out critical analysis and evaluation of experimental data.• Develop high quality generic communication and time

management skills.• Competently describe the work in a poster format.• Present the research work in the form of an abstract for a scientific

meeting.• Use poster presentation format to maximize the visual impact of

presented information.• Present the research work in the form of a concise scientific

dissertation.

Learning Objectives

Are learning objectives clear even though they may not be defined explicitly?

0 10 20 30 40 50 60 70 80 90 100

Perceived Objectives of the UG Project

Literature searching, data analysis and interpretation

Working independently

Major objective (%)

Putting theory of lectures into practice

Laboratory techniques

Communication skills

Problem solving

Writing

Students

Staff

0% 20% 40% 60% 80% 100%

MajorObjective

MinorObjective

Not anObjective

Percentage Response

Lab. (n=327)

Lit. (n=124)

Staff (n=150)

Learning Objectives

The purpose of the UGRP is to learn / teach laboratory skills.

Group v Individual UGRPs

Do students prefer to work as part of a group or individually?

Group v Individual UGRPs

0% 20% 40% 60% 80% 100%

StronglyAgree

Agree

Neutral

Disagree

StronglyDisagree

Percentage Response

Lab. (n=327)

Lit. (n=124)

Staff (n=150)

I would prefer working as part of a group rather than individually.

0% 20% 40% 60% 80% 100%

StronglyAgree

Agree

Neutral

Disagree

StronglyDisagree

Percentage Response

Group v Individual UGRPs

I would prefer working as part of a group rather than individually.

BMedSci students

45 %

Assessment / Successful outcomes of the UGRP

Are literature based projects of value and should they be assessed in the same way?

Value of Literature-based UGRPs

0% 20% 40% 60% 80% 100%

Strongly Agree

Agree

Neutral

Disagree

StronglyDisagree

Percentage Responses

Lab. (n=327)

Lit. (n=124)

Staff (n=150)

Wholly literature based projects have limited value.

Open Comments

“…literature projects should not exist for science based degrees, nothing is achieved by just reading literature.” [Student – Biosciences]

0% 20% 40% 60% 80% 100%

StronglyAgree

Agree

Neutral

Disagree

StronglyDisagree

Percentage Responses

Lab. (n=327)

Lit. (n=124)

Staff (n=150)

Literature and laboratory based projects should be assessed in the same way.

Assessment of Literature-based UGRPs

Learning outcomes

Do staff and students feel that the learning outcomes are being met?

0 10 20 30 40 50 60 70 80 90 100

Literature review

Working independently

Agree (%)

Experimental design

Laboratory techniques

Communication skills

Data analysis

Writing skills

Students

Staff

Learning outcomes

The UGRP’s influence on future career

Do staff and students feel that the UGRP influences career choice?

0 10 20 30 40 50 60 70 80 90 100

Agree (%)

Students

Staff

Is a useful experience regardless of future career

Is a good introduction to working in research

Has no relevance to future career

Discourages students from a career in research

Influences students’ career choice

Influences students to consider a career in research

The UGRP’s influence on future career

Open Comments

“Discouraging students from a career in research can be beneficial if they realise that they are not well suited.”

[Staff – Biosciences]

“This [the discouragement] can be a good thing…better to realise this in the context of the UGRP rather than during a PhD.” [Staff – Biosciences]

The UGRP’s influence on future career

There were differences between vocational (Pharmacy) v non-vocational (Biosciences) degrees.

22%

94%

46%

68%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

… had no relevance to what Iwant to do in the future.

… was still as useful experienceregardless of whether or not itinfluenced my career choice.

Str

ongly

agre

e +

Agre

e R

esponses

Biosciences

Pharmacy

Open Comments

“…the UGRP is entirely pointless…it is completely irrelevant to the degree and the Pharmacy job.” [Student - MPharm]

Conclusions

• Learning objectives seem to be clear to staff and students even in Schools that do not explicitly state them.

• Most, but not all, of the learning objectives are achieved in literature and laboratory-based projects.

• Students and staff feel that literature and laboratory-based projects should not be assessed in the same way.

• In general, students and staff would not welcome more group projects.

• The project is valued in its current format by staff and students independent of future career choices.

Limitations

• Methodology adopted ensured large sample size and consistency across Schools but…– Some questions may have influenced responses.

– Questionnaire may have been difficult for staff to answer if they had supervised both literature and laboratory based UGRP.

– Would have been useful to have surveyed students before the start of the UGRP to see if responses were influenced by the experience.

– Students may experience questionnaire fatigue (with the use of SEMs and SETs).

Recommendations

• All Schools must ensure that learning objectives are written in the module profile.

• Differences in skills acquired between literature and laboratory-based projects should be highlighted to students.

• The assessment criteria must be appropriate for the differential outcomes of the two types of project.

• If implementing group project work, Schools need to investigate methods to ensure that they are effective.

• The research projects are valued by staff and students and should therefore continue to be an important part of the curriculum with a high weighting of marks.

Open Comments

“The work I had to do in my UGRP has improved my overall ability to work in all areas” [Student – Biosciences].

“The UGRP should be standardised…there is no point in giving us invaluable (useless?) skills and torturing us…………..we are paying for our degree and surely as a consumer we want value for money” [Student - MPharm].

Acknowledgements

• Martin Willis, Learning Set Advisor.

• Ken Levine, Survey Unit.

• Staff and students for completing the questionnaires.

• MEADS Ltd.

• University of Nottingham Learning and Teaching Development Fund.

References• Ali J.A. and Seville P.C. (2005) Usefulness of final year MPharm projects:

supervisors’ opinions. J. Pharm. Pharmacol. 2005; 57 (Sept sppl.), S46.

• Cowie R. (2005) Practice within UK institutions. Presentation at LTSN Staff Development Event ‘Making the most of final year projects’ Durham, 8/2/05. Report available on line at ftp://www.bioscience.heacademy.ac.uk/events/dur05/cowie.pdf

• Hollingsworth M., Mahon M. and Thomas L. (2004) Web projects for Life Science students. Bioscience Education E-journal 4 paper 5. Available on line at http://www.bioscience.heacademy.ac.uk/journal/vol4/beej-4-5.htm.

• Mills P. (2003) Group Project Work with Undergraduate Veterinary Science Students. Assessment & Evaluation in Higher Education, Vol. 28, No. 5.

• Orsmond P., Merry S. and Reiling K. (2004) Undergraduate project work: can directed tutor support enhance skills development? Assessment & Evaluation in Higher Education Vol. 29, No. 5.

• Sandhu D. and Seville P.C. (2005) Usefulness of final year MPharm projects: students’ opinions. J. Pharm. Pharmacol. 2005; 57 (September supplement), S45-S46.