DR LOU COMERFORD BOYES

STEM EVALUATION MANAGER AND RESEARCH FELLOW

UNIVERSITY OF BRADFORD

Developing Evaluation as Institutional Research: a

STEM case study

Evaluation Workshop University of Birmingham

Sept 2011

Plan for workshop

Brief overview of the University of Bradford NE Spoke Legacy project ‘Greening STEM’;

Evaluation activities attached to strands;

Relationship between project evaluation activity and institutional research at the UoB;

Practical activities, discussion and feedback.

Plan for workshop cont.

Practical activity 1:- Experiences to date of evaluating your STEM project?

What supports you in this? - What STEM project evaluation would you like to do?

What’s stopping you?• Practical activity 2:- Finding the links between your STEM project evaluation

and your organisation’s ways and means of knowing itself (i.e. institutional research):

- Where are there existing institutional mechanisms and practices that can help you?

- How will your project level good evaluative practice help inform what institutional research should be?

N.E Spoke Legacy Project: Greening STEM

• Institutional rationale: the project integrates two major themes relevant to the UK economy: STEM and sustainable development........

• Connections between STEM and SD are often weak and fragmented..........

• project aim is to pilot, trial and evaluate a range of curriculum learning materials across a range of STEM subjects............

• in order to build support, capacity and leadership for STEM and sustainable development.

Levels of resolution

• Institutional demonstrator for the ‘greening’ of STEM which comprises as 3-aspected test bed and involves three academic schools:

• A curriculum stream that will yield self-generated learning materials that combine STEM and SD;

• A skills stream that enhances existing employer-led initiatives at the University: the creation of a Greening STEM Skills Forum to help identify the scale and nature of employer requirements with regard to STEM and SD - ‘the green skills gap’ - and to create resources;

• An operations stream to highlight and stimulate opportunities to minimise the environmental footprint and waste associated with STEM activities in order to start addressing the obvious disconnect.

Project strands in brief:

Green chemistry: environmental monitoring in a laboratory context – the relationship between the different ways of communicating energy usage and subsequent student behaviour: a 7 fume cupboards experiment;

Automotive engineering: multidisciplinary, practical project located in a high impact, substantial core module – a ‘green vehicle’ build to launch a national racing competition;

Computing: the creation of web-based resources to encourage students to think about and address the relationship between computing and sustainable development; comparison of an directed and intensively delivered versus light but continuous exposure to materials.

Evaluation map

Mixing up what’s tailor made and existing………..

What do these approaches have to do with institutional research?

The need to show what’s different and what’s changed in terms of intervention;

You can’t reinvent the wheel – the evaluation toolkit is a finite collection of instruments;

What are the best tools to use? Look first at the tools and practices which already exist in your institution and how they can be adopted, adapted and tweaked to fit;

Those practices that don’t exist – but should because they worked for you: how do you get them into the system?

Activity 1

Practical activity 1:

- Experiences to date of evaluating your STEM project? What supports you in this?

- What STEM project evaluation would you like to do? What’s stopping you?

Practical activity 2:

• Practical activity 2:- Finding the links between your STEM project

evaluation and your organisation’s ways and means of knowing itself (i.e. institutional research):

- Where are there existing institutional mechanisms and practices that can help you?

- How will your project level good evaluative practice help inform what institutional research should be?

Recommendations for project evaluations:

To align oneself with established institutional data capture mechanisms that work, to aim to improve others;

To use existing data banks (be they quantitative or qualitative) to good effect – to consciously know what we ‘know’ and make it the starting point;

To be creative and imaginative in the way that we create new data capture events – its less onerous!

To re-consider the relationship of researcher and researched: to rethink who the stakeholders are;

To revisit the research approaches that are used – from monitoring KPI to appreciative inquiry.....