Slicing the evaluation cake JISC Content programme mtg - 14-15 Nov 2011

Paola Marchionni, JISC Programme Manager p.marchionni@jisc.ac.uk

Before we start baking, we need to think of what we want to make, the ingredients we need and how we’re going to mix them.

So, what do we want to make?

What do we want to achieve? Why? What are the (evaluation and impact) questions we want to ask, and be able

to answer?

Need to consider and align:

JISC impact areasContent programme aims, objectives

and key issues Projects and institutions aims, objectives

and priorities

Think of slicing a layered cake…

JISC funders

JISC 5 impact areas

Content programme

Projects and institutions

JISC layer: 5 impact areas:

- Research excellence- Learning and teaching excellence

- Increasing effectiveness/saving money- Positive impact on society

- Helping with future technology needs

Content programme layer: aims, objectives and key issues

- Notion of critical mass- User engagement/responding to users needs (incl embedding in research and

teaching)- Partnerships and collaborations - benefit of- Innovation in any aspect of project life cycle

- Sustainability and business models- Resource discovery through multiple channels

- Skills development- QA: quality, technical standards and best practice (eg IPR and licencing...)

Projects layer:Evaluation and Impact questions

Evaluation (formative and summative): during and at the end of the project, more about how we have done things, processes, workflows, skills… and the quality of outputs created, their fitness for purpose

Impact (short – medium – long term): more about the kind of “change” or benefit that a project has brought about. Answers to the evaluation questions above will

feed into the measurement of impact.

Some examples of evaluation questions(projects in different strands may have different emphasis)

- Have projects created/clustered content? - Does this follow best practice for creating, delivering and sustaining digital resources?

- Is content fit for purpose, ie for use/re-use in teaching and research (eg licencing, formats,

usability…) - Can content be discovered through multiple

channels?- Have partnerships worked effectively?

- Have new skills been developed?

Some examples of impact questions, or the 5 cake slices

(mapped to JISC impact areas, Content programme and projects/inst concerns)

Slice 1: Learning and teaching excellence

- How have the new resources been embedded in teaching and learning (prog key issue)?

- Have users (prog key issue) (eg students) and other relevant stakeholders been involved in the process of creation?

- What change or benefit has this brought to learning and teaching?

- How does this align with institutional priorities (prog key issue and inst issue) in teaching and learning?

Slice 2:Research excellence

- Is the digitised content suitable for researchers needs (prog key issue) (eg quality, detail, reliability, critical mass, tools...?) - Has it opened up new research opportunities?

- Are there any specific activities or plans to embed content (prog key issue) in research activity?

- What does this content allow researchers to do that they couldn’t have done before?

- What change has it set in motion?- How does this align with institutional priorities (prog key issue)

in research?

Slice 3: Increasing effectiveness/saving money

- Has there been fruitful collaboration (prog key issue) within different departments and other external partners?

- What have been the benefits/dis-benefits (?) of this collaboration? - Is collaboration and partnership an effective process (in terms of

time and resources) for producing and sharing resources (use, re-use)?

- Has the project added value/critical mass (prog key issue) to existing repositories?

- Are innovative sustainability and business models (prog key issue and inst) contributing to increasing effectiveness of the resource?

Slice 4: Positive impact on society

- Is content accessible (prog key issue) beyond the formal education sector and usable by a variety of

users (prog key issue)the public at large? - Are there any innovative partnerships (prog key

issue, also inst) that help content reach wider audiences?

- Have different delivery channels (prog key issue) been used to provide access to these resources?

Slice 5: Helping with future technology needs

- Have any innovative technologies (prog key issue) been used to deliver the project?

- Do these have wider application within the sector?- Do they have the potential to benefit other

institutions?

In summary• Be aware of the bigger picture • Identify how your project is contributing to it• Chose your “slice/s” of cake: in what area is your

project going to make more impact? Project are likely to make impact in more than one area

• Formulate key evaluation and impact questions (at least short term impact)

• Identify your baseline so that you can document change more easily

• Plan for medium-long term impact and what you have to put in place in order to be able to measure that

• Look at JISC guidance for how to tell your impact stories

Guidance and tools

JISC Evaluation guidance - http://bit.ly/tkE5F1 Brief and concise guidance on formative and summative evaluation

JISC Impact guidelines - http://bit.ly/rzQ9lS Background to the JISC 5 impact areas and pragmatic approach to gathering

evidence and measuring impact during and after a project

Toolkit for the Impacts of Scholarly Digitised Resources - http://microsites.oii.ox.ac.uk/tidsr/

Very useful, includes qualitative and quantitative methodologies to measure online impact of resources and case studies. Useful also for planning measurement of long term impact.

Open Educational Resources: The value of reuse in higher education - http://bit.ly/rvAxwP Useful not just for OER but digital resources in general.

Image credits• Cover slide: Front display window of G. E. Adams cake shop, Brisbane

Arcade, ca. 1938 No known copyright restrictions - http://bit.ly/vlo0mL • Slide 3: Cake and mold pans, Cornell University Library, No known

copyright restrictions http://flic.kr/p/6SJBQR • Slide 6: Photograph of Newlyweds Ronald Reagan and Nancy Reagan

cutting their wedding cake, 03/04/1952 - 03/04/1952, Ronald Reagan Library, US National Archives, No known copyright restrictions - http://flic.kr/p/9nJ1Np

• Slide 11 and subsequent, Slice o cake, thepinkpeppercorn CC-BY http://flic.kr/p/6mJyiU