Consider the Evidence

www.minedu.govt.nz

© New Zealand Ministry of Education 2009 - copying restricted to use by New Zealand education sector.Page 1

Consider the EvidenceEvidence-driven decision making

for secondary schools

A resource to assist schools to review their use of data and other evidence

7

Changing and Evaluating

www.minedu.govt.nz


Evidence-driven decision making

This module is part of a resource about how we use data and other evidence to improve teaching, learning and student achievementToday we are looking at the final stage of this process – changing our practice and evaluating the impact of that change

www.minedu.govt.nz


The evidence-driven decision making cycle

ExploreCheck data and evidence to explore the issue Question

Clarify the issue and ask a question

Assemble Decide what data and evidence might be useful

Analyse data and evidence

TriggerData indicate a possible issue that could impact on student achievement

Evaluate the impact on the intervention

IntervenePlan an action aimed at improving student achievement

Interpret Insights that answer your question

SpeculateA teacher has a hunch about a problem or a possible action

ActCarry out the intervention

Reflecton what has been learned, how practice will change

www.minedu.govt.nz


Changing and Evaluating

Trigger Clues found in data, hunchesExplore Is there really an issue?Question What do you want to know?Assemble Get all useful evidence togetherAnalyse Process data and other evidenceInterpret What information do you have?Intervene Design and carry out actionEvaluate What was the impact?Reflect What will we change?

www.minedu.govt.nz



Trigger Clues found in data, hunchesExplore Is there really an issue?Question What do you want to know?Assemble Get all useful evidence togetherAnalyse Process data and other evidenceInterpret What information do you have?

> Intervene Design and carry out actionEvaluate What was the impact?Reflect What will we change?

www.minedu.govt.nz


Professionals making decisions

How do we decide what action to take as result of the information we get from the analysis?We use our professional judgment

www.minedu.govt.nz


Professional decision making

We have evidence-based information that we see as reliable and validWhat do we do about it?If the information indicates a need for action, we use our collective experience to make a professional decision

www.minedu.govt.nz



Have my students not achieved a particular history standard because they have poor formal writing skills, rather than poor history knowledge?The answer was Yes ... so I need to think about how to improve their writing skills. How will I do that?

www.minedu.govt.nz



Do any particular groups of year 11 students attend less regularly than average for the whole cohort? The analysis identified two groups – so I need to think about how to deal with irregular attendance for each group. How will I do that?

www.minedu.govt.nz



You asked what factors are related to poor student performance in formal writing. The analysis suggested that poor homework habits have a significant impact on student writing. You make some professional judgements and decide•Students who do little homework don’t write enough•You could take action to improve homework habits – but you’ve tried that before and the success rate is low•You have more control over other factors – like how much time you give students to write in class So you conclude – the real need is to get students to write more often

www.minedu.govt.nz


Deciding on an action

Information will often suggest a number of options for action. How do we decide which action to choose?We need to consider •what control we have over the action•the likely impact of the action•the resources needed

www.minedu.govt.nz


Planning for action

• Is this a major change to policy or processes? • What other changes are being proposed• How soon can you make this change?• How will you achieve wide buy-in? • What time and resources will you need?• Who will co-ordinate and monitor implementation?

www.minedu.govt.nz


Planning for action

• Is this an incremental change? Or are you just tweaking how you do things?

• How will you fit the change into your regular work? • When can you start the intervention?• Will you need extra resources? • How will this change affect other things you do?• How will you monitor implementation?

www.minedu.govt.nz


Timing is all

• How long should we run the intervention before we evaluate it?

• When is the best time of the year to start (and finish) in terms of measuring changes in student achievement?

• How much preparation time will we need to get maximum benefit?

www.minedu.govt.nz


Planning for evaluation

We are carrying out this action to see what impact it has on student achievementWe need to decide exactly how we’ll know how successful the intervention has been To do this we will need good baseline data

www.minedu.govt.nz


Planning for evaluation

• What evidence do we need to collect before we start?• Do we need to collect evidence along the way, or just at

the end? • How can we be sure that any assessment at the end of

the process will be comparable with assessment at the outset?

• How will we monitor any unintended effects? Don’t forget evidence such as timetables, student

opinions, teacher observations …

www.minedu.govt.nz



Trigger Clues found in data, hunchesExplore Is there really an issue?Question What do you want to know?Assemble Get all useful evidence togetherAnalyse Process data and other evidenceInterpret What information do you have?Intervene Design and carry out action

> Evaluate What was the impact?Reflect What will we change?

www.minedu.govt.nz


Evaluate the impact of our action

Did the intervention improve the situation that triggered the process? If the aim was to improve student achievement, did that happen?

www.minedu.govt.nz



Was any change in student achievement significant? What else happened that we didn’t expect? How do our results compare with other similar studies we can find?Does the result give us the confidence to make the change permanent?

www.minedu.govt.nz



A school created a new year 13 art programme. In the past students had been offered standard Design and Painting programmes, internally and externally assessed against the full range of achievement standards. Some students had to produce two folios for assessment and were unsure of where to take their art after leaving school. The new programme blended drawing, design and painting concepts and focused on electronic media. Assessment was against internally assessed standards only.

www.minedu.govt.nz



• Did students complete more assessments?• Were students gain more national assessment credits?• How did student perceptions of workload and

satisfaction compare with teacher perceptions from the previous year?

• Did students leave school with clearer intentions about where to go next with their art than the previous cohort?

• How did teachers and parents feel about the change?

www.minedu.govt.nz


Evaluate the intervention

• How well did we design and carry out the intervention? Would we do anything differently if we did it again?

• Were our results affected by anything that happened during the intervention period - within or beyond our control?

• Did we ask the right question in the first place? How useful was our question?

• How adequate were our evaluation data?

www.minedu.govt.nz


Think about the process

• Did we ask the right question in the first place? How useful was our question?

• Did we select the right data? Could we have used other evidence?

• Did the intervention work well? Could we have done anything differently?

• Did we interpret the data-based information correctly?• How adequate were our evaluation data?• Did the outcome justify the effort we put into it?

www.minedu.govt.nz



Trigger Clues found in data, hunchesExplore Is there really an issue?Question What do you want to know?Assemble Get all useful evidence togetherAnalyse Process data and other evidenceInterpret What information do you have?Intervene Design and carry out actionEvaluate What was the impact?

> Reflect What will we change?

www.minedu.govt.nz


Future practice

• What aspects of the intervention will we embed in future practice?

• What aspects of the intervention will have the greatest impact?

• What aspects of the intervention can we maintain over time?

• What changes can we build into the way we do things in our school?

• Would there be any side-effects?

www.minedu.govt.nz


Future directions

• What professional learning is needed? Who would most benefit from it?

• Do we have the expertise we need in-house or do we need external help?

• What other resources do we need? • What disadvantages could there be?• When will we evaluate this change again?

Documents

Consider the Evidence