THE USE OF EVIDENCE TO IMPROVE EDUCATION AND SERVE THE PUBLIC GOOD?

UNIVERSITY OF CAMBRIDGE1 NOVEMBER 2012

Adrienne Alton-Lee, PhD, Dip Tchg (Distinction) Iterative Best Evidence Synthesis Programme

Hei Kete Raukura Ministry of Education

New Zealand www.educationcounts.govt.nz/goto/BES

BEYOND PLATO’S MYTH OF THE METALS’

• the role of teachers as one of maintaining inequalities in society by educating children differently

• gold, brass or iron education for gold, iron and brass children

• Sorting via education as a peaceful alternative to a military regime in maintaining differential status and access to material wealth in society

What works for valued outcomes for diverse

(all) learners?

Why?

How?

What doesn’t work?

First do no harm in education

What makes a bigger difference in areas of

need?

FIT-FOR-PURPOSE METHODOLOGY

• Influences on valued student outcomes

• Trustworthy bodies of evidence

• Rigorous eclecticism – iterative process

• Comparative magnitude of impact

• Effect sizes – impact & equity index

• Critical role of theory

• Case & vignette

• Context matters

INQUIRY & KNOWLEDGE BUILDING CYCLES

EARLY BEST EVIDENCE SYNTHESIS ITERATIONS

The complexity of community & family influences e.g.

• Poverty

• Pre-school access to experiences

• Undiagnosed hearing loss

• Family interactions

• Television

• Effective school-home partnerships

COLLABORATIVE KNOWLEDGE BUILDING & USE FOR ONGOING IMPROVEMENT

Research

Practice Policy?

PRIMARY TEACHERS’ UNION

NZEI Te Riu Roa welcomes the opportunity to comment on this Best Evidence Synthesis.

Drawing widely and systematically from national and international research on social sciences education, its authors have sought evidence of what works, for which students, and in what circumstances. The synthesis of findings contributes to our understanding of the relationship between pedagogy and outcomes and the importance of context. Teachers will welcome the many practical ways in which to strengthen practice – a particular feature of the BES Programme.

National President, New Zealand Educational Institute, 2008.

SECONDARY PRINCIPALS’ ASSOCIATION

The Leadership BES already has significant traction in New Zealand secondary schools and is well regarded by the profession as being both aspirational and practical in content. We are proud to have been involved with it from the beginning and commend it to you as a well-researched, clear and detailed way forward for leaders at any level of the schooling system. We hope it gives principals in particular a focus for their work as well as being a useful tool to help us all find ways to improve student outcomes.

President of the New Zealand Secondary Principals’ Association, 2011.

12,472

SCHOOL TRUSTEES’ ASSOCIATION (SCHOOL GOVERNANCE)

...the final document reflects only a small proportion of the influence that this work has had for those of us who have been involved in its development. The process of developing the BES has triggered new learnings. It has built stronger links, within and across the sector, between academics and practitioners, and it has provided springboards for new initiatives in leadership...best of all, we found that ‘unpicking each section , as we reviewed it, raised questions in our own minds and those of our colleagues and helped us to think in a more disciplined way about what matters; our students, and how leadership contributes to enhancing their achievements...President, Manager Training and Development & Manager, Service Delivery, New Zealand School Trustees Association

SCHOOL LEADERSHIP & STUDENT OUTCOMES: IDENTIFYING WHAT WORKS AND WHY: BEST

EVIDENCE SYNTHESIS

Promoting and participating in teacher learning

Creating educationally powerful connections

Establishing goals and expectations

Planning, coordinating and evaluating teaching & the curriculum

Resourcing strategically

Ensuring an orderly & supportive environment

-0.2 0 0.2 0.4 0.6 0.8 1

0.84

-0.04

0.42

0.42

0.31

0.27

0.280.6

Promoting and participating in teacher learning

Creating educationally powerful connections

Establishing goals and expectations

Planning, coordinating and evaluating teaching & the curriculum

Resourcing strategically

Ensuring an orderly & supportive environment

-0.2 0 0.2 0.4 0.6 0.8 1

0.84

-0.04

0.42

0.42

0.31

0.27

0.280.6

BACKWARD MAPPING

• Ensure administrative decisions are informed by knowledge about effective pedagogy

• Engage in constructive problem talk • Engage in open-to-learning conversations • Analyse and solve complex problems • Select, develop and use smart tools • Build relational trust• Create a community that learns how to improve

student success

FORMATIVE QUALITY ASSURERS’ WARNING

We believe that any school leader, or system leader, or anyone with an interest in education improvement or leadership, will find this report stimulating and valuable. Certainly both of us did.  Having such a high quality review is an important accomplishment, but it is not enough…The challenge for all partners in New Zealand (and beyond) will be to make sure that the lessons and implications of this review leap off the pages and become part of the fabric of education in the country.

The BES report on leadership will be for nought unless there is a concerted plan to develop the core capacities of effective leadership in all New Zealand schools. This for us would not mean merely deriving a plan from the findings, but rather taking a concrete problem, such as raising the bar and closing the gap in literacy in New Zealand schools and incorporating the key leadership capacities into the implementation of the very problem to be addressed. It is always better to start with the concrete and incorporate what is needed to make specific improvements.

TEACHER PROFESSIONAL LEARNING & DEVELOPMENT BES

Timperley, H., & Alton-Lee, A. (2008). Reframing teacher professional learning: An alternative policy approach to strengthening valued outcomes for diverse learners. Review of Research in Education 32, 328 – 369.

1. Much teacher professional development has no effect on valued student outcomes

2. Some professional development has negative effects 3. A comparative magnitude of impact analysis shows some

outstanding professional development can lift achievement for all and accelerate progress of lowest 20% of achievers 3-6 times the rate of business-as-usual practice

http://www.ibe.unesco.org/en/services/online-materials/publications/educational-practices.html

14,145

Teacher Professional Learning and Development

What works What doesn’t work

Focus on valued student outcomes

Ensure worthwhile content

Assume rather than check influences on valued outcomes for diverse (all) learner

Integrate knowledge and skills Use a skills-only focus without theory

Use assessment for professional inquiry

Use assessment for sorting or labelling Fail to develop adaptive expertise

Ensure multiple opportunities to learn and apply in ways that ensure trust & challenge

1-2 years for deep change

‘Sit and get’ US Teacher: “I hope I die during an in-service session because the transition between life and death would be so subtle.”

Ensure responsiveness to teachers By-pass teachers’ theories

Teacher Professional Learning and Development

What works What doesn’t work

Provide opportunities to process new learning with others

Little opportunity to learn with others

Professional communities that are not engaging with impacts on learners

Knowledgeable expertise – external to group of participating teachers

Ignore complexity of teachingLow or narrow expertise risks –ve effects

Ensure active leadership of designated leaders to create conditions for change

Fail to involve active support of leadership for ongoing improvement

Maintain momentum through self-regulation Where am I going? How am I doing? Where to next?

Move on to the next fad without integrating the learning into practice

A PERSPECTIVE FROM THE AUDITOR GENERAL

3.21 Various stakeholders told us that the Ministry did not consistently base its decisions about funding and providing professional development initiatives on the evidence it has.3.22 In our view, it would be helpful for the Ministry to review the professional development initiatives it funds against its BES evidence and any other relevant evidence on effective professional development.Office of the Auditor General 2008

QUALITY TEACHING FOR DIVERSE (ALL) LEARNERS

• www.educationcounts.govt.nz/goto/BES• See ‘Overview of Findings’ Cross-curricula • ‘The use of evidence to improve education and serve the public

good’• 19,276 copies

requested in NZ 2500 schools

To 140 countries from UNESCO

EFFECTIVE PEDAGOGY IN SOCIAL SCIENCES 10,221 copies requested in NZ

HIGH IMPACT PEDAGOGIES FOR MULTIPLE VALUED OUTCOMES

• Highly effective pedagogies in areas of need

• Approaches that simultaneously advance a range of

valued outcomes – cognitive, well-being, social…

• Developed through several iterations of collaborative

research & development

• Disciplined innovation by the profession as a resource

for improvement

BES EXEMPLARS

• Response to feedback from the profession

• Explain the complexity of teaching and the how

• Use of student and teacher ‘voice’

• Not just teaching - professional learning and leadership supports

• An inquiry and knowledge building approach to improvement

• Implementation alerts

• Access to trusted resources, smart tools, DVDs, websites

• Access to Research Behind BES

Think alouds

HIGH IMPACT PEDAGOGIES

ACCELERATED IMPROVEMENT IN AREAS OF NEED FOR UNDERSERVED OR DISADVANTAGED

LEARNERS BES Exemplar 1: Developing Communities of

Mathematical Inquiry Dr Roberta Hunter • Students are scaffolded to engage with the

teacher and peers in mathematical inquiry, reasoning & argumentation

• 4-5 years of achievement acceleration in one year • ‘Don’t ‘dis her, man, when she’s taking a risk.”

BES EXEMPLAR 1: DEVELOPING COMMUNITIES OF MATHEMATICAL INQUIRY

Signific

antly

lower

One le

vel b

elow

At ave

rage a

ge-gr

oup l

evel

Above

avera

ge

Well

abov

e ave

rage

0204060

24

55

210 00 0

42 46

12

Ava’s Class

% Before % After

BES EXEMPLAR 5 LEARNING LOGS – JENNIFER GLENN

THAMES HIGH SCHOOL Teacher feedback Then – students read and respond to these questions….

What do you think I’ve said about your writing?How do you feel about the outcomes/comment?Set three goals for the next similar piece of writing?

Wendy: ‘I like using this blue book [learning log] because it makes you really think about what you need to improve John: ‘This is the best thing that ever happened to my writing’ Brian: ‘We cover things in class that are much more useful to me and everyone else’

BES EXEMPLAR 5 LEARNING LOGS

Not achieved Achieved Merit Excellence 0

10

20

30

40

50

60

21

46

27

5

56

38

52

47

37

13

3

46

39

12

3

Aggregate Achievement NCEA Level 1 English

Learning Logs Class Comparison Class School National

LEARNING LOGS IN PHYSICAL EDUCATION XANTHE SULZBERGER, AQUINAS COLLEGE

Over time, the emails became electronic “learning logs” that provided evidence of learning and resources for ongoing improvement, for me as well as the students. They provided me with rich information as to how much the students understood, and what I may have to re-address in class. I could see whether questions needed to be re-framed or concepts needed to be re-taught. This sometimes meant that I had to adapt my questioning or how I presented the question. The interaction became a two-way process, with students asking me for clarification that they may not have asked during class time’

We also feel more involved as parents from reading the logs, and it has led to some great discussions around the dinner table with the family all being interested in the topic. This all contributed, I believe, to our son getting his first Excellence in a written assignment.” Parent of student in Xanthe’s class Xanthe SulzbergerPhysical Education Achievement Standard 2.2: The Badminton Smash Shot

XANTHE’S USE OF LEARNING LOGS

STUDENT FEEDBACK

EDUCATIONAL POLICY, PRACTICE & EVIDENCE USE

Meta-analyses ES 0.53 0.54 0.59 for cooperationES 0.36 0.23 for competitive or individualistic efforts Stanne et al (1999); Hattie (2009)

In comparison with schooling practices that are often supported by governments – such as tutoring, technology use and school restructuring – co-operative learning is relatively inexpensive and easily adopted. Yet, thirty years after much of the foundational research was completed, it remains at the edge of school policy. This does not have to remain the case: as governments come to support the larger concept of evidence-based reform, the strong evidence base for co-operative learning may lead to a greater focus on this set of approaches at the core of instructional practice. In the learning environments of the 21st century, co-operative learning should play a central role. (Slavin, 2010, p. 174).

Galton, M., & Hargreaves, L. (2009). (Eds). Group work: still a neglected art? Cambridge Journal of Education, 39(1) 1-6.

THE NEW FRONTIER: THE USE OF EVIDENCE IN POLICY

“Research evidence is perceived as “softly spoken” because, “empirical inquiry simply cannot make its voice heard amidst the clatter of other, political imperatives on policy making” (Pawson, 2002).

RESEARCH INTO HOW EVIDENCE IS USED IN POLICY

COBURN, TOURĒ & YAMASHITA (2009)• A strong tendency to discount evidence when it does not support the beliefs decision-makers hold

• Orchestrated consensus to over-ride action on the available evidence

• Extent of use of evidence depends on individuals with positional authority

• Evidence brought to the resolution of disputes ‘rarely addressed assumptions about high quality instruction or about how children learn’

• In times of decision-making under pressure and/or contracting resources ‘decision-making trajectories became more interrupted, last minute and symbolic’

• ‘symbolic’ = the use of evidence to give legitimacy to a policy position rather than to specifically guide improvement efforts

POLICY FOR THE USE OF EVIDENCE IN EDUCATIONAL POLICY

• Little research on linking research to policy in education • Passive dissemination of research is generally ineffective whilst

multi-faceted interventions targeting multiple barriers more effective Recommendations e.g.• Knowledge, awareness and skills capacity building in all parts of

the research evidence production-to-use system• Policy decisions to develop evidence informed policy in education• Increasing capacity in research on research generation and use

AN EFFECTIVE, SYSTEM-WIDE CHANGE STRATEGY REQUIRES THE FOLLOWING ELEMENTS:

1. A small number of ambitious yet achievable and well-grounded goals, publicly stated2. A positive stance on improving all schools and success for all students 3. An emphasis on capacity building and a focus on results 4. Multi-level engagement with strong leadership and a ‘guiding coalition 5. Continuous learning through innovation and effective use of research and data6. A focus on key strategies while also managing other interests and

issues7. Effective use of resources8. A strong implementation effort to support the change process.

USE EVIDENCE TO IMPROVE EDUCATION AND SERVE PUBLIC GOOD

Use evidence to strategically resource improvement in education

Focus on improvement in valued outcomes for diverse (all) learners & accelerated improvement for those underserved or disadvantaged

Establish goals and expectations for improvement

Use trustworthy evidence about the what and the how of improvement

In times of fiscal crisis, give priority to leveraging the evidence of what makes a bigger difference for multiple valued outcomes

Ensure knowledge of effective pedagogy drives improvement

Do no harm

USE EVIDENCE TO IMPROVE EDUCATION AND SERVE PUBLIC GOOD

Foster constructive problem talk, build relational trust & ensure effective supports for improvement

Build commitment across stakeholders for aligned action to support ongoing improvement in teaching and learning

Leverage all four areas of influence for accelerated improvement: pedagogy, educationally powerful connections, professional learning and leadership

Invest in collaborative R & D expertise as a driver for accelerated systemic improvement in areas of need.

Develop adaptive expertise & smart tools

Scale up effective implementation of high-impact pedagogies across the system.